Nov 01 03:05:51.840936 ip-10-0-133-54 systemd[1]: Starting Kubernetes Kubelet...
Nov 01 03:05:52.223479 ip-10-0-133-54 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 03:05:52.223479 ip-10-0-133-54 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 01 03:05:52.223479 ip-10-0-133-54 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 03:05:52.223479 ip-10-0-133-54 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 03:05:52.226731 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.226645    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 03:05:52.226731 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.226731    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 03:05:52.226731 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.226735    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 03:05:52.226820 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.226738    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 03:05:52.226820 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.226741    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 03:05:52.226820 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.226745    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 03:05:52.226820 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.226747    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 03:05:52.226820 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.226750    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 03:05:52.226820 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.226752    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 03:05:52.226820 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.226755    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 03:05:52.226820 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.226757    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 03:05:52.226820 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.226760    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 03:05:52.226820 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.226763    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 03:05:52.226820 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.226765    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 03:05:52.226820 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.226768    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 03:05:52.226820 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.226776    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 03:05:52.226820 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.226778    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 03:05:52.226820 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.226781    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 03:05:52.226820 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.226784    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 03:05:52.226820 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.226786    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 03:05:52.226820 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.226789    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 03:05:52.226820 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.226791    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 03:05:52.226820 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.226794    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 03:05:52.227276 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.226796    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 03:05:52.227276 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.226799    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 03:05:52.227276 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.226801    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 03:05:52.227276 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.226803    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 03:05:52.227276 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.226806    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 03:05:52.227276 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.226808    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 03:05:52.227276 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.226811    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 03:05:52.227276 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.226813    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 03:05:52.227276 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.226818    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 03:05:52.227276 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.226822    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 03:05:52.227276 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.226826    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 03:05:52.227276 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.226829    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 03:05:52.227276 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.226832    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 03:05:52.227276 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.226835    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 03:05:52.227276 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.226837    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 03:05:52.227276 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.226840    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 03:05:52.227276 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.226842    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 03:05:52.227276 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.226845    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 03:05:52.227276 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.226847    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 03:05:52.227741 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.226849    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 03:05:52.227741 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.226852    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 03:05:52.227741 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.226854    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 03:05:52.227741 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.226857    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 03:05:52.227741 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.226859    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 03:05:52.227741 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.226862    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 03:05:52.227741 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.226866    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 03:05:52.227741 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.226870    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 03:05:52.227741 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.226873    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 03:05:52.227741 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.226876    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 03:05:52.227741 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.226879    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 03:05:52.227741 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.226882    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 03:05:52.227741 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.226884    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 03:05:52.227741 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.226887    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 03:05:52.227741 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.226890    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 03:05:52.227741 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.226892    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 03:05:52.227741 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.226895    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 03:05:52.227741 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.226898    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 03:05:52.227741 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.226901    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 03:05:52.228177 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.226903    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 03:05:52.228177 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.226906    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 03:05:52.228177 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.226908    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 03:05:52.228177 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.226910    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 03:05:52.228177 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.226913    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 03:05:52.228177 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.226916    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 03:05:52.228177 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.226918    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 03:05:52.228177 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.226921    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 03:05:52.228177 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.226923    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 03:05:52.228177 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.226927    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 03:05:52.228177 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.226929    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 03:05:52.228177 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.226932    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 03:05:52.228177 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.226934    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 03:05:52.228177 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.226936    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 03:05:52.228177 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.226939    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 03:05:52.228177 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.226941    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 03:05:52.228177 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.226944    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 03:05:52.228177 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.226946    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 03:05:52.228177 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.226949    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 03:05:52.228644 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.226951    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 03:05:52.228644 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.226955    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 03:05:52.228644 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.226957    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 03:05:52.228644 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.226960    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 03:05:52.228644 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.226962    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 03:05:52.228644 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.226964    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 03:05:52.228644 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.226967    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 03:05:52.228644 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.226970    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 03:05:52.228644 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.226973    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 03:05:52.228644 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.226975    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 03:05:52.228644 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.226978    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 03:05:52.228644 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.226980    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 03:05:52.228644 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.226983    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 03:05:52.228644 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.226986    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 03:05:52.228644 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.226988    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 03:05:52.228644 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.226991    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 03:05:52.228644 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.226993    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 03:05:52.228644 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.226996    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 03:05:52.228644 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.226998    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 03:05:52.228644 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.227001    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 03:05:52.229168 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.227003    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 03:05:52.229168 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.227006    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 03:05:52.229168 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.227009    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 03:05:52.229168 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.227011    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 03:05:52.229168 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.227013    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 03:05:52.229168 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.227016    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 03:05:52.229168 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.227801    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 03:05:52.229168 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.227810    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 03:05:52.229168 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.227814    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 03:05:52.229168 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.227817    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 03:05:52.229168 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.227820    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 03:05:52.229168 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.227822    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 03:05:52.229168 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.227825    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 03:05:52.229168 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.227828    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 03:05:52.229168 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.227831    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 03:05:52.229168 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.227834    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 03:05:52.229168 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.227836    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 03:05:52.229168 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.227839    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 03:05:52.229168 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.227841    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 03:05:52.229168 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.227845    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 03:05:52.229639 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.227848    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 03:05:52.229639 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.227851    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 03:05:52.229639 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.227854    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 03:05:52.229639 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.227856    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 03:05:52.229639 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.227859    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 03:05:52.229639 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.227861    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 03:05:52.229639 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.227864    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 03:05:52.229639 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.227866    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 03:05:52.229639 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.227869    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 03:05:52.229639 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.227871    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 03:05:52.229639 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.227874    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 03:05:52.229639 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.227876    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 03:05:52.229639 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.227879    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 03:05:52.229639 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.227881    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 03:05:52.229639 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.227885    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 03:05:52.229639 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.227889    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 03:05:52.229639 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.227892    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 03:05:52.229639 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.227895    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 03:05:52.229639 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.227897    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 03:05:52.230112 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.227900    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 03:05:52.230112 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.227902    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 03:05:52.230112 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.227905    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 03:05:52.230112 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.227907    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 03:05:52.230112 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.227910    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 03:05:52.230112 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.227912    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 03:05:52.230112 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.227915    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 03:05:52.230112 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.227917    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 03:05:52.230112 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.227920    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 03:05:52.230112 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.227922    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 03:05:52.230112 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.227925    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 03:05:52.230112 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.227927    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 03:05:52.230112 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.227931    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 03:05:52.230112 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.227933    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 03:05:52.230112 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.227936    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 03:05:52.230112 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.227938    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 03:05:52.230112 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.227941    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 03:05:52.230112 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.227943    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 03:05:52.230112 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.227945    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 03:05:52.230112 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.227948    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 03:05:52.230572 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.227950    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 03:05:52.230572 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.227952    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 03:05:52.230572 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.227955    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 03:05:52.230572 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.227957    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 03:05:52.230572 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.227960    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 03:05:52.230572 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.227962    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 03:05:52.230572 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.227965    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 03:05:52.230572 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.227967    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 03:05:52.230572 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.227969    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 03:05:52.230572 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.227972    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 03:05:52.230572 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.227974    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 03:05:52.230572 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.227977    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 03:05:52.230572 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.227980    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 03:05:52.230572 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.227982    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 03:05:52.230572 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.227985    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 03:05:52.230572 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.227987    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 03:05:52.230572 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.227989    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 03:05:52.230572 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.227992    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 03:05:52.230572 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.227994    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 03:05:52.231028 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.227996    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 03:05:52.231028 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.227999    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 03:05:52.231028 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228002    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 03:05:52.231028 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228005    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 03:05:52.231028 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228007    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 03:05:52.231028 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228010    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 03:05:52.231028 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228013    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 03:05:52.231028 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228016    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 03:05:52.231028 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228018    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 03:05:52.231028 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228021    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 03:05:52.231028 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228023    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 03:05:52.231028 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228026    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 03:05:52.231028 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228028    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 03:05:52.231028 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228031    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 03:05:52.231028 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228033    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 03:05:52.231028 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228036    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 03:05:52.231028 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228038    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 03:05:52.231028 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228041    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 03:05:52.231028 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228044    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 03:05:52.231028 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228048    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 03:05:52.231504 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228051    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 03:05:52.231504 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228054    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 03:05:52.231504 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228056    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 03:05:52.231504 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228059    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 03:05:52.231504 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228061    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 03:05:52.231504 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228063    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 03:05:52.231504 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228066    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 03:05:52.231504 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228068    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 03:05:52.231504 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228071    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 03:05:52.231504 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228073    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 03:05:52.231504 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228076    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 03:05:52.231504 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228080    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 03:05:52.231504 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228082    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 03:05:52.231504 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228085    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 03:05:52.231504 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228560    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 03:05:52.231504 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228565    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 03:05:52.231504 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228569    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 03:05:52.231504 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228574    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 03:05:52.231504 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228578    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 03:05:52.231970 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228582    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 03:05:52.231970 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228585    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 03:05:52.231970 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228588    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 03:05:52.231970 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228590    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 03:05:52.231970 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228593    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 03:05:52.231970 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228596    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 03:05:52.231970 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228598    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 03:05:52.231970 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228601    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 03:05:52.231970 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228603    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 03:05:52.231970 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228606    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 03:05:52.231970 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228608    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 03:05:52.231970 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228610    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 03:05:52.231970 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228613    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 03:05:52.231970 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228615    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 03:05:52.231970 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228618    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 03:05:52.231970 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228620    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 03:05:52.231970 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228623    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 03:05:52.231970 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228625    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 03:05:52.231970 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228627    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 03:05:52.232427 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228630    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 03:05:52.232427 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228632    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 03:05:52.232427 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228635    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 03:05:52.232427 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228637    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 03:05:52.232427 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228640    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 03:05:52.232427 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228642    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 03:05:52.232427 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228645    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 03:05:52.232427 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228649    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 03:05:52.232427 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228652    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 03:05:52.232427 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228655    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 03:05:52.232427 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228658    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 03:05:52.232427 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228661    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 03:05:52.232427 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228663    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 03:05:52.232427 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228666    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 03:05:52.232427 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228668    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 03:05:52.232427 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228671    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 03:05:52.232427 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228674    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 03:05:52.232427 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228676    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 03:05:52.232427 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228679    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 03:05:52.232427 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228697    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 03:05:52.232909 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228699    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 03:05:52.232909 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228702    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 03:05:52.232909 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228704    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 03:05:52.232909 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228707    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 03:05:52.232909 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228709    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 03:05:52.232909 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228712    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 03:05:52.232909 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228714    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 03:05:52.232909 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228717    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 03:05:52.232909 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228719    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 03:05:52.232909 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228722    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 03:05:52.232909 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228724    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 03:05:52.232909 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228727    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 03:05:52.232909 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228729    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 03:05:52.232909 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228732    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 03:05:52.232909 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228734    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 03:05:52.232909 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228737    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 03:05:52.232909 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228740    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 03:05:52.232909 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228742    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 03:05:52.232909 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228745    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 03:05:52.233342 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228747    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 03:05:52.233342 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228750    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 03:05:52.233342 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228752    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 03:05:52.233342 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228756    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 03:05:52.233342 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228759    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 03:05:52.233342 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228762    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 03:05:52.233342 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228764    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 03:05:52.233342 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228767    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 03:05:52.233342 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228770    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 03:05:52.233342 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228772    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 03:05:52.233342 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228774    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 03:05:52.233342 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228777    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 03:05:52.233342 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228779    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 03:05:52.233342 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228782    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 03:05:52.233342 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228784    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 03:05:52.233342 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228786    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 03:05:52.233342 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228789    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 03:05:52.233342 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228791    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 03:05:52.233342 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228793    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 03:05:52.233342 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228796    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 03:05:52.233851 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228798    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 03:05:52.233851 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228800    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 03:05:52.233851 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228803    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 03:05:52.233851 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228805    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 03:05:52.233851 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228807    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 03:05:52.233851 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228810    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 03:05:52.233851 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228812    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 03:05:52.233851 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228818    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 03:05:52.233851 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228821    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 03:05:52.233851 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228823    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 03:05:52.233851 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228826    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 03:05:52.233851 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228828    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 03:05:52.233851 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228831    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 03:05:52.233851 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228833    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 03:05:52.233851 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228835    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 03:05:52.233851 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228838    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 03:05:52.233851 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228841    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 03:05:52.233851 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228844    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 03:05:52.233851 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228846    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 03:05:52.233851 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228849    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 03:05:52.234305 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228851    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 03:05:52.234305 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228853    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 03:05:52.234305 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.228856    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 03:05:52.234305 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230128    2576 flags.go:64] FLAG: --address="0.0.0.0"
Nov 01 03:05:52.234305 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230136    2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 01 03:05:52.234305 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230144    2576 flags.go:64] FLAG: --anonymous-auth="true"
Nov 01 03:05:52.234305 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230148    2576 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 01 03:05:52.234305 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230157    2576 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 01 03:05:52.234305 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230160    2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 01 03:05:52.234305 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230165    2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 01 03:05:52.234305 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230169    2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 01 03:05:52.234305 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230172    2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 01 03:05:52.234305 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230175    2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 01 03:05:52.234305 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230178    2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 01 03:05:52.234305 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230182    2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 01 03:05:52.234305 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230185    2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 01 03:05:52.234305 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230187    2576 flags.go:64] FLAG: --cgroup-root=""
Nov 01 03:05:52.234305 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230190    2576 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 01 03:05:52.234305 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230193    2576 flags.go:64] FLAG: --client-ca-file=""
Nov 01 03:05:52.234305 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230198    2576 flags.go:64] FLAG: --cloud-provider="external"
Nov 01 03:05:52.234305 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230201    2576 flags.go:64] FLAG: --cluster-dns="[]"
Nov 01 03:05:52.234305 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230205    2576 flags.go:64] FLAG: --cluster-domain=""
Nov 01 03:05:52.234305 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230209    2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 01 03:05:52.234305 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230212    2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 01 03:05:52.234909 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230215    2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 01 03:05:52.234909 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230219    2576 flags.go:64] FLAG: --container-log-max-files="5"
Nov 01 03:05:52.234909 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230223    2576 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 01 03:05:52.234909 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230225    2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 01 03:05:52.234909 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230228    2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 01 03:05:52.234909 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230232    2576 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 01 03:05:52.234909 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230235    2576 flags.go:64] FLAG: --contention-profiling="false"
Nov 01 03:05:52.234909 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230238    2576 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 01 03:05:52.234909 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230241    2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 01 03:05:52.234909 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230244    2576 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 01 03:05:52.234909 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230247    2576 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 01 03:05:52.234909 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230251    2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 01 03:05:52.234909 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230254    2576 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 01 03:05:52.234909 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230257    2576 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 01 03:05:52.234909 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230259    2576 flags.go:64] FLAG: --enable-load-reader="false"
Nov 01 03:05:52.234909 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230262    2576 flags.go:64] FLAG: --enable-server="true"
Nov 01 03:05:52.234909 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230265    2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 01 03:05:52.234909 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230270    2576 flags.go:64] FLAG: --event-burst="100"
Nov 01 03:05:52.234909 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230273    2576 flags.go:64] FLAG: --event-qps="50"
Nov 01 03:05:52.234909 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230276    2576 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 01 03:05:52.234909 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230278    2576 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 01 03:05:52.234909 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230281    2576 flags.go:64] FLAG: --eviction-hard=""
Nov 01 03:05:52.234909 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230286    2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 01 03:05:52.234909 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230289    2576 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 01 03:05:52.235489 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230292    2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 01 03:05:52.235489 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230294    2576 flags.go:64] FLAG: --eviction-soft=""
Nov 01 03:05:52.235489 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230297    2576 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 01 03:05:52.235489 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230302    2576 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 01 03:05:52.235489 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230305    2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 01 03:05:52.235489 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230308    2576 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 01 03:05:52.235489 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230310    2576 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 01 03:05:52.235489 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230313    2576 flags.go:64] FLAG: --fail-swap-on="true"
Nov 01 03:05:52.235489 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230316    2576 flags.go:64] FLAG: --feature-gates=""
Nov 01 03:05:52.235489 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230320    2576 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 01 03:05:52.235489 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230323    2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 01 03:05:52.235489 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230326    2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 01 03:05:52.235489 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230329    2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 01 03:05:52.235489 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230332    2576 flags.go:64] FLAG: --healthz-port="10248"
Nov 01 03:05:52.235489 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230335    2576 flags.go:64] FLAG: --help="false"
Nov 01 03:05:52.235489 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230338    2576 flags.go:64] FLAG: --hostname-override="ip-10-0-133-54.us-west-1.compute.internal"
Nov 01 03:05:52.235489 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230341    2576 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 01 03:05:52.235489 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230344    2576 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 01 03:05:52.235489 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230347    2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 01 03:05:52.235489 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230351    2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 01 03:05:52.235489 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230354    2576 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 01 03:05:52.235489 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230356    2576 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 01 03:05:52.235489 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230359    2576 flags.go:64] FLAG: --image-service-endpoint=""
Nov 01 03:05:52.236039 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230362    2576 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 01 03:05:52.236039 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230365    2576 flags.go:64] FLAG: --kube-api-burst="100"
Nov 01 03:05:52.236039 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230368    2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 01 03:05:52.236039 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230370    2576 flags.go:64] FLAG: --kube-api-qps="50"
Nov 01 03:05:52.236039 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230373    2576 flags.go:64] FLAG: --kube-reserved=""
Nov 01 03:05:52.236039 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230376    2576 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 01 03:05:52.236039 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230378    2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 01 03:05:52.236039 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230381    2576 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 01 03:05:52.236039 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230384    2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 01 03:05:52.236039 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230387    2576 flags.go:64] FLAG: --lock-file=""
Nov 01 03:05:52.236039 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230389    2576 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 01 03:05:52.236039 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230392    2576 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 01 03:05:52.236039 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230396    2576 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 01 03:05:52.236039 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230402    2576 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 01 03:05:52.236039 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230405    2576 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 01 03:05:52.236039 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230408    2576 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 01 03:05:52.236039 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230411    2576 flags.go:64] FLAG: --logging-format="text"
Nov 01 03:05:52.236039 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230414    2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 01 03:05:52.236039 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230417    2576 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 01 03:05:52.236039 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230420    2576 flags.go:64] FLAG: --manifest-url=""
Nov 01 03:05:52.236039 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230422    2576 flags.go:64] FLAG: --manifest-url-header=""
Nov 01 03:05:52.236039 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230427    2576 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 01 03:05:52.236039 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230429    2576 flags.go:64] FLAG: --max-open-files="1000000"
Nov 01 03:05:52.236039 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230433    2576 flags.go:64] FLAG: --max-pods="110"
Nov 01 03:05:52.236039 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230437    2576 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 01 03:05:52.236643 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230440    2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 01 03:05:52.236643 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230442    2576 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 01 03:05:52.236643 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230445    2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 01 03:05:52.236643 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230448    2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 01 03:05:52.236643 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230451    2576 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 01 03:05:52.236643 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230453    2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 01 03:05:52.236643 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230461    2576 flags.go:64] FLAG: --node-status-max-images="50"
Nov 01 03:05:52.236643 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230464    2576 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 01 03:05:52.236643 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230466    2576 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 01 03:05:52.236643 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230469    2576 flags.go:64] FLAG: --pod-cidr=""
Nov 01 03:05:52.236643 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230472    2576 flags.go:64] FLAG: --pod-manifest-path=""
Nov 01 03:05:52.236643 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230475    2576 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 01 03:05:52.236643 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230478    2576 flags.go:64] FLAG: --pods-per-core="0"
Nov 01 03:05:52.236643 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230480    2576 flags.go:64] FLAG: --port="10250"
Nov 01 03:05:52.236643 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230483    2576 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 01 03:05:52.236643 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230486    2576 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0b938cf80efcff5b7"
Nov 01 03:05:52.236643 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230489    2576 flags.go:64] FLAG: --qos-reserved=""
Nov 01 03:05:52.236643 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230492    2576 flags.go:64] FLAG: --read-only-port="10255"
Nov 01 03:05:52.236643 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230495    2576 flags.go:64] FLAG: --register-node="true"
Nov 01 03:05:52.236643 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230499    2576 flags.go:64] FLAG: --register-with-taints=""
Nov 01 03:05:52.236643 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230502    2576 flags.go:64] FLAG: --registry-burst="10"
Nov 01 03:05:52.236643 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230505    2576 flags.go:64] FLAG: --registry-qps="5"
Nov 01 03:05:52.236643 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230508    2576 flags.go:64] FLAG: --reserved-cpus=""
Nov 01 03:05:52.236643 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230511    2576 flags.go:64] FLAG: --reserved-memory=""
Nov 01 03:05:52.236643 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230514    2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 01 03:05:52.236643 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230517    2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 01 03:05:52.237289 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230520    2576 flags.go:64] FLAG: --rotate-certificates="false"
Nov 01 03:05:52.237289 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230523    2576 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 01 03:05:52.237289 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230526    2576 flags.go:64] FLAG: --runonce="false"
Nov 01 03:05:52.237289 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230529    2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 01 03:05:52.237289 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230532    2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 01 03:05:52.237289 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230534    2576 flags.go:64] FLAG: --seccomp-default="false"
Nov 01 03:05:52.237289 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230538    2576 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 01 03:05:52.237289 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230541    2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 01 03:05:52.237289 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230544    2576 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 01 03:05:52.237289 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230546    2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 01 03:05:52.237289 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230549    2576 flags.go:64] FLAG: --storage-driver-password="root"
Nov 01 03:05:52.237289 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230552    2576 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 01 03:05:52.237289 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230555    2576 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 01 03:05:52.237289 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230557    2576 flags.go:64] FLAG: --storage-driver-user="root"
Nov 01 03:05:52.237289 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230560    2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 01 03:05:52.237289 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230563    2576 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 01 03:05:52.237289 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230566    2576 flags.go:64] FLAG: --system-cgroups=""
Nov 01 03:05:52.237289 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230568    2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 01 03:05:52.237289 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230573    2576 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 01 03:05:52.237289 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230576    2576 flags.go:64] FLAG: --tls-cert-file=""
Nov 01 03:05:52.237289 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230579    2576 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 01 03:05:52.237289 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230583    2576 flags.go:64] FLAG: --tls-min-version=""
Nov 01 03:05:52.237289 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230586    2576 flags.go:64] FLAG: --tls-private-key-file=""
Nov 01 03:05:52.237289 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230589    2576 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 01 03:05:52.237289 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230592    2576 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 01 03:05:52.238031 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230596    2576 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 01 03:05:52.238031 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230599    2576 flags.go:64] FLAG: --v="2"
Nov 01 03:05:52.238031 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230605    2576 flags.go:64] FLAG: --version="false"
Nov 01 03:05:52.238031 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230609    2576 flags.go:64] FLAG: --vmodule=""
Nov 01 03:05:52.238031 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230613    2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 01 03:05:52.238031 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.230616    2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 01 03:05:52.238031 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.230767    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 03:05:52.238031 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.230771    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 03:05:52.238031 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.230774    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 03:05:52.238031 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.230777    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 03:05:52.238031 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.230779    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 03:05:52.238031 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.230782    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 03:05:52.238031 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.230784    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 03:05:52.238031 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.230787    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 03:05:52.238031 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.230790    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 03:05:52.238031 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.230793    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 03:05:52.238031 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.230795    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 03:05:52.238031 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.230798    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 03:05:52.238031 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.230801    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 03:05:52.238031 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.230803    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 03:05:52.238031 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.230806    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 03:05:52.238553 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.230808    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 03:05:52.238553 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.230810    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 03:05:52.238553 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.230813    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 03:05:52.238553 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.230815    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 03:05:52.238553 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.230818    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 03:05:52.238553 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.230820    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 03:05:52.238553 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.230823    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 03:05:52.238553 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.230825    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 03:05:52.238553 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.230828    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 03:05:52.238553 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.230830    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 03:05:52.238553 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.230832    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 03:05:52.238553 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.230836    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 03:05:52.238553 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.230839    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 03:05:52.238553 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.230842    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 03:05:52.238553 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.230845    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 03:05:52.238553 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.230847    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 03:05:52.238553 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.230850    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 03:05:52.238553 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.230852    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 03:05:52.238553 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.230855    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 03:05:52.239025 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.230857    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 03:05:52.239025 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.230860    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 03:05:52.239025 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.230862    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 03:05:52.239025 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.230866    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 03:05:52.239025 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.230869    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 03:05:52.239025 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.230873    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 03:05:52.239025 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.230877    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 03:05:52.239025 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.230880    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 03:05:52.239025 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.230883    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 03:05:52.239025 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.230885    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 03:05:52.239025 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.230888    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 03:05:52.239025 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.230890    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 03:05:52.239025 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.230893    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 03:05:52.239025 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.230895    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 03:05:52.239025 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.230898    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 03:05:52.239025 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.230901    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 03:05:52.239025 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.230903    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 03:05:52.239025 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.230906    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 03:05:52.239025 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.230908    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 03:05:52.239475 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.230910    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 03:05:52.239475 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.230913    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 03:05:52.239475 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.230916    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 03:05:52.239475 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.230918    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 03:05:52.239475 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.230920    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 03:05:52.239475 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.230924    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 03:05:52.239475 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.230926    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 03:05:52.239475 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.230930    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 03:05:52.239475 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.230932    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 03:05:52.239475 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.230935    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 03:05:52.239475 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.230937    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 03:05:52.239475 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.230940    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 03:05:52.239475 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.230942    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 03:05:52.239475 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.230944    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 03:05:52.239475 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.230947    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 03:05:52.239475 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.230949    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 03:05:52.239475 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.230952    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 03:05:52.239475 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.230954    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 03:05:52.239475 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.230957    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 03:05:52.239475 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.230959    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 03:05:52.239952 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.230962    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 03:05:52.239952 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.230965    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 03:05:52.239952 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.230968    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 03:05:52.239952 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.230970    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 03:05:52.239952 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.230973    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 03:05:52.239952 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.230975    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 03:05:52.239952 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.230978    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 03:05:52.239952 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.230980    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 03:05:52.239952 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.230982    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 03:05:52.239952 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.230985    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 03:05:52.239952 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.230987    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 03:05:52.239952 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.230990    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 03:05:52.239952 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.230992    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 03:05:52.239952 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.230994    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 03:05:52.239952 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.230997    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 03:05:52.239952 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.230999    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 03:05:52.239952 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231002    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 03:05:52.239952 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231005    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 03:05:52.239952 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231008    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 03:05:52.239952 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231011    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 03:05:52.240419 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231014    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 03:05:52.240419 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231016    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 03:05:52.240419 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231019    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 03:05:52.240419 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231021    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 03:05:52.240419 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231024    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 03:05:52.240419 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231026    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 03:05:52.240419 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231029    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 03:05:52.240419 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231031    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 03:05:52.240419 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231034    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 03:05:52.240419 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231036    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 03:05:52.240419 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231039    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 03:05:52.240419 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231042    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 03:05:52.240419 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231044    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 03:05:52.240419 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.231078    2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 03:05:52.240419 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231119    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 03:05:52.240824 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231123    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 03:05:52.240824 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231126    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 03:05:52.240824 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231129    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 03:05:52.240824 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231131    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 03:05:52.240824 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231134    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 03:05:52.240824 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231137    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 03:05:52.240824 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231139    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 03:05:52.240824 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231141    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 03:05:52.240824 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231144    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 03:05:52.240824 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231146    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 03:05:52.240824 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231149    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 03:05:52.240824 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231151    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 03:05:52.240824 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231155    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 03:05:52.240824 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231159    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 03:05:52.240824 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231163    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 03:05:52.240824 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231167    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 03:05:52.240824 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231170    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 03:05:52.240824 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231174    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 03:05:52.240824 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231177    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 03:05:52.241268 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231180    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 03:05:52.241268 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231182    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 03:05:52.241268 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231185    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 03:05:52.241268 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231187    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 03:05:52.241268 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231190    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 03:05:52.241268 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231192    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 03:05:52.241268 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231195    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 03:05:52.241268 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231197    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 03:05:52.241268 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231200    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 03:05:52.241268 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231202    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 03:05:52.241268 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231205    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 03:05:52.241268 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231207    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 03:05:52.241268 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231210    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 03:05:52.241268 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231212    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 03:05:52.241268 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231215    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 03:05:52.241268 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231218    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 03:05:52.241268 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231220    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 03:05:52.241268 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231223    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 03:05:52.241268 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231225    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 03:05:52.241268 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231227    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 03:05:52.241748 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231230    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 03:05:52.241748 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231232    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 03:05:52.241748 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231234    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 03:05:52.241748 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231237    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 03:05:52.241748 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231239    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 03:05:52.241748 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231242    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 03:05:52.241748 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231244    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 03:05:52.241748 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231247    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 03:05:52.241748 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231249    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 03:05:52.241748 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231252    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 03:05:52.241748 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231256    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 03:05:52.241748 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231258    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 03:05:52.241748 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231261    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 03:05:52.241748 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231264    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 03:05:52.241748 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231266    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 03:05:52.241748 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231269    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 03:05:52.241748 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231272    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 03:05:52.241748 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231274    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 03:05:52.241748 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231277    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 03:05:52.241748 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231279    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 03:05:52.242208 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231282    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 03:05:52.242208 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231284    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 03:05:52.242208 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231286    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 03:05:52.242208 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231289    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 03:05:52.242208 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231291    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 03:05:52.242208 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231294    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 03:05:52.242208 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231297    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 03:05:52.242208 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231300    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 03:05:52.242208 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231302    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 03:05:52.242208 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231304    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 03:05:52.242208 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231307    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 03:05:52.242208 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231309    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 03:05:52.242208 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231312    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 03:05:52.242208 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231314    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 03:05:52.242208 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231317    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 03:05:52.242208 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231319    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 03:05:52.242208 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231321    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 03:05:52.242208 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231324    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 03:05:52.242208 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231326    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 03:05:52.242633 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231329    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 03:05:52.242633 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231331    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 03:05:52.242633 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231333    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 03:05:52.242633 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231337    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 03:05:52.242633 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231340    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 03:05:52.242633 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231342    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 03:05:52.242633 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231344    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 03:05:52.242633 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231347    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 03:05:52.242633 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231350    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 03:05:52.242633 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231352    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 03:05:52.242633 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231355    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 03:05:52.242633 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231357    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 03:05:52.242633 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231359    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 03:05:52.242633 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231362    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 03:05:52.242633 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231364    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 03:05:52.242633 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231368    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 03:05:52.242633 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231370    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 03:05:52.242633 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231373    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 03:05:52.242633 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231376    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 03:05:52.242633 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231378    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 03:05:52.243118 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231381    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 03:05:52.243118 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231383    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 03:05:52.243118 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231385    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 03:05:52.243118 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231388    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 03:05:52.243118 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231390    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 03:05:52.243118 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231393    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 03:05:52.243118 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.231395    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 03:05:52.243118 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.238850    2576 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 01 03:05:52.243118 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.238865    2576 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 01 03:05:52.243118 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.238879    2576 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 01 03:05:52.243118 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.238891    2576 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 01 03:05:52.243118 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.238946    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 03:05:52.243118 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.238952    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 03:05:52.243118 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.238955    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 03:05:52.243118 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.238957    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 03:05:52.243118 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.238960    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 03:05:52.243118 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.238963    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 03:05:52.243118 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.238965    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 03:05:52.243118 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.238968    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 03:05:52.243118 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.238970    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 03:05:52.243582 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.238973    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 03:05:52.243582 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.238975    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 03:05:52.243582 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.238978    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 03:05:52.243582 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.238980    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 03:05:52.243582 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.238983    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 03:05:52.243582 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.238985    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 03:05:52.243582 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.238987    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 03:05:52.243582 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.238990    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 03:05:52.243582 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.238993    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 03:05:52.243582 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.238995    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 03:05:52.243582 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239011    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 03:05:52.243582 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239015    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 03:05:52.243582 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239018    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 03:05:52.243582 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239021    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 03:05:52.243582 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239024    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 03:05:52.243582 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239027    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 03:05:52.243582 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239031    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 03:05:52.243582 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239033    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 03:05:52.243582 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239036    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 03:05:52.244058 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239039    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 03:05:52.244058 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239048    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 03:05:52.244058 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239051    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 03:05:52.244058 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239062    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 03:05:52.244058 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239066    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 03:05:52.244058 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239068    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 03:05:52.244058 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239071    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 03:05:52.244058 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239074    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 03:05:52.244058 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239076    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 03:05:52.244058 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239079    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 03:05:52.244058 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239082    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 03:05:52.244058 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239084    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 03:05:52.244058 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239086    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 03:05:52.244058 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239089    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 03:05:52.244058 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239091    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 03:05:52.244058 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239094    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 03:05:52.244058 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239096    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 03:05:52.244058 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239099    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 03:05:52.244058 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239101    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 03:05:52.244493 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239104    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 03:05:52.244493 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239106    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 03:05:52.244493 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239109    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 03:05:52.244493 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239111    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 03:05:52.244493 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239114    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 03:05:52.244493 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239116    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 03:05:52.244493 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239119    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 03:05:52.244493 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239123    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 03:05:52.244493 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239127    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 03:05:52.244493 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239130    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 03:05:52.244493 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239139    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 03:05:52.244493 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239141    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 03:05:52.244493 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239144    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 03:05:52.244493 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239147    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 03:05:52.244493 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239150    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 03:05:52.244493 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239158    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 03:05:52.244493 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239161    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 03:05:52.244493 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239163    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 03:05:52.244493 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239166    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 03:05:52.245058 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239168    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 03:05:52.245058 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239171    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 03:05:52.245058 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239181    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 03:05:52.245058 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239186    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 03:05:52.245058 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239189    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 03:05:52.245058 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239191    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 03:05:52.245058 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239194    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 03:05:52.245058 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239197    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 03:05:52.245058 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239199    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 03:05:52.245058 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239203    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 03:05:52.245058 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239208    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 03:05:52.245058 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239211    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 03:05:52.245058 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239214    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 03:05:52.245058 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239217    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 03:05:52.245058 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239220    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 03:05:52.245058 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239223    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 03:05:52.245058 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239225    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 03:05:52.245058 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239228    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 03:05:52.245058 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239230    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 03:05:52.245486 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239233    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 03:05:52.245486 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239236    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 03:05:52.245486 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239238    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 03:05:52.245486 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239241    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 03:05:52.245486 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239243    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 03:05:52.245486 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239246    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 03:05:52.245486 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239248    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 03:05:52.245486 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239250    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 03:05:52.245486 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239253    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 03:05:52.245486 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239256    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 03:05:52.245486 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239264    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 03:05:52.245486 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239266    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 03:05:52.245486 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239269    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 03:05:52.245486 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239271    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 03:05:52.245486 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239274    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 03:05:52.245486 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239276    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 03:05:52.245486 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239279    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 03:05:52.245486 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239281    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 03:05:52.245486 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239284    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 03:05:52.245486 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239286    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 03:05:52.245994 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239289    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 03:05:52.245994 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.239316    2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 03:05:52.245994 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239502    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 03:05:52.245994 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239506    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 03:05:52.245994 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239509    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 03:05:52.245994 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239512    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 03:05:52.245994 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239515    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 03:05:52.245994 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239517    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 03:05:52.245994 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239519    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 03:05:52.245994 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239522    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 03:05:52.245994 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239524    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 03:05:52.245994 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239527    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 03:05:52.245994 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239530    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 03:05:52.245994 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239533    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 03:05:52.245994 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239536    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 03:05:52.245994 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239540    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 03:05:52.246360 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239542    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 03:05:52.246360 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239545    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 03:05:52.246360 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239547    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 03:05:52.246360 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239550    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 03:05:52.246360 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239553    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 03:05:52.246360 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239555    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 03:05:52.246360 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239558    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 03:05:52.246360 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239567    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 03:05:52.246360 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239569    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 03:05:52.246360 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239572    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 03:05:52.246360 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239575    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 03:05:52.246360 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239577    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 03:05:52.246360 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239580    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 03:05:52.246360 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239583    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 03:05:52.246360 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239585    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 03:05:52.246360 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239587    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 03:05:52.246360 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239590    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 03:05:52.246360 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239592    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 03:05:52.246360 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239595    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 03:05:52.246360 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239597    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 03:05:52.246836 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239600    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 03:05:52.246836 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239602    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 03:05:52.246836 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239605    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 03:05:52.246836 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239607    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 03:05:52.246836 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239610    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 03:05:52.246836 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239612    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 03:05:52.246836 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239615    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 03:05:52.246836 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239617    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 03:05:52.246836 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239620    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 03:05:52.246836 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239622    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 03:05:52.246836 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239624    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 03:05:52.246836 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239627    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 03:05:52.246836 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239630    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 03:05:52.246836 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239633    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 03:05:52.246836 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239635    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 03:05:52.246836 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239637    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 03:05:52.246836 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239640    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 03:05:52.246836 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239643    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 03:05:52.246836 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239645    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 03:05:52.247315 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239647    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 03:05:52.247315 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239655    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 03:05:52.247315 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239658    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 03:05:52.247315 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239661    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 03:05:52.247315 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239663    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 03:05:52.247315 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239666    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 03:05:52.247315 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239668    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 03:05:52.247315 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239670    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 03:05:52.247315 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239673    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 03:05:52.247315 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239675    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 03:05:52.247315 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239677    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 03:05:52.247315 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239700    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 03:05:52.247315 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239704    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 03:05:52.247315 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239706    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 03:05:52.247315 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239709    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 03:05:52.247315 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239711    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 03:05:52.247315 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239714    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 03:05:52.247315 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239716    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 03:05:52.247315 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239719    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 03:05:52.247789 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239722    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 03:05:52.247789 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239724    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 03:05:52.247789 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239728    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 03:05:52.247789 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239732    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 03:05:52.247789 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239735    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 03:05:52.247789 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239737    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 03:05:52.247789 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239740    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 03:05:52.247789 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239742    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 03:05:52.247789 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239745    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 03:05:52.247789 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239747    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 03:05:52.247789 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239750    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 03:05:52.247789 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239752    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 03:05:52.247789 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239754    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 03:05:52.247789 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239757    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 03:05:52.247789 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239759    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 03:05:52.247789 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239767    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 03:05:52.247789 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239770    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 03:05:52.247789 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239773    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 03:05:52.247789 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239775    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 03:05:52.247789 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239777    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 03:05:52.248243 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239780    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 03:05:52.248243 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239782    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 03:05:52.248243 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239785    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 03:05:52.248243 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239787    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 03:05:52.248243 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239789    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 03:05:52.248243 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239793    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 03:05:52.248243 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239795    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 03:05:52.248243 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239797    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 03:05:52.248243 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239800    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 03:05:52.248243 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239802    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 03:05:52.248243 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239805    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 03:05:52.248243 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239807    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 03:05:52.248243 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239809    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 03:05:52.248243 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239812    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 03:05:52.248243 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.239836    2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 03:05:52.248591 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239874    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 03:05:52.248591 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239878    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 03:05:52.248591 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239881    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 03:05:52.248591 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239884    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 03:05:52.248591 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239886    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 03:05:52.248591 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239889    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 03:05:52.248591 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239891    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 03:05:52.248591 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239894    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 03:05:52.248591 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239896    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 03:05:52.248591 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239899    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 03:05:52.248591 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239901    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 03:05:52.248591 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239904    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 03:05:52.248591 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239906    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 03:05:52.248591 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239914    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 03:05:52.248591 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239917    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 03:05:52.248591 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239919    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 03:05:52.248591 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239922    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 03:05:52.248591 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239924    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 03:05:52.248591 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239927    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 03:05:52.248591 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239929    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 03:05:52.249073 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239931    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 03:05:52.249073 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239934    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 03:05:52.249073 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239939    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 03:05:52.249073 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239941    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 03:05:52.249073 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239943    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 03:05:52.249073 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239946    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 03:05:52.249073 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239948    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 03:05:52.249073 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239951    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 03:05:52.249073 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239953    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 03:05:52.249073 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239956    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 03:05:52.249073 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239959    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 03:05:52.249073 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239961    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 03:05:52.249073 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239964    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 03:05:52.249073 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239966    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 03:05:52.249073 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239969    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 03:05:52.249073 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239971    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 03:05:52.249073 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239974    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 03:05:52.249073 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239976    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 03:05:52.249073 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239979    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 03:05:52.249073 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239981    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 03:05:52.249530 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239984    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 03:05:52.249530 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239987    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 03:05:52.249530 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239991    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 03:05:52.249530 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239994    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 03:05:52.249530 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.239997    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 03:05:52.249530 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.240000    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 03:05:52.249530 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.240008    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 03:05:52.249530 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.240011    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 03:05:52.249530 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.240013    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 03:05:52.249530 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.240015    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 03:05:52.249530 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.240018    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 03:05:52.249530 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.240020    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 03:05:52.249530 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.240023    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 03:05:52.249530 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.240026    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 03:05:52.249530 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.240028    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 03:05:52.249530 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.240031    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 03:05:52.249530 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.240034    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 03:05:52.249530 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.240036    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 03:05:52.249530 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.240039    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 03:05:52.249974 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.240041    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 03:05:52.249974 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.240043    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 03:05:52.249974 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.240046    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 03:05:52.249974 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.240048    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 03:05:52.249974 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.240051    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 03:05:52.249974 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.240053    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 03:05:52.249974 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.240055    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 03:05:52.249974 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.240058    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 03:05:52.249974 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.240060    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 03:05:52.249974 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.240070    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 03:05:52.249974 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.240072    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 03:05:52.249974 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.240075    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 03:05:52.249974 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.240077    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 03:05:52.249974 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.240079    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 03:05:52.249974 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.240083    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 03:05:52.249974 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.240085    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 03:05:52.249974 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.240087    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 03:05:52.249974 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.240090    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 03:05:52.249974 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.240092    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 03:05:52.249974 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.240095    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 03:05:52.250459 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.240102    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 03:05:52.250459 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.240105    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 03:05:52.250459 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.240108    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 03:05:52.250459 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.240110    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 03:05:52.250459 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.240112    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 03:05:52.250459 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.240115    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 03:05:52.250459 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.240117    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 03:05:52.250459 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.240120    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 03:05:52.250459 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.240122    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 03:05:52.250459 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.240124    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 03:05:52.250459 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.240127    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 03:05:52.250459 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.240130    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 03:05:52.250459 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.240132    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 03:05:52.250459 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.240134    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 03:05:52.250459 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.240137    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 03:05:52.250459 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.240139    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 03:05:52.250459 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.240141    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 03:05:52.250459 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.240143    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 03:05:52.250459 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.240146    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 03:05:52.250909 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.240149    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 03:05:52.250909 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.240151    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 03:05:52.250909 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.240153    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 03:05:52.250909 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.240156    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 03:05:52.250909 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.240159    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 03:05:52.250909 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.240161    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 03:05:52.250909 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.240165    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 03:05:52.250909 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:52.240169    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 03:05:52.250909 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.240220    2576 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 01 03:05:52.250909 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.242373    2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 01 03:05:52.250909 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.243278    2576 server.go:1014] "Starting client certificate rotation"
Nov 01 03:05:52.250909 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.243372    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 01 03:05:52.250909 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.244059    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 01 03:05:52.250909 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.245542    2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 01 03:05:52.250909 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.246852    2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 01 03:05:52.257939 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.257921    2576 log.go:25] "Validated CRI v1 runtime API"
Nov 01 03:05:52.261593 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.261578    2576 log.go:25] "Validated CRI v1 image API"
Nov 01 03:05:52.262445 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.262415    2576 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 01 03:05:52.266140 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.266116    2576 fs.go:135] Filesystem UUIDs: map[2b49dd07-e358-4aa8-88e7-94f420cbf1f5:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 88d8d6f1-66d5-44a0-9804-56478685e0dd:/dev/nvme0n1p3]
Nov 01 03:05:52.266204 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.266140    2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 01 03:05:52.272071 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.271970    2576 manager.go:217] Machine: {Timestamp:2026-11-01 03:05:52.270514082 +0000 UTC m=+0.324655756 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3062349 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e0161c0260d3b36d00164f51ea9c1 SystemUUID:ec2e0161-c026-0d3b-36d0-0164f51ea9c1 BootID:27b2e57b-db8c-4735-8450-2b0385021ed3 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:7e:7e:3b:4e:e5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:7e:7e:3b:4e:e5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ee:d5:ea:90:14:ca Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 01 03:05:52.272071 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.272069    2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 01 03:05:52.272179 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.272164    2576 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 01 03:05:52.276765 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.276744    2576 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 01 03:05:52.276900 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.276766    2576 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-54.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 01 03:05:52.276941 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.276902    2576 topology_manager.go:143] "Creating topology manager with none policy"
Nov 01 03:05:52.276941 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.276909    2576 container_manager_linux.go:311] "Creating device plugin manager"
Nov 01 03:05:52.276991 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.276948    2576 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 01 03:05:52.276991 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.276965    2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 01 03:05:52.278054 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.278044    2576 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 01 03:05:52.282129 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.282117    2576 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 01 03:05:52.282242 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.282232    2576 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 01 03:05:52.282996 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.282986    2576 kubelet.go:493] "Attempting to sync node with API server"
Nov 01 03:05:52.283026 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.283002    2576 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 01 03:05:52.283026 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.283013    2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 01 03:05:52.283026 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.283022    2576 kubelet.go:405] "Adding apiserver pod source"
Nov 01 03:05:52.283136 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.283029    2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 01 03:05:52.283308 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.283297    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 01 03:05:52.285498 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.285478    2576 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 01 03:05:52.292336 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.292320    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 01 03:05:52.293320 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.293309    2576 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 01 03:05:52.293360 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.293345    2576 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 01 03:05:52.293496 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.293486    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 01 03:05:52.293524 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.293506    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 01 03:05:52.293524 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.293515    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 01 03:05:52.293524 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.293524    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 01 03:05:52.293596 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.293530    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 01 03:05:52.293596 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.293535    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 01 03:05:52.294248 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.294239    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 01 03:05:52.294276 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.294256    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 01 03:05:52.294276 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.294264    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 01 03:05:52.294276 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.294270    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 01 03:05:52.294349 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.294282    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 01 03:05:52.294349 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.294290    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 01 03:05:52.295404 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.295395    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 01 03:05:52.295436 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.295407    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 01 03:05:52.298977 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.298965    2576 server.go:1263] "Started kubelet"
Nov 01 03:05:52.299169 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.299096    2576 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 01 03:05:52.299568 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.299086    2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 01 03:05:52.299642 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.299610    2576 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 01 03:05:52.299785 ip-10-0-133-54 systemd[1]: Started Kubernetes Kubelet.
Nov 01 03:05:52.302257 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.302242    2576 server.go:317] "Adding debug handlers to kubelet server"
Nov 01 03:05:52.303088 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.303069    2576 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 01 03:05:52.305622 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:05:52.305605    2576 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 01 03:05:52.309064 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.308899    2576 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 01 03:05:52.309064 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.308926    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 01 03:05:52.309189 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:05:52.309063    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-54.us-west-1.compute.internal\" not found"
Nov 01 03:05:52.309593 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.309572    2576 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 01 03:05:52.309839 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.309596    2576 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 01 03:05:52.309839 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.309674    2576 reconstruct.go:97] "Volume reconstruction finished"
Nov 01 03:05:52.309839 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.309705    2576 reconciler.go:29] "Reconciler: start to sync state"
Nov 01 03:05:52.310033 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.309583    2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 01 03:05:52.310194 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.310172    2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 01 03:05:52.310194 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.310194    2576 factory.go:55] Registering systemd factory
Nov 01 03:05:52.310329 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.310202    2576 factory.go:223] Registration of the systemd container factory successfully
Nov 01 03:05:52.310445 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.310427    2576 factory.go:153] Registering CRI-O factory
Nov 01 03:05:52.310491 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.310447    2576 factory.go:223] Registration of the crio container factory successfully
Nov 01 03:05:52.310491 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.310466    2576 factory.go:103] Registering Raw factory
Nov 01 03:05:52.310491 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.310475    2576 manager.go:1199] Started watching for new ooms in manager
Nov 01 03:05:52.311408 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.311389    2576 manager.go:319] Starting recovery of all containers
Nov 01 03:05:52.321186 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.321170    2576 manager.go:324] Recovery completed
Nov 01 03:05:52.325603 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.325589    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 03:05:52.326929 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.326908    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-54.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 03:05:52.326979 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.326941    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-54.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 03:05:52.326979 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.326952    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-54.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 03:05:52.327429 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.327417    2576 cpu_manager.go:226] "Starting" policy="none"
Nov 01 03:05:52.327429 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.327427    2576 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 01 03:05:52.327536 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.327442    2576 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 01 03:05:52.327536 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.327450    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 03:05:52.328071 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.328058    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-54.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 03:05:52.328121 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.328082    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-54.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 03:05:52.328121 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.328093    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-54.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 03:05:52.329521 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.329510    2576 policy_none.go:50] "Start"
Nov 01 03:05:52.329556 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.329526    2576 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 01 03:05:52.329556 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.329536    2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 01 03:05:52.331608 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.331597    2576 policy_none.go:44] "Start"
Nov 01 03:05:52.396968 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.377795    2576 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 01 03:05:52.396968 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.386343    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 01 03:05:52.396968 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.387571    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 01 03:05:52.396968 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.387600    2576 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 01 03:05:52.396968 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.387623    2576 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 01 03:05:52.396968 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:05:52.387666    2576 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 01 03:05:52.396968 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.393736    2576 manager.go:341] "Starting Device Plugin manager"
Nov 01 03:05:52.396968 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:05:52.393766    2576 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 01 03:05:52.396968 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.393775    2576 server.go:87] "Starting device plugin registration server"
Nov 01 03:05:52.396968 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.393984    2576 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 01 03:05:52.396968 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.393994    2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 01 03:05:52.396968 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.394123    2576 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 01 03:05:52.396968 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.394264    2576 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 01 03:05:52.396968 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.394307    2576 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 01 03:05:52.396968 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:05:52.394625    2576 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 01 03:05:52.396968 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:05:52.394670    2576 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-54.us-west-1.compute.internal\" not found"
Nov 01 03:05:52.447079 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.447045    2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-54.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 01 03:05:52.447079 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:05:52.447058    2576 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-54.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Nov 01 03:05:52.448067 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:05:52.447048    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-54.us-west-1.compute.internal.18e3cce3149b1274  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-54.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-54.us-west-1.compute.internal,},FirstTimestamp:2026-11-01 03:05:52.298938996 +0000 UTC m=+0.353080669,LastTimestamp:2026-11-01 03:05:52.298938996 +0000 UTC m=+0.353080669,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-54.us-west-1.compute.internal,}"
Nov 01 03:05:52.488411 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.488368    2576 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-54.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-54.us-west-1.compute.internal"]
Nov 01 03:05:52.488475 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.488418    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 03:05:52.489249 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.489234    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-54.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 03:05:52.489303 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.489261    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-54.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 03:05:52.489303 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.489274    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-54.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 03:05:52.491462 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.491449    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 03:05:52.491605 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.491592    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-54.us-west-1.compute.internal"
Nov 01 03:05:52.491641 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.491619    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 03:05:52.492160 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.492144    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-54.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 03:05:52.492252 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.492169    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-54.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 03:05:52.492252 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.492179    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-54.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 03:05:52.492252 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.492181    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-54.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 03:05:52.492252 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.492203    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-54.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 03:05:52.492252 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.492218    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-54.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 03:05:52.494059 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.494044    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 03:05:52.494330 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.494317    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-54.us-west-1.compute.internal"
Nov 01 03:05:52.494399 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.494340    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 03:05:52.494799 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.494779    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-54.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 03:05:52.494891 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.494814    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-54.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 03:05:52.494891 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.494828    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-54.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 03:05:52.494891 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.494851    2576 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-54.us-west-1.compute.internal"
Nov 01 03:05:52.495196 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.495178    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-54.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 03:05:52.495278 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.495205    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-54.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 03:05:52.495278 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.495221    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-54.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 03:05:52.511251 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.511235    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f068e0e9089d99d5fd3ec23628aa7d71-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-54.us-west-1.compute.internal\" (UID: \"f068e0e9089d99d5fd3ec23628aa7d71\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-54.us-west-1.compute.internal"
Nov 01 03:05:52.511333 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.511258    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f068e0e9089d99d5fd3ec23628aa7d71-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-54.us-west-1.compute.internal\" (UID: \"f068e0e9089d99d5fd3ec23628aa7d71\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-54.us-west-1.compute.internal"
Nov 01 03:05:52.511333 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.511275    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8cf98fd570993ef6ee2b41a9af4899c8-config\") pod \"kube-apiserver-proxy-ip-10-0-133-54.us-west-1.compute.internal\" (UID: \"8cf98fd570993ef6ee2b41a9af4899c8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-54.us-west-1.compute.internal"
Nov 01 03:05:52.519165 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.519150    2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 01 03:05:52.519212 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.519196    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bpmlx"
Nov 01 03:05:52.532101 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:05:52.532086    2576 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-54.us-west-1.compute.internal\" not found" node="ip-10-0-133-54.us-west-1.compute.internal"
Nov 01 03:05:52.535084 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:05:52.535067    2576 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-54.us-west-1.compute.internal\" not found" node="ip-10-0-133-54.us-west-1.compute.internal"
Nov 01 03:05:52.547963 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:05:52.547940    2576 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Nov 01 03:05:52.548029 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:05:52.548002    2576 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass"
Nov 01 03:05:52.548216 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:05:52.548198    2576 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-54.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Nov 01 03:05:52.548332 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:05:52.548269    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-54.us-west-1.compute.internal.18e3cce316462a95  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-54.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-54.us-west-1.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-54.us-west-1.compute.internal,},FirstTimestamp:2026-11-01 03:05:52.326929045 +0000 UTC m=+0.381070718,LastTimestamp:2026-11-01 03:05:52.326929045 +0000 UTC m=+0.381070718,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-54.us-west-1.compute.internal,}"
Nov 01 03:05:52.596910 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:05:52.596888    2576 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Nov 01 03:05:52.611954 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.611934    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f068e0e9089d99d5fd3ec23628aa7d71-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-54.us-west-1.compute.internal\" (UID: \"f068e0e9089d99d5fd3ec23628aa7d71\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-54.us-west-1.compute.internal"
Nov 01 03:05:52.612017 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.611959    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f068e0e9089d99d5fd3ec23628aa7d71-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-54.us-west-1.compute.internal\" (UID: \"f068e0e9089d99d5fd3ec23628aa7d71\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-54.us-west-1.compute.internal"
Nov 01 03:05:52.612017 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.611976    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8cf98fd570993ef6ee2b41a9af4899c8-config\") pod \"kube-apiserver-proxy-ip-10-0-133-54.us-west-1.compute.internal\" (UID: \"8cf98fd570993ef6ee2b41a9af4899c8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-54.us-west-1.compute.internal"
Nov 01 03:05:52.612088 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.612026    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8cf98fd570993ef6ee2b41a9af4899c8-config\") pod \"kube-apiserver-proxy-ip-10-0-133-54.us-west-1.compute.internal\" (UID: \"8cf98fd570993ef6ee2b41a9af4899c8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-54.us-west-1.compute.internal"
Nov 01 03:05:52.612088 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.612041    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f068e0e9089d99d5fd3ec23628aa7d71-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-54.us-west-1.compute.internal\" (UID: \"f068e0e9089d99d5fd3ec23628aa7d71\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-54.us-west-1.compute.internal"
Nov 01 03:05:52.612088 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.612048    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f068e0e9089d99d5fd3ec23628aa7d71-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-54.us-west-1.compute.internal\" (UID: \"f068e0e9089d99d5fd3ec23628aa7d71\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-54.us-west-1.compute.internal"
Nov 01 03:05:52.648607 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:05:52.648589    2576 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-133-54.us-west-1.compute.internal\" not found" node="ip-10-0-133-54.us-west-1.compute.internal"
Nov 01 03:05:52.750849 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:05:52.750806    2576 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-133-54.us-west-1.compute.internal\" not found" node="ip-10-0-133-54.us-west-1.compute.internal"
Nov 01 03:05:52.836573 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.836556    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-54.us-west-1.compute.internal"
Nov 01 03:05:52.839742 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.839726    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-54.us-west-1.compute.internal"
Nov 01 03:05:52.849029 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.849012    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 03:05:52.849865 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.849849    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-54.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 03:05:52.849948 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.849877    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-54.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 03:05:52.849948 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.849888    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-54.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 03:05:52.849948 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.849907    2576 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-54.us-west-1.compute.internal"
Nov 01 03:05:52.907253 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:52.907236    2576 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-133-54.us-west-1.compute.internal"
Nov 01 03:05:52.907253 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:05:52.907254    2576 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-54.us-west-1.compute.internal\": node \"ip-10-0-133-54.us-west-1.compute.internal\" not found"
Nov 01 03:05:53.018168 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:05:53.018114    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-54.us-west-1.compute.internal\" not found"
Nov 01 03:05:53.119159 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:05:53.119137    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-54.us-west-1.compute.internal\" not found"
Nov 01 03:05:53.219577 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:05:53.219554    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-54.us-west-1.compute.internal\" not found"
Nov 01 03:05:53.243278 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:53.243260    2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 01 03:05:53.243613 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:05:53.243372    2576 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a28e94d7245cf43b9a1c835832cbb3a3-03ecbaf5388fcb74.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-133-54.us-west-1.compute.internal.18e3cce316466fd6\": read tcp 10.0.133.54:46698->52.15.232.74:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-133-54.us-west-1.compute.internal.18e3cce316466fd6  default   5651 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-54.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-133-54.us-west-1.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-133-54.us-west-1.compute.internal,},FirstTimestamp:2026-11-01 03:05:52 +0000 UTC,LastTimestamp:2026-11-01 03:05:52.492174783 +0000 UTC m=+0.546316458,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-54.us-west-1.compute.internal,}"
Nov 01 03:05:53.320109 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:05:53.320070    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-54.us-west-1.compute.internal\" not found"
Nov 01 03:05:53.420565 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:05:53.420547    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-54.us-west-1.compute.internal\" not found"
Nov 01 03:05:53.521073 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:05:53.521051    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-54.us-west-1.compute.internal\" not found"
Nov 01 03:05:53.525210 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:53.525189    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-31 03:00:52 +0000 UTC" deadline="2028-05-19 16:06:16.046828865 +0000 UTC"
Nov 01 03:05:53.525269 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:53.525212    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13573h0m22.521620517s"
Nov 01 03:05:53.621072 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:53.621033    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 03:05:53.621158 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:05:53.621101    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-54.us-west-1.compute.internal\" not found"
Nov 01 03:05:53.621884 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:53.621867    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-54.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 03:05:53.621954 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:53.621895    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-54.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 03:05:53.621954 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:53.621907    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-54.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 03:05:53.714164 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:53.714139    2576 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 03:05:53.721372 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:05:53.721355    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-54.us-west-1.compute.internal\" not found"
Nov 01 03:05:53.731241 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:53.731224    2576 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 03:05:53.789596 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:53.789579    2576 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 03:05:53.809657 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:53.809639    2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-54.us-west-1.compute.internal"
Nov 01 03:05:53.875143 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:53.875085    2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 03:05:53.876349 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:53.876312    2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-54.us-west-1.compute.internal"
Nov 01 03:05:53.878754 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:53.878737    2576 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 03:05:53.938040 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:53.938017    2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 03:05:53.948596 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:53.948568    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cf98fd570993ef6ee2b41a9af4899c8.slice/crio-d84526df1da0ed7187d5efa80942b6e3936499085b86291746b62ea9e3886525 WatchSource:0}: Error finding container d84526df1da0ed7187d5efa80942b6e3936499085b86291746b62ea9e3886525: Status 404 returned error can't find the container with id d84526df1da0ed7187d5efa80942b6e3936499085b86291746b62ea9e3886525
Nov 01 03:05:53.949216 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:53.949199    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf068e0e9089d99d5fd3ec23628aa7d71.slice/crio-7f86d26dee5abd851a44f92a5c77c0c1738cfb836092f63bd0db9c8840f8def0 WatchSource:0}: Error finding container 7f86d26dee5abd851a44f92a5c77c0c1738cfb836092f63bd0db9c8840f8def0: Status 404 returned error can't find the container with id 7f86d26dee5abd851a44f92a5c77c0c1738cfb836092f63bd0db9c8840f8def0
Nov 01 03:05:53.953979 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:53.953965    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 03:05:54.285658 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.285627    2576 apiserver.go:52] "Watching apiserver"
Nov 01 03:05:54.310517 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.310493    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 01 03:05:54.393539 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.393457    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-54.us-west-1.compute.internal" event={"ID":"f068e0e9089d99d5fd3ec23628aa7d71","Type":"ContainerStarted","Data":"7f86d26dee5abd851a44f92a5c77c0c1738cfb836092f63bd0db9c8840f8def0"}
Nov 01 03:05:54.394544 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.394517    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-54.us-west-1.compute.internal" event={"ID":"8cf98fd570993ef6ee2b41a9af4899c8","Type":"ContainerStarted","Data":"d84526df1da0ed7187d5efa80942b6e3936499085b86291746b62ea9e3886525"}
Nov 01 03:05:54.399930 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.399914    2576 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 01 03:05:54.400320 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.400299    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-lkbk6","openshift-image-registry/node-ca-wq27m","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-54.us-west-1.compute.internal","openshift-multus/multus-qzl9n","openshift-multus/network-metrics-daemon-dnqjp","kube-system/global-pull-secret-syncer-rrdzd","kube-system/konnectivity-agent-fh7sf","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vq4dj","openshift-multus/multus-additional-cni-plugins-qnrf8","openshift-network-diagnostics/network-check-target-mx48d","openshift-network-operator/iptables-alerter-wfg7k","openshift-ovn-kubernetes/ovnkube-node-bcqk9","kube-system/kube-apiserver-proxy-ip-10-0-133-54.us-west-1.compute.internal","openshift-dns/node-resolver-pzt6v"]
Nov 01 03:05:54.404279 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.404262    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-fh7sf"
Nov 01 03:05:54.406476 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.406453    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wfg7k"
Nov 01 03:05:54.408589 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.408571    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-pzt6v"
Nov 01 03:05:54.410866 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.410844    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qzl9n"
Nov 01 03:05:54.413190 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.413167    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dnqjp"
Nov 01 03:05:54.413304 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:05:54.413279    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dnqjp" podUID="e48a634f-ab31-435c-96c1-c3633c5bc061"
Nov 01 03:05:54.415520 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.415502    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rrdzd"
Nov 01 03:05:54.415591 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:05:54.415576    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rrdzd" podUID="0303d6c6-c4ec-4403-af74-f4bcb5cbf2a4"
Nov 01 03:05:54.417790 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.417773    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qnrf8"
Nov 01 03:05:54.420156 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.420138    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vq4dj"
Nov 01 03:05:54.422540 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.422524    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-wq27m"
Nov 01 03:05:54.422673 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.422652    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0ede655d-c20d-41cc-b5f0-2db4338c1ba7-multus-conf-dir\") pod \"multus-qzl9n\" (UID: \"0ede655d-c20d-41cc-b5f0-2db4338c1ba7\") " pod="openshift-multus/multus-qzl9n"
Nov 01 03:05:54.422740 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.422688    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0ede655d-c20d-41cc-b5f0-2db4338c1ba7-etc-kubernetes\") pod \"multus-qzl9n\" (UID: \"0ede655d-c20d-41cc-b5f0-2db4338c1ba7\") " pod="openshift-multus/multus-qzl9n"
Nov 01 03:05:54.422740 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.422715    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gljgq\" (UniqueName: \"kubernetes.io/projected/c9e89456-9acd-408f-87b5-f5fd968c82c3-kube-api-access-gljgq\") pod \"iptables-alerter-wfg7k\" (UID: \"c9e89456-9acd-408f-87b5-f5fd968c82c3\") " pod="openshift-network-operator/iptables-alerter-wfg7k"
Nov 01 03:05:54.422866 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.422799    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/994175bb-d192-47f7-b162-a840eb0f9be1-hosts-file\") pod \"node-resolver-pzt6v\" (UID: \"994175bb-d192-47f7-b162-a840eb0f9be1\") " pod="openshift-dns/node-resolver-pzt6v"
Nov 01 03:05:54.422901 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.422856    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0ede655d-c20d-41cc-b5f0-2db4338c1ba7-multus-socket-dir-parent\") pod \"multus-qzl9n\" (UID: \"0ede655d-c20d-41cc-b5f0-2db4338c1ba7\") " pod="openshift-multus/multus-qzl9n"
Nov 01 03:05:54.422901 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.422893    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0ede655d-c20d-41cc-b5f0-2db4338c1ba7-multus-daemon-config\") pod \"multus-qzl9n\" (UID: \"0ede655d-c20d-41cc-b5f0-2db4338c1ba7\") " pod="openshift-multus/multus-qzl9n"
Nov 01 03:05:54.422963 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.422932    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/bd8260bb-3811-4cdc-91d0-c6ddc693891e-agent-certs\") pod \"konnectivity-agent-fh7sf\" (UID: \"bd8260bb-3811-4cdc-91d0-c6ddc693891e\") " pod="kube-system/konnectivity-agent-fh7sf"
Nov 01 03:05:54.422995 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.422965    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c9e89456-9acd-408f-87b5-f5fd968c82c3-host-slash\") pod \"iptables-alerter-wfg7k\" (UID: \"c9e89456-9acd-408f-87b5-f5fd968c82c3\") " pod="openshift-network-operator/iptables-alerter-wfg7k"
Nov 01 03:05:54.422995 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.422986    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0ede655d-c20d-41cc-b5f0-2db4338c1ba7-host-var-lib-cni-bin\") pod \"multus-qzl9n\" (UID: \"0ede655d-c20d-41cc-b5f0-2db4338c1ba7\") " pod="openshift-multus/multus-qzl9n"
Nov 01 03:05:54.423061 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.423004    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0ede655d-c20d-41cc-b5f0-2db4338c1ba7-host-var-lib-cni-multus\") pod \"multus-qzl9n\" (UID: \"0ede655d-c20d-41cc-b5f0-2db4338c1ba7\") " pod="openshift-multus/multus-qzl9n"
Nov 01 03:05:54.423061 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.423021    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0ede655d-c20d-41cc-b5f0-2db4338c1ba7-host-var-lib-kubelet\") pod \"multus-qzl9n\" (UID: \"0ede655d-c20d-41cc-b5f0-2db4338c1ba7\") " pod="openshift-multus/multus-qzl9n"
Nov 01 03:05:54.423061 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.423036    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0ede655d-c20d-41cc-b5f0-2db4338c1ba7-host-run-multus-certs\") pod \"multus-qzl9n\" (UID: \"0ede655d-c20d-41cc-b5f0-2db4338c1ba7\") " pod="openshift-multus/multus-qzl9n"
Nov 01 03:05:54.423061 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.423052    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/bd8260bb-3811-4cdc-91d0-c6ddc693891e-konnectivity-ca\") pod \"konnectivity-agent-fh7sf\" (UID: \"bd8260bb-3811-4cdc-91d0-c6ddc693891e\") " pod="kube-system/konnectivity-agent-fh7sf"
Nov 01 03:05:54.423179 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.423072    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c9e89456-9acd-408f-87b5-f5fd968c82c3-iptables-alerter-script\") pod \"iptables-alerter-wfg7k\" (UID: \"c9e89456-9acd-408f-87b5-f5fd968c82c3\") " pod="openshift-network-operator/iptables-alerter-wfg7k"
Nov 01 03:05:54.423179 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.423111    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/994175bb-d192-47f7-b162-a840eb0f9be1-tmp-dir\") pod \"node-resolver-pzt6v\" (UID: \"994175bb-d192-47f7-b162-a840eb0f9be1\") " pod="openshift-dns/node-resolver-pzt6v"
Nov 01 03:05:54.423179 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.423129    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0ede655d-c20d-41cc-b5f0-2db4338c1ba7-system-cni-dir\") pod \"multus-qzl9n\" (UID: \"0ede655d-c20d-41cc-b5f0-2db4338c1ba7\") " pod="openshift-multus/multus-qzl9n"
Nov 01 03:05:54.423179 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.423149    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0ede655d-c20d-41cc-b5f0-2db4338c1ba7-multus-cni-dir\") pod \"multus-qzl9n\" (UID: \"0ede655d-c20d-41cc-b5f0-2db4338c1ba7\") " pod="openshift-multus/multus-qzl9n"
Nov 01 03:05:54.423179 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.423165    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0ede655d-c20d-41cc-b5f0-2db4338c1ba7-cni-binary-copy\") pod \"multus-qzl9n\" (UID: \"0ede655d-c20d-41cc-b5f0-2db4338c1ba7\") " pod="openshift-multus/multus-qzl9n"
Nov 01 03:05:54.423375 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.423188    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0ede655d-c20d-41cc-b5f0-2db4338c1ba7-host-run-k8s-cni-cncf-io\") pod \"multus-qzl9n\" (UID: \"0ede655d-c20d-41cc-b5f0-2db4338c1ba7\") " pod="openshift-multus/multus-qzl9n"
Nov 01 03:05:54.423375 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.423210    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slnpd\" (UniqueName: \"kubernetes.io/projected/0ede655d-c20d-41cc-b5f0-2db4338c1ba7-kube-api-access-slnpd\") pod \"multus-qzl9n\" (UID: \"0ede655d-c20d-41cc-b5f0-2db4338c1ba7\") " pod="openshift-multus/multus-qzl9n"
Nov 01 03:05:54.423375 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.423245    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtjjf\" (UniqueName: \"kubernetes.io/projected/994175bb-d192-47f7-b162-a840eb0f9be1-kube-api-access-xtjjf\") pod \"node-resolver-pzt6v\" (UID: \"994175bb-d192-47f7-b162-a840eb0f9be1\") " pod="openshift-dns/node-resolver-pzt6v"
Nov 01 03:05:54.423375 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.423261    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0ede655d-c20d-41cc-b5f0-2db4338c1ba7-cnibin\") pod \"multus-qzl9n\" (UID: \"0ede655d-c20d-41cc-b5f0-2db4338c1ba7\") " pod="openshift-multus/multus-qzl9n"
Nov 01 03:05:54.423375 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.423276    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0ede655d-c20d-41cc-b5f0-2db4338c1ba7-host-run-netns\") pod \"multus-qzl9n\" (UID: \"0ede655d-c20d-41cc-b5f0-2db4338c1ba7\") " pod="openshift-multus/multus-qzl9n"
Nov 01 03:05:54.423375 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.423289    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0ede655d-c20d-41cc-b5f0-2db4338c1ba7-hostroot\") pod \"multus-qzl9n\" (UID: \"0ede655d-c20d-41cc-b5f0-2db4338c1ba7\") " pod="openshift-multus/multus-qzl9n"
Nov 01 03:05:54.424912 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.424897    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mx48d"
Nov 01 03:05:54.424985 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:05:54.424957    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mx48d" podUID="6e302b40-9c43-4822-8c8b-4d021a0590ab"
Nov 01 03:05:54.427310 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.427293    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lkbk6"
Nov 01 03:05:54.429950 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.429934    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:05:54.464232 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.464212    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 01 03:05:54.465231 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.465209    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-kbjpv\""
Nov 01 03:05:54.465231 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.465224    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 01 03:05:54.465394 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.465311    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 01 03:05:54.465478 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.465464    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 01 03:05:54.465540 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.465470    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-6592m\""
Nov 01 03:05:54.465595 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.465582    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 01 03:05:54.467189 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.467171    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-77n5m\""
Nov 01 03:05:54.467623 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.467605    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 01 03:05:54.467745 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.467701    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 01 03:05:54.473058 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.473014    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 01 03:05:54.473145 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.473093    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 01 03:05:54.473210 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.473159    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-vzcml\""
Nov 01 03:05:54.473210 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.473164    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 01 03:05:54.473906 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.473892    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 01 03:05:54.475640 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.475626    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 01 03:05:54.475640 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.475633    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 01 03:05:54.475745 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.475640    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-hrmfk\""
Nov 01 03:05:54.478442 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.478428    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 01 03:05:54.479644 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.479630    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 01 03:05:54.480399 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.480384    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 01 03:05:54.481096 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.481079    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 01 03:05:54.481177 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.481118    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-62njk\""
Nov 01 03:05:54.481177 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.481131    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-mt5gq\""
Nov 01 03:05:54.481965 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.481950    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 01 03:05:54.482231 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.482220    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 01 03:05:54.486058 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.486042    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 01 03:05:54.486183 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.486170    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 01 03:05:54.486343 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.486329    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-4hc7l\""
Nov 01 03:05:54.495317 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.495295    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 01 03:05:54.495317 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.495297    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-ksxvj\""
Nov 01 03:05:54.495730 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.495712    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 01 03:05:54.495730 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.495723    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 01 03:05:54.495868 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.495752    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 01 03:05:54.511065 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.511046    2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 01 03:05:54.519028 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.519004    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 01 03:05:54.519028 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.519007    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 01 03:05:54.519135 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.519097    2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 01 03:05:54.519186 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.519138    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-5ntmh"
Nov 01 03:05:54.524512 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.524496    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e8f2a24b-e770-4615-a2c7-b2d065e14f77-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qnrf8\" (UID: \"e8f2a24b-e770-4615-a2c7-b2d065e14f77\") " pod="openshift-multus/multus-additional-cni-plugins-qnrf8"
Nov 01 03:05:54.524565 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.524528    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/994175bb-d192-47f7-b162-a840eb0f9be1-tmp-dir\") pod \"node-resolver-pzt6v\" (UID: \"994175bb-d192-47f7-b162-a840eb0f9be1\") " pod="openshift-dns/node-resolver-pzt6v"
Nov 01 03:05:54.524565 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.524552    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0ede655d-c20d-41cc-b5f0-2db4338c1ba7-multus-cni-dir\") pod \"multus-qzl9n\" (UID: \"0ede655d-c20d-41cc-b5f0-2db4338c1ba7\") " pod="openshift-multus/multus-qzl9n"
Nov 01 03:05:54.524666 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.524569    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0ede655d-c20d-41cc-b5f0-2db4338c1ba7-host-run-k8s-cni-cncf-io\") pod \"multus-qzl9n\" (UID: \"0ede655d-c20d-41cc-b5f0-2db4338c1ba7\") " pod="openshift-multus/multus-qzl9n"
Nov 01 03:05:54.524666 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.524597    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0ede655d-c20d-41cc-b5f0-2db4338c1ba7-host-run-k8s-cni-cncf-io\") pod \"multus-qzl9n\" (UID: \"0ede655d-c20d-41cc-b5f0-2db4338c1ba7\") " pod="openshift-multus/multus-qzl9n"
Nov 01 03:05:54.524666 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.524625    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slnpd\" (UniqueName: \"kubernetes.io/projected/0ede655d-c20d-41cc-b5f0-2db4338c1ba7-kube-api-access-slnpd\") pod \"multus-qzl9n\" (UID: \"0ede655d-c20d-41cc-b5f0-2db4338c1ba7\") " pod="openshift-multus/multus-qzl9n"
Nov 01 03:05:54.524836 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.524661    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/de5f469f-1029-4161-bf51-6ad18a3ac9e2-host-run-netns\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:05:54.524836 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.524709    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/de5f469f-1029-4161-bf51-6ad18a3ac9e2-ovn-node-metrics-cert\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:05:54.524836 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.524739    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e8f2a24b-e770-4615-a2c7-b2d065e14f77-cni-binary-copy\") pod \"multus-additional-cni-plugins-qnrf8\" (UID: \"e8f2a24b-e770-4615-a2c7-b2d065e14f77\") " pod="openshift-multus/multus-additional-cni-plugins-qnrf8"
Nov 01 03:05:54.524836 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.524769    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85r94\" (UniqueName: \"kubernetes.io/projected/e8f2a24b-e770-4615-a2c7-b2d065e14f77-kube-api-access-85r94\") pod \"multus-additional-cni-plugins-qnrf8\" (UID: \"e8f2a24b-e770-4615-a2c7-b2d065e14f77\") " pod="openshift-multus/multus-additional-cni-plugins-qnrf8"
Nov 01 03:05:54.524836 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.524777    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0ede655d-c20d-41cc-b5f0-2db4338c1ba7-multus-cni-dir\") pod \"multus-qzl9n\" (UID: \"0ede655d-c20d-41cc-b5f0-2db4338c1ba7\") " pod="openshift-multus/multus-qzl9n"
Nov 01 03:05:54.525064 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.524838    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0ede655d-c20d-41cc-b5f0-2db4338c1ba7-cnibin\") pod \"multus-qzl9n\" (UID: \"0ede655d-c20d-41cc-b5f0-2db4338c1ba7\") " pod="openshift-multus/multus-qzl9n"
Nov 01 03:05:54.525064 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.524863    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/994175bb-d192-47f7-b162-a840eb0f9be1-tmp-dir\") pod \"node-resolver-pzt6v\" (UID: \"994175bb-d192-47f7-b162-a840eb0f9be1\") " pod="openshift-dns/node-resolver-pzt6v"
Nov 01 03:05:54.525064 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.524875    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/11fb87e4-ef01-4209-8207-80b66ab09f5e-tmp\") pod \"tuned-lkbk6\" (UID: \"11fb87e4-ef01-4209-8207-80b66ab09f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-lkbk6"
Nov 01 03:05:54.525064 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.524922    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0303d6c6-c4ec-4403-af74-f4bcb5cbf2a4-kubelet-config\") pod \"global-pull-secret-syncer-rrdzd\" (UID: \"0303d6c6-c4ec-4403-af74-f4bcb5cbf2a4\") " pod="kube-system/global-pull-secret-syncer-rrdzd"
Nov 01 03:05:54.525064 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.524931    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0ede655d-c20d-41cc-b5f0-2db4338c1ba7-cnibin\") pod \"multus-qzl9n\" (UID: \"0ede655d-c20d-41cc-b5f0-2db4338c1ba7\") " pod="openshift-multus/multus-qzl9n"
Nov 01 03:05:54.525064 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.524946    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/de5f469f-1029-4161-bf51-6ad18a3ac9e2-ovnkube-script-lib\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:05:54.525064 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.524980    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gljgq\" (UniqueName: \"kubernetes.io/projected/c9e89456-9acd-408f-87b5-f5fd968c82c3-kube-api-access-gljgq\") pod \"iptables-alerter-wfg7k\" (UID: \"c9e89456-9acd-408f-87b5-f5fd968c82c3\") " pod="openshift-network-operator/iptables-alerter-wfg7k"
Nov 01 03:05:54.525064 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.525009    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0ede655d-c20d-41cc-b5f0-2db4338c1ba7-multus-daemon-config\") pod \"multus-qzl9n\" (UID: \"0ede655d-c20d-41cc-b5f0-2db4338c1ba7\") " pod="openshift-multus/multus-qzl9n"
Nov 01 03:05:54.525064 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.525036    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/11fb87e4-ef01-4209-8207-80b66ab09f5e-host\") pod \"tuned-lkbk6\" (UID: \"11fb87e4-ef01-4209-8207-80b66ab09f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-lkbk6"
Nov 01 03:05:54.525064 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.525063    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bf9dac1d-d9fc-455a-be03-ea34f8838730-socket-dir\") pod \"aws-ebs-csi-driver-node-vq4dj\" (UID: \"bf9dac1d-d9fc-455a-be03-ea34f8838730\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vq4dj"
Nov 01 03:05:54.525455 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.525087    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/de5f469f-1029-4161-bf51-6ad18a3ac9e2-run-ovn\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:05:54.525455 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.525112    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/de5f469f-1029-4161-bf51-6ad18a3ac9e2-log-socket\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:05:54.525455 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.525140    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d7cef651-3047-4e64-ae7f-b0fa71fbd0fe-serviceca\") pod \"node-ca-wq27m\" (UID: \"d7cef651-3047-4e64-ae7f-b0fa71fbd0fe\") " pod="openshift-image-registry/node-ca-wq27m"
Nov 01 03:05:54.525455 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.525165    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/11fb87e4-ef01-4209-8207-80b66ab09f5e-run\") pod \"tuned-lkbk6\" (UID: \"11fb87e4-ef01-4209-8207-80b66ab09f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-lkbk6"
Nov 01 03:05:54.525455 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.525211    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/11fb87e4-ef01-4209-8207-80b66ab09f5e-sys\") pod \"tuned-lkbk6\" (UID: \"11fb87e4-ef01-4209-8207-80b66ab09f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-lkbk6"
Nov 01 03:05:54.525455 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.525237    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h58g6\" (UniqueName: \"kubernetes.io/projected/11fb87e4-ef01-4209-8207-80b66ab09f5e-kube-api-access-h58g6\") pod \"tuned-lkbk6\" (UID: \"11fb87e4-ef01-4209-8207-80b66ab09f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-lkbk6"
Nov 01 03:05:54.525455 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.525261    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e8f2a24b-e770-4615-a2c7-b2d065e14f77-system-cni-dir\") pod \"multus-additional-cni-plugins-qnrf8\" (UID: \"e8f2a24b-e770-4615-a2c7-b2d065e14f77\") " pod="openshift-multus/multus-additional-cni-plugins-qnrf8"
Nov 01 03:05:54.525455 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.525275    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e8f2a24b-e770-4615-a2c7-b2d065e14f77-cnibin\") pod \"multus-additional-cni-plugins-qnrf8\" (UID: \"e8f2a24b-e770-4615-a2c7-b2d065e14f77\") " pod="openshift-multus/multus-additional-cni-plugins-qnrf8"
Nov 01 03:05:54.525455 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.525293    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/bd8260bb-3811-4cdc-91d0-c6ddc693891e-konnectivity-ca\") pod \"konnectivity-agent-fh7sf\" (UID: \"bd8260bb-3811-4cdc-91d0-c6ddc693891e\") " pod="kube-system/konnectivity-agent-fh7sf"
Nov 01 03:05:54.525455 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.525314    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c9e89456-9acd-408f-87b5-f5fd968c82c3-iptables-alerter-script\") pod \"iptables-alerter-wfg7k\" (UID: \"c9e89456-9acd-408f-87b5-f5fd968c82c3\") " pod="openshift-network-operator/iptables-alerter-wfg7k"
Nov 01 03:05:54.525455 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.525339    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de5f469f-1029-4161-bf51-6ad18a3ac9e2-var-lib-openvswitch\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:05:54.525455 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.525363    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/de5f469f-1029-4161-bf51-6ad18a3ac9e2-env-overrides\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:05:54.525455 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.525390    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e48a634f-ab31-435c-96c1-c3633c5bc061-metrics-certs\") pod \"network-metrics-daemon-dnqjp\" (UID: \"e48a634f-ab31-435c-96c1-c3633c5bc061\") " pod="openshift-multus/network-metrics-daemon-dnqjp"
Nov 01 03:05:54.525455 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.525416    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0ede655d-c20d-41cc-b5f0-2db4338c1ba7-multus-conf-dir\") pod \"multus-qzl9n\" (UID: \"0ede655d-c20d-41cc-b5f0-2db4338c1ba7\") " pod="openshift-multus/multus-qzl9n"
Nov 01 03:05:54.526016 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.525461    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0ede655d-c20d-41cc-b5f0-2db4338c1ba7-multus-conf-dir\") pod \"multus-qzl9n\" (UID: \"0ede655d-c20d-41cc-b5f0-2db4338c1ba7\") " pod="openshift-multus/multus-qzl9n"
Nov 01 03:05:54.526016 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.525459    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/11fb87e4-ef01-4209-8207-80b66ab09f5e-etc-kubernetes\") pod \"tuned-lkbk6\" (UID: \"11fb87e4-ef01-4209-8207-80b66ab09f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-lkbk6"
Nov 01 03:05:54.526016 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.525497    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/11fb87e4-ef01-4209-8207-80b66ab09f5e-etc-systemd\") pod \"tuned-lkbk6\" (UID: \"11fb87e4-ef01-4209-8207-80b66ab09f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-lkbk6"
Nov 01 03:05:54.526016 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.525539    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/de5f469f-1029-4161-bf51-6ad18a3ac9e2-host-cni-bin\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:05:54.526016 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.525566    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/de5f469f-1029-4161-bf51-6ad18a3ac9e2-host-cni-netd\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:05:54.526016 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.525589    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d7cef651-3047-4e64-ae7f-b0fa71fbd0fe-host\") pod \"node-ca-wq27m\" (UID: \"d7cef651-3047-4e64-ae7f-b0fa71fbd0fe\") " pod="openshift-image-registry/node-ca-wq27m"
Nov 01 03:05:54.526016 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.525590    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0ede655d-c20d-41cc-b5f0-2db4338c1ba7-multus-daemon-config\") pod \"multus-qzl9n\" (UID: \"0ede655d-c20d-41cc-b5f0-2db4338c1ba7\") " pod="openshift-multus/multus-qzl9n"
Nov 01 03:05:54.526016 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.525629    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/994175bb-d192-47f7-b162-a840eb0f9be1-hosts-file\") pod \"node-resolver-pzt6v\" (UID: \"994175bb-d192-47f7-b162-a840eb0f9be1\") " pod="openshift-dns/node-resolver-pzt6v"
Nov 01 03:05:54.526016 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.525660    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bf9dac1d-d9fc-455a-be03-ea34f8838730-etc-selinux\") pod \"aws-ebs-csi-driver-node-vq4dj\" (UID: \"bf9dac1d-d9fc-455a-be03-ea34f8838730\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vq4dj"
Nov 01 03:05:54.526016 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.525701    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/994175bb-d192-47f7-b162-a840eb0f9be1-hosts-file\") pod \"node-resolver-pzt6v\" (UID: \"994175bb-d192-47f7-b162-a840eb0f9be1\") " pod="openshift-dns/node-resolver-pzt6v"
Nov 01 03:05:54.526016 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.525703    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/bd8260bb-3811-4cdc-91d0-c6ddc693891e-agent-certs\") pod \"konnectivity-agent-fh7sf\" (UID: \"bd8260bb-3811-4cdc-91d0-c6ddc693891e\") " pod="kube-system/konnectivity-agent-fh7sf"
Nov 01 03:05:54.526016 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.525790    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0ede655d-c20d-41cc-b5f0-2db4338c1ba7-host-var-lib-kubelet\") pod \"multus-qzl9n\" (UID: \"0ede655d-c20d-41cc-b5f0-2db4338c1ba7\") " pod="openshift-multus/multus-qzl9n"
Nov 01 03:05:54.526016 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.525820    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0ede655d-c20d-41cc-b5f0-2db4338c1ba7-host-run-multus-certs\") pod \"multus-qzl9n\" (UID: \"0ede655d-c20d-41cc-b5f0-2db4338c1ba7\") " pod="openshift-multus/multus-qzl9n"
Nov 01 03:05:54.526016 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.525848    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/de5f469f-1029-4161-bf51-6ad18a3ac9e2-systemd-units\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:05:54.526016 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.525870    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0ede655d-c20d-41cc-b5f0-2db4338c1ba7-host-var-lib-kubelet\") pod \"multus-qzl9n\" (UID: \"0ede655d-c20d-41cc-b5f0-2db4338c1ba7\") " pod="openshift-multus/multus-qzl9n"
Nov 01 03:05:54.526016 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.525874    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0ede655d-c20d-41cc-b5f0-2db4338c1ba7-system-cni-dir\") pod \"multus-qzl9n\" (UID: \"0ede655d-c20d-41cc-b5f0-2db4338c1ba7\") " pod="openshift-multus/multus-qzl9n"
Nov 01 03:05:54.526016 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.525890    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0ede655d-c20d-41cc-b5f0-2db4338c1ba7-host-run-multus-certs\") pod \"multus-qzl9n\" (UID: \"0ede655d-c20d-41cc-b5f0-2db4338c1ba7\") " pod="openshift-multus/multus-qzl9n"
Nov 01 03:05:54.526659 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.525892    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/bd8260bb-3811-4cdc-91d0-c6ddc693891e-konnectivity-ca\") pod \"konnectivity-agent-fh7sf\" (UID: \"bd8260bb-3811-4cdc-91d0-c6ddc693891e\") " pod="kube-system/konnectivity-agent-fh7sf"
Nov 01 03:05:54.526659 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.525901    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/11fb87e4-ef01-4209-8207-80b66ab09f5e-etc-sysctl-conf\") pod \"tuned-lkbk6\" (UID: \"11fb87e4-ef01-4209-8207-80b66ab09f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-lkbk6"
Nov 01 03:05:54.526659 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.525905    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c9e89456-9acd-408f-87b5-f5fd968c82c3-iptables-alerter-script\") pod \"iptables-alerter-wfg7k\" (UID: \"c9e89456-9acd-408f-87b5-f5fd968c82c3\") " pod="openshift-network-operator/iptables-alerter-wfg7k"
Nov 01 03:05:54.526659 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.525924    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/11fb87e4-ef01-4209-8207-80b66ab09f5e-etc-tuned\") pod \"tuned-lkbk6\" (UID: \"11fb87e4-ef01-4209-8207-80b66ab09f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-lkbk6"
Nov 01 03:05:54.526659 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.525930    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0ede655d-c20d-41cc-b5f0-2db4338c1ba7-system-cni-dir\") pod \"multus-qzl9n\" (UID: \"0ede655d-c20d-41cc-b5f0-2db4338c1ba7\") " pod="openshift-multus/multus-qzl9n"
Nov 01 03:05:54.526659 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.525964    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de5f469f-1029-4161-bf51-6ad18a3ac9e2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:05:54.526659 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.525999    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2krw\" (UniqueName: \"kubernetes.io/projected/e48a634f-ab31-435c-96c1-c3633c5bc061-kube-api-access-b2krw\") pod \"network-metrics-daemon-dnqjp\" (UID: \"e48a634f-ab31-435c-96c1-c3633c5bc061\") " pod="openshift-multus/network-metrics-daemon-dnqjp"
Nov 01 03:05:54.526659 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.526002    2576 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 01 03:05:54.526659 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.526027    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0ede655d-c20d-41cc-b5f0-2db4338c1ba7-host-run-netns\") pod \"multus-qzl9n\" (UID: \"0ede655d-c20d-41cc-b5f0-2db4338c1ba7\") " pod="openshift-multus/multus-qzl9n"
Nov 01 03:05:54.526659 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.526050    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0ede655d-c20d-41cc-b5f0-2db4338c1ba7-host-run-netns\") pod \"multus-qzl9n\" (UID: \"0ede655d-c20d-41cc-b5f0-2db4338c1ba7\") " pod="openshift-multus/multus-qzl9n"
Nov 01 03:05:54.526659 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.526043    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0ede655d-c20d-41cc-b5f0-2db4338c1ba7-hostroot\") pod \"multus-qzl9n\" (UID: \"0ede655d-c20d-41cc-b5f0-2db4338c1ba7\") " pod="openshift-multus/multus-qzl9n"
Nov 01 03:05:54.526659 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.526078    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0ede655d-c20d-41cc-b5f0-2db4338c1ba7-hostroot\") pod \"multus-qzl9n\" (UID: \"0ede655d-c20d-41cc-b5f0-2db4338c1ba7\") " pod="openshift-multus/multus-qzl9n"
Nov 01 03:05:54.526659 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.526081    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0ede655d-c20d-41cc-b5f0-2db4338c1ba7-etc-kubernetes\") pod \"multus-qzl9n\" (UID: \"0ede655d-c20d-41cc-b5f0-2db4338c1ba7\") " pod="openshift-multus/multus-qzl9n"
Nov 01 03:05:54.526659 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.526094    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0ede655d-c20d-41cc-b5f0-2db4338c1ba7-etc-kubernetes\") pod \"multus-qzl9n\" (UID: \"0ede655d-c20d-41cc-b5f0-2db4338c1ba7\") " pod="openshift-multus/multus-qzl9n"
Nov 01 03:05:54.526659 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.526106    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/11fb87e4-ef01-4209-8207-80b66ab09f5e-etc-modprobe-d\") pod \"tuned-lkbk6\" (UID: \"11fb87e4-ef01-4209-8207-80b66ab09f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-lkbk6"
Nov 01 03:05:54.526659 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.526121    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/11fb87e4-ef01-4209-8207-80b66ab09f5e-etc-sysctl-d\") pod \"tuned-lkbk6\" (UID: \"11fb87e4-ef01-4209-8207-80b66ab09f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-lkbk6"
Nov 01 03:05:54.526659 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.526137    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0303d6c6-c4ec-4403-af74-f4bcb5cbf2a4-dbus\") pod \"global-pull-secret-syncer-rrdzd\" (UID: \"0303d6c6-c4ec-4403-af74-f4bcb5cbf2a4\") " pod="kube-system/global-pull-secret-syncer-rrdzd"
Nov 01 03:05:54.526659 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.526161    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bf9dac1d-d9fc-455a-be03-ea34f8838730-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vq4dj\" (UID: \"bf9dac1d-d9fc-455a-be03-ea34f8838730\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vq4dj"
Nov 01 03:05:54.527437 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.526200    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bf9dac1d-d9fc-455a-be03-ea34f8838730-sys-fs\") pod \"aws-ebs-csi-driver-node-vq4dj\" (UID: \"bf9dac1d-d9fc-455a-be03-ea34f8838730\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vq4dj"
Nov 01 03:05:54.527437 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.526251    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de5f469f-1029-4161-bf51-6ad18a3ac9e2-run-openvswitch\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:05:54.527437 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.526282    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t86zn\" (UniqueName: \"kubernetes.io/projected/d7cef651-3047-4e64-ae7f-b0fa71fbd0fe-kube-api-access-t86zn\") pod \"node-ca-wq27m\" (UID: \"d7cef651-3047-4e64-ae7f-b0fa71fbd0fe\") " pod="openshift-image-registry/node-ca-wq27m"
Nov 01 03:05:54.527437 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.526305    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxqx8\" (UniqueName: \"kubernetes.io/projected/6e302b40-9c43-4822-8c8b-4d021a0590ab-kube-api-access-nxqx8\") pod \"network-check-target-mx48d\" (UID: \"6e302b40-9c43-4822-8c8b-4d021a0590ab\") " pod="openshift-network-diagnostics/network-check-target-mx48d"
Nov 01 03:05:54.527437 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.526328    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/11fb87e4-ef01-4209-8207-80b66ab09f5e-lib-modules\") pod \"tuned-lkbk6\" (UID: \"11fb87e4-ef01-4209-8207-80b66ab09f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-lkbk6"
Nov 01 03:05:54.527437 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.526343    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/de5f469f-1029-4161-bf51-6ad18a3ac9e2-host-kubelet\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:05:54.527437 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.526391    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/de5f469f-1029-4161-bf51-6ad18a3ac9e2-ovnkube-config\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:05:54.527437 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.526414    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e8f2a24b-e770-4615-a2c7-b2d065e14f77-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qnrf8\" (UID: \"e8f2a24b-e770-4615-a2c7-b2d065e14f77\") " pod="openshift-multus/multus-additional-cni-plugins-qnrf8"
Nov 01 03:05:54.527437 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.526451    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0ede655d-c20d-41cc-b5f0-2db4338c1ba7-cni-binary-copy\") pod \"multus-qzl9n\" (UID: \"0ede655d-c20d-41cc-b5f0-2db4338c1ba7\") " pod="openshift-multus/multus-qzl9n"
Nov 01 03:05:54.527437 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.526485    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/de5f469f-1029-4161-bf51-6ad18a3ac9e2-run-systemd\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:05:54.527437 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.526519    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de5f469f-1029-4161-bf51-6ad18a3ac9e2-etc-openvswitch\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:05:54.527437 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.526549    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtjjf\" (UniqueName: \"kubernetes.io/projected/994175bb-d192-47f7-b162-a840eb0f9be1-kube-api-access-xtjjf\") pod \"node-resolver-pzt6v\" (UID: \"994175bb-d192-47f7-b162-a840eb0f9be1\") " pod="openshift-dns/node-resolver-pzt6v"
Nov 01 03:05:54.527437 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.526625    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/11fb87e4-ef01-4209-8207-80b66ab09f5e-etc-sysconfig\") pod \"tuned-lkbk6\" (UID: \"11fb87e4-ef01-4209-8207-80b66ab09f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-lkbk6"
Nov 01 03:05:54.527437 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.526683    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bf9dac1d-d9fc-455a-be03-ea34f8838730-registration-dir\") pod \"aws-ebs-csi-driver-node-vq4dj\" (UID: \"bf9dac1d-d9fc-455a-be03-ea34f8838730\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vq4dj"
Nov 01 03:05:54.527437 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.526731    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhdnz\" (UniqueName: \"kubernetes.io/projected/bf9dac1d-d9fc-455a-be03-ea34f8838730-kube-api-access-jhdnz\") pod \"aws-ebs-csi-driver-node-vq4dj\" (UID: \"bf9dac1d-d9fc-455a-be03-ea34f8838730\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vq4dj"
Nov 01 03:05:54.527437 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.526756    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/de5f469f-1029-4161-bf51-6ad18a3ac9e2-node-log\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:05:54.528050 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.526787    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0ede655d-c20d-41cc-b5f0-2db4338c1ba7-multus-socket-dir-parent\") pod \"multus-qzl9n\" (UID: \"0ede655d-c20d-41cc-b5f0-2db4338c1ba7\") " pod="openshift-multus/multus-qzl9n"
Nov 01 03:05:54.528050 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.526823    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/11fb87e4-ef01-4209-8207-80b66ab09f5e-var-lib-kubelet\") pod \"tuned-lkbk6\" (UID: \"11fb87e4-ef01-4209-8207-80b66ab09f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-lkbk6"
Nov 01 03:05:54.528050 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.526833    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0ede655d-c20d-41cc-b5f0-2db4338c1ba7-multus-socket-dir-parent\") pod \"multus-qzl9n\" (UID: \"0ede655d-c20d-41cc-b5f0-2db4338c1ba7\") " pod="openshift-multus/multus-qzl9n"
Nov 01 03:05:54.528050 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.526854    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0303d6c6-c4ec-4403-af74-f4bcb5cbf2a4-original-pull-secret\") pod \"global-pull-secret-syncer-rrdzd\" (UID: \"0303d6c6-c4ec-4403-af74-f4bcb5cbf2a4\") " pod="kube-system/global-pull-secret-syncer-rrdzd"
Nov 01 03:05:54.528050 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.526868    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0ede655d-c20d-41cc-b5f0-2db4338c1ba7-cni-binary-copy\") pod \"multus-qzl9n\" (UID: \"0ede655d-c20d-41cc-b5f0-2db4338c1ba7\") " pod="openshift-multus/multus-qzl9n"
Nov 01 03:05:54.528050 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.526879    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bf9dac1d-d9fc-455a-be03-ea34f8838730-device-dir\") pod \"aws-ebs-csi-driver-node-vq4dj\" (UID: \"bf9dac1d-d9fc-455a-be03-ea34f8838730\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vq4dj"
Nov 01 03:05:54.528050 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.526932    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/de5f469f-1029-4161-bf51-6ad18a3ac9e2-host-slash\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:05:54.528050 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.526984    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e8f2a24b-e770-4615-a2c7-b2d065e14f77-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qnrf8\" (UID: \"e8f2a24b-e770-4615-a2c7-b2d065e14f77\") " pod="openshift-multus/multus-additional-cni-plugins-qnrf8"
Nov 01 03:05:54.528050 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.527015    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c9e89456-9acd-408f-87b5-f5fd968c82c3-host-slash\") pod \"iptables-alerter-wfg7k\" (UID: \"c9e89456-9acd-408f-87b5-f5fd968c82c3\") " pod="openshift-network-operator/iptables-alerter-wfg7k"
Nov 01 03:05:54.528050 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.527048    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0ede655d-c20d-41cc-b5f0-2db4338c1ba7-host-var-lib-cni-bin\") pod \"multus-qzl9n\" (UID: \"0ede655d-c20d-41cc-b5f0-2db4338c1ba7\") " pod="openshift-multus/multus-qzl9n"
Nov 01 03:05:54.528050 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.527095    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c9e89456-9acd-408f-87b5-f5fd968c82c3-host-slash\") pod \"iptables-alerter-wfg7k\" (UID: \"c9e89456-9acd-408f-87b5-f5fd968c82c3\") " pod="openshift-network-operator/iptables-alerter-wfg7k"
Nov 01 03:05:54.528050 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.527101    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0ede655d-c20d-41cc-b5f0-2db4338c1ba7-host-var-lib-cni-bin\") pod \"multus-qzl9n\" (UID: \"0ede655d-c20d-41cc-b5f0-2db4338c1ba7\") " pod="openshift-multus/multus-qzl9n"
Nov 01 03:05:54.528050 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.527215    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0ede655d-c20d-41cc-b5f0-2db4338c1ba7-host-var-lib-cni-multus\") pod \"multus-qzl9n\" (UID: \"0ede655d-c20d-41cc-b5f0-2db4338c1ba7\") " pod="openshift-multus/multus-qzl9n"
Nov 01 03:05:54.528050 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.527269    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0ede655d-c20d-41cc-b5f0-2db4338c1ba7-host-var-lib-cni-multus\") pod \"multus-qzl9n\" (UID: \"0ede655d-c20d-41cc-b5f0-2db4338c1ba7\") " pod="openshift-multus/multus-qzl9n"
Nov 01 03:05:54.528050 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.527277    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de5f469f-1029-4161-bf51-6ad18a3ac9e2-host-run-ovn-kubernetes\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:05:54.528050 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.527303    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckfr8\" (UniqueName: \"kubernetes.io/projected/de5f469f-1029-4161-bf51-6ad18a3ac9e2-kube-api-access-ckfr8\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:05:54.529608 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.529590    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/bd8260bb-3811-4cdc-91d0-c6ddc693891e-agent-certs\") pod \"konnectivity-agent-fh7sf\" (UID: \"bd8260bb-3811-4cdc-91d0-c6ddc693891e\") " pod="kube-system/konnectivity-agent-fh7sf"
Nov 01 03:05:54.597248 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.597229    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gljgq\" (UniqueName: \"kubernetes.io/projected/c9e89456-9acd-408f-87b5-f5fd968c82c3-kube-api-access-gljgq\") pod \"iptables-alerter-wfg7k\" (UID: \"c9e89456-9acd-408f-87b5-f5fd968c82c3\") " pod="openshift-network-operator/iptables-alerter-wfg7k"
Nov 01 03:05:54.597502 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.597478    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slnpd\" (UniqueName: \"kubernetes.io/projected/0ede655d-c20d-41cc-b5f0-2db4338c1ba7-kube-api-access-slnpd\") pod \"multus-qzl9n\" (UID: \"0ede655d-c20d-41cc-b5f0-2db4338c1ba7\") " pod="openshift-multus/multus-qzl9n"
Nov 01 03:05:54.597692 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.597674    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtjjf\" (UniqueName: \"kubernetes.io/projected/994175bb-d192-47f7-b162-a840eb0f9be1-kube-api-access-xtjjf\") pod \"node-resolver-pzt6v\" (UID: \"994175bb-d192-47f7-b162-a840eb0f9be1\") " pod="openshift-dns/node-resolver-pzt6v"
Nov 01 03:05:54.628671 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.628649    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/11fb87e4-ef01-4209-8207-80b66ab09f5e-etc-modprobe-d\") pod \"tuned-lkbk6\" (UID: \"11fb87e4-ef01-4209-8207-80b66ab09f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-lkbk6"
Nov 01 03:05:54.628746 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.628674    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/11fb87e4-ef01-4209-8207-80b66ab09f5e-etc-sysctl-d\") pod \"tuned-lkbk6\" (UID: \"11fb87e4-ef01-4209-8207-80b66ab09f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-lkbk6"
Nov 01 03:05:54.628746 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.628689    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0303d6c6-c4ec-4403-af74-f4bcb5cbf2a4-dbus\") pod \"global-pull-secret-syncer-rrdzd\" (UID: \"0303d6c6-c4ec-4403-af74-f4bcb5cbf2a4\") " pod="kube-system/global-pull-secret-syncer-rrdzd"
Nov 01 03:05:54.628746 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.628720    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bf9dac1d-d9fc-455a-be03-ea34f8838730-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vq4dj\" (UID: \"bf9dac1d-d9fc-455a-be03-ea34f8838730\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vq4dj"
Nov 01 03:05:54.628746 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.628734    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bf9dac1d-d9fc-455a-be03-ea34f8838730-sys-fs\") pod \"aws-ebs-csi-driver-node-vq4dj\" (UID: \"bf9dac1d-d9fc-455a-be03-ea34f8838730\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vq4dj"
Nov 01 03:05:54.628956 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.628750    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de5f469f-1029-4161-bf51-6ad18a3ac9e2-run-openvswitch\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:05:54.628956 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.628765    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t86zn\" (UniqueName: \"kubernetes.io/projected/d7cef651-3047-4e64-ae7f-b0fa71fbd0fe-kube-api-access-t86zn\") pod \"node-ca-wq27m\" (UID: \"d7cef651-3047-4e64-ae7f-b0fa71fbd0fe\") " pod="openshift-image-registry/node-ca-wq27m"
Nov 01 03:05:54.628956 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.628781    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxqx8\" (UniqueName: \"kubernetes.io/projected/6e302b40-9c43-4822-8c8b-4d021a0590ab-kube-api-access-nxqx8\") pod \"network-check-target-mx48d\" (UID: \"6e302b40-9c43-4822-8c8b-4d021a0590ab\") " pod="openshift-network-diagnostics/network-check-target-mx48d"
Nov 01 03:05:54.628956 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.628784    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/11fb87e4-ef01-4209-8207-80b66ab09f5e-etc-modprobe-d\") pod \"tuned-lkbk6\" (UID: \"11fb87e4-ef01-4209-8207-80b66ab09f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-lkbk6"
Nov 01 03:05:54.628956 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.628788    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0303d6c6-c4ec-4403-af74-f4bcb5cbf2a4-dbus\") pod \"global-pull-secret-syncer-rrdzd\" (UID: \"0303d6c6-c4ec-4403-af74-f4bcb5cbf2a4\") " pod="kube-system/global-pull-secret-syncer-rrdzd"
Nov 01 03:05:54.628956 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.628795    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bf9dac1d-d9fc-455a-be03-ea34f8838730-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vq4dj\" (UID: \"bf9dac1d-d9fc-455a-be03-ea34f8838730\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vq4dj"
Nov 01 03:05:54.628956 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.628806    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/11fb87e4-ef01-4209-8207-80b66ab09f5e-lib-modules\") pod \"tuned-lkbk6\" (UID: \"11fb87e4-ef01-4209-8207-80b66ab09f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-lkbk6"
Nov 01 03:05:54.628956 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.628807    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de5f469f-1029-4161-bf51-6ad18a3ac9e2-run-openvswitch\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:05:54.628956 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.628849    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bf9dac1d-d9fc-455a-be03-ea34f8838730-sys-fs\") pod \"aws-ebs-csi-driver-node-vq4dj\" (UID: \"bf9dac1d-d9fc-455a-be03-ea34f8838730\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vq4dj"
Nov 01 03:05:54.628956 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.628858    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/de5f469f-1029-4161-bf51-6ad18a3ac9e2-host-kubelet\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:05:54.628956 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.628873    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/11fb87e4-ef01-4209-8207-80b66ab09f5e-etc-sysctl-d\") pod \"tuned-lkbk6\" (UID: \"11fb87e4-ef01-4209-8207-80b66ab09f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-lkbk6"
Nov 01 03:05:54.628956 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.628881    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/11fb87e4-ef01-4209-8207-80b66ab09f5e-lib-modules\") pod \"tuned-lkbk6\" (UID: \"11fb87e4-ef01-4209-8207-80b66ab09f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-lkbk6"
Nov 01 03:05:54.628956 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.628883    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/de5f469f-1029-4161-bf51-6ad18a3ac9e2-host-kubelet\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:05:54.628956 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.628905    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/de5f469f-1029-4161-bf51-6ad18a3ac9e2-ovnkube-config\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:05:54.628956 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.628931    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e8f2a24b-e770-4615-a2c7-b2d065e14f77-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qnrf8\" (UID: \"e8f2a24b-e770-4615-a2c7-b2d065e14f77\") " pod="openshift-multus/multus-additional-cni-plugins-qnrf8"
Nov 01 03:05:54.628956 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.628949    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/de5f469f-1029-4161-bf51-6ad18a3ac9e2-run-systemd\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:05:54.628956 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.628968    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/de5f469f-1029-4161-bf51-6ad18a3ac9e2-run-systemd\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:05:54.629623 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.628995    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de5f469f-1029-4161-bf51-6ad18a3ac9e2-etc-openvswitch\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:05:54.629623 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.629011    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/11fb87e4-ef01-4209-8207-80b66ab09f5e-etc-sysconfig\") pod \"tuned-lkbk6\" (UID: \"11fb87e4-ef01-4209-8207-80b66ab09f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-lkbk6"
Nov 01 03:05:54.629623 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.629029    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bf9dac1d-d9fc-455a-be03-ea34f8838730-registration-dir\") pod \"aws-ebs-csi-driver-node-vq4dj\" (UID: \"bf9dac1d-d9fc-455a-be03-ea34f8838730\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vq4dj"
Nov 01 03:05:54.629623 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.629046    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhdnz\" (UniqueName: \"kubernetes.io/projected/bf9dac1d-d9fc-455a-be03-ea34f8838730-kube-api-access-jhdnz\") pod \"aws-ebs-csi-driver-node-vq4dj\" (UID: \"bf9dac1d-d9fc-455a-be03-ea34f8838730\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vq4dj"
Nov 01 03:05:54.629623 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.629060    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/de5f469f-1029-4161-bf51-6ad18a3ac9e2-node-log\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:05:54.629623 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.629075    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/11fb87e4-ef01-4209-8207-80b66ab09f5e-var-lib-kubelet\") pod \"tuned-lkbk6\" (UID: \"11fb87e4-ef01-4209-8207-80b66ab09f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-lkbk6"
Nov 01 03:05:54.629623 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.629093    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/11fb87e4-ef01-4209-8207-80b66ab09f5e-etc-sysconfig\") pod \"tuned-lkbk6\" (UID: \"11fb87e4-ef01-4209-8207-80b66ab09f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-lkbk6"
Nov 01 03:05:54.629623 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.629133    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/de5f469f-1029-4161-bf51-6ad18a3ac9e2-node-log\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:05:54.629623 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.629158    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/11fb87e4-ef01-4209-8207-80b66ab09f5e-var-lib-kubelet\") pod \"tuned-lkbk6\" (UID: \"11fb87e4-ef01-4209-8207-80b66ab09f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-lkbk6"
Nov 01 03:05:54.629623 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.629167    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de5f469f-1029-4161-bf51-6ad18a3ac9e2-etc-openvswitch\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:05:54.629623 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.629098    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0303d6c6-c4ec-4403-af74-f4bcb5cbf2a4-original-pull-secret\") pod \"global-pull-secret-syncer-rrdzd\" (UID: \"0303d6c6-c4ec-4403-af74-f4bcb5cbf2a4\") " pod="kube-system/global-pull-secret-syncer-rrdzd"
Nov 01 03:05:54.629623 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:05:54.629185    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 03:05:54.629623 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.629204    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bf9dac1d-d9fc-455a-be03-ea34f8838730-device-dir\") pod \"aws-ebs-csi-driver-node-vq4dj\" (UID: \"bf9dac1d-d9fc-455a-be03-ea34f8838730\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vq4dj"
Nov 01 03:05:54.629623 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.629213    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bf9dac1d-d9fc-455a-be03-ea34f8838730-registration-dir\") pod \"aws-ebs-csi-driver-node-vq4dj\" (UID: \"bf9dac1d-d9fc-455a-be03-ea34f8838730\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vq4dj"
Nov 01 03:05:54.629623 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.629228    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/de5f469f-1029-4161-bf51-6ad18a3ac9e2-host-slash\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:05:54.629623 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:05:54.629245    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0303d6c6-c4ec-4403-af74-f4bcb5cbf2a4-original-pull-secret podName:0303d6c6-c4ec-4403-af74-f4bcb5cbf2a4 nodeName:}" failed. No retries permitted until 2026-11-01 03:05:55.129224613 +0000 UTC m=+3.183366303 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0303d6c6-c4ec-4403-af74-f4bcb5cbf2a4-original-pull-secret") pod "global-pull-secret-syncer-rrdzd" (UID: "0303d6c6-c4ec-4403-af74-f4bcb5cbf2a4") : object "kube-system"/"original-pull-secret" not registered
Nov 01 03:05:54.629623 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.629254    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/de5f469f-1029-4161-bf51-6ad18a3ac9e2-host-slash\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:05:54.630473 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.629257    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bf9dac1d-d9fc-455a-be03-ea34f8838730-device-dir\") pod \"aws-ebs-csi-driver-node-vq4dj\" (UID: \"bf9dac1d-d9fc-455a-be03-ea34f8838730\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vq4dj"
Nov 01 03:05:54.630473 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.629293    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e8f2a24b-e770-4615-a2c7-b2d065e14f77-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qnrf8\" (UID: \"e8f2a24b-e770-4615-a2c7-b2d065e14f77\") " pod="openshift-multus/multus-additional-cni-plugins-qnrf8"
Nov 01 03:05:54.630473 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.629312    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de5f469f-1029-4161-bf51-6ad18a3ac9e2-host-run-ovn-kubernetes\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:05:54.630473 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.629328    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckfr8\" (UniqueName: \"kubernetes.io/projected/de5f469f-1029-4161-bf51-6ad18a3ac9e2-kube-api-access-ckfr8\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:05:54.630473 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.629350    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e8f2a24b-e770-4615-a2c7-b2d065e14f77-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qnrf8\" (UID: \"e8f2a24b-e770-4615-a2c7-b2d065e14f77\") " pod="openshift-multus/multus-additional-cni-plugins-qnrf8"
Nov 01 03:05:54.630473 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.629398    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de5f469f-1029-4161-bf51-6ad18a3ac9e2-host-run-ovn-kubernetes\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:05:54.630473 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.629455    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/de5f469f-1029-4161-bf51-6ad18a3ac9e2-ovnkube-config\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:05:54.630473 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.629483    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e8f2a24b-e770-4615-a2c7-b2d065e14f77-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qnrf8\" (UID: \"e8f2a24b-e770-4615-a2c7-b2d065e14f77\") " pod="openshift-multus/multus-additional-cni-plugins-qnrf8"
Nov 01 03:05:54.630473 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.629499    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e8f2a24b-e770-4615-a2c7-b2d065e14f77-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qnrf8\" (UID: \"e8f2a24b-e770-4615-a2c7-b2d065e14f77\") " pod="openshift-multus/multus-additional-cni-plugins-qnrf8"
Nov 01 03:05:54.630473 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.629583    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/de5f469f-1029-4161-bf51-6ad18a3ac9e2-host-run-netns\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:05:54.630473 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.629601    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/de5f469f-1029-4161-bf51-6ad18a3ac9e2-ovn-node-metrics-cert\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:05:54.630473 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.629616    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e8f2a24b-e770-4615-a2c7-b2d065e14f77-cni-binary-copy\") pod \"multus-additional-cni-plugins-qnrf8\" (UID: \"e8f2a24b-e770-4615-a2c7-b2d065e14f77\") " pod="openshift-multus/multus-additional-cni-plugins-qnrf8"
Nov 01 03:05:54.630473 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.629638    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85r94\" (UniqueName: \"kubernetes.io/projected/e8f2a24b-e770-4615-a2c7-b2d065e14f77-kube-api-access-85r94\") pod \"multus-additional-cni-plugins-qnrf8\" (UID: \"e8f2a24b-e770-4615-a2c7-b2d065e14f77\") " pod="openshift-multus/multus-additional-cni-plugins-qnrf8"
Nov 01 03:05:54.630473 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.629653    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/11fb87e4-ef01-4209-8207-80b66ab09f5e-tmp\") pod \"tuned-lkbk6\" (UID: \"11fb87e4-ef01-4209-8207-80b66ab09f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-lkbk6"
Nov 01 03:05:54.630473 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.629673    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0303d6c6-c4ec-4403-af74-f4bcb5cbf2a4-kubelet-config\") pod \"global-pull-secret-syncer-rrdzd\" (UID: \"0303d6c6-c4ec-4403-af74-f4bcb5cbf2a4\") " pod="kube-system/global-pull-secret-syncer-rrdzd"
Nov 01 03:05:54.630473 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.629711    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/de5f469f-1029-4161-bf51-6ad18a3ac9e2-ovnkube-script-lib\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:05:54.630473 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.629715    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/de5f469f-1029-4161-bf51-6ad18a3ac9e2-host-run-netns\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:05:54.631283 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.629739    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/11fb87e4-ef01-4209-8207-80b66ab09f5e-host\") pod \"tuned-lkbk6\" (UID: \"11fb87e4-ef01-4209-8207-80b66ab09f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-lkbk6"
Nov 01 03:05:54.631283 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.629792    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bf9dac1d-d9fc-455a-be03-ea34f8838730-socket-dir\") pod \"aws-ebs-csi-driver-node-vq4dj\" (UID: \"bf9dac1d-d9fc-455a-be03-ea34f8838730\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vq4dj"
Nov 01 03:05:54.631283 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.629824    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/de5f469f-1029-4161-bf51-6ad18a3ac9e2-run-ovn\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:05:54.631283 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.629852    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/de5f469f-1029-4161-bf51-6ad18a3ac9e2-log-socket\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:05:54.631283 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.629881    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d7cef651-3047-4e64-ae7f-b0fa71fbd0fe-serviceca\") pod \"node-ca-wq27m\" (UID: \"d7cef651-3047-4e64-ae7f-b0fa71fbd0fe\") " pod="openshift-image-registry/node-ca-wq27m"
Nov 01 03:05:54.631283 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.629914    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/11fb87e4-ef01-4209-8207-80b66ab09f5e-run\") pod \"tuned-lkbk6\" (UID: \"11fb87e4-ef01-4209-8207-80b66ab09f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-lkbk6"
Nov 01 03:05:54.631283 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.629941    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/11fb87e4-ef01-4209-8207-80b66ab09f5e-sys\") pod \"tuned-lkbk6\" (UID: \"11fb87e4-ef01-4209-8207-80b66ab09f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-lkbk6"
Nov 01 03:05:54.631283 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.629956    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bf9dac1d-d9fc-455a-be03-ea34f8838730-socket-dir\") pod \"aws-ebs-csi-driver-node-vq4dj\" (UID: \"bf9dac1d-d9fc-455a-be03-ea34f8838730\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vq4dj"
Nov 01 03:05:54.631283 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.629972    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h58g6\" (UniqueName: \"kubernetes.io/projected/11fb87e4-ef01-4209-8207-80b66ab09f5e-kube-api-access-h58g6\") pod \"tuned-lkbk6\" (UID: \"11fb87e4-ef01-4209-8207-80b66ab09f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-lkbk6"
Nov 01 03:05:54.631283 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.630002    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e8f2a24b-e770-4615-a2c7-b2d065e14f77-system-cni-dir\") pod \"multus-additional-cni-plugins-qnrf8\" (UID: \"e8f2a24b-e770-4615-a2c7-b2d065e14f77\") " pod="openshift-multus/multus-additional-cni-plugins-qnrf8"
Nov 01 03:05:54.631283 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.630026    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/11fb87e4-ef01-4209-8207-80b66ab09f5e-host\") pod \"tuned-lkbk6\" (UID: \"11fb87e4-ef01-4209-8207-80b66ab09f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-lkbk6"
Nov 01 03:05:54.631283 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.630031    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e8f2a24b-e770-4615-a2c7-b2d065e14f77-cnibin\") pod \"multus-additional-cni-plugins-qnrf8\" (UID: \"e8f2a24b-e770-4615-a2c7-b2d065e14f77\") " pod="openshift-multus/multus-additional-cni-plugins-qnrf8"
Nov 01 03:05:54.631283 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.630065    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de5f469f-1029-4161-bf51-6ad18a3ac9e2-var-lib-openvswitch\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:05:54.631283 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.630094    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/de5f469f-1029-4161-bf51-6ad18a3ac9e2-env-overrides\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:05:54.631283 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.630125    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/de5f469f-1029-4161-bf51-6ad18a3ac9e2-log-socket\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:05:54.631283 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.630126    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e48a634f-ab31-435c-96c1-c3633c5bc061-metrics-certs\") pod \"network-metrics-daemon-dnqjp\" (UID: \"e48a634f-ab31-435c-96c1-c3633c5bc061\") " pod="openshift-multus/network-metrics-daemon-dnqjp"
Nov 01 03:05:54.631283 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.630144    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e8f2a24b-e770-4615-a2c7-b2d065e14f77-cni-binary-copy\") pod \"multus-additional-cni-plugins-qnrf8\" (UID: \"e8f2a24b-e770-4615-a2c7-b2d065e14f77\") " pod="openshift-multus/multus-additional-cni-plugins-qnrf8"
Nov 01 03:05:54.631816 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.630172    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/11fb87e4-ef01-4209-8207-80b66ab09f5e-etc-kubernetes\") pod \"tuned-lkbk6\" (UID: \"11fb87e4-ef01-4209-8207-80b66ab09f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-lkbk6"
Nov 01 03:05:54.631816 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.630201    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/11fb87e4-ef01-4209-8207-80b66ab09f5e-etc-systemd\") pod \"tuned-lkbk6\" (UID: \"11fb87e4-ef01-4209-8207-80b66ab09f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-lkbk6"
Nov 01 03:05:54.631816 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.630217    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e8f2a24b-e770-4615-a2c7-b2d065e14f77-cnibin\") pod \"multus-additional-cni-plugins-qnrf8\" (UID: \"e8f2a24b-e770-4615-a2c7-b2d065e14f77\") " pod="openshift-multus/multus-additional-cni-plugins-qnrf8"
Nov 01 03:05:54.631816 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.630228    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/de5f469f-1029-4161-bf51-6ad18a3ac9e2-host-cni-bin\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:05:54.631816 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.630256    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/de5f469f-1029-4161-bf51-6ad18a3ac9e2-host-cni-netd\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:05:54.631816 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.630282    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d7cef651-3047-4e64-ae7f-b0fa71fbd0fe-host\") pod \"node-ca-wq27m\" (UID: \"d7cef651-3047-4e64-ae7f-b0fa71fbd0fe\") " pod="openshift-image-registry/node-ca-wq27m"
Nov 01 03:05:54.631816 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.630291    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/11fb87e4-ef01-4209-8207-80b66ab09f5e-sys\") pod \"tuned-lkbk6\" (UID: \"11fb87e4-ef01-4209-8207-80b66ab09f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-lkbk6"
Nov 01 03:05:54.631816 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.630299    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/de5f469f-1029-4161-bf51-6ad18a3ac9e2-run-ovn\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:05:54.631816 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.630289    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0303d6c6-c4ec-4403-af74-f4bcb5cbf2a4-kubelet-config\") pod \"global-pull-secret-syncer-rrdzd\" (UID: \"0303d6c6-c4ec-4403-af74-f4bcb5cbf2a4\") " pod="kube-system/global-pull-secret-syncer-rrdzd"
Nov 01 03:05:54.631816 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.630309    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bf9dac1d-d9fc-455a-be03-ea34f8838730-etc-selinux\") pod \"aws-ebs-csi-driver-node-vq4dj\" (UID: \"bf9dac1d-d9fc-455a-be03-ea34f8838730\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vq4dj"
Nov 01 03:05:54.631816 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.630344    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/de5f469f-1029-4161-bf51-6ad18a3ac9e2-host-cni-netd\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:05:54.631816 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.630357    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/11fb87e4-ef01-4209-8207-80b66ab09f5e-etc-kubernetes\") pod \"tuned-lkbk6\" (UID: \"11fb87e4-ef01-4209-8207-80b66ab09f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-lkbk6"
Nov 01 03:05:54.631816 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.630366    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e8f2a24b-e770-4615-a2c7-b2d065e14f77-system-cni-dir\") pod \"multus-additional-cni-plugins-qnrf8\" (UID: \"e8f2a24b-e770-4615-a2c7-b2d065e14f77\") " pod="openshift-multus/multus-additional-cni-plugins-qnrf8"
Nov 01 03:05:54.631816 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.630451    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bf9dac1d-d9fc-455a-be03-ea34f8838730-etc-selinux\") pod \"aws-ebs-csi-driver-node-vq4dj\" (UID: \"bf9dac1d-d9fc-455a-be03-ea34f8838730\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vq4dj"
Nov 01 03:05:54.631816 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.630439    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/de5f469f-1029-4161-bf51-6ad18a3ac9e2-host-cni-bin\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:05:54.631816 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.630526    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d7cef651-3047-4e64-ae7f-b0fa71fbd0fe-host\") pod \"node-ca-wq27m\" (UID: \"d7cef651-3047-4e64-ae7f-b0fa71fbd0fe\") " pod="openshift-image-registry/node-ca-wq27m"
Nov 01 03:05:54.631816 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.630532    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/de5f469f-1029-4161-bf51-6ad18a3ac9e2-systemd-units\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:05:54.631816 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.630562    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/11fb87e4-ef01-4209-8207-80b66ab09f5e-etc-sysctl-conf\") pod \"tuned-lkbk6\" (UID: \"11fb87e4-ef01-4209-8207-80b66ab09f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-lkbk6"
Nov 01 03:05:54.632316 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.630506    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/11fb87e4-ef01-4209-8207-80b66ab09f5e-etc-systemd\") pod \"tuned-lkbk6\" (UID: \"11fb87e4-ef01-4209-8207-80b66ab09f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-lkbk6"
Nov 01 03:05:54.632316 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.630573    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de5f469f-1029-4161-bf51-6ad18a3ac9e2-var-lib-openvswitch\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:05:54.632316 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.630603    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/11fb87e4-ef01-4209-8207-80b66ab09f5e-etc-tuned\") pod \"tuned-lkbk6\" (UID: \"11fb87e4-ef01-4209-8207-80b66ab09f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-lkbk6"
Nov 01 03:05:54.632316 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.630613    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/11fb87e4-ef01-4209-8207-80b66ab09f5e-run\") pod \"tuned-lkbk6\" (UID: \"11fb87e4-ef01-4209-8207-80b66ab09f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-lkbk6"
Nov 01 03:05:54.632316 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.630631    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de5f469f-1029-4161-bf51-6ad18a3ac9e2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:05:54.632316 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.630649    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/de5f469f-1029-4161-bf51-6ad18a3ac9e2-systemd-units\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:05:54.632316 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.630661    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2krw\" (UniqueName: \"kubernetes.io/projected/e48a634f-ab31-435c-96c1-c3633c5bc061-kube-api-access-b2krw\") pod \"network-metrics-daemon-dnqjp\" (UID: \"e48a634f-ab31-435c-96c1-c3633c5bc061\") " pod="openshift-multus/network-metrics-daemon-dnqjp"
Nov 01 03:05:54.632316 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.630715    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/de5f469f-1029-4161-bf51-6ad18a3ac9e2-ovnkube-script-lib\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:05:54.632316 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.630719    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de5f469f-1029-4161-bf51-6ad18a3ac9e2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:05:54.632316 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.630882    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/11fb87e4-ef01-4209-8207-80b66ab09f5e-etc-sysctl-conf\") pod \"tuned-lkbk6\" (UID: \"11fb87e4-ef01-4209-8207-80b66ab09f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-lkbk6"
Nov 01 03:05:54.632316 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.631064    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d7cef651-3047-4e64-ae7f-b0fa71fbd0fe-serviceca\") pod \"node-ca-wq27m\" (UID: \"d7cef651-3047-4e64-ae7f-b0fa71fbd0fe\") " pod="openshift-image-registry/node-ca-wq27m"
Nov 01 03:05:54.632316 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.631092    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/de5f469f-1029-4161-bf51-6ad18a3ac9e2-env-overrides\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:05:54.632316 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.631159    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e8f2a24b-e770-4615-a2c7-b2d065e14f77-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qnrf8\" (UID: \"e8f2a24b-e770-4615-a2c7-b2d065e14f77\") " pod="openshift-multus/multus-additional-cni-plugins-qnrf8"
Nov 01 03:05:54.632316 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:05:54.631367    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 03:05:54.632316 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:05:54.631426    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e48a634f-ab31-435c-96c1-c3633c5bc061-metrics-certs podName:e48a634f-ab31-435c-96c1-c3633c5bc061 nodeName:}" failed. No retries permitted until 2026-11-01 03:05:55.131409204 +0000 UTC m=+3.185550882 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e48a634f-ab31-435c-96c1-c3633c5bc061-metrics-certs") pod "network-metrics-daemon-dnqjp" (UID: "e48a634f-ab31-435c-96c1-c3633c5bc061") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 03:05:54.632316 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.631994    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/11fb87e4-ef01-4209-8207-80b66ab09f5e-tmp\") pod \"tuned-lkbk6\" (UID: \"11fb87e4-ef01-4209-8207-80b66ab09f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-lkbk6"
Nov 01 03:05:54.632316 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.632136    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/de5f469f-1029-4161-bf51-6ad18a3ac9e2-ovn-node-metrics-cert\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:05:54.632821 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.632674    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/11fb87e4-ef01-4209-8207-80b66ab09f5e-etc-tuned\") pod \"tuned-lkbk6\" (UID: \"11fb87e4-ef01-4209-8207-80b66ab09f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-lkbk6"
Nov 01 03:05:54.700183 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.700157    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85r94\" (UniqueName: \"kubernetes.io/projected/e8f2a24b-e770-4615-a2c7-b2d065e14f77-kube-api-access-85r94\") pod \"multus-additional-cni-plugins-qnrf8\" (UID: \"e8f2a24b-e770-4615-a2c7-b2d065e14f77\") " pod="openshift-multus/multus-additional-cni-plugins-qnrf8"
Nov 01 03:05:54.707301 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:05:54.707278    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 03:05:54.707301 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:05:54.707299    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 03:05:54.707407 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:05:54.707309    2576 projected.go:196] Error preparing data for projected volume kube-api-access-nxqx8 for pod openshift-network-diagnostics/network-check-target-mx48d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 03:05:54.707407 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:05:54.707356    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6e302b40-9c43-4822-8c8b-4d021a0590ab-kube-api-access-nxqx8 podName:6e302b40-9c43-4822-8c8b-4d021a0590ab nodeName:}" failed. No retries permitted until 2026-11-01 03:05:55.207343576 +0000 UTC m=+3.261485281 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-nxqx8" (UniqueName: "kubernetes.io/projected/6e302b40-9c43-4822-8c8b-4d021a0590ab-kube-api-access-nxqx8") pod "network-check-target-mx48d" (UID: "6e302b40-9c43-4822-8c8b-4d021a0590ab") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 03:05:54.708384 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.708361    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhdnz\" (UniqueName: \"kubernetes.io/projected/bf9dac1d-d9fc-455a-be03-ea34f8838730-kube-api-access-jhdnz\") pod \"aws-ebs-csi-driver-node-vq4dj\" (UID: \"bf9dac1d-d9fc-455a-be03-ea34f8838730\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vq4dj"
Nov 01 03:05:54.710168 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.710148    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t86zn\" (UniqueName: \"kubernetes.io/projected/d7cef651-3047-4e64-ae7f-b0fa71fbd0fe-kube-api-access-t86zn\") pod \"node-ca-wq27m\" (UID: \"d7cef651-3047-4e64-ae7f-b0fa71fbd0fe\") " pod="openshift-image-registry/node-ca-wq27m"
Nov 01 03:05:54.710268 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.710190    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckfr8\" (UniqueName: \"kubernetes.io/projected/de5f469f-1029-4161-bf51-6ad18a3ac9e2-kube-api-access-ckfr8\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:05:54.710824 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.710800    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h58g6\" (UniqueName: \"kubernetes.io/projected/11fb87e4-ef01-4209-8207-80b66ab09f5e-kube-api-access-h58g6\") pod \"tuned-lkbk6\" (UID: \"11fb87e4-ef01-4209-8207-80b66ab09f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-lkbk6"
Nov 01 03:05:54.710901 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.710829    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2krw\" (UniqueName: \"kubernetes.io/projected/e48a634f-ab31-435c-96c1-c3633c5bc061-kube-api-access-b2krw\") pod \"network-metrics-daemon-dnqjp\" (UID: \"e48a634f-ab31-435c-96c1-c3633c5bc061\") " pod="openshift-multus/network-metrics-daemon-dnqjp"
Nov 01 03:05:54.716848 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.716834    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-fh7sf"
Nov 01 03:05:54.724305 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.724272    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wfg7k"
Nov 01 03:05:54.726955 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:54.726934    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd8260bb_3811_4cdc_91d0_c6ddc693891e.slice/crio-b13508327d908895943975963c426c766b021361d87a9196f741e7bdd976509e WatchSource:0}: Error finding container b13508327d908895943975963c426c766b021361d87a9196f741e7bdd976509e: Status 404 returned error can't find the container with id b13508327d908895943975963c426c766b021361d87a9196f741e7bdd976509e
Nov 01 03:05:54.731956 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.731754    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-pzt6v"
Nov 01 03:05:54.734228 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:54.734207    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9e89456_9acd_408f_87b5_f5fd968c82c3.slice/crio-38e73bd0c99d3b29aac95c81a2e64aa64d2f964154b942bfecd271af059f2c56 WatchSource:0}: Error finding container 38e73bd0c99d3b29aac95c81a2e64aa64d2f964154b942bfecd271af059f2c56: Status 404 returned error can't find the container with id 38e73bd0c99d3b29aac95c81a2e64aa64d2f964154b942bfecd271af059f2c56
Nov 01 03:05:54.737997 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.737978    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qzl9n"
Nov 01 03:05:54.741151 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:54.741129    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod994175bb_d192_47f7_b162_a840eb0f9be1.slice/crio-e109df44865a3f092a8d3d3680d8eee4ecf3f68aaf32da287d971df39293cbda WatchSource:0}: Error finding container e109df44865a3f092a8d3d3680d8eee4ecf3f68aaf32da287d971df39293cbda: Status 404 returned error can't find the container with id e109df44865a3f092a8d3d3680d8eee4ecf3f68aaf32da287d971df39293cbda
Nov 01 03:05:54.748320 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:54.748300    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ede655d_c20d_41cc_b5f0_2db4338c1ba7.slice/crio-02b4fc07486757aadcb2800218a1fc8a3076f38fa1f4185193c2aa7e5ef8e7f6 WatchSource:0}: Error finding container 02b4fc07486757aadcb2800218a1fc8a3076f38fa1f4185193c2aa7e5ef8e7f6: Status 404 returned error can't find the container with id 02b4fc07486757aadcb2800218a1fc8a3076f38fa1f4185193c2aa7e5ef8e7f6
Nov 01 03:05:54.761538 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.761515    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-wq27m"
Nov 01 03:05:54.763850 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.763831    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vq4dj"
Nov 01 03:05:54.766354 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.766340    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qnrf8"
Nov 01 03:05:54.768460 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.768441    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lkbk6"
Nov 01 03:05:54.771038 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:54.771019    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7cef651_3047_4e64_ae7f_b0fa71fbd0fe.slice/crio-c2f90ee59606115f6de4c50b793a548e266a500e6c80b26671d87b883f9d7fec WatchSource:0}: Error finding container c2f90ee59606115f6de4c50b793a548e266a500e6c80b26671d87b883f9d7fec: Status 404 returned error can't find the container with id c2f90ee59606115f6de4c50b793a548e266a500e6c80b26671d87b883f9d7fec
Nov 01 03:05:54.774033 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:54.774013    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf9dac1d_d9fc_455a_be03_ea34f8838730.slice/crio-8eda87d96f5445f76e7e3a31f88f1b1cfb59bd38489e93b16b05392b80c42192 WatchSource:0}: Error finding container 8eda87d96f5445f76e7e3a31f88f1b1cfb59bd38489e93b16b05392b80c42192: Status 404 returned error can't find the container with id 8eda87d96f5445f76e7e3a31f88f1b1cfb59bd38489e93b16b05392b80c42192
Nov 01 03:05:54.777773 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:54.777754    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8f2a24b_e770_4615_a2c7_b2d065e14f77.slice/crio-a4edacea31fe2dce66095cabb9f2f3de51aa76066e1021f1a6fb8287ba47933f WatchSource:0}: Error finding container a4edacea31fe2dce66095cabb9f2f3de51aa76066e1021f1a6fb8287ba47933f: Status 404 returned error can't find the container with id a4edacea31fe2dce66095cabb9f2f3de51aa76066e1021f1a6fb8287ba47933f
Nov 01 03:05:54.781025 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:05:54.781006    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11fb87e4_ef01_4209_8207_80b66ab09f5e.slice/crio-1c21a168b73fa874c4996e1b192a4c090cb60dfdb88f0eb5a18a28301b217c66 WatchSource:0}: Error finding container 1c21a168b73fa874c4996e1b192a4c090cb60dfdb88f0eb5a18a28301b217c66: Status 404 returned error can't find the container with id 1c21a168b73fa874c4996e1b192a4c090cb60dfdb88f0eb5a18a28301b217c66
Nov 01 03:05:54.786417 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:54.786401    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:05:55.135213 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:55.135125    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e48a634f-ab31-435c-96c1-c3633c5bc061-metrics-certs\") pod \"network-metrics-daemon-dnqjp\" (UID: \"e48a634f-ab31-435c-96c1-c3633c5bc061\") " pod="openshift-multus/network-metrics-daemon-dnqjp"
Nov 01 03:05:55.135379 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:55.135211    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0303d6c6-c4ec-4403-af74-f4bcb5cbf2a4-original-pull-secret\") pod \"global-pull-secret-syncer-rrdzd\" (UID: \"0303d6c6-c4ec-4403-af74-f4bcb5cbf2a4\") " pod="kube-system/global-pull-secret-syncer-rrdzd"
Nov 01 03:05:55.135379 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:05:55.135316    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 03:05:55.135379 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:05:55.135334    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 03:05:55.135523 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:05:55.135381    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e48a634f-ab31-435c-96c1-c3633c5bc061-metrics-certs podName:e48a634f-ab31-435c-96c1-c3633c5bc061 nodeName:}" failed. No retries permitted until 2026-11-01 03:05:56.135361592 +0000 UTC m=+4.189503271 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e48a634f-ab31-435c-96c1-c3633c5bc061-metrics-certs") pod "network-metrics-daemon-dnqjp" (UID: "e48a634f-ab31-435c-96c1-c3633c5bc061") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 03:05:55.135523 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:05:55.135402    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0303d6c6-c4ec-4403-af74-f4bcb5cbf2a4-original-pull-secret podName:0303d6c6-c4ec-4403-af74-f4bcb5cbf2a4 nodeName:}" failed. No retries permitted until 2026-11-01 03:05:56.135391558 +0000 UTC m=+4.189533220 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0303d6c6-c4ec-4403-af74-f4bcb5cbf2a4-original-pull-secret") pod "global-pull-secret-syncer-rrdzd" (UID: "0303d6c6-c4ec-4403-af74-f4bcb5cbf2a4") : object "kube-system"/"original-pull-secret" not registered
Nov 01 03:05:55.236055 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:55.236021    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxqx8\" (UniqueName: \"kubernetes.io/projected/6e302b40-9c43-4822-8c8b-4d021a0590ab-kube-api-access-nxqx8\") pod \"network-check-target-mx48d\" (UID: \"6e302b40-9c43-4822-8c8b-4d021a0590ab\") " pod="openshift-network-diagnostics/network-check-target-mx48d"
Nov 01 03:05:55.236244 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:05:55.236217    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 03:05:55.236244 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:05:55.236240    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 03:05:55.236367 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:05:55.236253    2576 projected.go:196] Error preparing data for projected volume kube-api-access-nxqx8 for pod openshift-network-diagnostics/network-check-target-mx48d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 03:05:55.236367 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:05:55.236319    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6e302b40-9c43-4822-8c8b-4d021a0590ab-kube-api-access-nxqx8 podName:6e302b40-9c43-4822-8c8b-4d021a0590ab nodeName:}" failed. No retries permitted until 2026-11-01 03:05:56.2363 +0000 UTC m=+4.290441669 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-nxqx8" (UniqueName: "kubernetes.io/projected/6e302b40-9c43-4822-8c8b-4d021a0590ab-kube-api-access-nxqx8") pod "network-check-target-mx48d" (UID: "6e302b40-9c43-4822-8c8b-4d021a0590ab") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 03:05:55.397967 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:55.397815    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qnrf8" event={"ID":"e8f2a24b-e770-4615-a2c7-b2d065e14f77","Type":"ContainerStarted","Data":"a4edacea31fe2dce66095cabb9f2f3de51aa76066e1021f1a6fb8287ba47933f"}
Nov 01 03:05:55.399533 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:55.399495    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vq4dj" event={"ID":"bf9dac1d-d9fc-455a-be03-ea34f8838730","Type":"ContainerStarted","Data":"8eda87d96f5445f76e7e3a31f88f1b1cfb59bd38489e93b16b05392b80c42192"}
Nov 01 03:05:55.400982 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:55.400950    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wfg7k" event={"ID":"c9e89456-9acd-408f-87b5-f5fd968c82c3","Type":"ContainerStarted","Data":"38e73bd0c99d3b29aac95c81a2e64aa64d2f964154b942bfecd271af059f2c56"}
Nov 01 03:05:55.402848 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:55.402823    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-fh7sf" event={"ID":"bd8260bb-3811-4cdc-91d0-c6ddc693891e","Type":"ContainerStarted","Data":"b13508327d908895943975963c426c766b021361d87a9196f741e7bdd976509e"}
Nov 01 03:05:55.404216 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:55.404189    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lkbk6" event={"ID":"11fb87e4-ef01-4209-8207-80b66ab09f5e","Type":"ContainerStarted","Data":"1c21a168b73fa874c4996e1b192a4c090cb60dfdb88f0eb5a18a28301b217c66"}
Nov 01 03:05:55.405490 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:55.405459    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-wq27m" event={"ID":"d7cef651-3047-4e64-ae7f-b0fa71fbd0fe","Type":"ContainerStarted","Data":"c2f90ee59606115f6de4c50b793a548e266a500e6c80b26671d87b883f9d7fec"}
Nov 01 03:05:55.406824 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:55.406803    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qzl9n" event={"ID":"0ede655d-c20d-41cc-b5f0-2db4338c1ba7","Type":"ContainerStarted","Data":"02b4fc07486757aadcb2800218a1fc8a3076f38fa1f4185193c2aa7e5ef8e7f6"}
Nov 01 03:05:55.408237 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:55.408204    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-pzt6v" event={"ID":"994175bb-d192-47f7-b162-a840eb0f9be1","Type":"ContainerStarted","Data":"e109df44865a3f092a8d3d3680d8eee4ecf3f68aaf32da287d971df39293cbda"}
Nov 01 03:05:55.409660 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:55.409636    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9" event={"ID":"de5f469f-1029-4161-bf51-6ad18a3ac9e2","Type":"ContainerStarted","Data":"65bca1ddcd7a850283c27829fa3fb5a6b5f48f97e5bad23b011d4a2f20816cb8"}
Nov 01 03:05:55.522823 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:55.522783    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 03:00:54 +0000 UTC" deadline="2028-05-18 18:59:42.830697543 +0000 UTC"
Nov 01 03:05:55.522823 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:55.522816    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13551h53m47.30788561s"
Nov 01 03:05:56.142465 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:56.142421    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0303d6c6-c4ec-4403-af74-f4bcb5cbf2a4-original-pull-secret\") pod \"global-pull-secret-syncer-rrdzd\" (UID: \"0303d6c6-c4ec-4403-af74-f4bcb5cbf2a4\") " pod="kube-system/global-pull-secret-syncer-rrdzd"
Nov 01 03:05:56.142670 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:56.142498    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e48a634f-ab31-435c-96c1-c3633c5bc061-metrics-certs\") pod \"network-metrics-daemon-dnqjp\" (UID: \"e48a634f-ab31-435c-96c1-c3633c5bc061\") " pod="openshift-multus/network-metrics-daemon-dnqjp"
Nov 01 03:05:56.142670 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:05:56.142649    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 03:05:56.142803 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:05:56.142711    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e48a634f-ab31-435c-96c1-c3633c5bc061-metrics-certs podName:e48a634f-ab31-435c-96c1-c3633c5bc061 nodeName:}" failed. No retries permitted until 2026-11-01 03:05:58.142692311 +0000 UTC m=+6.196833991 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e48a634f-ab31-435c-96c1-c3633c5bc061-metrics-certs") pod "network-metrics-daemon-dnqjp" (UID: "e48a634f-ab31-435c-96c1-c3633c5bc061") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 03:05:56.143310 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:05:56.143120    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 03:05:56.143310 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:05:56.143170    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0303d6c6-c4ec-4403-af74-f4bcb5cbf2a4-original-pull-secret podName:0303d6c6-c4ec-4403-af74-f4bcb5cbf2a4 nodeName:}" failed. No retries permitted until 2026-11-01 03:05:58.143155517 +0000 UTC m=+6.197297196 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0303d6c6-c4ec-4403-af74-f4bcb5cbf2a4-original-pull-secret") pod "global-pull-secret-syncer-rrdzd" (UID: "0303d6c6-c4ec-4403-af74-f4bcb5cbf2a4") : object "kube-system"/"original-pull-secret" not registered
Nov 01 03:05:56.243413 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:56.243357    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxqx8\" (UniqueName: \"kubernetes.io/projected/6e302b40-9c43-4822-8c8b-4d021a0590ab-kube-api-access-nxqx8\") pod \"network-check-target-mx48d\" (UID: \"6e302b40-9c43-4822-8c8b-4d021a0590ab\") " pod="openshift-network-diagnostics/network-check-target-mx48d"
Nov 01 03:05:56.243604 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:05:56.243580    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 03:05:56.243604 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:05:56.243602    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 03:05:56.243808 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:05:56.243615    2576 projected.go:196] Error preparing data for projected volume kube-api-access-nxqx8 for pod openshift-network-diagnostics/network-check-target-mx48d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 03:05:56.243808 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:05:56.243672    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6e302b40-9c43-4822-8c8b-4d021a0590ab-kube-api-access-nxqx8 podName:6e302b40-9c43-4822-8c8b-4d021a0590ab nodeName:}" failed. No retries permitted until 2026-11-01 03:05:58.243655534 +0000 UTC m=+6.297797214 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-nxqx8" (UniqueName: "kubernetes.io/projected/6e302b40-9c43-4822-8c8b-4d021a0590ab-kube-api-access-nxqx8") pod "network-check-target-mx48d" (UID: "6e302b40-9c43-4822-8c8b-4d021a0590ab") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 03:05:56.392312 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:56.392277    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dnqjp"
Nov 01 03:05:56.392503 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:05:56.392425    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dnqjp" podUID="e48a634f-ab31-435c-96c1-c3633c5bc061"
Nov 01 03:05:56.392975 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:56.392953    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rrdzd"
Nov 01 03:05:56.393088 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:05:56.393065    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rrdzd" podUID="0303d6c6-c4ec-4403-af74-f4bcb5cbf2a4"
Nov 01 03:05:56.393284 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:56.393245    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mx48d"
Nov 01 03:05:56.393362 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:05:56.393340    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mx48d" podUID="6e302b40-9c43-4822-8c8b-4d021a0590ab"
Nov 01 03:05:56.523825 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:56.523374    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 03:00:54 +0000 UTC" deadline="2028-06-07 18:56:38.425428353 +0000 UTC"
Nov 01 03:05:56.523825 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:56.523437    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14031h50m41.901995292s"
Nov 01 03:05:57.438579 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:57.438381    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-54.us-west-1.compute.internal" event={"ID":"f068e0e9089d99d5fd3ec23628aa7d71","Type":"ContainerStarted","Data":"d5b61e0bd6f02fdbc487a22e3a2d4c28bdb394f605a9d71854d455f712280907"}
Nov 01 03:05:58.161362 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:58.161323    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0303d6c6-c4ec-4403-af74-f4bcb5cbf2a4-original-pull-secret\") pod \"global-pull-secret-syncer-rrdzd\" (UID: \"0303d6c6-c4ec-4403-af74-f4bcb5cbf2a4\") " pod="kube-system/global-pull-secret-syncer-rrdzd"
Nov 01 03:05:58.161852 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:58.161398    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e48a634f-ab31-435c-96c1-c3633c5bc061-metrics-certs\") pod \"network-metrics-daemon-dnqjp\" (UID: \"e48a634f-ab31-435c-96c1-c3633c5bc061\") " pod="openshift-multus/network-metrics-daemon-dnqjp"
Nov 01 03:05:58.161852 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:05:58.161528    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 03:05:58.161852 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:05:58.161592    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e48a634f-ab31-435c-96c1-c3633c5bc061-metrics-certs podName:e48a634f-ab31-435c-96c1-c3633c5bc061 nodeName:}" failed. No retries permitted until 2026-11-01 03:06:02.161573999 +0000 UTC m=+10.215715683 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e48a634f-ab31-435c-96c1-c3633c5bc061-metrics-certs") pod "network-metrics-daemon-dnqjp" (UID: "e48a634f-ab31-435c-96c1-c3633c5bc061") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 03:05:58.162060 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:05:58.162018    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 03:05:58.162060 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:05:58.162067    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0303d6c6-c4ec-4403-af74-f4bcb5cbf2a4-original-pull-secret podName:0303d6c6-c4ec-4403-af74-f4bcb5cbf2a4 nodeName:}" failed. No retries permitted until 2026-11-01 03:06:02.16205218 +0000 UTC m=+10.216193842 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0303d6c6-c4ec-4403-af74-f4bcb5cbf2a4-original-pull-secret") pod "global-pull-secret-syncer-rrdzd" (UID: "0303d6c6-c4ec-4403-af74-f4bcb5cbf2a4") : object "kube-system"/"original-pull-secret" not registered
Nov 01 03:05:58.262595 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:58.262285    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxqx8\" (UniqueName: \"kubernetes.io/projected/6e302b40-9c43-4822-8c8b-4d021a0590ab-kube-api-access-nxqx8\") pod \"network-check-target-mx48d\" (UID: \"6e302b40-9c43-4822-8c8b-4d021a0590ab\") " pod="openshift-network-diagnostics/network-check-target-mx48d"
Nov 01 03:05:58.262595 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:05:58.262466    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 03:05:58.262595 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:05:58.262492    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 03:05:58.262595 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:05:58.262503    2576 projected.go:196] Error preparing data for projected volume kube-api-access-nxqx8 for pod openshift-network-diagnostics/network-check-target-mx48d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 03:05:58.262595 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:05:58.262560    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6e302b40-9c43-4822-8c8b-4d021a0590ab-kube-api-access-nxqx8 podName:6e302b40-9c43-4822-8c8b-4d021a0590ab nodeName:}" failed. No retries permitted until 2026-11-01 03:06:02.262542684 +0000 UTC m=+10.316684349 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-nxqx8" (UniqueName: "kubernetes.io/projected/6e302b40-9c43-4822-8c8b-4d021a0590ab-kube-api-access-nxqx8") pod "network-check-target-mx48d" (UID: "6e302b40-9c43-4822-8c8b-4d021a0590ab") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 03:05:58.388388 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:58.387961    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rrdzd"
Nov 01 03:05:58.388388 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:05:58.388085    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rrdzd" podUID="0303d6c6-c4ec-4403-af74-f4bcb5cbf2a4"
Nov 01 03:05:58.388388 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:58.388132    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dnqjp"
Nov 01 03:05:58.388388 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:05:58.388259    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dnqjp" podUID="e48a634f-ab31-435c-96c1-c3633c5bc061"
Nov 01 03:05:58.388388 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:58.388288    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mx48d"
Nov 01 03:05:58.388388 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:05:58.388362    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mx48d" podUID="6e302b40-9c43-4822-8c8b-4d021a0590ab"
Nov 01 03:05:58.444114 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:58.444018    2576 generic.go:356] "Generic (PLEG): container finished" podID="f068e0e9089d99d5fd3ec23628aa7d71" containerID="d5b61e0bd6f02fdbc487a22e3a2d4c28bdb394f605a9d71854d455f712280907" exitCode=0
Nov 01 03:05:58.444231 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:58.444117    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-54.us-west-1.compute.internal" event={"ID":"f068e0e9089d99d5fd3ec23628aa7d71","Type":"ContainerDied","Data":"d5b61e0bd6f02fdbc487a22e3a2d4c28bdb394f605a9d71854d455f712280907"}
Nov 01 03:05:58.468414 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:58.468389    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qnrf8" event={"ID":"e8f2a24b-e770-4615-a2c7-b2d065e14f77","Type":"ContainerStarted","Data":"dbaa0818d7e1dce276a6f3de3ca017c5753fbea77adadded69796601fe2ac532"}
Nov 01 03:05:59.486883 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:59.486806    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-fh7sf" event={"ID":"bd8260bb-3811-4cdc-91d0-c6ddc693891e","Type":"ContainerStarted","Data":"48e117845eb7cf3d660efff2c312b45e75d8b191cacfba012bb4e2a63c7e79de"}
Nov 01 03:05:59.496433 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:59.496402    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-54.us-west-1.compute.internal" event={"ID":"f068e0e9089d99d5fd3ec23628aa7d71","Type":"ContainerStarted","Data":"b6f2d1fcb60f2c8916321d6db336f078cb62e96c75c5d09329168b0be1027ffc"}
Nov 01 03:05:59.499424 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:59.499394    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-54.us-west-1.compute.internal" event={"ID":"8cf98fd570993ef6ee2b41a9af4899c8","Type":"ContainerStarted","Data":"4a0cf118fcd331a054fa79cc9f2b03a569b1ee27a9f76a3c15da4c3407d6a68b"}
Nov 01 03:05:59.502391 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:59.502360    2576 generic.go:356] "Generic (PLEG): container finished" podID="e8f2a24b-e770-4615-a2c7-b2d065e14f77" containerID="dbaa0818d7e1dce276a6f3de3ca017c5753fbea77adadded69796601fe2ac532" exitCode=0
Nov 01 03:05:59.502498 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:59.502402    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qnrf8" event={"ID":"e8f2a24b-e770-4615-a2c7-b2d065e14f77","Type":"ContainerDied","Data":"dbaa0818d7e1dce276a6f3de3ca017c5753fbea77adadded69796601fe2ac532"}
Nov 01 03:05:59.609902 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:59.609844    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-fh7sf" podStartSLOduration=2.229441589 podStartE2EDuration="6.609824657s" totalImagesPullingTime="4.380383068s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:05:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:05:59.608659462 +0000 UTC m=+7.662801146" watchObservedRunningTime="2026-11-01 03:05:59.609824657 +0000 UTC m=+7.663966341"
Nov 01 03:05:59.725719 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:05:59.725256    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-54.us-west-1.compute.internal" podStartSLOduration=5.725237233 podStartE2EDuration="5.725237233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 03:05:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:05:59.725148742 +0000 UTC m=+7.779290422" watchObservedRunningTime="2026-11-01 03:05:59.725237233 +0000 UTC m=+7.779378926"
Nov 01 03:06:00.390582 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:00.389809    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mx48d"
Nov 01 03:06:00.390582 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:06:00.389935    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mx48d" podUID="6e302b40-9c43-4822-8c8b-4d021a0590ab"
Nov 01 03:06:00.390582 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:00.390316    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rrdzd"
Nov 01 03:06:00.390582 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:06:00.390407    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rrdzd" podUID="0303d6c6-c4ec-4403-af74-f4bcb5cbf2a4"
Nov 01 03:06:00.390582 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:00.390468    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dnqjp"
Nov 01 03:06:00.390582 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:06:00.390538    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dnqjp" podUID="e48a634f-ab31-435c-96c1-c3633c5bc061"
Nov 01 03:06:01.513582 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:01.513507    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-wq27m" event={"ID":"d7cef651-3047-4e64-ae7f-b0fa71fbd0fe","Type":"ContainerStarted","Data":"f9848fe7e1a937add617a70004ea7e7c6da40c4622f7b07f31077984d8c0bfba"}
Nov 01 03:06:01.516719 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:01.516673    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vq4dj" event={"ID":"bf9dac1d-d9fc-455a-be03-ea34f8838730","Type":"ContainerStarted","Data":"0a287d03d20b65e90e48be51efb80548d3ef665d81ef35f65db09115e5f88ee4"}
Nov 01 03:06:01.635460 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:01.635414    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-wq27m" podStartSLOduration=2.641047778 podStartE2EDuration="8.635398732s" totalImagesPullingTime="5.994350954s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 03:05:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:06:01.634464444 +0000 UTC m=+9.688606128" watchObservedRunningTime="2026-11-01 03:06:01.635398732 +0000 UTC m=+9.689540414"
Nov 01 03:06:01.635594 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:01.635483    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-54.us-west-1.compute.internal" podStartSLOduration=7.635476613 podStartE2EDuration="7.635476613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 03:05:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:05:59.844597193 +0000 UTC m=+7.898738875" watchObservedRunningTime="2026-11-01 03:06:01.635476613 +0000 UTC m=+9.689618295"
Nov 01 03:06:01.795851 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:01.795257    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-fh7sf"
Nov 01 03:06:02.197413 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:02.196640    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e48a634f-ab31-435c-96c1-c3633c5bc061-metrics-certs\") pod \"network-metrics-daemon-dnqjp\" (UID: \"e48a634f-ab31-435c-96c1-c3633c5bc061\") " pod="openshift-multus/network-metrics-daemon-dnqjp"
Nov 01 03:06:02.197413 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:02.196737    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0303d6c6-c4ec-4403-af74-f4bcb5cbf2a4-original-pull-secret\") pod \"global-pull-secret-syncer-rrdzd\" (UID: \"0303d6c6-c4ec-4403-af74-f4bcb5cbf2a4\") " pod="kube-system/global-pull-secret-syncer-rrdzd"
Nov 01 03:06:02.197413 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:06:02.196859    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 03:06:02.197413 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:06:02.196917    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0303d6c6-c4ec-4403-af74-f4bcb5cbf2a4-original-pull-secret podName:0303d6c6-c4ec-4403-af74-f4bcb5cbf2a4 nodeName:}" failed. No retries permitted until 2026-11-01 03:06:10.19690015 +0000 UTC m=+18.251041824 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0303d6c6-c4ec-4403-af74-f4bcb5cbf2a4-original-pull-secret") pod "global-pull-secret-syncer-rrdzd" (UID: "0303d6c6-c4ec-4403-af74-f4bcb5cbf2a4") : object "kube-system"/"original-pull-secret" not registered
Nov 01 03:06:02.197413 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:06:02.197291    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 03:06:02.197413 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:06:02.197336    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e48a634f-ab31-435c-96c1-c3633c5bc061-metrics-certs podName:e48a634f-ab31-435c-96c1-c3633c5bc061 nodeName:}" failed. No retries permitted until 2026-11-01 03:06:10.197319876 +0000 UTC m=+18.251461554 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e48a634f-ab31-435c-96c1-c3633c5bc061-metrics-certs") pod "network-metrics-daemon-dnqjp" (UID: "e48a634f-ab31-435c-96c1-c3633c5bc061") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 03:06:02.297823 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:02.297789    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxqx8\" (UniqueName: \"kubernetes.io/projected/6e302b40-9c43-4822-8c8b-4d021a0590ab-kube-api-access-nxqx8\") pod \"network-check-target-mx48d\" (UID: \"6e302b40-9c43-4822-8c8b-4d021a0590ab\") " pod="openshift-network-diagnostics/network-check-target-mx48d"
Nov 01 03:06:02.298001 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:06:02.297935    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 03:06:02.298001 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:06:02.297955    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 03:06:02.298001 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:06:02.297966    2576 projected.go:196] Error preparing data for projected volume kube-api-access-nxqx8 for pod openshift-network-diagnostics/network-check-target-mx48d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 03:06:02.298162 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:06:02.298021    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6e302b40-9c43-4822-8c8b-4d021a0590ab-kube-api-access-nxqx8 podName:6e302b40-9c43-4822-8c8b-4d021a0590ab nodeName:}" failed. No retries permitted until 2026-11-01 03:06:10.298004834 +0000 UTC m=+18.352146496 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-nxqx8" (UniqueName: "kubernetes.io/projected/6e302b40-9c43-4822-8c8b-4d021a0590ab-kube-api-access-nxqx8") pod "network-check-target-mx48d" (UID: "6e302b40-9c43-4822-8c8b-4d021a0590ab") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 03:06:02.404558 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:02.403780    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rrdzd"
Nov 01 03:06:02.404558 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:06:02.403943    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rrdzd" podUID="0303d6c6-c4ec-4403-af74-f4bcb5cbf2a4"
Nov 01 03:06:02.404558 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:02.404316    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dnqjp"
Nov 01 03:06:02.404558 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:06:02.404506    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dnqjp" podUID="e48a634f-ab31-435c-96c1-c3633c5bc061"
Nov 01 03:06:02.407596 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:02.407401    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mx48d"
Nov 01 03:06:02.407596 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:06:02.407521    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mx48d" podUID="6e302b40-9c43-4822-8c8b-4d021a0590ab"
Nov 01 03:06:02.769304 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:02.768302    2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-fh7sf"
Nov 01 03:06:02.770505 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:02.770284    2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-fh7sf"
Nov 01 03:06:03.350662 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:03.350620    2576 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 01 03:06:03.400669 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:03.400288    2576 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-01T03:06:03.350641054Z","UUID":"f7375064-9d0c-4c57-ad0b-55b62f83608a","Handler":null,"Name":"","Endpoint":""}
Nov 01 03:06:03.403794 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:03.403332    2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 01 03:06:03.403794 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:03.403373    2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 01 03:06:03.524456 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:03.524413    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vq4dj" event={"ID":"bf9dac1d-d9fc-455a-be03-ea34f8838730","Type":"ContainerStarted","Data":"0658db641a1965f7979fc4e9afcc39049988999d9c32a4336c11f254d78d17e1"}
Nov 01 03:06:03.526803 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:03.526774    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qzl9n" event={"ID":"0ede655d-c20d-41cc-b5f0-2db4338c1ba7","Type":"ContainerStarted","Data":"feff69479764090abcdcd83424a41c48e9021d8829279cc6cc09a76fbc817a40"}
Nov 01 03:06:03.527308 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:03.527289    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-fh7sf"
Nov 01 03:06:03.766267 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:03.766214    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-qzl9n" podStartSLOduration=2.439265191 podStartE2EDuration="10.766200671s" totalImagesPullingTime="8.32693548s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 03:05:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:06:03.647546438 +0000 UTC m=+11.701688123" watchObservedRunningTime="2026-11-01 03:06:03.766200671 +0000 UTC m=+11.820342353"
Nov 01 03:06:04.388214 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:04.388129    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rrdzd"
Nov 01 03:06:04.388214 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:04.388198    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dnqjp"
Nov 01 03:06:04.388742 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:04.388216    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mx48d"
Nov 01 03:06:04.388742 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:06:04.388312    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dnqjp" podUID="e48a634f-ab31-435c-96c1-c3633c5bc061"
Nov 01 03:06:04.388742 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:06:04.388417    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rrdzd" podUID="0303d6c6-c4ec-4403-af74-f4bcb5cbf2a4"
Nov 01 03:06:04.388742 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:06:04.388501    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mx48d" podUID="6e302b40-9c43-4822-8c8b-4d021a0590ab"
Nov 01 03:06:04.530555 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:04.530512    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-pzt6v" event={"ID":"994175bb-d192-47f7-b162-a840eb0f9be1","Type":"ContainerStarted","Data":"d632cc2a37da5346c367ba0b56fad9d2d3466552b3fe66b093ba85c3f596a40e"}
Nov 01 03:06:04.532699 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:04.532640    2576 generic.go:356] "Generic (PLEG): container finished" podID="e8f2a24b-e770-4615-a2c7-b2d065e14f77" containerID="06cabc60d357d7f0b9685a86c5469302f4fcb57b97986a1fece1b2e010839133" exitCode=0
Nov 01 03:06:04.532839 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:04.532761    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qnrf8" event={"ID":"e8f2a24b-e770-4615-a2c7-b2d065e14f77","Type":"ContainerDied","Data":"06cabc60d357d7f0b9685a86c5469302f4fcb57b97986a1fece1b2e010839133"}
Nov 01 03:06:04.537025 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:04.536996    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wfg7k" event={"ID":"c9e89456-9acd-408f-87b5-f5fd968c82c3","Type":"ContainerStarted","Data":"e093764fd9002a6f171b4bee5826e4792d4853442c85a59bbe5ad445159d8743"}
Nov 01 03:06:04.648526 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:04.648421    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-pzt6v" podStartSLOduration=2.900210517 podStartE2EDuration="11.648402321s" totalImagesPullingTime="8.748191804s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 03:05:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:06:04.647817994 +0000 UTC m=+12.701959678" watchObservedRunningTime="2026-11-01 03:06:04.648402321 +0000 UTC m=+12.702544006"
Nov 01 03:06:04.895973 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:04.895921    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-wfg7k" podStartSLOduration=3.635232608 podStartE2EDuration="11.895906768s" totalImagesPullingTime="8.26067416s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 03:05:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:06:04.895387111 +0000 UTC m=+12.949528795" watchObservedRunningTime="2026-11-01 03:06:04.895906768 +0000 UTC m=+12.950048451"
Nov 01 03:06:05.541135 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:05.541054    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vq4dj" event={"ID":"bf9dac1d-d9fc-455a-be03-ea34f8838730","Type":"ContainerStarted","Data":"2e3ddefc8aa35cac44e5748d2bbf5295d4aeacf90739e55ef269c8ff201e8f47"}
Nov 01 03:06:05.661532 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:05.661479    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vq4dj" podStartSLOduration=2.6044975729999997 podStartE2EDuration="12.661465029s" totalImagesPullingTime="10.056967456s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 03:05:53 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:06:05.661374899 +0000 UTC m=+13.715516584" watchObservedRunningTime="2026-11-01 03:06:05.661465029 +0000 UTC m=+13.715606712"
Nov 01 03:06:06.387987 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:06.387924    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mx48d"
Nov 01 03:06:06.387987 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:06.387948    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dnqjp"
Nov 01 03:06:06.387987 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:06.387965    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rrdzd"
Nov 01 03:06:06.388166 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:06:06.388042    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mx48d" podUID="6e302b40-9c43-4822-8c8b-4d021a0590ab"
Nov 01 03:06:06.388166 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:06:06.388112    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dnqjp" podUID="e48a634f-ab31-435c-96c1-c3633c5bc061"
Nov 01 03:06:06.388166 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:06:06.388160    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rrdzd" podUID="0303d6c6-c4ec-4403-af74-f4bcb5cbf2a4"
Nov 01 03:06:06.544791 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:06.544760    2576 generic.go:356] "Generic (PLEG): container finished" podID="e8f2a24b-e770-4615-a2c7-b2d065e14f77" containerID="2e7969b36ffd11c38396cb69f58aa57521a56690c344a881ec17f42a14e3cd56" exitCode=0
Nov 01 03:06:06.545410 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:06.544842    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qnrf8" event={"ID":"e8f2a24b-e770-4615-a2c7-b2d065e14f77","Type":"ContainerDied","Data":"2e7969b36ffd11c38396cb69f58aa57521a56690c344a881ec17f42a14e3cd56"}
Nov 01 03:06:06.546544 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:06.546522    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lkbk6" event={"ID":"11fb87e4-ef01-4209-8207-80b66ab09f5e","Type":"ContainerStarted","Data":"ecc9437a060b64c2836d277742d4c16b66fc5ee0b570db2b49f1e0aa509cfe07"}
Nov 01 03:06:06.787614 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:06.787522    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-lkbk6" podStartSLOduration=2.488338817 podStartE2EDuration="13.787507805s" totalImagesPullingTime="11.299168988s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 03:05:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:06:06.787321568 +0000 UTC m=+14.841463250" watchObservedRunningTime="2026-11-01 03:06:06.787507805 +0000 UTC m=+14.841649488"
Nov 01 03:06:08.388106 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:08.387918    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dnqjp"
Nov 01 03:06:08.388622 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:08.387918    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rrdzd"
Nov 01 03:06:08.388622 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:06:08.388217    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dnqjp" podUID="e48a634f-ab31-435c-96c1-c3633c5bc061"
Nov 01 03:06:08.388622 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:06:08.388304    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rrdzd" podUID="0303d6c6-c4ec-4403-af74-f4bcb5cbf2a4"
Nov 01 03:06:08.388622 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:08.387918    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mx48d"
Nov 01 03:06:08.388622 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:06:08.388408    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mx48d" podUID="6e302b40-9c43-4822-8c8b-4d021a0590ab"
Nov 01 03:06:08.555438 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:08.554490    2576 generic.go:356] "Generic (PLEG): container finished" podID="e8f2a24b-e770-4615-a2c7-b2d065e14f77" containerID="7a7a6f067ff3ff337f9d2ed6c781228d058576d5f90fe2568fbf31960eda720e" exitCode=0
Nov 01 03:06:08.555438 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:08.554546    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qnrf8" event={"ID":"e8f2a24b-e770-4615-a2c7-b2d065e14f77","Type":"ContainerDied","Data":"7a7a6f067ff3ff337f9d2ed6c781228d058576d5f90fe2568fbf31960eda720e"}
Nov 01 03:06:10.277896 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:10.277859    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0303d6c6-c4ec-4403-af74-f4bcb5cbf2a4-original-pull-secret\") pod \"global-pull-secret-syncer-rrdzd\" (UID: \"0303d6c6-c4ec-4403-af74-f4bcb5cbf2a4\") " pod="kube-system/global-pull-secret-syncer-rrdzd"
Nov 01 03:06:10.278379 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:10.277924    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e48a634f-ab31-435c-96c1-c3633c5bc061-metrics-certs\") pod \"network-metrics-daemon-dnqjp\" (UID: \"e48a634f-ab31-435c-96c1-c3633c5bc061\") " pod="openshift-multus/network-metrics-daemon-dnqjp"
Nov 01 03:06:10.278379 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:06:10.278017    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 03:06:10.278379 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:06:10.278047    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 03:06:10.278379 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:06:10.278088    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0303d6c6-c4ec-4403-af74-f4bcb5cbf2a4-original-pull-secret podName:0303d6c6-c4ec-4403-af74-f4bcb5cbf2a4 nodeName:}" failed. No retries permitted until 2026-11-01 03:06:26.278069729 +0000 UTC m=+34.332211404 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0303d6c6-c4ec-4403-af74-f4bcb5cbf2a4-original-pull-secret") pod "global-pull-secret-syncer-rrdzd" (UID: "0303d6c6-c4ec-4403-af74-f4bcb5cbf2a4") : object "kube-system"/"original-pull-secret" not registered
Nov 01 03:06:10.278379 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:06:10.278103    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e48a634f-ab31-435c-96c1-c3633c5bc061-metrics-certs podName:e48a634f-ab31-435c-96c1-c3633c5bc061 nodeName:}" failed. No retries permitted until 2026-11-01 03:06:26.278096577 +0000 UTC m=+34.332238241 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e48a634f-ab31-435c-96c1-c3633c5bc061-metrics-certs") pod "network-metrics-daemon-dnqjp" (UID: "e48a634f-ab31-435c-96c1-c3633c5bc061") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 03:06:10.379055 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:10.379021    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxqx8\" (UniqueName: \"kubernetes.io/projected/6e302b40-9c43-4822-8c8b-4d021a0590ab-kube-api-access-nxqx8\") pod \"network-check-target-mx48d\" (UID: \"6e302b40-9c43-4822-8c8b-4d021a0590ab\") " pod="openshift-network-diagnostics/network-check-target-mx48d"
Nov 01 03:06:10.379216 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:06:10.379178    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 03:06:10.379216 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:06:10.379202    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 03:06:10.379216 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:06:10.379214    2576 projected.go:196] Error preparing data for projected volume kube-api-access-nxqx8 for pod openshift-network-diagnostics/network-check-target-mx48d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 03:06:10.379343 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:06:10.379261    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6e302b40-9c43-4822-8c8b-4d021a0590ab-kube-api-access-nxqx8 podName:6e302b40-9c43-4822-8c8b-4d021a0590ab nodeName:}" failed. No retries permitted until 2026-11-01 03:06:26.379248361 +0000 UTC m=+34.433390026 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-nxqx8" (UniqueName: "kubernetes.io/projected/6e302b40-9c43-4822-8c8b-4d021a0590ab-kube-api-access-nxqx8") pod "network-check-target-mx48d" (UID: "6e302b40-9c43-4822-8c8b-4d021a0590ab") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 03:06:10.388140 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:10.388115    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mx48d"
Nov 01 03:06:10.388234 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:10.388143    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dnqjp"
Nov 01 03:06:10.388234 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:06:10.388215    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mx48d" podUID="6e302b40-9c43-4822-8c8b-4d021a0590ab"
Nov 01 03:06:10.388327 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:10.388115    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rrdzd"
Nov 01 03:06:10.388327 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:06:10.388304    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dnqjp" podUID="e48a634f-ab31-435c-96c1-c3633c5bc061"
Nov 01 03:06:10.388400 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:06:10.388362    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rrdzd" podUID="0303d6c6-c4ec-4403-af74-f4bcb5cbf2a4"
Nov 01 03:06:11.563141 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:11.563111    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9" event={"ID":"de5f469f-1029-4161-bf51-6ad18a3ac9e2","Type":"ContainerStarted","Data":"b898e434e1f9430fd4bcffcc3b4bc49da3dae015561cf6fac1ac6f2c2047ba16"}
Nov 01 03:06:11.566031 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:11.565865    2576 generic.go:356] "Generic (PLEG): container finished" podID="e8f2a24b-e770-4615-a2c7-b2d065e14f77" containerID="acf22f9816062b882b8699ed1e3095a7da75acd5fae7f1787afb7da634d5b67d" exitCode=0
Nov 01 03:06:11.566031 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:11.565920    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qnrf8" event={"ID":"e8f2a24b-e770-4615-a2c7-b2d065e14f77","Type":"ContainerDied","Data":"acf22f9816062b882b8699ed1e3095a7da75acd5fae7f1787afb7da634d5b67d"}
Nov 01 03:06:12.388678 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:12.388477    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dnqjp"
Nov 01 03:06:12.388811 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:12.388528    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mx48d"
Nov 01 03:06:12.388811 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:06:12.388729    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dnqjp" podUID="e48a634f-ab31-435c-96c1-c3633c5bc061"
Nov 01 03:06:12.388873 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:12.388551    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rrdzd"
Nov 01 03:06:12.388873 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:06:12.388827    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mx48d" podUID="6e302b40-9c43-4822-8c8b-4d021a0590ab"
Nov 01 03:06:12.388925 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:06:12.388878    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rrdzd" podUID="0303d6c6-c4ec-4403-af74-f4bcb5cbf2a4"
Nov 01 03:06:12.570061 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:12.570037    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9" event={"ID":"de5f469f-1029-4161-bf51-6ad18a3ac9e2","Type":"ContainerStarted","Data":"1eef5e97379f580c99d4e82de34f1df64204d6cc2281c8e9d14c5e2f03b2094e"}
Nov 01 03:06:12.570561 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:12.570095    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9" event={"ID":"de5f469f-1029-4161-bf51-6ad18a3ac9e2","Type":"ContainerStarted","Data":"fd2d5a7e39b2bd47b17be994ea5f9046dc1aa537de0fd90fd41ef211a4f7d420"}
Nov 01 03:06:12.570561 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:12.570137    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9" event={"ID":"de5f469f-1029-4161-bf51-6ad18a3ac9e2","Type":"ContainerStarted","Data":"073ca91b4cdb167bc46db14fc663cf7f40f9546d2b4511ced8cb04aaa775196c"}
Nov 01 03:06:12.570561 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:12.570176    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9" event={"ID":"de5f469f-1029-4161-bf51-6ad18a3ac9e2","Type":"ContainerStarted","Data":"894bef23402bbcbe05b1d3abdf91476dc3d003873ef95b63b79332a7597a9012"}
Nov 01 03:06:12.570561 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:12.570215    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9" event={"ID":"de5f469f-1029-4161-bf51-6ad18a3ac9e2","Type":"ContainerStarted","Data":"5934ba8114621c4bba12fdb17e8d5295fddec4855fd8d01c112acb35c7f30e2a"}
Nov 01 03:06:12.572890 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:12.572869    2576 generic.go:356] "Generic (PLEG): container finished" podID="e8f2a24b-e770-4615-a2c7-b2d065e14f77" containerID="0207c500b12b0ef1bf2f9fb73381ca1dfecfd909373ce84f8ca1488bebd78cca" exitCode=0
Nov 01 03:06:12.572974 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:12.572914    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qnrf8" event={"ID":"e8f2a24b-e770-4615-a2c7-b2d065e14f77","Type":"ContainerDied","Data":"0207c500b12b0ef1bf2f9fb73381ca1dfecfd909373ce84f8ca1488bebd78cca"}
Nov 01 03:06:13.577972 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:13.577940    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qnrf8" event={"ID":"e8f2a24b-e770-4615-a2c7-b2d065e14f77","Type":"ContainerStarted","Data":"4616c7cfe19b492dd7268e8c9439fff1500769e852d9eb92f94dcacb8c7749aa"}
Nov 01 03:06:13.720817 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:13.720772    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-qnrf8" podStartSLOduration=6.749495768 podStartE2EDuration="20.720759406s" totalImagesPullingTime="13.971263638s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 03:05:53 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:06:13.720602261 +0000 UTC m=+21.774743944" watchObservedRunningTime="2026-11-01 03:06:13.720759406 +0000 UTC m=+21.774901089"
Nov 01 03:06:14.388615 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:14.388589    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mx48d"
Nov 01 03:06:14.388615 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:14.388608    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dnqjp"
Nov 01 03:06:14.388847 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:06:14.388697    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mx48d" podUID="6e302b40-9c43-4822-8c8b-4d021a0590ab"
Nov 01 03:06:14.388847 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:14.388731    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rrdzd"
Nov 01 03:06:14.388847 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:06:14.388807    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rrdzd" podUID="0303d6c6-c4ec-4403-af74-f4bcb5cbf2a4"
Nov 01 03:06:14.388997 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:06:14.388889    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dnqjp" podUID="e48a634f-ab31-435c-96c1-c3633c5bc061"
Nov 01 03:06:14.582741 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:14.582715    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9" event={"ID":"de5f469f-1029-4161-bf51-6ad18a3ac9e2","Type":"ContainerStarted","Data":"a26066df3c4b7d14b9bde479d74b11eadf05d2963b77a5bd524183dfeb7f25c6"}
Nov 01 03:06:16.388078 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:16.387997    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dnqjp"
Nov 01 03:06:16.388593 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:16.388001    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mx48d"
Nov 01 03:06:16.388593 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:16.388005    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rrdzd"
Nov 01 03:06:16.388593 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:06:16.388398    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dnqjp" podUID="e48a634f-ab31-435c-96c1-c3633c5bc061"
Nov 01 03:06:16.388593 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:06:16.388521    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mx48d" podUID="6e302b40-9c43-4822-8c8b-4d021a0590ab"
Nov 01 03:06:16.388835 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:06:16.388607    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rrdzd" podUID="0303d6c6-c4ec-4403-af74-f4bcb5cbf2a4"
Nov 01 03:06:16.589995 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:16.589818    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9" event={"ID":"de5f469f-1029-4161-bf51-6ad18a3ac9e2","Type":"ContainerStarted","Data":"6e4749cb3d4fe61ae463b5b0f293721d5a9af954616efc7a4b86325903db83a9"}
Nov 01 03:06:16.590143 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:16.590120    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:06:16.590266 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:16.590251    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:06:16.590339 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:16.590314    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:06:16.606359 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:16.606338    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:06:16.606481 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:16.606459    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:06:16.873714 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:16.873613    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9" podStartSLOduration=7.186641248 podStartE2EDuration="23.873602747s" totalImagesPullingTime="16.686961499s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 03:05:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:06:16.873115507 +0000 UTC m=+24.927257200" watchObservedRunningTime="2026-11-01 03:06:16.873602747 +0000 UTC m=+24.927744436"
Nov 01 03:06:18.388788 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:18.388751    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dnqjp"
Nov 01 03:06:18.389167 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:18.388804    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rrdzd"
Nov 01 03:06:18.389167 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:18.388811    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mx48d"
Nov 01 03:06:18.389167 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:06:18.388916    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dnqjp" podUID="e48a634f-ab31-435c-96c1-c3633c5bc061"
Nov 01 03:06:18.389167 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:06:18.389029    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rrdzd" podUID="0303d6c6-c4ec-4403-af74-f4bcb5cbf2a4"
Nov 01 03:06:18.389167 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:06:18.389090    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mx48d" podUID="6e302b40-9c43-4822-8c8b-4d021a0590ab"
Nov 01 03:06:18.631461 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:18.631431    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rrdzd"]
Nov 01 03:06:18.631600 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:18.631525    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rrdzd"
Nov 01 03:06:18.631655 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:06:18.631602    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rrdzd" podUID="0303d6c6-c4ec-4403-af74-f4bcb5cbf2a4"
Nov 01 03:06:18.634932 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:18.634908    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mx48d"]
Nov 01 03:06:18.635069 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:18.634991    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mx48d"
Nov 01 03:06:18.635125 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:06:18.635096    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mx48d" podUID="6e302b40-9c43-4822-8c8b-4d021a0590ab"
Nov 01 03:06:18.635498 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:18.635478    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dnqjp"]
Nov 01 03:06:18.635581 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:18.635569    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dnqjp"
Nov 01 03:06:18.635669 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:06:18.635650    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dnqjp" podUID="e48a634f-ab31-435c-96c1-c3633c5bc061"
Nov 01 03:06:20.391585 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:20.391514    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mx48d"
Nov 01 03:06:20.391972 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:20.391601    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dnqjp"
Nov 01 03:06:20.391972 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:20.391699    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rrdzd"
Nov 01 03:06:20.391972 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:06:20.391703    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mx48d" podUID="6e302b40-9c43-4822-8c8b-4d021a0590ab"
Nov 01 03:06:20.391972 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:06:20.391793    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dnqjp" podUID="e48a634f-ab31-435c-96c1-c3633c5bc061"
Nov 01 03:06:20.391972 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:06:20.391837    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rrdzd" podUID="0303d6c6-c4ec-4403-af74-f4bcb5cbf2a4"
Nov 01 03:06:22.388808 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:22.388742    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mx48d"
Nov 01 03:06:22.389176 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:06:22.388816    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mx48d" podUID="6e302b40-9c43-4822-8c8b-4d021a0590ab"
Nov 01 03:06:22.389176 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:22.388834    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rrdzd"
Nov 01 03:06:22.389176 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:06:22.388950    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rrdzd" podUID="0303d6c6-c4ec-4403-af74-f4bcb5cbf2a4"
Nov 01 03:06:22.389176 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:22.389002    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dnqjp"
Nov 01 03:06:22.389176 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:06:22.389082    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dnqjp" podUID="e48a634f-ab31-435c-96c1-c3633c5bc061"
Nov 01 03:06:24.133013 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:24.132980    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-54.us-west-1.compute.internal" event="NodeReady"
Nov 01 03:06:24.133317 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:24.133072    2576 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 01 03:06:24.266670 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:24.266645    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-nzbnl"]
Nov 01 03:06:24.348674 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:24.348624    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-h6696"]
Nov 01 03:06:24.348787 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:24.348747    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nzbnl"
Nov 01 03:06:24.365072 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:24.365054    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-h6696"
Nov 01 03:06:24.388579 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:24.388552    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rrdzd"
Nov 01 03:06:24.388579 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:24.388569    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dnqjp"
Nov 01 03:06:24.388757 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:24.388585    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mx48d"
Nov 01 03:06:24.411697 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:24.411652    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-nzbnl"]
Nov 01 03:06:24.418462 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:24.418446    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 01 03:06:24.420309 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:24.420286    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-h6696"]
Nov 01 03:06:24.461139 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:24.461120    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 01 03:06:24.461405 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:24.461377    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 01 03:06:24.461497 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:24.461424    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 01 03:06:24.461637 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:24.461623    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 01 03:06:24.461873 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:24.461855    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 01 03:06:24.489367 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:24.489344    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4c1dc667-d774-4db6-b221-f4e7de423260-cert\") pod \"ingress-canary-h6696\" (UID: \"4c1dc667-d774-4db6-b221-f4e7de423260\") " pod="openshift-ingress-canary/ingress-canary-h6696"
Nov 01 03:06:24.489458 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:24.489392    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56aed8ff-9b41-400a-90fb-8c283ce0c820-tmp-dir\") pod \"dns-default-nzbnl\" (UID: \"56aed8ff-9b41-400a-90fb-8c283ce0c820\") " pod="openshift-dns/dns-default-nzbnl"
Nov 01 03:06:24.489502 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:24.489487    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2459t\" (UniqueName: \"kubernetes.io/projected/4c1dc667-d774-4db6-b221-f4e7de423260-kube-api-access-2459t\") pod \"ingress-canary-h6696\" (UID: \"4c1dc667-d774-4db6-b221-f4e7de423260\") " pod="openshift-ingress-canary/ingress-canary-h6696"
Nov 01 03:06:24.489534 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:24.489512    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/56aed8ff-9b41-400a-90fb-8c283ce0c820-metrics-tls\") pod \"dns-default-nzbnl\" (UID: \"56aed8ff-9b41-400a-90fb-8c283ce0c820\") " pod="openshift-dns/dns-default-nzbnl"
Nov 01 03:06:24.489579 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:24.489545    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/56aed8ff-9b41-400a-90fb-8c283ce0c820-config-volume\") pod \"dns-default-nzbnl\" (UID: \"56aed8ff-9b41-400a-90fb-8c283ce0c820\") " pod="openshift-dns/dns-default-nzbnl"
Nov 01 03:06:24.489634 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:24.489590    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9wst\" (UniqueName: \"kubernetes.io/projected/56aed8ff-9b41-400a-90fb-8c283ce0c820-kube-api-access-r9wst\") pod \"dns-default-nzbnl\" (UID: \"56aed8ff-9b41-400a-90fb-8c283ce0c820\") " pod="openshift-dns/dns-default-nzbnl"
Nov 01 03:06:24.511186 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:24.511161    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 01 03:06:24.511264 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:24.511188    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 01 03:06:24.511735 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:24.511721    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-fdfjz\""
Nov 01 03:06:24.512436 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:24.512415    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-kkkrb\""
Nov 01 03:06:24.512512 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:24.512494    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-74fv2\""
Nov 01 03:06:24.512705 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:24.512671    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 01 03:06:24.512746 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:24.512700    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-wgrq6\""
Nov 01 03:06:24.512746 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:24.512725    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-z9gxp\""
Nov 01 03:06:24.590286 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:24.590259    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2459t\" (UniqueName: \"kubernetes.io/projected/4c1dc667-d774-4db6-b221-f4e7de423260-kube-api-access-2459t\") pod \"ingress-canary-h6696\" (UID: \"4c1dc667-d774-4db6-b221-f4e7de423260\") " pod="openshift-ingress-canary/ingress-canary-h6696"
Nov 01 03:06:24.590286 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:24.590287    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/56aed8ff-9b41-400a-90fb-8c283ce0c820-metrics-tls\") pod \"dns-default-nzbnl\" (UID: \"56aed8ff-9b41-400a-90fb-8c283ce0c820\") " pod="openshift-dns/dns-default-nzbnl"
Nov 01 03:06:24.590435 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:24.590316    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/56aed8ff-9b41-400a-90fb-8c283ce0c820-config-volume\") pod \"dns-default-nzbnl\" (UID: \"56aed8ff-9b41-400a-90fb-8c283ce0c820\") " pod="openshift-dns/dns-default-nzbnl"
Nov 01 03:06:24.590435 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:24.590331    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9wst\" (UniqueName: \"kubernetes.io/projected/56aed8ff-9b41-400a-90fb-8c283ce0c820-kube-api-access-r9wst\") pod \"dns-default-nzbnl\" (UID: \"56aed8ff-9b41-400a-90fb-8c283ce0c820\") " pod="openshift-dns/dns-default-nzbnl"
Nov 01 03:06:24.590435 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:06:24.590420    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 03:06:24.590539 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:06:24.590483    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56aed8ff-9b41-400a-90fb-8c283ce0c820-metrics-tls podName:56aed8ff-9b41-400a-90fb-8c283ce0c820 nodeName:}" failed. No retries permitted until 2026-11-01 03:06:25.090463821 +0000 UTC m=+33.144605502 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/56aed8ff-9b41-400a-90fb-8c283ce0c820-metrics-tls") pod "dns-default-nzbnl" (UID: "56aed8ff-9b41-400a-90fb-8c283ce0c820") : secret "dns-default-metrics-tls" not found
Nov 01 03:06:24.590539 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:24.590503    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4c1dc667-d774-4db6-b221-f4e7de423260-cert\") pod \"ingress-canary-h6696\" (UID: \"4c1dc667-d774-4db6-b221-f4e7de423260\") " pod="openshift-ingress-canary/ingress-canary-h6696"
Nov 01 03:06:24.590633 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:24.590564    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56aed8ff-9b41-400a-90fb-8c283ce0c820-tmp-dir\") pod \"dns-default-nzbnl\" (UID: \"56aed8ff-9b41-400a-90fb-8c283ce0c820\") " pod="openshift-dns/dns-default-nzbnl"
Nov 01 03:06:24.590898 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:24.590877    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56aed8ff-9b41-400a-90fb-8c283ce0c820-tmp-dir\") pod \"dns-default-nzbnl\" (UID: \"56aed8ff-9b41-400a-90fb-8c283ce0c820\") " pod="openshift-dns/dns-default-nzbnl"
Nov 01 03:06:24.590950 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:24.590925    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/56aed8ff-9b41-400a-90fb-8c283ce0c820-config-volume\") pod \"dns-default-nzbnl\" (UID: \"56aed8ff-9b41-400a-90fb-8c283ce0c820\") " pod="openshift-dns/dns-default-nzbnl"
Nov 01 03:06:24.594137 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:24.594120    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4c1dc667-d774-4db6-b221-f4e7de423260-cert\") pod \"ingress-canary-h6696\" (UID: \"4c1dc667-d774-4db6-b221-f4e7de423260\") " pod="openshift-ingress-canary/ingress-canary-h6696"
Nov 01 03:06:24.653613 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:24.653570    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2459t\" (UniqueName: \"kubernetes.io/projected/4c1dc667-d774-4db6-b221-f4e7de423260-kube-api-access-2459t\") pod \"ingress-canary-h6696\" (UID: \"4c1dc667-d774-4db6-b221-f4e7de423260\") " pod="openshift-ingress-canary/ingress-canary-h6696"
Nov 01 03:06:24.654813 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:24.654792    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9wst\" (UniqueName: \"kubernetes.io/projected/56aed8ff-9b41-400a-90fb-8c283ce0c820-kube-api-access-r9wst\") pod \"dns-default-nzbnl\" (UID: \"56aed8ff-9b41-400a-90fb-8c283ce0c820\") " pod="openshift-dns/dns-default-nzbnl"
Nov 01 03:06:24.698480 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:24.698460    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-h6696"
Nov 01 03:06:25.094212 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:25.094147    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/56aed8ff-9b41-400a-90fb-8c283ce0c820-metrics-tls\") pod \"dns-default-nzbnl\" (UID: \"56aed8ff-9b41-400a-90fb-8c283ce0c820\") " pod="openshift-dns/dns-default-nzbnl"
Nov 01 03:06:25.094327 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:06:25.094292    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 03:06:25.094364 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:06:25.094348    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56aed8ff-9b41-400a-90fb-8c283ce0c820-metrics-tls podName:56aed8ff-9b41-400a-90fb-8c283ce0c820 nodeName:}" failed. No retries permitted until 2026-11-01 03:06:26.094332315 +0000 UTC m=+34.148473994 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/56aed8ff-9b41-400a-90fb-8c283ce0c820-metrics-tls") pod "dns-default-nzbnl" (UID: "56aed8ff-9b41-400a-90fb-8c283ce0c820") : secret "dns-default-metrics-tls" not found
Nov 01 03:06:25.108158 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:25.108138    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-h6696"]
Nov 01 03:06:25.110502 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:06:25.110477    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c1dc667_d774_4db6_b221_f4e7de423260.slice/crio-ca80f4e18c06e6002a072be7097cb83d48d6fe56ab99647cb2d9d8894402e92e WatchSource:0}: Error finding container ca80f4e18c06e6002a072be7097cb83d48d6fe56ab99647cb2d9d8894402e92e: Status 404 returned error can't find the container with id ca80f4e18c06e6002a072be7097cb83d48d6fe56ab99647cb2d9d8894402e92e
Nov 01 03:06:25.610341 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:25.610310    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-h6696" event={"ID":"4c1dc667-d774-4db6-b221-f4e7de423260","Type":"ContainerStarted","Data":"ca80f4e18c06e6002a072be7097cb83d48d6fe56ab99647cb2d9d8894402e92e"}
Nov 01 03:06:26.101710 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:26.101619    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/56aed8ff-9b41-400a-90fb-8c283ce0c820-metrics-tls\") pod \"dns-default-nzbnl\" (UID: \"56aed8ff-9b41-400a-90fb-8c283ce0c820\") " pod="openshift-dns/dns-default-nzbnl"
Nov 01 03:06:26.101842 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:06:26.101763    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 03:06:26.101842 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:06:26.101815    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56aed8ff-9b41-400a-90fb-8c283ce0c820-metrics-tls podName:56aed8ff-9b41-400a-90fb-8c283ce0c820 nodeName:}" failed. No retries permitted until 2026-11-01 03:06:28.101802769 +0000 UTC m=+36.155944429 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/56aed8ff-9b41-400a-90fb-8c283ce0c820-metrics-tls") pod "dns-default-nzbnl" (UID: "56aed8ff-9b41-400a-90fb-8c283ce0c820") : secret "dns-default-metrics-tls" not found
Nov 01 03:06:26.303842 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:26.303805    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0303d6c6-c4ec-4403-af74-f4bcb5cbf2a4-original-pull-secret\") pod \"global-pull-secret-syncer-rrdzd\" (UID: \"0303d6c6-c4ec-4403-af74-f4bcb5cbf2a4\") " pod="kube-system/global-pull-secret-syncer-rrdzd"
Nov 01 03:06:26.304017 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:26.303871    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e48a634f-ab31-435c-96c1-c3633c5bc061-metrics-certs\") pod \"network-metrics-daemon-dnqjp\" (UID: \"e48a634f-ab31-435c-96c1-c3633c5bc061\") " pod="openshift-multus/network-metrics-daemon-dnqjp"
Nov 01 03:06:26.304017 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:06:26.303981    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 01 03:06:26.304132 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:06:26.304042    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e48a634f-ab31-435c-96c1-c3633c5bc061-metrics-certs podName:e48a634f-ab31-435c-96c1-c3633c5bc061 nodeName:}" failed. No retries permitted until 2026-11-01 03:06:58.304023776 +0000 UTC m=+66.358165449 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e48a634f-ab31-435c-96c1-c3633c5bc061-metrics-certs") pod "network-metrics-daemon-dnqjp" (UID: "e48a634f-ab31-435c-96c1-c3633c5bc061") : secret "metrics-daemon-secret" not found
Nov 01 03:06:26.312589 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:26.312553    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0303d6c6-c4ec-4403-af74-f4bcb5cbf2a4-original-pull-secret\") pod \"global-pull-secret-syncer-rrdzd\" (UID: \"0303d6c6-c4ec-4403-af74-f4bcb5cbf2a4\") " pod="kube-system/global-pull-secret-syncer-rrdzd"
Nov 01 03:06:26.404650 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:26.404590    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxqx8\" (UniqueName: \"kubernetes.io/projected/6e302b40-9c43-4822-8c8b-4d021a0590ab-kube-api-access-nxqx8\") pod \"network-check-target-mx48d\" (UID: \"6e302b40-9c43-4822-8c8b-4d021a0590ab\") " pod="openshift-network-diagnostics/network-check-target-mx48d"
Nov 01 03:06:26.407035 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:26.407014    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxqx8\" (UniqueName: \"kubernetes.io/projected/6e302b40-9c43-4822-8c8b-4d021a0590ab-kube-api-access-nxqx8\") pod \"network-check-target-mx48d\" (UID: \"6e302b40-9c43-4822-8c8b-4d021a0590ab\") " pod="openshift-network-diagnostics/network-check-target-mx48d"
Nov 01 03:06:26.521494 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:26.521468    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rrdzd"
Nov 01 03:06:26.545809 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:26.545783    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mx48d"
Nov 01 03:06:26.905923 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:26.905894    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rrdzd"]
Nov 01 03:06:26.946419 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:06:26.946388    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0303d6c6_c4ec_4403_af74_f4bcb5cbf2a4.slice/crio-656a993b19b5fb0b91c6c5940fe8ed4ed810a56d28407cb6a0bf8d1ed48f10c4 WatchSource:0}: Error finding container 656a993b19b5fb0b91c6c5940fe8ed4ed810a56d28407cb6a0bf8d1ed48f10c4: Status 404 returned error can't find the container with id 656a993b19b5fb0b91c6c5940fe8ed4ed810a56d28407cb6a0bf8d1ed48f10c4
Nov 01 03:06:26.968037 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:26.968013    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mx48d"]
Nov 01 03:06:26.968572 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:06:26.968548    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e302b40_9c43_4822_8c8b_4d021a0590ab.slice/crio-c3970e088aa262baf754bfedaeebe86bb4f33f854f9f5d5fbec0b6fbc0aa084d WatchSource:0}: Error finding container c3970e088aa262baf754bfedaeebe86bb4f33f854f9f5d5fbec0b6fbc0aa084d: Status 404 returned error can't find the container with id c3970e088aa262baf754bfedaeebe86bb4f33f854f9f5d5fbec0b6fbc0aa084d
Nov 01 03:06:27.615816 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:27.615784    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mx48d" event={"ID":"6e302b40-9c43-4822-8c8b-4d021a0590ab","Type":"ContainerStarted","Data":"c3970e088aa262baf754bfedaeebe86bb4f33f854f9f5d5fbec0b6fbc0aa084d"}
Nov 01 03:06:27.616823 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:27.616801    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rrdzd" event={"ID":"0303d6c6-c4ec-4403-af74-f4bcb5cbf2a4","Type":"ContainerStarted","Data":"656a993b19b5fb0b91c6c5940fe8ed4ed810a56d28407cb6a0bf8d1ed48f10c4"}
Nov 01 03:06:28.115830 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:28.115792    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/56aed8ff-9b41-400a-90fb-8c283ce0c820-metrics-tls\") pod \"dns-default-nzbnl\" (UID: \"56aed8ff-9b41-400a-90fb-8c283ce0c820\") " pod="openshift-dns/dns-default-nzbnl"
Nov 01 03:06:28.116267 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:06:28.115962    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 03:06:28.116267 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:06:28.116036    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56aed8ff-9b41-400a-90fb-8c283ce0c820-metrics-tls podName:56aed8ff-9b41-400a-90fb-8c283ce0c820 nodeName:}" failed. No retries permitted until 2026-11-01 03:06:32.116014635 +0000 UTC m=+40.170156309 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/56aed8ff-9b41-400a-90fb-8c283ce0c820-metrics-tls") pod "dns-default-nzbnl" (UID: "56aed8ff-9b41-400a-90fb-8c283ce0c820") : secret "dns-default-metrics-tls" not found
Nov 01 03:06:29.624701 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:29.624604    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-h6696" event={"ID":"4c1dc667-d774-4db6-b221-f4e7de423260","Type":"ContainerStarted","Data":"38bce215d0c151961ec7f7d0ba30d257f737e031fd0a5c659e47acec27bb257e"}
Nov 01 03:06:29.747375 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:29.747325    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-h6696" podStartSLOduration=1.713970648 podStartE2EDuration="5.747310917s" totalImagesPullingTime="4.033340269s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:06:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:06:29.74601115 +0000 UTC m=+37.800152834" watchObservedRunningTime="2026-11-01 03:06:29.747310917 +0000 UTC m=+37.801452642"
Nov 01 03:06:31.291894 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:31.291816    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-6vv92"]
Nov 01 03:06:31.311383 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:31.311358    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6vv92"
Nov 01 03:06:31.341070 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:31.341037    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-httfj\" (UniqueName: \"kubernetes.io/projected/73b27ccb-7d7a-40c1-8479-8379b4fb50ed-kube-api-access-httfj\") pod \"insights-runtime-extractor-6vv92\" (UID: \"73b27ccb-7d7a-40c1-8479-8379b4fb50ed\") " pod="openshift-insights/insights-runtime-extractor-6vv92"
Nov 01 03:06:31.341178 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:31.341093    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/73b27ccb-7d7a-40c1-8479-8379b4fb50ed-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6vv92\" (UID: \"73b27ccb-7d7a-40c1-8479-8379b4fb50ed\") " pod="openshift-insights/insights-runtime-extractor-6vv92"
Nov 01 03:06:31.341239 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:31.341195    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/73b27ccb-7d7a-40c1-8479-8379b4fb50ed-crio-socket\") pod \"insights-runtime-extractor-6vv92\" (UID: \"73b27ccb-7d7a-40c1-8479-8379b4fb50ed\") " pod="openshift-insights/insights-runtime-extractor-6vv92"
Nov 01 03:06:31.341239 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:31.341227    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/73b27ccb-7d7a-40c1-8479-8379b4fb50ed-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6vv92\" (UID: \"73b27ccb-7d7a-40c1-8479-8379b4fb50ed\") " pod="openshift-insights/insights-runtime-extractor-6vv92"
Nov 01 03:06:31.341344 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:31.341327    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/73b27ccb-7d7a-40c1-8479-8379b4fb50ed-data-volume\") pod \"insights-runtime-extractor-6vv92\" (UID: \"73b27ccb-7d7a-40c1-8479-8379b4fb50ed\") " pod="openshift-insights/insights-runtime-extractor-6vv92"
Nov 01 03:06:31.359536 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:31.359508    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6vv92"]
Nov 01 03:06:31.409208 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:31.409179    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-dg4zg\""
Nov 01 03:06:31.409457 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:31.409421    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 01 03:06:31.409571 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:31.409459    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 01 03:06:31.409571 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:31.409504    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 01 03:06:31.442086 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:31.442062    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/73b27ccb-7d7a-40c1-8479-8379b4fb50ed-data-volume\") pod \"insights-runtime-extractor-6vv92\" (UID: \"73b27ccb-7d7a-40c1-8479-8379b4fb50ed\") " pod="openshift-insights/insights-runtime-extractor-6vv92"
Nov 01 03:06:31.442218 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:31.442099    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-httfj\" (UniqueName: \"kubernetes.io/projected/73b27ccb-7d7a-40c1-8479-8379b4fb50ed-kube-api-access-httfj\") pod \"insights-runtime-extractor-6vv92\" (UID: \"73b27ccb-7d7a-40c1-8479-8379b4fb50ed\") " pod="openshift-insights/insights-runtime-extractor-6vv92"
Nov 01 03:06:31.442218 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:31.442136    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/73b27ccb-7d7a-40c1-8479-8379b4fb50ed-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6vv92\" (UID: \"73b27ccb-7d7a-40c1-8479-8379b4fb50ed\") " pod="openshift-insights/insights-runtime-extractor-6vv92"
Nov 01 03:06:31.442369 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:06:31.442352    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 01 03:06:31.442435 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:31.442418    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/73b27ccb-7d7a-40c1-8479-8379b4fb50ed-data-volume\") pod \"insights-runtime-extractor-6vv92\" (UID: \"73b27ccb-7d7a-40c1-8479-8379b4fb50ed\") " pod="openshift-insights/insights-runtime-extractor-6vv92"
Nov 01 03:06:31.442487 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:06:31.442425    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/73b27ccb-7d7a-40c1-8479-8379b4fb50ed-kube-rbac-proxy-cm podName:73b27ccb-7d7a-40c1-8479-8379b4fb50ed nodeName:}" failed. No retries permitted until 2026-11-01 03:06:31.942405184 +0000 UTC m=+39.996546864 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/73b27ccb-7d7a-40c1-8479-8379b4fb50ed-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6vv92" (UID: "73b27ccb-7d7a-40c1-8479-8379b4fb50ed") : configmap "kube-rbac-proxy" not found
Nov 01 03:06:31.442487 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:31.442475    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/73b27ccb-7d7a-40c1-8479-8379b4fb50ed-crio-socket\") pod \"insights-runtime-extractor-6vv92\" (UID: \"73b27ccb-7d7a-40c1-8479-8379b4fb50ed\") " pod="openshift-insights/insights-runtime-extractor-6vv92"
Nov 01 03:06:31.442590 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:31.442509    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/73b27ccb-7d7a-40c1-8479-8379b4fb50ed-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6vv92\" (UID: \"73b27ccb-7d7a-40c1-8479-8379b4fb50ed\") " pod="openshift-insights/insights-runtime-extractor-6vv92"
Nov 01 03:06:31.442699 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:31.442662    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/73b27ccb-7d7a-40c1-8479-8379b4fb50ed-crio-socket\") pod \"insights-runtime-extractor-6vv92\" (UID: \"73b27ccb-7d7a-40c1-8479-8379b4fb50ed\") " pod="openshift-insights/insights-runtime-extractor-6vv92"
Nov 01 03:06:31.468150 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:31.468124    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 01 03:06:31.473233 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:06:31.473209    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 03:06:31.473331 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:06:31.473260    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73b27ccb-7d7a-40c1-8479-8379b4fb50ed-insights-runtime-extractor-tls podName:73b27ccb-7d7a-40c1-8479-8379b4fb50ed nodeName:}" failed. No retries permitted until 2026-11-01 03:06:31.973247748 +0000 UTC m=+40.027389408 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/73b27ccb-7d7a-40c1-8479-8379b4fb50ed-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6vv92" (UID: "73b27ccb-7d7a-40c1-8479-8379b4fb50ed") : secret "insights-runtime-extractor-tls" not found
Nov 01 03:06:31.516234 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:31.516209    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-httfj\" (UniqueName: \"kubernetes.io/projected/73b27ccb-7d7a-40c1-8479-8379b4fb50ed-kube-api-access-httfj\") pod \"insights-runtime-extractor-6vv92\" (UID: \"73b27ccb-7d7a-40c1-8479-8379b4fb50ed\") " pod="openshift-insights/insights-runtime-extractor-6vv92"
Nov 01 03:06:31.568293 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:31.568232    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-5rxlb-bg459"]
Nov 01 03:06:31.589962 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:31.589937    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-5rxlb-bg459"
Nov 01 03:06:31.631502 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:31.631475    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-5rxlb-bg459"]
Nov 01 03:06:31.631609 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:31.631522    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mx48d" event={"ID":"6e302b40-9c43-4822-8c8b-4d021a0590ab","Type":"ContainerStarted","Data":"a0d157a88c4fde44dd571cf7299950c20483b5fb8bcb32ddfc80067c482d16a3"}
Nov 01 03:06:31.631707 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:31.631671    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-mx48d"
Nov 01 03:06:31.644287 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:31.644252    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/68932dfe-7e25-4f1b-b485-77c0519203db-archives-path\") pod \"periodic-gathering-5rxlb-bg459\" (UID: \"68932dfe-7e25-4f1b-b485-77c0519203db\") " pod="openshift-insights/periodic-gathering-5rxlb-bg459"
Nov 01 03:06:31.644427 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:31.644402    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnrhp\" (UniqueName: \"kubernetes.io/projected/68932dfe-7e25-4f1b-b485-77c0519203db-kube-api-access-qnrhp\") pod \"periodic-gathering-5rxlb-bg459\" (UID: \"68932dfe-7e25-4f1b-b485-77c0519203db\") " pod="openshift-insights/periodic-gathering-5rxlb-bg459"
Nov 01 03:06:31.644482 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:31.644452    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68932dfe-7e25-4f1b-b485-77c0519203db-service-ca-bundle\") pod \"periodic-gathering-5rxlb-bg459\" (UID: \"68932dfe-7e25-4f1b-b485-77c0519203db\") " pod="openshift-insights/periodic-gathering-5rxlb-bg459"
Nov 01 03:06:31.644534 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:31.644484    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68932dfe-7e25-4f1b-b485-77c0519203db-trusted-ca-bundle\") pod \"periodic-gathering-5rxlb-bg459\" (UID: \"68932dfe-7e25-4f1b-b485-77c0519203db\") " pod="openshift-insights/periodic-gathering-5rxlb-bg459"
Nov 01 03:06:31.646769 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:31.646748    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-bt25w\""
Nov 01 03:06:31.646877 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:31.646822    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 01 03:06:31.699317 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:31.699288    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 01 03:06:31.745305 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:31.745276    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnrhp\" (UniqueName: \"kubernetes.io/projected/68932dfe-7e25-4f1b-b485-77c0519203db-kube-api-access-qnrhp\") pod \"periodic-gathering-5rxlb-bg459\" (UID: \"68932dfe-7e25-4f1b-b485-77c0519203db\") " pod="openshift-insights/periodic-gathering-5rxlb-bg459"
Nov 01 03:06:31.745405 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:31.745310    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68932dfe-7e25-4f1b-b485-77c0519203db-service-ca-bundle\") pod \"periodic-gathering-5rxlb-bg459\" (UID: \"68932dfe-7e25-4f1b-b485-77c0519203db\") " pod="openshift-insights/periodic-gathering-5rxlb-bg459"
Nov 01 03:06:31.745405 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:31.745332    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68932dfe-7e25-4f1b-b485-77c0519203db-trusted-ca-bundle\") pod \"periodic-gathering-5rxlb-bg459\" (UID: \"68932dfe-7e25-4f1b-b485-77c0519203db\") " pod="openshift-insights/periodic-gathering-5rxlb-bg459"
Nov 01 03:06:31.745405 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:31.745374    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/68932dfe-7e25-4f1b-b485-77c0519203db-archives-path\") pod \"periodic-gathering-5rxlb-bg459\" (UID: \"68932dfe-7e25-4f1b-b485-77c0519203db\") " pod="openshift-insights/periodic-gathering-5rxlb-bg459"
Nov 01 03:06:31.745887 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:31.745865    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/68932dfe-7e25-4f1b-b485-77c0519203db-archives-path\") pod \"periodic-gathering-5rxlb-bg459\" (UID: \"68932dfe-7e25-4f1b-b485-77c0519203db\") " pod="openshift-insights/periodic-gathering-5rxlb-bg459"
Nov 01 03:06:31.746039 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:31.746021    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68932dfe-7e25-4f1b-b485-77c0519203db-service-ca-bundle\") pod \"periodic-gathering-5rxlb-bg459\" (UID: \"68932dfe-7e25-4f1b-b485-77c0519203db\") " pod="openshift-insights/periodic-gathering-5rxlb-bg459"
Nov 01 03:06:31.746280 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:31.746252    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68932dfe-7e25-4f1b-b485-77c0519203db-trusted-ca-bundle\") pod \"periodic-gathering-5rxlb-bg459\" (UID: \"68932dfe-7e25-4f1b-b485-77c0519203db\") " pod="openshift-insights/periodic-gathering-5rxlb-bg459"
Nov 01 03:06:31.809294 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:31.809259    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnrhp\" (UniqueName: \"kubernetes.io/projected/68932dfe-7e25-4f1b-b485-77c0519203db-kube-api-access-qnrhp\") pod \"periodic-gathering-5rxlb-bg459\" (UID: \"68932dfe-7e25-4f1b-b485-77c0519203db\") " pod="openshift-insights/periodic-gathering-5rxlb-bg459"
Nov 01 03:06:31.915269 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:31.915188    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-5rxlb-bg459"
Nov 01 03:06:31.947453 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:31.947373    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/73b27ccb-7d7a-40c1-8479-8379b4fb50ed-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6vv92\" (UID: \"73b27ccb-7d7a-40c1-8479-8379b4fb50ed\") " pod="openshift-insights/insights-runtime-extractor-6vv92"
Nov 01 03:06:31.947560 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:06:31.947452    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 01 03:06:31.947560 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:06:31.947537    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/73b27ccb-7d7a-40c1-8479-8379b4fb50ed-kube-rbac-proxy-cm podName:73b27ccb-7d7a-40c1-8479-8379b4fb50ed nodeName:}" failed. No retries permitted until 2026-11-01 03:06:32.947514538 +0000 UTC m=+41.001656200 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/73b27ccb-7d7a-40c1-8479-8379b4fb50ed-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6vv92" (UID: "73b27ccb-7d7a-40c1-8479-8379b4fb50ed") : configmap "kube-rbac-proxy" not found
Nov 01 03:06:32.048393 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:32.048359    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/73b27ccb-7d7a-40c1-8479-8379b4fb50ed-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6vv92\" (UID: \"73b27ccb-7d7a-40c1-8479-8379b4fb50ed\") " pod="openshift-insights/insights-runtime-extractor-6vv92"
Nov 01 03:06:32.048580 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:06:32.048541    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 03:06:32.048621 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:06:32.048600    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73b27ccb-7d7a-40c1-8479-8379b4fb50ed-insights-runtime-extractor-tls podName:73b27ccb-7d7a-40c1-8479-8379b4fb50ed nodeName:}" failed. No retries permitted until 2026-11-01 03:06:33.048586309 +0000 UTC m=+41.102727971 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/73b27ccb-7d7a-40c1-8479-8379b4fb50ed-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6vv92" (UID: "73b27ccb-7d7a-40c1-8479-8379b4fb50ed") : secret "insights-runtime-extractor-tls" not found
Nov 01 03:06:32.149623 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:32.149587    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/56aed8ff-9b41-400a-90fb-8c283ce0c820-metrics-tls\") pod \"dns-default-nzbnl\" (UID: \"56aed8ff-9b41-400a-90fb-8c283ce0c820\") " pod="openshift-dns/dns-default-nzbnl"
Nov 01 03:06:32.149813 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:06:32.149719    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 03:06:32.149813 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:06:32.149794    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56aed8ff-9b41-400a-90fb-8c283ce0c820-metrics-tls podName:56aed8ff-9b41-400a-90fb-8c283ce0c820 nodeName:}" failed. No retries permitted until 2026-11-01 03:06:40.149775461 +0000 UTC m=+48.203917121 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/56aed8ff-9b41-400a-90fb-8c283ce0c820-metrics-tls") pod "dns-default-nzbnl" (UID: "56aed8ff-9b41-400a-90fb-8c283ce0c820") : secret "dns-default-metrics-tls" not found
Nov 01 03:06:32.308061 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:32.307955    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-mx48d" podStartSLOduration=35.269108146 podStartE2EDuration="39.307937938s" totalImagesPullingTime="4.038829792s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:05:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:06:31.853821478 +0000 UTC m=+39.907963182" watchObservedRunningTime="2026-11-01 03:06:32.307937938 +0000 UTC m=+40.362079622"
Nov 01 03:06:32.309178 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:32.309044    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-5rxlb-bg459"]
Nov 01 03:06:32.312196 ip-10-0-133-54 kubenswrapper[2576]: W1101 03:06:32.312156    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68932dfe_7e25_4f1b_b485_77c0519203db.slice/crio-410bd40c1eca1aaaeeea283ba9e2c2a185c749e790f96ef1fdb69feacf71c338 WatchSource:0}: Error finding container 410bd40c1eca1aaaeeea283ba9e2c2a185c749e790f96ef1fdb69feacf71c338: Status 404 returned error can't find the container with id 410bd40c1eca1aaaeeea283ba9e2c2a185c749e790f96ef1fdb69feacf71c338
Nov 01 03:06:32.635506 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:32.635428    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-5rxlb-bg459" event={"ID":"68932dfe-7e25-4f1b-b485-77c0519203db","Type":"ContainerStarted","Data":"410bd40c1eca1aaaeeea283ba9e2c2a185c749e790f96ef1fdb69feacf71c338"}
Nov 01 03:06:32.956335 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:32.956270    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/73b27ccb-7d7a-40c1-8479-8379b4fb50ed-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6vv92\" (UID: \"73b27ccb-7d7a-40c1-8479-8379b4fb50ed\") " pod="openshift-insights/insights-runtime-extractor-6vv92"
Nov 01 03:06:32.956472 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:06:32.956382    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 01 03:06:32.956472 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:06:32.956460    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/73b27ccb-7d7a-40c1-8479-8379b4fb50ed-kube-rbac-proxy-cm podName:73b27ccb-7d7a-40c1-8479-8379b4fb50ed nodeName:}" failed. No retries permitted until 2026-11-01 03:06:34.956445479 +0000 UTC m=+43.010587144 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/73b27ccb-7d7a-40c1-8479-8379b4fb50ed-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6vv92" (UID: "73b27ccb-7d7a-40c1-8479-8379b4fb50ed") : configmap "kube-rbac-proxy" not found
Nov 01 03:06:33.056942 ip-10-0-133-54 kubenswrapper[2576]: I1101 03:06:33.056914    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/73b27ccb-7d7a-40c1-8479-8379b4fb50ed-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6vv92\" (UID: \"73b27ccb-7d7a-40c1-8479-8379b4fb50ed\") " pod="openshift-insights/insights-runtime-extractor-6vv92"
Nov 01 03:06:33.057077 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:06:33.057024    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 03:06:33.057077 ip-10-0-133-54 kubenswrapper[2576]: E1101 03:06:33.057067    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73b27ccb-7d7a-40c1-8479-8379b4fb50ed-insights-runtime-extractor-tls podName:73b27ccb-7d7a-40c1-8479-8379b4fb50ed nodeName:}" failed. No retries permitted until 2026-11-01 03:06:35.057056543 +0000 UTC m=+43.111198204 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/73b27ccb-7d7a-40c1-8479-8379b4fb50ed-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6vv92" (UID: "73b27ccb-7d7a-40c1-8479-8379b4fb50ed") : secret "insights-runtime-extractor-tls" not found
Nov 01 03:06:33.406662 ip-10-0-133-54 systemd[1]: Stopping Kubernetes Kubelet...
Nov 01 03:06:33.421133 ip-10-0-133-54 systemd[1]: kubelet.service: Deactivated successfully.
Nov 01 03:06:33.421323 ip-10-0-133-54 systemd[1]: Stopped Kubernetes Kubelet.
Nov 01 03:06:33.421377 ip-10-0-133-54 systemd[1]: kubelet.service: Consumed 2.225s CPU time.
Nov 01 03:06:33.436382 ip-10-0-133-54 systemd[1]: Starting Kubernetes Kubelet...
Nov 01 03:06:33.525673 ip-10-0-133-54 kubenswrapper[4400]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 03:06:33.525673 ip-10-0-133-54 kubenswrapper[4400]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 01 03:06:33.525673 ip-10-0-133-54 kubenswrapper[4400]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 03:06:33.525673 ip-10-0-133-54 kubenswrapper[4400]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 03:06:33.527660 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527589    4400 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 03:06:33.527660 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527660    4400 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 03:06:33.527755 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527664    4400 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 03:06:33.527755 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527668    4400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 03:06:33.527755 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527670    4400 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 03:06:33.527755 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527674    4400 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 03:06:33.527755 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527677    4400 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 03:06:33.527755 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527694    4400 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 03:06:33.527755 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527697    4400 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 03:06:33.527755 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527700    4400 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 03:06:33.527755 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527703    4400 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 03:06:33.527755 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527705    4400 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 03:06:33.527755 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527711    4400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 03:06:33.527755 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527714    4400 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 03:06:33.527755 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527717    4400 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 03:06:33.527755 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527719    4400 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 03:06:33.527755 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527722    4400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 03:06:33.527755 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527724    4400 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 03:06:33.527755 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527726    4400 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 03:06:33.527755 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527729    4400 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 03:06:33.527755 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527731    4400 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 03:06:33.527755 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527734    4400 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 03:06:33.528248 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527736    4400 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 03:06:33.528248 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527739    4400 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 03:06:33.528248 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527741    4400 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 03:06:33.528248 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527744    4400 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 03:06:33.528248 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527746    4400 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 03:06:33.528248 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527749    4400 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 03:06:33.528248 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527751    4400 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 03:06:33.528248 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527753    4400 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 03:06:33.528248 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527756    4400 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 03:06:33.528248 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527760    4400 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 03:06:33.528248 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527763    4400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 03:06:33.528248 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527765    4400 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 03:06:33.528248 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527768    4400 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 03:06:33.528248 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527771    4400 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 03:06:33.528248 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527773    4400 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 03:06:33.528248 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527776    4400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 03:06:33.528248 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527778    4400 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 03:06:33.528248 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527780    4400 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 03:06:33.528248 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527784    4400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 03:06:33.528248 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527786    4400 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 03:06:33.528846 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527789    4400 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 03:06:33.528846 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527792    4400 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 03:06:33.528846 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527797    4400 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 03:06:33.528846 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527801    4400 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 03:06:33.528846 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527805    4400 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 03:06:33.528846 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527807    4400 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 03:06:33.528846 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527810    4400 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 03:06:33.528846 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527812    4400 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 03:06:33.528846 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527815    4400 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 03:06:33.528846 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527817    4400 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 03:06:33.528846 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527820    4400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 03:06:33.528846 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527822    4400 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 03:06:33.528846 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527825    4400 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 03:06:33.528846 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527827    4400 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 03:06:33.528846 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527830    4400 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 03:06:33.528846 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527832    4400 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 03:06:33.528846 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527835    4400 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 03:06:33.528846 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527838    4400 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 03:06:33.528846 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527840    4400 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 03:06:33.529315 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527843    4400 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 03:06:33.529315 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527845    4400 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 03:06:33.529315 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527848    4400 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 03:06:33.529315 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527851    4400 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 03:06:33.529315 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527855    4400 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 03:06:33.529315 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527859    4400 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 03:06:33.529315 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527862    4400 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 03:06:33.529315 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527865    4400 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 03:06:33.529315 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527867    4400 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 03:06:33.529315 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527870    4400 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 03:06:33.529315 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527872    4400 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 03:06:33.529315 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527875    4400 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 03:06:33.529315 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527877    4400 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 03:06:33.529315 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527880    4400 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 03:06:33.529315 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527882    4400 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 03:06:33.529315 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527885    4400 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 03:06:33.529315 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527888    4400 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 03:06:33.529315 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527890    4400 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 03:06:33.529315 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527893    4400 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 03:06:33.529786 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527896    4400 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 03:06:33.529786 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527898    4400 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 03:06:33.529786 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527901    4400 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 03:06:33.529786 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527903    4400 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 03:06:33.529786 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527906    4400 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 03:06:33.529786 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527908    4400 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 03:06:33.529786 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527911    4400 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 03:06:33.529786 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527913    4400 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 03:06:33.529786 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527915    4400 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 03:06:33.529786 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527918    4400 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 03:06:33.529786 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527920    4400 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 03:06:33.529786 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527923    4400 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 03:06:33.529786 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527925    4400 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 03:06:33.529786 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527927    4400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 03:06:33.529786 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527930    4400 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 03:06:33.529786 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527933    4400 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 03:06:33.529786 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527935    4400 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 03:06:33.529786 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527938    4400 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 03:06:33.529786 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527940    4400 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 03:06:33.530245 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527943    4400 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 03:06:33.530245 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527945    4400 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 03:06:33.530245 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527948    4400 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 03:06:33.530245 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527950    4400 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 03:06:33.530245 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527953    4400 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 03:06:33.530245 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527956    4400 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 03:06:33.530245 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.527958    4400 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 03:06:33.530245 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.528755    4400 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 03:06:33.530245 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.528763    4400 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 03:06:33.530245 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.528767    4400 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 03:06:33.530245 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.528770    4400 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 03:06:33.530245 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.528773    4400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 03:06:33.530245 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.528776    4400 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 03:06:33.530245 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.528778    4400 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 03:06:33.530245 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.528781    4400 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 03:06:33.530245 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.528784    4400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 03:06:33.530245 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.528786    4400 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 03:06:33.530245 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.528789    4400 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 03:06:33.530245 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.528792    4400 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 03:06:33.530245 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.528795    4400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 03:06:33.530743 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.528797    4400 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 03:06:33.530743 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.528800    4400 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 03:06:33.530743 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.528802    4400 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 03:06:33.530743 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.528805    4400 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 03:06:33.530743 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.528807    4400 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 03:06:33.530743 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.528810    4400 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 03:06:33.530743 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.528813    4400 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 03:06:33.530743 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.528815    4400 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 03:06:33.530743 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.528818    4400 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 03:06:33.530743 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.528821    4400 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 03:06:33.530743 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.528823    4400 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 03:06:33.530743 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.528826    4400 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 03:06:33.530743 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.528828    4400 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 03:06:33.530743 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.528831    4400 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 03:06:33.530743 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.528833    4400 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 03:06:33.530743 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.528836    4400 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 03:06:33.530743 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.528838    4400 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 03:06:33.530743 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.528841    4400 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 03:06:33.530743 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.528843    4400 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 03:06:33.530743 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.528849    4400 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 03:06:33.531511 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.528851    4400 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 03:06:33.531511 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.528854    4400 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 03:06:33.531511 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.528856    4400 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 03:06:33.531511 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.528859    4400 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 03:06:33.531511 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.528862    4400 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 03:06:33.531511 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.528865    4400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 03:06:33.531511 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.528868    4400 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 03:06:33.531511 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.528870    4400 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 03:06:33.531511 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.528873    4400 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 03:06:33.531511 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.528875    4400 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 03:06:33.531511 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.528878    4400 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 03:06:33.531511 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.528880    4400 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 03:06:33.531511 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.528883    4400 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 03:06:33.531511 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.528885    4400 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 03:06:33.531511 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.528888    4400 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 03:06:33.531511 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.528891    4400 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 03:06:33.531511 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.528893    4400 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 03:06:33.531511 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.528895    4400 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 03:06:33.531511 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.528898    4400 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 03:06:33.531511 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.528901    4400 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 03:06:33.532135 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.528904    4400 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 03:06:33.532135 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.528906    4400 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 03:06:33.532135 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.528909    4400 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 03:06:33.532135 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.528912    4400 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 03:06:33.532135 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.528914    4400 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 03:06:33.532135 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.528917    4400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 03:06:33.532135 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.528919    4400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 03:06:33.532135 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.528924    4400 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 03:06:33.532135 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.528928    4400 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 03:06:33.532135 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.528931    4400 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 03:06:33.532135 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.528934    4400 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 03:06:33.532135 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.528938    4400 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 03:06:33.532135 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.528940    4400 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 03:06:33.532135 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.528943    4400 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 03:06:33.532135 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.528947    4400 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 03:06:33.532135 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.528950    4400 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 03:06:33.532135 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.528954    4400 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 03:06:33.532135 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.528957    4400 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 03:06:33.532135 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.528959    4400 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 03:06:33.532604 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.528962    4400 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 03:06:33.532604 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.528964    4400 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 03:06:33.532604 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.528967    4400 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 03:06:33.532604 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.528969    4400 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 03:06:33.532604 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.528972    4400 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 03:06:33.532604 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.528975    4400 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 03:06:33.532604 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.528977    4400 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 03:06:33.532604 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.528980    4400 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 03:06:33.532604 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.528982    4400 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 03:06:33.532604 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.528985    4400 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 03:06:33.532604 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.528988    4400 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 03:06:33.532604 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.528990    4400 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 03:06:33.532604 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.528992    4400 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 03:06:33.532604 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.528995    4400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 03:06:33.532604 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.528997    4400 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 03:06:33.532604 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.528999    4400 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 03:06:33.532604 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529002    4400 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 03:06:33.532604 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529005    4400 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 03:06:33.532604 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529007    4400 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 03:06:33.532604 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529009    4400 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 03:06:33.533102 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529012    4400 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 03:06:33.533102 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529014    4400 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 03:06:33.533102 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529017    4400 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 03:06:33.533102 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529021    4400 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 03:06:33.533102 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529024    4400 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 03:06:33.533102 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529027    4400 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 03:06:33.533102 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529029    4400 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 03:06:33.533102 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529032    4400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 03:06:33.533102 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529034    4400 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 03:06:33.533102 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529037    4400 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 03:06:33.533102 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529040    4400 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 03:06:33.533102 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529046    4400 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 03:06:33.533102 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529050    4400 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 03:06:33.533102 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529054    4400 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 03:06:33.533102 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529527    4400 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 03:06:33.533102 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529532    4400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 03:06:33.533102 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529535    4400 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 03:06:33.533102 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529538    4400 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 03:06:33.533102 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529540    4400 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 03:06:33.533102 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529543    4400 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 03:06:33.533613 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529546    4400 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 03:06:33.533613 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529549    4400 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 03:06:33.533613 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529551    4400 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 03:06:33.533613 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529554    4400 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 03:06:33.533613 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529556    4400 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 03:06:33.533613 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529561    4400 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 03:06:33.533613 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529564    4400 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 03:06:33.533613 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529567    4400 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 03:06:33.533613 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529570    4400 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 03:06:33.533613 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529572    4400 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 03:06:33.533613 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529575    4400 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 03:06:33.533613 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529577    4400 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 03:06:33.533613 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529579    4400 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 03:06:33.533613 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529582    4400 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 03:06:33.533613 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529584    4400 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 03:06:33.533613 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529587    4400 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 03:06:33.533613 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529592    4400 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 03:06:33.533613 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529594    4400 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 03:06:33.533613 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529596    4400 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 03:06:33.534121 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529599    4400 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 03:06:33.534121 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529602    4400 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 03:06:33.534121 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529605    4400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 03:06:33.534121 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529608    4400 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 03:06:33.534121 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529618    4400 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 03:06:33.534121 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529620    4400 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 03:06:33.534121 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529623    4400 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 03:06:33.534121 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529626    4400 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 03:06:33.534121 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529628    4400 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 03:06:33.534121 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529631    4400 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 03:06:33.534121 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529633    4400 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 03:06:33.534121 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529636    4400 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 03:06:33.534121 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529638    4400 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 03:06:33.534121 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529641    4400 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 03:06:33.534121 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529643    4400 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 03:06:33.534121 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529645    4400 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 03:06:33.534121 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529648    4400 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 03:06:33.534121 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529650    4400 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 03:06:33.534121 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529653    4400 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 03:06:33.534121 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529655    4400 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 03:06:33.534988 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529657    4400 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 03:06:33.534988 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529660    4400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 03:06:33.534988 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529662    4400 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 03:06:33.534988 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529665    4400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 03:06:33.534988 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529667    4400 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 03:06:33.534988 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529670    4400 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 03:06:33.534988 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529672    4400 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 03:06:33.534988 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529674    4400 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 03:06:33.534988 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529677    4400 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 03:06:33.534988 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529694    4400 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 03:06:33.534988 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529698    4400 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 03:06:33.534988 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529711    4400 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 03:06:33.534988 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529714    4400 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 03:06:33.534988 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529717    4400 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 03:06:33.534988 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529720    4400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 03:06:33.534988 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529723    4400 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 03:06:33.534988 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529725    4400 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 03:06:33.534988 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529728    4400 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 03:06:33.534988 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529731    4400 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 03:06:33.534988 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529734    4400 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 03:06:33.535878 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529736    4400 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 03:06:33.535878 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529738    4400 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 03:06:33.535878 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529741    4400 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 03:06:33.535878 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529743    4400 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 03:06:33.535878 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529746    4400 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 03:06:33.535878 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529748    4400 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 03:06:33.535878 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529751    4400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 03:06:33.535878 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529753    4400 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 03:06:33.535878 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529756    4400 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 03:06:33.535878 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529758    4400 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 03:06:33.535878 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529760    4400 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 03:06:33.535878 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529763    4400 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 03:06:33.535878 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529765    4400 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 03:06:33.535878 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529767    4400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 03:06:33.535878 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529770    4400 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 03:06:33.535878 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529772    4400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 03:06:33.535878 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529775    4400 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 03:06:33.535878 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529777    4400 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 03:06:33.535878 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529780    4400 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 03:06:33.536703 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529782    4400 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 03:06:33.536703 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529785    4400 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 03:06:33.536703 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529788    4400 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 03:06:33.536703 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529792    4400 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 03:06:33.536703 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529795    4400 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 03:06:33.536703 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529799    4400 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 03:06:33.536703 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529804    4400 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 03:06:33.536703 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529807    4400 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 03:06:33.536703 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529810    4400 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 03:06:33.536703 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529813    4400 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 03:06:33.536703 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529816    4400 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 03:06:33.536703 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529820    4400 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 03:06:33.536703 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529822    4400 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 03:06:33.536703 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529825    4400 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 03:06:33.536703 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529827    4400 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 03:06:33.536703 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529830    4400 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 03:06:33.536703 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529832    4400 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 03:06:33.536703 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529834    4400 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 03:06:33.536703 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529837    4400 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 03:06:33.537544 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529839    4400 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 03:06:33.537544 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529842    4400 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 03:06:33.537544 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.529844    4400 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 03:06:33.537544 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.529934    4400 flags.go:64] FLAG: --address="0.0.0.0"
Nov 01 03:06:33.537544 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.529942    4400 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 01 03:06:33.537544 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.529953    4400 flags.go:64] FLAG: --anonymous-auth="true"
Nov 01 03:06:33.537544 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.529957    4400 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 01 03:06:33.537544 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.529978    4400 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 01 03:06:33.537544 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.529984    4400 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 01 03:06:33.537544 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.529988    4400 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 01 03:06:33.537544 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.529992    4400 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 01 03:06:33.537544 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.529996    4400 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 01 03:06:33.537544 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.529999    4400 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 01 03:06:33.537544 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530002    4400 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 01 03:06:33.537544 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530005    4400 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 01 03:06:33.537544 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530008    4400 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 01 03:06:33.537544 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530013    4400 flags.go:64] FLAG: --cgroup-root=""
Nov 01 03:06:33.537544 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530016    4400 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 01 03:06:33.537544 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530019    4400 flags.go:64] FLAG: --client-ca-file=""
Nov 01 03:06:33.537544 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530022    4400 flags.go:64] FLAG: --cloud-provider="external"
Nov 01 03:06:33.537544 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530025    4400 flags.go:64] FLAG: --cluster-dns="[]"
Nov 01 03:06:33.537544 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530030    4400 flags.go:64] FLAG: --cluster-domain=""
Nov 01 03:06:33.537544 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530032    4400 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 01 03:06:33.537544 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530036    4400 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 01 03:06:33.538603 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530038    4400 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 01 03:06:33.538603 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530042    4400 flags.go:64] FLAG: --container-log-max-files="5"
Nov 01 03:06:33.538603 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530046    4400 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 01 03:06:33.538603 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530049    4400 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 01 03:06:33.538603 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530052    4400 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 01 03:06:33.538603 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530055    4400 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 01 03:06:33.538603 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530058    4400 flags.go:64] FLAG: --contention-profiling="false"
Nov 01 03:06:33.538603 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530061    4400 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 01 03:06:33.538603 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530064    4400 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 01 03:06:33.538603 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530067    4400 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 01 03:06:33.538603 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530070    4400 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 01 03:06:33.538603 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530074    4400 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 01 03:06:33.538603 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530077    4400 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 01 03:06:33.538603 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530080    4400 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 01 03:06:33.538603 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530083    4400 flags.go:64] FLAG: --enable-load-reader="false"
Nov 01 03:06:33.538603 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530086    4400 flags.go:64] FLAG: --enable-server="true"
Nov 01 03:06:33.538603 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530092    4400 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 01 03:06:33.538603 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530096    4400 flags.go:64] FLAG: --event-burst="100"
Nov 01 03:06:33.538603 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530099    4400 flags.go:64] FLAG: --event-qps="50"
Nov 01 03:06:33.538603 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530101    4400 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 01 03:06:33.538603 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530104    4400 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 01 03:06:33.538603 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530107    4400 flags.go:64] FLAG: --eviction-hard=""
Nov 01 03:06:33.538603 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530111    4400 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 01 03:06:33.538603 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530114    4400 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 01 03:06:33.539647 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530118    4400 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 01 03:06:33.539647 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530121    4400 flags.go:64] FLAG: --eviction-soft=""
Nov 01 03:06:33.539647 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530124    4400 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 01 03:06:33.539647 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530127    4400 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 01 03:06:33.539647 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530130    4400 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 01 03:06:33.539647 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530133    4400 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 01 03:06:33.539647 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530136    4400 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 01 03:06:33.539647 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530138    4400 flags.go:64] FLAG: --fail-swap-on="true"
Nov 01 03:06:33.539647 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530141    4400 flags.go:64] FLAG: --feature-gates=""
Nov 01 03:06:33.539647 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530145    4400 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 01 03:06:33.539647 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530149    4400 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 01 03:06:33.539647 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530152    4400 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 01 03:06:33.539647 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530155    4400 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 01 03:06:33.539647 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530157    4400 flags.go:64] FLAG: --healthz-port="10248"
Nov 01 03:06:33.539647 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530160    4400 flags.go:64] FLAG: --help="false"
Nov 01 03:06:33.539647 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530163    4400 flags.go:64] FLAG: --hostname-override="ip-10-0-133-54.us-west-1.compute.internal"
Nov 01 03:06:33.539647 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530166    4400 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 01 03:06:33.539647 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530169    4400 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 01 03:06:33.539647 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530172    4400 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 01 03:06:33.539647 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530175    4400 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 01 03:06:33.539647 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530178    4400 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 01 03:06:33.539647 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530181    4400 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 01 03:06:33.539647 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530184    4400 flags.go:64] FLAG: --image-service-endpoint=""
Nov 01 03:06:33.540656 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530186    4400 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 01 03:06:33.540656 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530190    4400 flags.go:64] FLAG: --kube-api-burst="100"
Nov 01 03:06:33.540656 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530193    4400 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 01 03:06:33.540656 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530197    4400 flags.go:64] FLAG: --kube-api-qps="50"
Nov 01 03:06:33.540656 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530200    4400 flags.go:64] FLAG: --kube-reserved=""
Nov 01 03:06:33.540656 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530203    4400 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 01 03:06:33.540656 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530206    4400 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 01 03:06:33.540656 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530209    4400 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 01 03:06:33.540656 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530212    4400 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 01 03:06:33.540656 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530216    4400 flags.go:64] FLAG: --lock-file=""
Nov 01 03:06:33.540656 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530219    4400 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 01 03:06:33.540656 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530222    4400 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 01 03:06:33.540656 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530225    4400 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 01 03:06:33.540656 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530230    4400 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 01 03:06:33.540656 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530233    4400 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 01 03:06:33.540656 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530236    4400 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 01 03:06:33.540656 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530239    4400 flags.go:64] FLAG: --logging-format="text"
Nov 01 03:06:33.540656 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530241    4400 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 01 03:06:33.540656 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530245    4400 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 01 03:06:33.540656 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530248    4400 flags.go:64] FLAG: --manifest-url=""
Nov 01 03:06:33.540656 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530251    4400 flags.go:64] FLAG: --manifest-url-header=""
Nov 01 03:06:33.540656 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530256    4400 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 01 03:06:33.540656 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530259    4400 flags.go:64] FLAG: --max-open-files="1000000"
Nov 01 03:06:33.540656 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530263    4400 flags.go:64] FLAG: --max-pods="110"
Nov 01 03:06:33.540656 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530266    4400 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 01 03:06:33.541803 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530269    4400 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 01 03:06:33.541803 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530272    4400 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 01 03:06:33.541803 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530274    4400 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 01 03:06:33.541803 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530277    4400 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 01 03:06:33.541803 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530280    4400 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 01 03:06:33.541803 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530282    4400 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 01 03:06:33.541803 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530289    4400 flags.go:64] FLAG: --node-status-max-images="50"
Nov 01 03:06:33.541803 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530292    4400 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 01 03:06:33.541803 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530296    4400 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 01 03:06:33.541803 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530299    4400 flags.go:64] FLAG: --pod-cidr=""
Nov 01 03:06:33.541803 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530302    4400 flags.go:64] FLAG: --pod-manifest-path=""
Nov 01 03:06:33.541803 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530304    4400 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 01 03:06:33.541803 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530307    4400 flags.go:64] FLAG: --pods-per-core="0"
Nov 01 03:06:33.541803 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530310    4400 flags.go:64] FLAG: --port="10250"
Nov 01 03:06:33.541803 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530313    4400 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 01 03:06:33.541803 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530316    4400 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0b938cf80efcff5b7"
Nov 01 03:06:33.541803 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530320    4400 flags.go:64] FLAG: --qos-reserved=""
Nov 01 03:06:33.541803 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530323    4400 flags.go:64] FLAG: --read-only-port="10255"
Nov 01 03:06:33.541803 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530325    4400 flags.go:64] FLAG: --register-node="true"
Nov 01 03:06:33.541803 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530329    4400 flags.go:64] FLAG: --register-with-taints=""
Nov 01 03:06:33.541803 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530332    4400 flags.go:64] FLAG: --registry-burst="10"
Nov 01 03:06:33.541803 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530335    4400 flags.go:64] FLAG: --registry-qps="5"
Nov 01 03:06:33.541803 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530338    4400 flags.go:64] FLAG: --reserved-cpus=""
Nov 01 03:06:33.541803 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530341    4400 flags.go:64] FLAG: --reserved-memory=""
Nov 01 03:06:33.541803 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530344    4400 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 01 03:06:33.541803 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530347    4400 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 01 03:06:33.542982 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530350    4400 flags.go:64] FLAG: --rotate-certificates="false"
Nov 01 03:06:33.542982 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530353    4400 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 01 03:06:33.542982 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530356    4400 flags.go:64] FLAG: --runonce="false"
Nov 01 03:06:33.542982 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530359    4400 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 01 03:06:33.542982 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530362    4400 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 01 03:06:33.542982 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530365    4400 flags.go:64] FLAG: --seccomp-default="false"
Nov 01 03:06:33.542982 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530368    4400 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 01 03:06:33.542982 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530370    4400 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 01 03:06:33.542982 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530373    4400 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 01 03:06:33.542982 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530376    4400 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 01 03:06:33.542982 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530379    4400 flags.go:64] FLAG: --storage-driver-password="root"
Nov 01 03:06:33.542982 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530382    4400 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 01 03:06:33.542982 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530385    4400 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 01 03:06:33.542982 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530387    4400 flags.go:64] FLAG: --storage-driver-user="root"
Nov 01 03:06:33.542982 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530391    4400 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 01 03:06:33.542982 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530394    4400 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 01 03:06:33.542982 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530397    4400 flags.go:64] FLAG: --system-cgroups=""
Nov 01 03:06:33.542982 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530400    4400 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 01 03:06:33.542982 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530405    4400 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 01 03:06:33.542982 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530408    4400 flags.go:64] FLAG: --tls-cert-file=""
Nov 01 03:06:33.542982 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530411    4400 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 01 03:06:33.542982 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530415    4400 flags.go:64] FLAG: --tls-min-version=""
Nov 01 03:06:33.542982 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530418    4400 flags.go:64] FLAG: --tls-private-key-file=""
Nov 01 03:06:33.542982 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530421    4400 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 01 03:06:33.542982 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530424    4400 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 01 03:06:33.543701 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530426    4400 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 01 03:06:33.543701 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530429    4400 flags.go:64] FLAG: --v="2"
Nov 01 03:06:33.543701 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530434    4400 flags.go:64] FLAG: --version="false"
Nov 01 03:06:33.543701 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530438    4400 flags.go:64] FLAG: --vmodule=""
Nov 01 03:06:33.543701 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530442    4400 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 01 03:06:33.543701 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.530445    4400 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 01 03:06:33.543701 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530569    4400 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 03:06:33.543701 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530573    4400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 03:06:33.543701 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530575    4400 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 03:06:33.543701 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530578    4400 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 03:06:33.543701 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530581    4400 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 03:06:33.543701 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530584    4400 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 03:06:33.543701 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530586    4400 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 03:06:33.543701 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530589    4400 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 03:06:33.543701 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530591    4400 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 03:06:33.543701 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530593    4400 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 03:06:33.543701 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530596    4400 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 03:06:33.543701 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530598    4400 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 03:06:33.543701 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530601    4400 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 03:06:33.543701 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530603    4400 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 03:06:33.543701 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530605    4400 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 03:06:33.544284 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530611    4400 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 03:06:33.544284 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530614    4400 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 03:06:33.544284 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530617    4400 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 03:06:33.544284 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530620    4400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 03:06:33.544284 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530622    4400 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 03:06:33.544284 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530625    4400 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 03:06:33.544284 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530627    4400 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 03:06:33.544284 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530630    4400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 03:06:33.544284 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530634    4400 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 03:06:33.544284 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530638    4400 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 03:06:33.544284 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530641    4400 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 03:06:33.544284 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530643    4400 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 03:06:33.544284 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530646    4400 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 03:06:33.544284 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530648    4400 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 03:06:33.544284 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530651    4400 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 03:06:33.544284 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530653    4400 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 03:06:33.544284 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530656    4400 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 03:06:33.544284 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530659    4400 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 03:06:33.544833 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530661    4400 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 03:06:33.544833 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530664    4400 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 03:06:33.544833 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530666    4400 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 03:06:33.544833 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530669    4400 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 03:06:33.544833 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530672    4400 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 03:06:33.544833 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530674    4400 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 03:06:33.544833 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530677    4400 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 03:06:33.544833 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530693    4400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 03:06:33.544833 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530696    4400 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 03:06:33.544833 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530698    4400 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 03:06:33.544833 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530701    4400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 03:06:33.544833 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530704    4400 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 03:06:33.544833 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530706    4400 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 03:06:33.544833 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530709    4400 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 03:06:33.544833 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530712    4400 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 03:06:33.544833 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530715    4400 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 03:06:33.544833 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530717    4400 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 03:06:33.544833 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530720    4400 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 03:06:33.544833 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530722    4400 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 03:06:33.544833 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530725    4400 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 03:06:33.545427 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530727    4400 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 03:06:33.545427 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530730    4400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 03:06:33.545427 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530733    4400 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 03:06:33.545427 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530735    4400 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 03:06:33.545427 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530738    4400 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 03:06:33.545427 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530740    4400 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 03:06:33.545427 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530743    4400 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 03:06:33.545427 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530745    4400 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 03:06:33.545427 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530748    4400 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 03:06:33.545427 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530751    4400 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 03:06:33.545427 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530762    4400 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 03:06:33.545427 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530765    4400 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 03:06:33.545427 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530767    4400 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 03:06:33.545427 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530770    4400 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 03:06:33.545427 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530772    4400 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 03:06:33.545427 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530775    4400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 03:06:33.545427 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530777    4400 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 03:06:33.545427 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530781    4400 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 03:06:33.545427 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530783    4400 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 03:06:33.546318 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530786    4400 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 03:06:33.546318 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530788    4400 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 03:06:33.546318 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530793    4400 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 03:06:33.546318 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530796    4400 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 03:06:33.546318 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530798    4400 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 03:06:33.546318 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530801    4400 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 03:06:33.546318 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530803    4400 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 03:06:33.546318 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530807    4400 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 03:06:33.546318 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530809    4400 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 03:06:33.546318 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530812    4400 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 03:06:33.546318 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530814    4400 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 03:06:33.546318 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530817    4400 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 03:06:33.546318 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530819    4400 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 03:06:33.546318 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530822    4400 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 03:06:33.546318 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530825    4400 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 03:06:33.546318 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530827    4400 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 03:06:33.546318 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530829    4400 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 03:06:33.546318 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530832    4400 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 03:06:33.546318 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530835    4400 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 03:06:33.546318 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530837    4400 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 03:06:33.547067 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530840    4400 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 03:06:33.547067 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530842    4400 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 03:06:33.547067 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530844    4400 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 03:06:33.547067 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530847    4400 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 03:06:33.547067 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530849    4400 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 03:06:33.547067 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530852    4400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 03:06:33.547067 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530854    4400 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 03:06:33.547067 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530857    4400 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 03:06:33.547067 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530859    4400 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 03:06:33.547067 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530862    4400 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 03:06:33.547067 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530865    4400 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 03:06:33.547067 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530896    4400 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 03:06:33.547067 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530928    4400 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 03:06:33.547067 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.530931    4400 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 03:06:33.547067 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.531055    4400 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 03:06:33.547067 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531198    4400 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 03:06:33.547495 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531209    4400 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 03:06:33.547495 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531215    4400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 03:06:33.547495 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531221    4400 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 03:06:33.547495 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531226    4400 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 03:06:33.547495 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531231    4400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 03:06:33.547495 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531236    4400 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 03:06:33.547495 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531240    4400 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 03:06:33.547495 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531244    4400 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 03:06:33.547495 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531254    4400 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 03:06:33.547495 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531259    4400 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 03:06:33.547495 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531263    4400 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 03:06:33.547495 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531267    4400 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 03:06:33.547495 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531271    4400 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 03:06:33.547495 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531275    4400 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 03:06:33.547495 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531292    4400 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 03:06:33.547495 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531296    4400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 03:06:33.547495 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531300    4400 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 03:06:33.547495 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531305    4400 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 03:06:33.547495 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531309    4400 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 03:06:33.547495 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531313    4400 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 03:06:33.548044 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531318    4400 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 03:06:33.548044 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531327    4400 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 03:06:33.548044 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531331    4400 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 03:06:33.548044 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531335    4400 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 03:06:33.548044 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531339    4400 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 03:06:33.548044 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531344    4400 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 03:06:33.548044 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531348    4400 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 03:06:33.548044 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531352    4400 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 03:06:33.548044 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531357    4400 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 03:06:33.548044 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531361    4400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 03:06:33.548044 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531365    4400 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 03:06:33.548044 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531371    4400 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 03:06:33.548044 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531379    4400 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 03:06:33.548044 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531383    4400 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 03:06:33.548044 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531392    4400 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 03:06:33.548044 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531396    4400 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 03:06:33.548044 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531401    4400 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 03:06:33.548044 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531405    4400 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 03:06:33.548044 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531409    4400 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 03:06:33.548754 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531413    4400 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 03:06:33.548754 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531417    4400 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 03:06:33.548754 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531421    4400 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 03:06:33.548754 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531426    4400 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 03:06:33.548754 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531430    4400 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 03:06:33.548754 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531434    4400 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 03:06:33.548754 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531438    4400 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 03:06:33.548754 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531443    4400 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 03:06:33.548754 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531452    4400 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 03:06:33.548754 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531456    4400 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 03:06:33.548754 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531460    4400 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 03:06:33.548754 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531464    4400 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 03:06:33.548754 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531468    4400 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 03:06:33.548754 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531472    4400 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 03:06:33.548754 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531476    4400 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 03:06:33.548754 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531480    4400 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 03:06:33.548754 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531485    4400 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 03:06:33.548754 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531489    4400 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 03:06:33.548754 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531494    4400 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 03:06:33.548754 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531498    4400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 03:06:33.549464 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531507    4400 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 03:06:33.549464 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531511    4400 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 03:06:33.549464 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531516    4400 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 03:06:33.549464 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531521    4400 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 03:06:33.549464 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531526    4400 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 03:06:33.549464 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531533    4400 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 03:06:33.549464 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531537    4400 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 03:06:33.549464 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531542    4400 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 03:06:33.549464 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531546    4400 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 03:06:33.549464 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531553    4400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 03:06:33.549464 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531557    4400 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 03:06:33.549464 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531561    4400 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 03:06:33.549464 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531570    4400 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 03:06:33.549464 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531575    4400 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 03:06:33.549464 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531579    4400 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 03:06:33.549464 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531583    4400 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 03:06:33.549464 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531587    4400 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 03:06:33.549464 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531591    4400 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 03:06:33.549464 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531595    4400 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 03:06:33.549464 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531600    4400 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 03:06:33.550017 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531604    4400 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 03:06:33.550017 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531608    4400 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 03:06:33.550017 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531613    4400 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 03:06:33.550017 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531619    4400 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 03:06:33.550017 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531625    4400 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 03:06:33.550017 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531634    4400 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 03:06:33.550017 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531639    4400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 03:06:33.550017 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531644    4400 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 03:06:33.550017 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531648    4400 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 03:06:33.550017 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531653    4400 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 03:06:33.550017 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531657    4400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 03:06:33.550017 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531663    4400 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 03:06:33.550017 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531669    4400 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 03:06:33.550017 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531674    4400 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 03:06:33.550017 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531678    4400 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 03:06:33.550017 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531706    4400 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 03:06:33.550017 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531711    4400 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 03:06:33.550017 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531723    4400 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 03:06:33.550451 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531728    4400 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 03:06:33.550451 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531736    4400 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 03:06:33.550451 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531740    4400 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 03:06:33.550451 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531745    4400 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 03:06:33.550451 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531749    4400 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 03:06:33.550451 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531753    4400 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 03:06:33.550451 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531757    4400 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 03:06:33.550451 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.531761    4400 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 03:06:33.550451 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.536624    4400 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 01 03:06:33.550451 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.536641    4400 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 01 03:06:33.550451 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.536665    4400 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 01 03:06:33.550451 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.536673    4400 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 01 03:06:33.550451 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.536776    4400 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 03:06:33.550451 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.536784    4400 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 03:06:33.550451 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.536789    4400 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 03:06:33.550451 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.536794    4400 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 03:06:33.550451 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.536798    4400 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 03:06:33.550451 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.536803    4400 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 03:06:33.550451 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.536807    4400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 03:06:33.550942 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.536811    4400 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 03:06:33.550942 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.536815    4400 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 03:06:33.550942 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.536819    4400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 03:06:33.550942 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.536824    4400 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 03:06:33.550942 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.536828    4400 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 03:06:33.550942 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.536845    4400 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 03:06:33.550942 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.536849    4400 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 03:06:33.550942 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.536854    4400 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 03:06:33.550942 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.536858    4400 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 03:06:33.550942 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.536863    4400 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 03:06:33.550942 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.536867    4400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 03:06:33.550942 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.536871    4400 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 03:06:33.550942 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.536877    4400 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 03:06:33.550942 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.536882    4400 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 03:06:33.550942 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.536886    4400 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 03:06:33.550942 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.536891    4400 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 03:06:33.550942 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.536895    4400 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 03:06:33.550942 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.536899    4400 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 03:06:33.550942 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.536903    4400 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 03:06:33.551396 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.536908    4400 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 03:06:33.551396 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.536915    4400 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 03:06:33.551396 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.536921    4400 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 03:06:33.551396 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.536926    4400 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 03:06:33.551396 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.536930    4400 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 03:06:33.551396 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.536934    4400 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 03:06:33.551396 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.536938    4400 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 03:06:33.551396 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.536942    4400 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 03:06:33.551396 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.536946    4400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 03:06:33.551396 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.536950    4400 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 03:06:33.551396 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.536955    4400 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 03:06:33.551396 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.536959    4400 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 03:06:33.551396 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.536963    4400 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 03:06:33.551396 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.536968    4400 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 03:06:33.551396 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.536972    4400 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 03:06:33.551396 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.536976    4400 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 03:06:33.551396 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.536980    4400 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 03:06:33.551396 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.536984    4400 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 03:06:33.551396 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.536988    4400 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 03:06:33.551396 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.536993    4400 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 03:06:33.551893 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.536998    4400 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 03:06:33.551893 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537002    4400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 03:06:33.551893 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537006    4400 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 03:06:33.551893 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537011    4400 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 03:06:33.551893 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537015    4400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 03:06:33.551893 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537019    4400 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 03:06:33.551893 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537023    4400 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 03:06:33.551893 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537027    4400 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 03:06:33.551893 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537032    4400 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 03:06:33.551893 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537036    4400 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 03:06:33.551893 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537041    4400 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 03:06:33.551893 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537045    4400 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 03:06:33.551893 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537049    4400 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 03:06:33.551893 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537053    4400 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 03:06:33.551893 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537057    4400 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 03:06:33.551893 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537061    4400 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 03:06:33.551893 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537066    4400 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 03:06:33.551893 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537071    4400 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 03:06:33.551893 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537075    4400 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 03:06:33.552334 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537079    4400 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 03:06:33.552334 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537083    4400 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 03:06:33.552334 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537087    4400 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 03:06:33.552334 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537091    4400 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 03:06:33.552334 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537095    4400 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 03:06:33.552334 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537099    4400 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 03:06:33.552334 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537104    4400 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 03:06:33.552334 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537108    4400 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 03:06:33.552334 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537112    4400 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 03:06:33.552334 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537116    4400 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 03:06:33.552334 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537120    4400 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 03:06:33.552334 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537124    4400 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 03:06:33.552334 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537128    4400 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 03:06:33.552334 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537133    4400 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 03:06:33.552334 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537137    4400 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 03:06:33.552334 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537141    4400 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 03:06:33.552334 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537146    4400 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 03:06:33.552334 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537150    4400 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 03:06:33.552334 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537154    4400 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 03:06:33.552334 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537158    4400 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 03:06:33.552866 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537162    4400 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 03:06:33.552866 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537166    4400 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 03:06:33.552866 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537170    4400 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 03:06:33.552866 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537174    4400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 03:06:33.552866 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537181    4400 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 03:06:33.552866 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537186    4400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 03:06:33.552866 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537191    4400 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 03:06:33.552866 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537195    4400 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 03:06:33.552866 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537199    4400 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 03:06:33.552866 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537203    4400 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 03:06:33.552866 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537207    4400 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 03:06:33.552866 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537211    4400 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 03:06:33.552866 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537215    4400 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 03:06:33.552866 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537219    4400 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 03:06:33.552866 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537223    4400 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 03:06:33.552866 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537227    4400 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 03:06:33.552866 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537231    4400 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 03:06:33.552866 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537235    4400 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 03:06:33.552866 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537239    4400 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 03:06:33.553317 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537244    4400 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 03:06:33.553317 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537248    4400 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 03:06:33.553317 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.537287    4400 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 03:06:33.553317 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537504    4400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 03:06:33.553317 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537512    4400 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 03:06:33.553317 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537518    4400 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 03:06:33.553317 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537522    4400 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 03:06:33.553317 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537526    4400 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 03:06:33.553317 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537530    4400 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 03:06:33.553317 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537535    4400 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 03:06:33.553317 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537539    4400 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 03:06:33.553317 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537543    4400 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 03:06:33.553317 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537546    4400 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 03:06:33.553317 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537550    4400 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 03:06:33.553317 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537555    4400 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 03:06:33.553317 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537559    4400 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 03:06:33.553715 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537562    4400 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 03:06:33.553715 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537566    4400 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 03:06:33.553715 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537570    4400 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 03:06:33.553715 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537574    4400 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 03:06:33.553715 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537578    4400 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 03:06:33.553715 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537583    4400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 03:06:33.553715 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537587    4400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 03:06:33.553715 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537591    4400 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 03:06:33.553715 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537595    4400 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 03:06:33.553715 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537599    4400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 03:06:33.553715 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537603    4400 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 03:06:33.553715 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537608    4400 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 03:06:33.553715 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537612    4400 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 03:06:33.553715 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537616    4400 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 03:06:33.553715 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537620    4400 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 03:06:33.553715 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537625    4400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 03:06:33.553715 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537629    4400 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 03:06:33.553715 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537633    4400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 03:06:33.553715 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537637    4400 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 03:06:33.553715 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537641    4400 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 03:06:33.554182 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537645    4400 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 03:06:33.554182 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537649    4400 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 03:06:33.554182 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537654    4400 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 03:06:33.554182 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537658    4400 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 03:06:33.554182 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537662    4400 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 03:06:33.554182 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537667    4400 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 03:06:33.554182 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537671    4400 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 03:06:33.554182 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537675    4400 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 03:06:33.554182 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537698    4400 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 03:06:33.554182 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537703    4400 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 03:06:33.554182 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537707    4400 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 03:06:33.554182 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537712    4400 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 03:06:33.554182 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537716    4400 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 03:06:33.554182 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537720    4400 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 03:06:33.554182 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537724    4400 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 03:06:33.554182 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537728    4400 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 03:06:33.554182 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537732    4400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 03:06:33.554182 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537736    4400 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 03:06:33.554182 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537745    4400 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 03:06:33.554182 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537749    4400 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 03:06:33.554643 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537753    4400 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 03:06:33.554643 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537759    4400 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 03:06:33.554643 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537766    4400 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 03:06:33.554643 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537772    4400 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 03:06:33.554643 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537776    4400 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 03:06:33.554643 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537780    4400 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 03:06:33.554643 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537784    4400 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 03:06:33.554643 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537789    4400 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 03:06:33.554643 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537793    4400 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 03:06:33.554643 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537797    4400 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 03:06:33.554643 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537801    4400 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 03:06:33.554643 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537805    4400 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 03:06:33.554643 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537809    4400 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 03:06:33.554643 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537813    4400 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 03:06:33.554643 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537820    4400 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 03:06:33.554643 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537825    4400 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 03:06:33.554643 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537830    4400 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 03:06:33.554643 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537834    4400 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 03:06:33.554643 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537838    4400 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 03:06:33.555148 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537842    4400 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 03:06:33.555148 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537846    4400 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 03:06:33.555148 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537850    4400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 03:06:33.555148 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537854    4400 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 03:06:33.555148 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537859    4400 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 03:06:33.555148 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537863    4400 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 03:06:33.555148 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537870    4400 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 03:06:33.555148 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537876    4400 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 03:06:33.555148 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537880    4400 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 03:06:33.555148 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537885    4400 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 03:06:33.555148 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537889    4400 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 03:06:33.555148 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537894    4400 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 03:06:33.555148 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537899    4400 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 03:06:33.555148 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537903    4400 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 03:06:33.555148 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537908    4400 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 03:06:33.555148 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537912    4400 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 03:06:33.555148 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537916    4400 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 03:06:33.555148 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537920    4400 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 03:06:33.555148 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537924    4400 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 03:06:33.555603 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537928    4400 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 03:06:33.555603 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537932    4400 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 03:06:33.555603 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537936    4400 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 03:06:33.555603 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537940    4400 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 03:06:33.555603 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537944    4400 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 03:06:33.555603 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537949    4400 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 03:06:33.555603 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537953    4400 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 03:06:33.555603 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537957    4400 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 03:06:33.555603 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537963    4400 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 03:06:33.555603 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537968    4400 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 03:06:33.555603 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537973    4400 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 03:06:33.555603 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537977    4400 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 03:06:33.555603 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537981    4400 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 03:06:33.555603 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537985    4400 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 03:06:33.555603 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.537989    4400 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 03:06:33.555603 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.538030    4400 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 03:06:33.556031 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538080    4400 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 03:06:33.556031 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538086    4400 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 03:06:33.556031 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538091    4400 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 03:06:33.556031 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538095    4400 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 03:06:33.556031 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538099    4400 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 03:06:33.556031 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538104    4400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 03:06:33.556031 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538108    4400 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 03:06:33.556031 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538112    4400 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 03:06:33.556031 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538117    4400 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 03:06:33.556031 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538124    4400 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 03:06:33.556031 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538131    4400 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 03:06:33.556031 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538137    4400 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 03:06:33.556031 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538141    4400 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 03:06:33.556031 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538145    4400 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 03:06:33.556031 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538149    4400 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 03:06:33.556031 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538154    4400 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 03:06:33.556031 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538158    4400 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 03:06:33.556031 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538163    4400 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 03:06:33.556031 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538167    4400 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 03:06:33.556511 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538171    4400 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 03:06:33.556511 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538175    4400 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 03:06:33.556511 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538179    4400 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 03:06:33.556511 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538183    4400 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 03:06:33.556511 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538187    4400 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 03:06:33.556511 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538194    4400 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 03:06:33.556511 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538198    4400 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 03:06:33.556511 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538202    4400 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 03:06:33.556511 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538207    4400 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 03:06:33.556511 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538211    4400 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 03:06:33.556511 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538215    4400 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 03:06:33.556511 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538220    4400 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 03:06:33.556511 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538224    4400 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 03:06:33.556511 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538228    4400 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 03:06:33.556511 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538233    4400 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 03:06:33.556511 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538237    4400 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 03:06:33.556511 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538241    4400 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 03:06:33.556511 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538245    4400 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 03:06:33.556511 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538250    4400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 03:06:33.556511 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538254    4400 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 03:06:33.557056 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538258    4400 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 03:06:33.557056 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538262    4400 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 03:06:33.557056 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538268    4400 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 03:06:33.557056 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538272    4400 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 03:06:33.557056 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538276    4400 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 03:06:33.557056 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538280    4400 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 03:06:33.557056 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538285    4400 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 03:06:33.557056 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538289    4400 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 03:06:33.557056 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538293    4400 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 03:06:33.557056 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538297    4400 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 03:06:33.557056 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538302    4400 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 03:06:33.557056 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538306    4400 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 03:06:33.557056 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538310    4400 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 03:06:33.557056 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538314    4400 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 03:06:33.557056 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538318    4400 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 03:06:33.557056 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538322    4400 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 03:06:33.557056 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538327    4400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 03:06:33.557056 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538332    4400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 03:06:33.557056 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538337    4400 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 03:06:33.557056 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538341    4400 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 03:06:33.557626 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538347    4400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 03:06:33.557626 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538352    4400 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 03:06:33.557626 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538356    4400 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 03:06:33.557626 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538360    4400 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 03:06:33.557626 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538365    4400 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 03:06:33.557626 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538369    4400 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 03:06:33.557626 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538374    4400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 03:06:33.557626 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538378    4400 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 03:06:33.557626 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538382    4400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 03:06:33.557626 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538386    4400 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 03:06:33.557626 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538390    4400 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 03:06:33.557626 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538394    4400 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 03:06:33.557626 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538398    4400 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 03:06:33.557626 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538403    4400 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 03:06:33.557626 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538408    4400 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 03:06:33.557626 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538413    4400 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 03:06:33.557626 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538417    4400 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 03:06:33.557626 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538421    4400 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 03:06:33.557626 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538425    4400 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 03:06:33.557626 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538429    4400 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 03:06:33.558139 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538434    4400 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 03:06:33.558139 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538438    4400 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 03:06:33.558139 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538442    4400 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 03:06:33.558139 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538446    4400 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 03:06:33.558139 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538451    4400 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 03:06:33.558139 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538455    4400 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 03:06:33.558139 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538459    4400 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 03:06:33.558139 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538463    4400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 03:06:33.558139 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538467    4400 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 03:06:33.558139 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538473    4400 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 03:06:33.558139 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538477    4400 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 03:06:33.558139 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538481    4400 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 03:06:33.558139 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538488    4400 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 03:06:33.558139 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538495    4400 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 03:06:33.558139 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538500    4400 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 03:06:33.558139 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538506    4400 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 03:06:33.558139 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538511    4400 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 03:06:33.558139 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538516    4400 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 03:06:33.558139 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538520    4400 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 03:06:33.558585 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538526    4400 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 03:06:33.558585 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538531    4400 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 03:06:33.558585 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538535    4400 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 03:06:33.558585 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538539    4400 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 03:06:33.558585 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538543    4400 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 03:06:33.558585 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538547    4400 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 03:06:33.558585 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538551    4400 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 03:06:33.558585 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:33.538557    4400 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 03:06:33.558585 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.538635    4400 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 01 03:06:33.558585 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.540111    4400 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 01 03:06:33.558585 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.540186    4400 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 01 03:06:33.558585 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.540459    4400 server.go:1014] "Starting client certificate rotation"
Nov 01 03:06:33.558585 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.540590    4400 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 01 03:06:33.558585 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.540635    4400 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-31 03:00:52 +0000 UTC" deadline="2028-06-20 06:21:34.32603248 +0000 UTC"
Nov 01 03:06:33.558585 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.540671    4400 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14331h15m0.785363373s"
Nov 01 03:06:33.558585 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.541204    4400 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 01 03:06:33.559023 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.542840    4400 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 01 03:06:33.559023 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.544277    4400 log.go:25] "Validated CRI v1 runtime API"
Nov 01 03:06:33.559023 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.546578    4400 log.go:25] "Validated CRI v1 image API"
Nov 01 03:06:33.559023 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.547339    4400 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 01 03:06:33.559023 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.549207    4400 fs.go:135] Filesystem UUIDs: map[2b49dd07-e358-4aa8-88e7-94f420cbf1f5:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 88d8d6f1-66d5-44a0-9804-56478685e0dd:/dev/nvme0n1p3]
Nov 01 03:06:33.559160 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.549256    4400 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/02b4fc07486757aadcb2800218a1fc8a3076f38fa1f4185193c2aa7e5ef8e7f6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/02b4fc07486757aadcb2800218a1fc8a3076f38fa1f4185193c2aa7e5ef8e7f6/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/38e73bd0c99d3b29aac95c81a2e64aa64d2f964154b942bfecd271af059f2c56/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/38e73bd0c99d3b29aac95c81a2e64aa64d2f964154b942bfecd271af059f2c56/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/410bd40c1eca1aaaeeea283ba9e2c2a185c749e790f96ef1fdb69feacf71c338/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/410bd40c1eca1aaaeeea283ba9e2c2a185c749e790f96ef1fdb69feacf71c338/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/656a993b19b5fb0b91c6c5940fe8ed4ed810a56d28407cb6a0bf8d1ed48f10c4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/656a993b19b5fb0b91c6c5940fe8ed4ed810a56d28407cb6a0bf8d1ed48f10c4/userdata/shm major:0 minor:208 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/65bca1ddcd7a850283c27829fa3fb5a6b5f48f97e5bad23b011d4a2f20816cb8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/65bca1ddcd7a850283c27829fa3fb5a6b5f48f97e5bad23b011d4a2f20816cb8/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7f86d26dee5abd851a44f92a5c77c0c1738cfb836092f63bd0db9c8840f8def0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7f86d26dee5abd851a44f92a5c77c0c1738cfb836092f63bd0db9c8840f8def0/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8eda87d96f5445f76e7e3a31f88f1b1cfb59bd38489e93b16b05392b80c42192/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8eda87d96f5445f76e7e3a31f88f1b1cfb59bd38489e93b16b05392b80c42192/userdata/shm major:0 minor:87 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a4edacea31fe2dce66095cabb9f2f3de51aa76066e1021f1a6fb8287ba47933f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a4edacea31fe2dce66095cabb9f2f3de51aa76066e1021f1a6fb8287ba47933f/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b13508327d908895943975963c426c766b021361d87a9196f741e7bdd976509e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b13508327d908895943975963c426c766b021361d87a9196f741e7bdd976509e/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c2f90ee59606115f6de4c50b793a548e266a500e6c80b26671d87b883f9d7fec/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c2f90ee59606115f6de4c50b793a548e266a500e6c80b26671d87b883f9d7fec/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c3970e088aa262baf754bfedaeebe86bb4f33f854f9f5d5fbec0b6fbc0aa084d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c3970e088aa262baf754bfedaeebe86bb4f33f854f9f5d5fbec0b6fbc0aa084d/userdata/shm major:0 minor:210 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ca80f4e18c06e6002a072be7097cb83d48d6fe56ab99647cb2d9d8894402e92e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ca80f4e18c06e6002a072be7097cb83d48d6fe56ab99647cb2d9d8894402e92e/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d84526df1da0ed7187d5efa80942b6e3936499085b86291746b62ea9e3886525/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d84526df1da0ed7187d5efa80942b6e3936499085b86291746b62ea9e3886525/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e109df44865a3f092a8d3d3680d8eee4ecf3f68aaf32da287d971df39293cbda/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e109df44865a3f092a8d3d3680d8eee4ecf3f68aaf32da287d971df39293cbda/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0303d6c6-c4ec-4403-af74-f4bcb5cbf2a4/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/0303d6c6-c4ec-4403-af74-f4bcb5cbf2a4/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0ede655d-c20d-41cc-b5f0-2db4338c1ba7/volumes/kubernetes.io~projected/kube-api-access-slnpd:{mountpoint:/var/lib/kubelet/pods/0ede655d-c20d-41cc-b5f0-2db4338c1ba7/volumes/kubernetes.io~projected/kube-api-access-slnpd major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/11fb87e4-ef01-4209-8207-80b66ab09f5e/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/11fb87e4-ef01-4209-8207-80b66ab09f5e/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/11fb87e4-ef01-4209-8207-80b66ab09f5e/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/11fb87e4-ef01-4209-8207-80b66ab09f5e/volumes/kubernetes.io~empty-dir/tmp major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/11fb87e4-ef01-4209-8207-80b66ab09f5e/volumes/kubernetes.io~projected/kube-api-access-h58g6:{mountpoint:/var/lib/kubelet/pods/11fb87e4-ef01-4209-8207-80b66ab09f5e/volumes/kubernetes.io~projected/kube-api-access-h58g6 major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4c1dc667-d774-4db6-b221-f4e7de423260/volumes/kubernetes.io~projected/kube-api-access-2459t:{mountpoint:/var/lib/kubelet/pods/4c1dc667-d774-4db6-b221-f4e7de423260/volumes/kubernetes.io~projected/kube-api-access-2459t major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4c1dc667-d774-4db6-b221-f4e7de423260/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/4c1dc667-d774-4db6-b221-f4e7de423260/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/56aed8ff-9b41-400a-90fb-8c283ce0c820/volumes/kubernetes.io~projected/kube-api-access-r9wst:{mountpoint:/var/lib/kubelet/pods/56aed8ff-9b41-400a-90fb-8c283ce0c820/volumes/kubernetes.io~projected/kube-api-access-r9wst major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/68932dfe-7e25-4f1b-b485-77c0519203db/volumes/kubernetes.io~projected/kube-api-access-qnrhp:{mountpoint:/var/lib/kubelet/pods/68932dfe-7e25-4f1b-b485-77c0519203db/volumes/kubernetes.io~projected/kube-api-access-qnrhp major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6e302b40-9c43-4822-8c8b-4d021a0590ab/volumes/kubernetes.io~projected/kube-api-access-nxqx8:{mountpoint:/var/lib/kubelet/pods/6e302b40-9c43-4822-8c8b-4d021a0590ab/volumes/kubernetes.io~projected/kube-api-access-nxqx8 major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/73b27ccb-7d7a-40c1-8479-8379b4fb50ed/volumes/kubernetes.io~projected/kube-api-access-httfj:{mountpoint:/var/lib/kubelet/pods/73b27ccb-7d7a-40c1-8479-8379b4fb50ed/volumes/kubernetes.io~projected/kube-api-access-httfj major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/994175bb-d192-47f7-b162-a840eb0f9be1/volumes/kubernetes.io~projected/kube-api-access-xtjjf:{mountpoint:/var/lib/kubelet/pods/994175bb-d192-47f7-b162-a840eb0f9be1/volumes/kubernetes.io~projected/kube-api-access-xtjjf major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bd8260bb-3811-4cdc-91d0-c6ddc693891e/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/bd8260bb-3811-4cdc-91d0-c6ddc693891e/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bf9dac1d-d9fc-455a-be03-ea34f8838730/volumes/kubernetes.io~projected/kube-api-access-jhdnz:{mountpoint:/var/lib/kubelet/pods/bf9dac1d-d9fc-455a-be03-ea34f8838730/volumes/kubernetes.io~projected/kube-api-access-jhdnz major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c9e89456-9acd-408f-87b5-f5fd968c82c3/volumes/kubernetes.io~projected/kube-api-access-gljgq:{mountpoint:/var/lib/kubelet/pods/c9e89456-9acd-408f-87b5-f5fd968c82c3/volumes/kubernetes.io~projected/kube-api-access-gljgq major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d7cef651-3047-4e64-ae7f-b0fa71fbd0fe/volumes/kubernetes.io~projected/kube-api-access-t86zn:{mountpoint:/var/lib/kubelet/pods/d7cef651-3047-4e64-ae7f-b0fa71fbd0fe/volumes/kubernetes.io~projected/kube-api-access-t86zn major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/de5f469f-1029-4161-bf51-6ad18a3ac9e2/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/de5f469f-1029-4161-bf51-6ad18a3ac9e2/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/de5f469f-1029-4161-bf51-6ad18a3ac9e2/volumes/kubernetes.io~projected/kube-api-access-ckfr8:{mountpoint:/var/lib/kubelet/pods/de5f469f-1029-4161-bf51-6ad18a3ac9e2/volumes/kubernetes.io~projected/kube-api-access-ckfr8 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/de5f469f-1029-4161-bf51-6ad18a3ac9e2/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/de5f469f-1029-4161-bf51-6ad18a3ac9e2/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e48a634f-ab31-435c-96c1-c3633c5bc061/volumes/kubernetes.io~projected/kube-api-access-b2krw:{mountpoint:/var/lib/kubelet/pods/e48a634f-ab31-435c-96c1-c3633c5bc061/volumes/kubernetes.io~projected/kube-api-access-b2krw major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e8f2a24b-e770-4615-a2c7-b2d065e14f77/volumes/kubernetes.io~projected/kube-api-access-85r94:{mountpoint:/var/lib/kubelet/pods/e8f2a24b-e770-4615-a2c7-b2d065e14f77/volumes/kubernetes.io~projected/kube-api-access-85r94 major:0 minor:62 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/eb34588f10d06198285539e567c677b0bcd6d5c0fa429f017091bdea777c7b65/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/c94bc7f78d009e7768021a52baeed7766dc292cea208f2bbee2536eed3f69c16/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/9dfa0cf463a270ac6cd457d8fc545bb3734b08f83d6148bd610d0b2861c094cd/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-105:{mountpoint:/var/lib/containers/storage/overlay/7e4a8ed32b3b04fcf4b813391322ca68c4e4b4e3142e7dcb24c5faaa2c705495/merged major:0 minor:105 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/1873c902a63968c3cda9b97d6a96c6d1d8da7dee00abd03a30ef43a20a4c9c54/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/bd2e4b6577fe742149ac57b234ae3909e55b127c42eab97c5db05eec128f88a5/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/b0bf723954543445deedf67d81ed059d7539a06ede0dfaf8875ed8359cf80b18/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/3b6e5f658a757dfc888d14356efae3bcb0c2c8852e227be2c41b55bb43711574/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/cb149aecfbe1473decdc99315e849f455ace115b65865c4b7074de8e3a676677/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/02533bc55cf262b4d801bf3c76b477ce2d90a3af312236373d1211999aea1677/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-138:{mountpoint:/var/lib/containers/storage/overlay/8bd5a82f5750f2f0973abc45b96d10eda2e6a84ee84302f224e2165e5b7c77d7/merged major:0 minor:138 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/e7ac5b7e611858bc23157d27496facfd0a2e950a5687ea5b8aca7a5225dcf796/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/949da68d96b2db23736f04d259acadd3c9cd453d9166dfb11f947f10599535b7/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-154:{mountpoint:/var/lib/containers/storage/overlay/77d4c8a92baddd7141db3ca2d3f971be62bcb5a612aa23426a2afe54f60a3da7/merged major:0 minor:154 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/b2dbeff71d25641a2be16f11ffd2276d56754bdcde6edfb0539897f2a067823b/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-169:{mountpoint:/var/lib/containers/storage/overlay/7244c4626b84c955e695819d62d08c9bf2238a456ca1672298b7b0d170682f7b/merged major:0 minor:169 fsType:overlay blockSize:0} overlay_0-174:{mountpoint:/var/lib/containers/storage/overlay/ef26036eaa8ba96df28b569c981a8b85df1bfbe6ee33a85909a88f31a68cdb63/merged major:0 minor:174 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/9a0f4f6e2d3f535aeaf6ca97192e5e18e3ea7ade89b035aca1f1bb8d34c9e979/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/e1b61d17cba75f288971ed82212d708c178208c5e02d6dfb1a0254529722551a/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/7d400b7fd18e1274f9b68d5cbafaba53baaa98e1741023581eae89d094a60a85/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/06bcf2c723ef076f26d6c4a481d87cc2fc0f0e37c7392854dfcd1fe7fa2cb89c/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/e25aefd37a664a518bfe079579df5c7da3037a1405b2c083b140328f8d435dbc/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-212:{mountpoint:/var/lib/containers/storage/overlay/835d4a4ccd0b14a361534c76d7bf781e131d4dd9c0ef5b53f63e80c988c272ca/merged major:0 minor:212 fsType:overlay blockSize:0} overlay_0-214:{mountpoint:/var/lib/containers/storage/overlay/8f43b91eac365ae9e47201c4056d6d16fc9f8d627b3b7ba3ecf357c83bb8dbf5/merged major:0 minor:214 fsType:overlay blockSize:0} overlay_0-216:{mountpoint:/var/lib/containers/storage/overlay/4b35d5f4d31debcf59cc95a09053c812e23fd3b0687b5ca4eb23c276afd34ea4/merged major:0 minor:216 fsType:overlay blockSize:0} overlay_0-218:{mountpoint:/var/lib/containers/storage/overlay/c4e128b05d96296d65036b9e0d84d636b6607b2afc18fe7b2c5f6febf3392715/merged major:0 minor:218 fsType:overlay blockSize:0} overlay_0-232:{mountpoint:/var/lib/containers/storage/overlay/0baacca15d918e819ab2c7d2ed79d66aa25da9913a27ccb934d5b88c3d9c934c/merged major:0 minor:232 fsType:overlay blockSize:0} overlay_0-234:{mountpoint:/var/lib/containers/storage/overlay/7694117ab77770e6c232409c1453b73b060aa214e51de64f5b598fc3deaa1937/merged major:0 minor:234 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/ced0ba12d8e6e925ceff9e8ebaef7f0489a82960389200e65fd88bba3a71d9f9/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/56a548444c8d0a05e553b5f08973489d2f86eb7c177656c8fea7db0392a1c963/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/03d36957e1c424542f925dee921e3ac16d0a42f1929138385e382742505dee47/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/3bcb598f43709fe11245047c9aee6434f97e138a7d953ebd471f64f6c7125a80/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/4b6853d01abe402d7cde44e0518ea4dc45ea098b73afe69cfd54af9221ef0be8/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/9a93dd197ca16048810cc5b9436649f141ed06c30678dc2368982f167ba2b1f2/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/9a449a31765ab5ae2bda3104142a97598b43a2d2641c580a172a18a9e89053d3/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/56f707ba5f49d116efea67c805da9059442afbc935f29a9dfdb622903963b64b/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/6de1b7039fbf7e279a3803ce4987da1980fe44203eb5c0c83c8c4e300100c8b0/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/57913b68c769c89bbddcdd529bbb0d69cf56e351af15129504b012518009ec75/merged major:0 minor:96 fsType:overlay blockSize:0}]
Nov 01 03:06:33.559305 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.555229    4400 manager.go:217] Machine: {Timestamp:2026-11-01 03:06:33.554790488 +0000 UTC m=+0.069436343 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101465 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e0161c0260d3b36d00164f51ea9c1 SystemUUID:ec2e0161-c026-0d3b-36d0-0164f51ea9c1 BootID:27b2e57b-db8c-4735-8450-2b0385021ed3 Filesystems:[{Device:/var/lib/kubelet/pods/c9e89456-9acd-408f-87b5-f5fd968c82c3/volumes/kubernetes.io~projected/kube-api-access-gljgq DeviceMajor:0 DeviceMinor:56 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/de5f469f-1029-4161-bf51-6ad18a3ac9e2/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/68932dfe-7e25-4f1b-b485-77c0519203db/volumes/kubernetes.io~projected/kube-api-access-qnrhp DeviceMajor:0 DeviceMinor:229 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/11fb87e4-ef01-4209-8207-80b66ab09f5e/volumes/kubernetes.io~projected/kube-api-access-h58g6 DeviceMajor:0 DeviceMinor:66 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b13508327d908895943975963c426c766b021361d87a9196f741e7bdd976509e/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c3970e088aa262baf754bfedaeebe86bb4f33f854f9f5d5fbec0b6fbc0aa084d/userdata/shm DeviceMajor:0 DeviceMinor:210 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d7cef651-3047-4e64-ae7f-b0fa71fbd0fe/volumes/kubernetes.io~projected/kube-api-access-t86zn DeviceMajor:0 DeviceMinor:65 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/994175bb-d192-47f7-b162-a840eb0f9be1/volumes/kubernetes.io~projected/kube-api-access-xtjjf DeviceMajor:0 DeviceMinor:58 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e109df44865a3f092a8d3d3680d8eee4ecf3f68aaf32da287d971df39293cbda/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/11fb87e4-ef01-4209-8207-80b66ab09f5e/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:59 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a4edacea31fe2dce66095cabb9f2f3de51aa76066e1021f1a6fb8287ba47933f/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-174 DeviceMajor:0 DeviceMinor:174 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4c1dc667-d774-4db6-b221-f4e7de423260/volumes/kubernetes.io~projected/kube-api-access-2459t DeviceMajor:0 DeviceMinor:197 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-216 DeviceMajor:0 DeviceMinor:216 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/e48a634f-ab31-435c-96c1-c3633c5bc061/volumes/kubernetes.io~projected/kube-api-access-b2krw DeviceMajor:0 DeviceMinor:67 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/65bca1ddcd7a850283c27829fa3fb5a6b5f48f97e5bad23b011d4a2f20816cb8/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-154 DeviceMajor:0 DeviceMinor:154 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-169 DeviceMajor:0 DeviceMinor:169 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4c1dc667-d774-4db6-b221-f4e7de423260/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0303d6c6-c4ec-4403-af74-f4bcb5cbf2a4/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:206 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/bd8260bb-3811-4cdc-91d0-c6ddc693891e/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/de5f469f-1029-4161-bf51-6ad18a3ac9e2/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:60 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8eda87d96f5445f76e7e3a31f88f1b1cfb59bd38489e93b16b05392b80c42192/userdata/shm DeviceMajor:0 DeviceMinor:87 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-234 DeviceMajor:0 DeviceMinor:234 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/11fb87e4-ef01-4209-8207-80b66ab09f5e/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:61 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/56aed8ff-9b41-400a-90fb-8c283ce0c820/volumes/kubernetes.io~projected/kube-api-access-r9wst DeviceMajor:0 DeviceMinor:201 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-212 DeviceMajor:0 DeviceMinor:212 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/02b4fc07486757aadcb2800218a1fc8a3076f38fa1f4185193c2aa7e5ef8e7f6/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e8f2a24b-e770-4615-a2c7-b2d065e14f77/volumes/kubernetes.io~projected/kube-api-access-85r94 DeviceMajor:0 DeviceMinor:62 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-105 DeviceMajor:0 DeviceMinor:105 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-214 DeviceMajor:0 DeviceMinor:214 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-218 DeviceMajor:0 DeviceMinor:218 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d84526df1da0ed7187d5efa80942b6e3936499085b86291746b62ea9e3886525/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/de5f469f-1029-4161-bf51-6ad18a3ac9e2/volumes/kubernetes.io~projected/kube-api-access-ckfr8 DeviceMajor:0 DeviceMinor:64 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/656a993b19b5fb0b91c6c5940fe8ed4ed810a56d28407cb6a0bf8d1ed48f10c4/userdata/shm DeviceMajor:0 DeviceMinor:208 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-232 DeviceMajor:0 DeviceMinor:232 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0ede655d-c20d-41cc-b5f0-2db4338c1ba7/volumes/kubernetes.io~projected/kube-api-access-slnpd DeviceMajor:0 DeviceMinor:57 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/bf9dac1d-d9fc-455a-be03-ea34f8838730/volumes/kubernetes.io~projected/kube-api-access-jhdnz DeviceMajor:0 DeviceMinor:63 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c2f90ee59606115f6de4c50b793a548e266a500e6c80b26671d87b883f9d7fec/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6e302b40-9c43-4822-8c8b-4d021a0590ab/volumes/kubernetes.io~projected/kube-api-access-nxqx8 DeviceMajor:0 DeviceMinor:207 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7f86d26dee5abd851a44f92a5c77c0c1738cfb836092f63bd0db9c8840f8def0/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ca80f4e18c06e6002a072be7097cb83d48d6fe56ab99647cb2d9d8894402e92e/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/410bd40c1eca1aaaeeea283ba9e2c2a185c749e790f96ef1fdb69feacf71c338/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/38e73bd0c99d3b29aac95c81a2e64aa64d2f964154b942bfecd271af059f2c56/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-138 DeviceMajor:0 DeviceMinor:138 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/73b27ccb-7d7a-40c1-8479-8379b4fb50ed/volumes/kubernetes.io~projected/kube-api-access-httfj DeviceMajor:0 DeviceMinor:224 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:410bd40c1eca1aa MacAddress:16:4d:12:be:d0:f9 Speed:10000 Mtu:8901} {Name:656a993b19b5fb0 MacAddress:c6:5b:d6:94:fd:d9 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:7e:7e:3b:4e:e5 Speed:0 Mtu:9001} {Name:br-int MacAddress:4a:aa:32:dd:f4:00 Speed:0 Mtu:8901} {Name:c3970e088aa262b MacAddress:7e:68:ef:ee:4a:7a Speed:10000 Mtu:8901} {Name:ca80f4e18c06e60 MacAddress:7e:df:4d:b8:70:0c Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:7e:7e:3b:4e:e5 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:2a:10:a7:6e:c9:e9 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:ee:d5:ea:90:14:ca Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 01 03:06:33.559305 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.555449    4400 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 01 03:06:33.559305 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.555489    4400 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 01 03:06:33.559305 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.555777    4400 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 01 03:06:33.559305 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.555802    4400 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-54.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 01 03:06:33.559305 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.555923    4400 topology_manager.go:143] "Creating topology manager with none policy"
Nov 01 03:06:33.559305 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.555930    4400 container_manager_linux.go:311] "Creating device plugin manager"
Nov 01 03:06:33.559305 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.555942    4400 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 01 03:06:33.559305 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.555962    4400 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 01 03:06:33.559305 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.556046    4400 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 01 03:06:33.559305 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.556197    4400 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 01 03:06:33.559305 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.556326    4400 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 01 03:06:33.559305 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.556362    4400 kubelet.go:493] "Attempting to sync node with API server"
Nov 01 03:06:33.559305 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.556380    4400 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 01 03:06:33.559305 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.556391    4400 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 01 03:06:33.559305 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.556399    4400 kubelet.go:405] "Adding apiserver pod source"
Nov 01 03:06:33.559305 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.556407    4400 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 01 03:06:33.559305 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.557158    4400 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 01 03:06:33.559305 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.557172    4400 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 01 03:06:33.559305 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.557621    4400 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 01 03:06:33.564810 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.564795    4400 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 01 03:06:33.564810 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.564808    4400 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 01 03:06:33.564934 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.564887    4400 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 01 03:06:33.565145 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.565078    4400 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 01 03:06:33.565145 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.565119    4400 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 01 03:06:33.565282 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.565217    4400 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 01 03:06:33.565282 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.565232    4400 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 01 03:06:33.565282 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.565241    4400 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 01 03:06:33.565282 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.565251    4400 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 01 03:06:33.565282 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.565259    4400 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 01 03:06:33.565282 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.565267    4400 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 01 03:06:33.565282 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.565276    4400 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 01 03:06:33.565513 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.565286    4400 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 01 03:06:33.565513 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.565297    4400 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 01 03:06:33.565513 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.565306    4400 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 01 03:06:33.565513 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.565318    4400 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 01 03:06:33.565513 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.565333    4400 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 01 03:06:33.565513 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.565366    4400 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 01 03:06:33.565513 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.565376    4400 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 01 03:06:33.568792 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.568778    4400 server.go:1263] "Started kubelet"
Nov 01 03:06:33.568890 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.568864    4400 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 01 03:06:33.568955 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.568874    4400 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 01 03:06:33.569005 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.568972    4400 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 01 03:06:33.569291 ip-10-0-133-54 systemd[1]: Started Kubernetes Kubelet.
Nov 01 03:06:33.573454 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.573412    4400 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 01 03:06:33.574900 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.574882    4400 server.go:317] "Adding debug handlers to kubelet server"
Nov 01 03:06:33.575926 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.575909    4400 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 01 03:06:33.576832 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.575926    4400 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 01 03:06:33.576832 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.576015    4400 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 03:00:54 +0000 UTC" deadline="2028-05-05 09:08:01.50248922 +0000 UTC"
Nov 01 03:06:33.576832 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.576047    4400 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13230h1m27.92644518s"
Nov 01 03:06:33.576832 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.576076    4400 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 01 03:06:33.576832 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.576084    4400 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 01 03:06:33.576832 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:06:33.576127    4400 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-54.us-west-1.compute.internal\" not found"
Nov 01 03:06:33.576832 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.576203    4400 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 01 03:06:33.577641 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.577575    4400 factory.go:55] Registering systemd factory
Nov 01 03:06:33.577641 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.577607    4400 factory.go:223] Registration of the systemd container factory successfully
Nov 01 03:06:33.577843 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.577825    4400 factory.go:153] Registering CRI-O factory
Nov 01 03:06:33.577905 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.577826    4400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="56aed8ff-9b41-400a-90fb-8c283ce0c820" volumeName="kubernetes.io/configmap/56aed8ff-9b41-400a-90fb-8c283ce0c820-config-volume" seLinuxMountContext=""
Nov 01 03:06:33.577905 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.577858    4400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="68932dfe-7e25-4f1b-b485-77c0519203db" volumeName="kubernetes.io/configmap/68932dfe-7e25-4f1b-b485-77c0519203db-service-ca-bundle" seLinuxMountContext=""
Nov 01 03:06:33.577905 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.577874    4400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd8260bb-3811-4cdc-91d0-c6ddc693891e" volumeName="kubernetes.io/configmap/bd8260bb-3811-4cdc-91d0-c6ddc693891e-konnectivity-ca" seLinuxMountContext=""
Nov 01 03:06:33.577905 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.577887    4400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e48a634f-ab31-435c-96c1-c3633c5bc061" volumeName="kubernetes.io/projected/e48a634f-ab31-435c-96c1-c3633c5bc061-kube-api-access-b2krw" seLinuxMountContext=""
Nov 01 03:06:33.577905 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.577898    4400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0ede655d-c20d-41cc-b5f0-2db4338c1ba7" volumeName="kubernetes.io/configmap/0ede655d-c20d-41cc-b5f0-2db4338c1ba7-cni-binary-copy" seLinuxMountContext=""
Nov 01 03:06:33.578196 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.577909    4400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11fb87e4-ef01-4209-8207-80b66ab09f5e" volumeName="kubernetes.io/projected/11fb87e4-ef01-4209-8207-80b66ab09f5e-kube-api-access-h58g6" seLinuxMountContext=""
Nov 01 03:06:33.578196 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.577920    4400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="68932dfe-7e25-4f1b-b485-77c0519203db" volumeName="kubernetes.io/configmap/68932dfe-7e25-4f1b-b485-77c0519203db-trusted-ca-bundle" seLinuxMountContext=""
Nov 01 03:06:33.578196 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.577930    4400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6e302b40-9c43-4822-8c8b-4d021a0590ab" volumeName="kubernetes.io/projected/6e302b40-9c43-4822-8c8b-4d021a0590ab-kube-api-access-nxqx8" seLinuxMountContext=""
Nov 01 03:06:33.578196 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.577944    4400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d7cef651-3047-4e64-ae7f-b0fa71fbd0fe" volumeName="kubernetes.io/configmap/d7cef651-3047-4e64-ae7f-b0fa71fbd0fe-serviceca" seLinuxMountContext=""
Nov 01 03:06:33.578196 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.577955    4400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e8f2a24b-e770-4615-a2c7-b2d065e14f77" volumeName="kubernetes.io/configmap/e8f2a24b-e770-4615-a2c7-b2d065e14f77-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 01 03:06:33.578196 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.577967    4400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e8f2a24b-e770-4615-a2c7-b2d065e14f77" volumeName="kubernetes.io/configmap/e8f2a24b-e770-4615-a2c7-b2d065e14f77-cni-binary-copy" seLinuxMountContext=""
Nov 01 03:06:33.578196 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.577985    4400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0303d6c6-c4ec-4403-af74-f4bcb5cbf2a4" volumeName="kubernetes.io/secret/0303d6c6-c4ec-4403-af74-f4bcb5cbf2a4-original-pull-secret" seLinuxMountContext=""
Nov 01 03:06:33.578196 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.577996    4400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="56aed8ff-9b41-400a-90fb-8c283ce0c820" volumeName="kubernetes.io/projected/56aed8ff-9b41-400a-90fb-8c283ce0c820-kube-api-access-r9wst" seLinuxMountContext=""
Nov 01 03:06:33.578196 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.578007    4400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73b27ccb-7d7a-40c1-8479-8379b4fb50ed" volumeName="kubernetes.io/empty-dir/73b27ccb-7d7a-40c1-8479-8379b4fb50ed-data-volume" seLinuxMountContext=""
Nov 01 03:06:33.578196 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.578021    4400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd8260bb-3811-4cdc-91d0-c6ddc693891e" volumeName="kubernetes.io/secret/bd8260bb-3811-4cdc-91d0-c6ddc693891e-agent-certs" seLinuxMountContext=""
Nov 01 03:06:33.578196 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.578033    4400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf9dac1d-d9fc-455a-be03-ea34f8838730" volumeName="kubernetes.io/projected/bf9dac1d-d9fc-455a-be03-ea34f8838730-kube-api-access-jhdnz" seLinuxMountContext=""
Nov 01 03:06:33.578196 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.578044    4400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e8f2a24b-e770-4615-a2c7-b2d065e14f77" volumeName="kubernetes.io/configmap/e8f2a24b-e770-4615-a2c7-b2d065e14f77-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 01 03:06:33.578196 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.578064    4400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11fb87e4-ef01-4209-8207-80b66ab09f5e" volumeName="kubernetes.io/empty-dir/11fb87e4-ef01-4209-8207-80b66ab09f5e-tmp" seLinuxMountContext=""
Nov 01 03:06:33.578196 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.578076    4400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4c1dc667-d774-4db6-b221-f4e7de423260" volumeName="kubernetes.io/secret/4c1dc667-d774-4db6-b221-f4e7de423260-cert" seLinuxMountContext=""
Nov 01 03:06:33.578196 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.578089    4400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="68932dfe-7e25-4f1b-b485-77c0519203db" volumeName="kubernetes.io/empty-dir/68932dfe-7e25-4f1b-b485-77c0519203db-archives-path" seLinuxMountContext=""
Nov 01 03:06:33.578196 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.578099    4400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d7cef651-3047-4e64-ae7f-b0fa71fbd0fe" volumeName="kubernetes.io/projected/d7cef651-3047-4e64-ae7f-b0fa71fbd0fe-kube-api-access-t86zn" seLinuxMountContext=""
Nov 01 03:06:33.578196 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.578106    4400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de5f469f-1029-4161-bf51-6ad18a3ac9e2" volumeName="kubernetes.io/secret/de5f469f-1029-4161-bf51-6ad18a3ac9e2-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 01 03:06:33.578196 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.578114    4400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0ede655d-c20d-41cc-b5f0-2db4338c1ba7" volumeName="kubernetes.io/configmap/0ede655d-c20d-41cc-b5f0-2db4338c1ba7-multus-daemon-config" seLinuxMountContext=""
Nov 01 03:06:33.578196 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.578123    4400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="68932dfe-7e25-4f1b-b485-77c0519203db" volumeName="kubernetes.io/projected/68932dfe-7e25-4f1b-b485-77c0519203db-kube-api-access-qnrhp" seLinuxMountContext=""
Nov 01 03:06:33.578196 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.578130    4400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="994175bb-d192-47f7-b162-a840eb0f9be1" volumeName="kubernetes.io/empty-dir/994175bb-d192-47f7-b162-a840eb0f9be1-tmp-dir" seLinuxMountContext=""
Nov 01 03:06:33.578196 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.577845    4400 factory.go:223] Registration of the crio container factory successfully
Nov 01 03:06:33.578196 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.578137    4400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c9e89456-9acd-408f-87b5-f5fd968c82c3" volumeName="kubernetes.io/configmap/c9e89456-9acd-408f-87b5-f5fd968c82c3-iptables-alerter-script" seLinuxMountContext=""
Nov 01 03:06:33.579169 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.578229    4400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de5f469f-1029-4161-bf51-6ad18a3ac9e2" volumeName="kubernetes.io/configmap/de5f469f-1029-4161-bf51-6ad18a3ac9e2-ovnkube-config" seLinuxMountContext=""
Nov 01 03:06:33.579169 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.578243    4400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de5f469f-1029-4161-bf51-6ad18a3ac9e2" volumeName="kubernetes.io/projected/de5f469f-1029-4161-bf51-6ad18a3ac9e2-kube-api-access-ckfr8" seLinuxMountContext=""
Nov 01 03:06:33.579169 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.578262    4400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4c1dc667-d774-4db6-b221-f4e7de423260" volumeName="kubernetes.io/projected/4c1dc667-d774-4db6-b221-f4e7de423260-kube-api-access-2459t" seLinuxMountContext=""
Nov 01 03:06:33.579169 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.578276    4400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de5f469f-1029-4161-bf51-6ad18a3ac9e2" volumeName="kubernetes.io/configmap/de5f469f-1029-4161-bf51-6ad18a3ac9e2-ovnkube-script-lib" seLinuxMountContext=""
Nov 01 03:06:33.579169 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.578208    4400 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 01 03:06:33.579169 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.578316    4400 factory.go:103] Registering Raw factory
Nov 01 03:06:33.579169 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.578333    4400 manager.go:1199] Started watching for new ooms in manager
Nov 01 03:06:33.579169 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.578289    4400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0ede655d-c20d-41cc-b5f0-2db4338c1ba7" volumeName="kubernetes.io/projected/0ede655d-c20d-41cc-b5f0-2db4338c1ba7-kube-api-access-slnpd" seLinuxMountContext=""
Nov 01 03:06:33.579169 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.578364    4400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11fb87e4-ef01-4209-8207-80b66ab09f5e" volumeName="kubernetes.io/empty-dir/11fb87e4-ef01-4209-8207-80b66ab09f5e-etc-tuned" seLinuxMountContext=""
Nov 01 03:06:33.579169 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.578386    4400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="56aed8ff-9b41-400a-90fb-8c283ce0c820" volumeName="kubernetes.io/empty-dir/56aed8ff-9b41-400a-90fb-8c283ce0c820-tmp-dir" seLinuxMountContext=""
Nov 01 03:06:33.579169 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.578397    4400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73b27ccb-7d7a-40c1-8479-8379b4fb50ed" volumeName="kubernetes.io/projected/73b27ccb-7d7a-40c1-8479-8379b4fb50ed-kube-api-access-httfj" seLinuxMountContext=""
Nov 01 03:06:33.579169 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.578408    4400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="994175bb-d192-47f7-b162-a840eb0f9be1" volumeName="kubernetes.io/projected/994175bb-d192-47f7-b162-a840eb0f9be1-kube-api-access-xtjjf" seLinuxMountContext=""
Nov 01 03:06:33.579169 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.578422    4400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c9e89456-9acd-408f-87b5-f5fd968c82c3" volumeName="kubernetes.io/projected/c9e89456-9acd-408f-87b5-f5fd968c82c3-kube-api-access-gljgq" seLinuxMountContext=""
Nov 01 03:06:33.579169 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.578433    4400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de5f469f-1029-4161-bf51-6ad18a3ac9e2" volumeName="kubernetes.io/configmap/de5f469f-1029-4161-bf51-6ad18a3ac9e2-env-overrides" seLinuxMountContext=""
Nov 01 03:06:33.579169 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.578441    4400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e8f2a24b-e770-4615-a2c7-b2d065e14f77" volumeName="kubernetes.io/projected/e8f2a24b-e770-4615-a2c7-b2d065e14f77-kube-api-access-85r94" seLinuxMountContext=""
Nov 01 03:06:33.579169 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.578449    4400 reconstruct.go:97] "Volume reconstruction finished"
Nov 01 03:06:33.579169 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.578453    4400 reconciler.go:29] "Reconciler: start to sync state"
Nov 01 03:06:33.579169 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.578614    4400 manager.go:319] Starting recovery of all containers
Nov 01 03:06:33.605288 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.605258    4400 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 01 03:06:33.606591 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.606556    4400 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 01 03:06:33.606591 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.606576    4400 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 01 03:06:33.606591 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.606595    4400 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 01 03:06:33.606769 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:06:33.606633    4400 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 01 03:06:33.630228 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.630196    4400 generic.go:356] "Generic (PLEG): container finished" podID="e8f2a24b-e770-4615-a2c7-b2d065e14f77" containerID="0207c500b12b0ef1bf2f9fb73381ca1dfecfd909373ce84f8ca1488bebd78cca" exitCode=0
Nov 01 03:06:33.630228 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.630230    4400 generic.go:356] "Generic (PLEG): container finished" podID="e8f2a24b-e770-4615-a2c7-b2d065e14f77" containerID="acf22f9816062b882b8699ed1e3095a7da75acd5fae7f1787afb7da634d5b67d" exitCode=0
Nov 01 03:06:33.630391 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.630240    4400 generic.go:356] "Generic (PLEG): container finished" podID="e8f2a24b-e770-4615-a2c7-b2d065e14f77" containerID="7a7a6f067ff3ff337f9d2ed6c781228d058576d5f90fe2568fbf31960eda720e" exitCode=0
Nov 01 03:06:33.630391 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.630248    4400 generic.go:356] "Generic (PLEG): container finished" podID="e8f2a24b-e770-4615-a2c7-b2d065e14f77" containerID="2e7969b36ffd11c38396cb69f58aa57521a56690c344a881ec17f42a14e3cd56" exitCode=0
Nov 01 03:06:33.630391 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.630257    4400 generic.go:356] "Generic (PLEG): container finished" podID="e8f2a24b-e770-4615-a2c7-b2d065e14f77" containerID="06cabc60d357d7f0b9685a86c5469302f4fcb57b97986a1fece1b2e010839133" exitCode=0
Nov 01 03:06:33.630391 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.630264    4400 generic.go:356] "Generic (PLEG): container finished" podID="e8f2a24b-e770-4615-a2c7-b2d065e14f77" containerID="dbaa0818d7e1dce276a6f3de3ca017c5753fbea77adadded69796601fe2ac532" exitCode=0
Nov 01 03:06:33.647432 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.647414    4400 manager.go:324] Recovery completed
Nov 01 03:06:33.647700 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.647647    4400 generic.go:356] "Generic (PLEG): container finished" podID="f068e0e9089d99d5fd3ec23628aa7d71" containerID="d5b61e0bd6f02fdbc487a22e3a2d4c28bdb394f605a9d71854d455f712280907" exitCode=0
Nov 01 03:06:33.659623 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.659607    4400 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 03:06:33.660606 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.660591    4400 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-54.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 03:06:33.660678 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.660619    4400 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-54.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 03:06:33.660678 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.660629    4400 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-54.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 03:06:33.661875 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.661862    4400 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 03:06:33.661950 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.661864    4400 cpu_manager.go:226] "Starting" policy="none"
Nov 01 03:06:33.661950 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.661921    4400 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 01 03:06:33.661950 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.661940    4400 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 01 03:06:33.662102 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.662087    4400 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 01 03:06:33.662141 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.662101    4400 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 01 03:06:33.662141 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.662116    4400 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 01 03:06:33.662141 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.662122    4400 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 01 03:06:33.662141 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.662130    4400 policy_none.go:50] "Start"
Nov 01 03:06:33.662244 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.662142    4400 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 01 03:06:33.662244 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.662153    4400 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 01 03:06:33.662293 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.662277    4400 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 01 03:06:33.662293 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.662284    4400 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 01 03:06:33.662293 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.662289    4400 policy_none.go:44] "Start"
Nov 01 03:06:33.662594 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.662580    4400 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-54.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 03:06:33.662636 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.662606    4400 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-54.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 03:06:33.662636 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.662616    4400 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-54.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 03:06:33.666329 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.666306    4400 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 01 03:06:33.668861 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.668848    4400 manager.go:341] "Starting Device Plugin manager"
Nov 01 03:06:33.668934 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:06:33.668875    4400 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 01 03:06:33.668934 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.668886    4400 server.go:87] "Starting device plugin registration server"
Nov 01 03:06:33.669165 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.669151    4400 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 01 03:06:33.669225 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.669165    4400 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 01 03:06:33.669291 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.669271    4400 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 01 03:06:33.669411 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.669376    4400 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 01 03:06:33.669472 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.669420    4400 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 01 03:06:33.669472 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.669427    4400 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 01 03:06:33.669572 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.669486    4400 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-01T03:06:33.6694078Z","UUID":"c2cfecd9-1d50-4dc9-b4d1-b7ee78d28ea6","Handler":null,"Name":"","Endpoint":""}
Nov 01 03:06:33.670612 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:06:33.670593    4400 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 01 03:06:33.670778 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:06:33.670763    4400 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-54.us-west-1.compute.internal\" not found"
Nov 01 03:06:33.671020 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.671004    4400 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 01 03:06:33.671020 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.671024    4400 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 01 03:06:33.707208 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.707175    4400 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-54.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-54.us-west-1.compute.internal"]
Nov 01 03:06:33.707303 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.707282    4400 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 03:06:33.708089 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.708072    4400 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-54.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 03:06:33.708160 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.708100    4400 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-54.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 03:06:33.708160 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.708110    4400 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-54.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 03:06:33.708323 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.708313    4400 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 03:06:33.708521 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.708507    4400 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 03:06:33.709027 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.709011    4400 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-54.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 03:06:33.709098 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.709034    4400 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-54.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 03:06:33.709098 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.709044    4400 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-54.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 03:06:33.709098 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.709057    4400 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-54.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 03:06:33.709098 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.709073    4400 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-54.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 03:06:33.709205 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.709060    4400 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-54.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 03:06:33.709259 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:06:33.709252    4400 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-54.us-west-1.compute.internal\" not found" node="ip-10-0-133-54.us-west-1.compute.internal"
Nov 01 03:06:33.709337 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.709305    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-54.us-west-1.compute.internal" event={"ID":"8cf98fd570993ef6ee2b41a9af4899c8","Type":"ContainerStarted","Data":"4a0cf118fcd331a054fa79cc9f2b03a569b1ee27a9f76a3c15da4c3407d6a68b"}
Nov 01 03:06:33.709366 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.709347    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-54.us-west-1.compute.internal" event={"ID":"8cf98fd570993ef6ee2b41a9af4899c8","Type":"ContainerStarted","Data":"d84526df1da0ed7187d5efa80942b6e3936499085b86291746b62ea9e3886525"}
Nov 01 03:06:33.709397 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.709364    4400 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 03:06:33.709447 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.709365    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-54.us-west-1.compute.internal" event={"ID":"f068e0e9089d99d5fd3ec23628aa7d71","Type":"ContainerStarted","Data":"b6f2d1fcb60f2c8916321d6db336f078cb62e96c75c5d09329168b0be1027ffc"}
Nov 01 03:06:33.709497 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.709478    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-54.us-west-1.compute.internal" event={"ID":"f068e0e9089d99d5fd3ec23628aa7d71","Type":"ContainerDied","Data":"d5b61e0bd6f02fdbc487a22e3a2d4c28bdb394f605a9d71854d455f712280907"}
Nov 01 03:06:33.709546 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.709508    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-54.us-west-1.compute.internal" event={"ID":"f068e0e9089d99d5fd3ec23628aa7d71","Type":"ContainerStarted","Data":"7f86d26dee5abd851a44f92a5c77c0c1738cfb836092f63bd0db9c8840f8def0"}
Nov 01 03:06:33.710106 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.710090    4400 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-54.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 03:06:33.710179 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.710114    4400 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-54.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 03:06:33.710179 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.710123    4400 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-54.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 03:06:33.710257 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:06:33.710237    4400 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-54.us-west-1.compute.internal\" not found" node="ip-10-0-133-54.us-west-1.compute.internal"
Nov 01 03:06:33.769473 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.769452    4400 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 03:06:33.770212 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.770192    4400 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-54.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 03:06:33.770288 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.770217    4400 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-54.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 03:06:33.770288 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.770227    4400 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-54.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 03:06:33.770288 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.770242    4400 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-54.us-west-1.compute.internal"
Nov 01 03:06:33.779293 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.779241    4400 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 03:06:33.780956 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.779590    4400 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 01 03:06:33.826356 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.826331    4400 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 03:06:33.826441 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.826402    4400 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 03:06:33.830563 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.830547    4400 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 03:06:33.876881 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.876861    4400 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-54.us-west-1.compute.internal"
Nov 01 03:06:33.880347 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.880330    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f068e0e9089d99d5fd3ec23628aa7d71-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-54.us-west-1.compute.internal\" (UID: \"f068e0e9089d99d5fd3ec23628aa7d71\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-54.us-west-1.compute.internal"
Nov 01 03:06:33.880410 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.880358    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8cf98fd570993ef6ee2b41a9af4899c8-config\") pod \"kube-apiserver-proxy-ip-10-0-133-54.us-west-1.compute.internal\" (UID: \"8cf98fd570993ef6ee2b41a9af4899c8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-54.us-west-1.compute.internal"
Nov 01 03:06:33.880410 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.880380    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f068e0e9089d99d5fd3ec23628aa7d71-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-54.us-west-1.compute.internal\" (UID: \"f068e0e9089d99d5fd3ec23628aa7d71\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-54.us-west-1.compute.internal"
Nov 01 03:06:33.883210 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.883196    4400 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-133-54.us-west-1.compute.internal"
Nov 01 03:06:33.883281 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.883240    4400 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-133-54.us-west-1.compute.internal"
Nov 01 03:06:33.883617 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.883603    4400 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-54.us-west-1.compute.internal" event="NodeNotReady"
Nov 01 03:06:33.883703 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.883633    4400 setters.go:546] "Node became not ready" node="ip-10-0-133-54.us-west-1.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-11-01T03:06:33Z","lastTransitionTime":"2026-11-01T03:06:33Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"}
Nov 01 03:06:33.930006 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.929983    4400 nodeinfomanager.go:471] existing CSINode "ip-10-0-133-54.us-west-1.compute.internal" is owned by different node (oldNodeID="a4459411-2eb5-4c2a-8675-fff1134e4884", newNodeID=""), cleaning up...
Nov 01 03:06:33.942110 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.942088    4400 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 03:06:33.942201 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:06:33.942128    4400 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-133-54.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-54.us-west-1.compute.internal"
Nov 01 03:06:33.942201 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.942142    4400 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-54.us-west-1.compute.internal"
Nov 01 03:06:33.980506 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.980484    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f068e0e9089d99d5fd3ec23628aa7d71-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-54.us-west-1.compute.internal\" (UID: \"f068e0e9089d99d5fd3ec23628aa7d71\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-54.us-west-1.compute.internal"
Nov 01 03:06:33.980605 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.980510    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f068e0e9089d99d5fd3ec23628aa7d71-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-54.us-west-1.compute.internal\" (UID: \"f068e0e9089d99d5fd3ec23628aa7d71\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-54.us-west-1.compute.internal"
Nov 01 03:06:33.980605 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.980530    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8cf98fd570993ef6ee2b41a9af4899c8-config\") pod \"kube-apiserver-proxy-ip-10-0-133-54.us-west-1.compute.internal\" (UID: \"8cf98fd570993ef6ee2b41a9af4899c8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-54.us-west-1.compute.internal"
Nov 01 03:06:33.980605 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.980579    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8cf98fd570993ef6ee2b41a9af4899c8-config\") pod \"kube-apiserver-proxy-ip-10-0-133-54.us-west-1.compute.internal\" (UID: \"8cf98fd570993ef6ee2b41a9af4899c8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-54.us-west-1.compute.internal"
Nov 01 03:06:33.980605 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.980592    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f068e0e9089d99d5fd3ec23628aa7d71-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-54.us-west-1.compute.internal\" (UID: \"f068e0e9089d99d5fd3ec23628aa7d71\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-54.us-west-1.compute.internal"
Nov 01 03:06:33.980746 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:33.980598    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f068e0e9089d99d5fd3ec23628aa7d71-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-54.us-west-1.compute.internal\" (UID: \"f068e0e9089d99d5fd3ec23628aa7d71\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-54.us-west-1.compute.internal"
Nov 01 03:06:34.037755 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.037708    4400 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 03:06:34.037755 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:06:34.037751    4400 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-133-54.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-54.us-west-1.compute.internal"
Nov 01 03:06:34.556840 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.556810    4400 apiserver.go:52] "Watching apiserver"
Nov 01 03:06:34.651595 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.651570    4400 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-54.us-west-1.compute.internal"
Nov 01 03:06:34.651739 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.651660    4400 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-54.us-west-1.compute.internal"
Nov 01 03:06:34.662785 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.662769    4400 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 01 03:06:34.663240 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.663222    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-wfg7k","openshift-dns/dns-default-nzbnl","openshift-network-diagnostics/network-check-target-mx48d","openshift-multus/multus-additional-cni-plugins-qnrf8","openshift-ingress-canary/ingress-canary-h6696","openshift-dns/node-resolver-pzt6v","kube-system/konnectivity-agent-fh7sf","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vq4dj","openshift-kube-storage-version-migrator/migrator-66df5d496b-g8ql6","openshift-cluster-node-tuning-operator/tuned-lkbk6","openshift-multus/network-metrics-daemon-dnqjp","openshift-ovn-kubernetes/ovnkube-node-bcqk9","openshift-multus/multus-qzl9n","kube-system/global-pull-secret-syncer-rrdzd","openshift-image-registry/node-ca-wq27m","openshift-insights/periodic-gathering-5rxlb-bg459","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-54.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-54.us-west-1.compute.internal","openshift-insights/insights-runtime-extractor-6vv92"]
Nov 01 03:06:34.666197 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.666178    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dnqjp"
Nov 01 03:06:34.670139 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.670114    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nzbnl"
Nov 01 03:06:34.670480 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.670462    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6vv92"
Nov 01 03:06:34.685948 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.685926    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/de5f469f-1029-4161-bf51-6ad18a3ac9e2-log-socket\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:06:34.686050 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.685952    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/56aed8ff-9b41-400a-90fb-8c283ce0c820-metrics-tls\") pod \"dns-default-nzbnl\" (UID: \"56aed8ff-9b41-400a-90fb-8c283ce0c820\") " pod="openshift-dns/dns-default-nzbnl"
Nov 01 03:06:34.686050 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.685967    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/11fb87e4-ef01-4209-8207-80b66ab09f5e-host\") pod \"tuned-lkbk6\" (UID: \"11fb87e4-ef01-4209-8207-80b66ab09f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-lkbk6"
Nov 01 03:06:34.686050 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.685986    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2459t\" (UniqueName: \"kubernetes.io/projected/4c1dc667-d774-4db6-b221-f4e7de423260-kube-api-access-2459t\") pod \"ingress-canary-h6696\" (UID: \"4c1dc667-d774-4db6-b221-f4e7de423260\") " pod="openshift-ingress-canary/ingress-canary-h6696"
Nov 01 03:06:34.686050 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.686003    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e8f2a24b-e770-4615-a2c7-b2d065e14f77-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qnrf8\" (UID: \"e8f2a24b-e770-4615-a2c7-b2d065e14f77\") " pod="openshift-multus/multus-additional-cni-plugins-qnrf8"
Nov 01 03:06:34.686213 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.686058    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0ede655d-c20d-41cc-b5f0-2db4338c1ba7-host-run-k8s-cni-cncf-io\") pod \"multus-qzl9n\" (UID: \"0ede655d-c20d-41cc-b5f0-2db4338c1ba7\") " pod="openshift-multus/multus-qzl9n"
Nov 01 03:06:34.686213 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.686092    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0ede655d-c20d-41cc-b5f0-2db4338c1ba7-etc-kubernetes\") pod \"multus-qzl9n\" (UID: \"0ede655d-c20d-41cc-b5f0-2db4338c1ba7\") " pod="openshift-multus/multus-qzl9n"
Nov 01 03:06:34.686213 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.686121    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhdnz\" (UniqueName: \"kubernetes.io/projected/bf9dac1d-d9fc-455a-be03-ea34f8838730-kube-api-access-jhdnz\") pod \"aws-ebs-csi-driver-node-vq4dj\" (UID: \"bf9dac1d-d9fc-455a-be03-ea34f8838730\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vq4dj"
Nov 01 03:06:34.686213 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.686145    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85r94\" (UniqueName: \"kubernetes.io/projected/e8f2a24b-e770-4615-a2c7-b2d065e14f77-kube-api-access-85r94\") pod \"multus-additional-cni-plugins-qnrf8\" (UID: \"e8f2a24b-e770-4615-a2c7-b2d065e14f77\") " pod="openshift-multus/multus-additional-cni-plugins-qnrf8"
Nov 01 03:06:34.686213 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.686169    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0ede655d-c20d-41cc-b5f0-2db4338c1ba7-host-var-lib-cni-bin\") pod \"multus-qzl9n\" (UID: \"0ede655d-c20d-41cc-b5f0-2db4338c1ba7\") " pod="openshift-multus/multus-qzl9n"
Nov 01 03:06:34.686213 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.686184    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0ede655d-c20d-41cc-b5f0-2db4338c1ba7-host-var-lib-cni-multus\") pod \"multus-qzl9n\" (UID: \"0ede655d-c20d-41cc-b5f0-2db4338c1ba7\") " pod="openshift-multus/multus-qzl9n"
Nov 01 03:06:34.686213 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.686199    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0ede655d-c20d-41cc-b5f0-2db4338c1ba7-multus-socket-dir-parent\") pod \"multus-qzl9n\" (UID: \"0ede655d-c20d-41cc-b5f0-2db4338c1ba7\") " pod="openshift-multus/multus-qzl9n"
Nov 01 03:06:34.686213 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.686213    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0ede655d-c20d-41cc-b5f0-2db4338c1ba7-multus-conf-dir\") pod \"multus-qzl9n\" (UID: \"0ede655d-c20d-41cc-b5f0-2db4338c1ba7\") " pod="openshift-multus/multus-qzl9n"
Nov 01 03:06:34.686542 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.686236    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c9e89456-9acd-408f-87b5-f5fd968c82c3-host-slash\") pod \"iptables-alerter-wfg7k\" (UID: \"c9e89456-9acd-408f-87b5-f5fd968c82c3\") " pod="openshift-network-operator/iptables-alerter-wfg7k"
Nov 01 03:06:34.686542 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.686279    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bf9dac1d-d9fc-455a-be03-ea34f8838730-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vq4dj\" (UID: \"bf9dac1d-d9fc-455a-be03-ea34f8838730\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vq4dj"
Nov 01 03:06:34.686542 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.686320    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slnpd\" (UniqueName: \"kubernetes.io/projected/0ede655d-c20d-41cc-b5f0-2db4338c1ba7-kube-api-access-slnpd\") pod \"multus-qzl9n\" (UID: \"0ede655d-c20d-41cc-b5f0-2db4338c1ba7\") " pod="openshift-multus/multus-qzl9n"
Nov 01 03:06:34.686542 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.686341    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de5f469f-1029-4161-bf51-6ad18a3ac9e2-host-run-ovn-kubernetes\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:06:34.686542 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.686357    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckfr8\" (UniqueName: \"kubernetes.io/projected/de5f469f-1029-4161-bf51-6ad18a3ac9e2-kube-api-access-ckfr8\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:06:34.686542 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.686376    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/11fb87e4-ef01-4209-8207-80b66ab09f5e-lib-modules\") pod \"tuned-lkbk6\" (UID: \"11fb87e4-ef01-4209-8207-80b66ab09f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-lkbk6"
Nov 01 03:06:34.686542 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.686412    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0303d6c6-c4ec-4403-af74-f4bcb5cbf2a4-kubelet-config\") pod \"global-pull-secret-syncer-rrdzd\" (UID: \"0303d6c6-c4ec-4403-af74-f4bcb5cbf2a4\") " pod="kube-system/global-pull-secret-syncer-rrdzd"
Nov 01 03:06:34.686542 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.686505    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0303d6c6-c4ec-4403-af74-f4bcb5cbf2a4-original-pull-secret\") pod \"global-pull-secret-syncer-rrdzd\" (UID: \"0303d6c6-c4ec-4403-af74-f4bcb5cbf2a4\") " pod="kube-system/global-pull-secret-syncer-rrdzd"
Nov 01 03:06:34.686542 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.686532    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t86zn\" (UniqueName: \"kubernetes.io/projected/d7cef651-3047-4e64-ae7f-b0fa71fbd0fe-kube-api-access-t86zn\") pod \"node-ca-wq27m\" (UID: \"d7cef651-3047-4e64-ae7f-b0fa71fbd0fe\") " pod="openshift-image-registry/node-ca-wq27m"
Nov 01 03:06:34.686909 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.686632    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e8f2a24b-e770-4615-a2c7-b2d065e14f77-system-cni-dir\") pod \"multus-additional-cni-plugins-qnrf8\" (UID: \"e8f2a24b-e770-4615-a2c7-b2d065e14f77\") " pod="openshift-multus/multus-additional-cni-plugins-qnrf8"
Nov 01 03:06:34.686909 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.686675    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/de5f469f-1029-4161-bf51-6ad18a3ac9e2-ovn-node-metrics-cert\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:06:34.686909 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.686773    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9wst\" (UniqueName: \"kubernetes.io/projected/56aed8ff-9b41-400a-90fb-8c283ce0c820-kube-api-access-r9wst\") pod \"dns-default-nzbnl\" (UID: \"56aed8ff-9b41-400a-90fb-8c283ce0c820\") " pod="openshift-dns/dns-default-nzbnl"
Nov 01 03:06:34.686909 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.686810    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/73b27ccb-7d7a-40c1-8479-8379b4fb50ed-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6vv92\" (UID: \"73b27ccb-7d7a-40c1-8479-8379b4fb50ed\") " pod="openshift-insights/insights-runtime-extractor-6vv92"
Nov 01 03:06:34.686909 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.686840    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/bd8260bb-3811-4cdc-91d0-c6ddc693891e-konnectivity-ca\") pod \"konnectivity-agent-fh7sf\" (UID: \"bd8260bb-3811-4cdc-91d0-c6ddc693891e\") " pod="kube-system/konnectivity-agent-fh7sf"
Nov 01 03:06:34.686909 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.686869    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68932dfe-7e25-4f1b-b485-77c0519203db-service-ca-bundle\") pod \"periodic-gathering-5rxlb-bg459\" (UID: \"68932dfe-7e25-4f1b-b485-77c0519203db\") " pod="openshift-insights/periodic-gathering-5rxlb-bg459"
Nov 01 03:06:34.686909 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.686898    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/de5f469f-1029-4161-bf51-6ad18a3ac9e2-host-slash\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:06:34.687135 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.686949    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/de5f469f-1029-4161-bf51-6ad18a3ac9e2-run-ovn\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:06:34.687135 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.686967    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/de5f469f-1029-4161-bf51-6ad18a3ac9e2-ovnkube-script-lib\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:06:34.687135 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.686982    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/56aed8ff-9b41-400a-90fb-8c283ce0c820-config-volume\") pod \"dns-default-nzbnl\" (UID: \"56aed8ff-9b41-400a-90fb-8c283ce0c820\") " pod="openshift-dns/dns-default-nzbnl"
Nov 01 03:06:34.687135 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.686997    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0ede655d-c20d-41cc-b5f0-2db4338c1ba7-system-cni-dir\") pod \"multus-qzl9n\" (UID: \"0ede655d-c20d-41cc-b5f0-2db4338c1ba7\") " pod="openshift-multus/multus-qzl9n"
Nov 01 03:06:34.687135 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.687011    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/de5f469f-1029-4161-bf51-6ad18a3ac9e2-run-systemd\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:06:34.687135 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.687024    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de5f469f-1029-4161-bf51-6ad18a3ac9e2-etc-openvswitch\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:06:34.687135 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.687039    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/11fb87e4-ef01-4209-8207-80b66ab09f5e-etc-kubernetes\") pod \"tuned-lkbk6\" (UID: \"11fb87e4-ef01-4209-8207-80b66ab09f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-lkbk6"
Nov 01 03:06:34.687135 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.687075    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gljgq\" (UniqueName: \"kubernetes.io/projected/c9e89456-9acd-408f-87b5-f5fd968c82c3-kube-api-access-gljgq\") pod \"iptables-alerter-wfg7k\" (UID: \"c9e89456-9acd-408f-87b5-f5fd968c82c3\") " pod="openshift-network-operator/iptables-alerter-wfg7k"
Nov 01 03:06:34.687135 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.687135    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e8f2a24b-e770-4615-a2c7-b2d065e14f77-cni-binary-copy\") pod \"multus-additional-cni-plugins-qnrf8\" (UID: \"e8f2a24b-e770-4615-a2c7-b2d065e14f77\") " pod="openshift-multus/multus-additional-cni-plugins-qnrf8"
Nov 01 03:06:34.687367 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.687212    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2krw\" (UniqueName: \"kubernetes.io/projected/e48a634f-ab31-435c-96c1-c3633c5bc061-kube-api-access-b2krw\") pod \"network-metrics-daemon-dnqjp\" (UID: \"e48a634f-ab31-435c-96c1-c3633c5bc061\") " pod="openshift-multus/network-metrics-daemon-dnqjp"
Nov 01 03:06:34.687367 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.687267    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/994175bb-d192-47f7-b162-a840eb0f9be1-hosts-file\") pod \"node-resolver-pzt6v\" (UID: \"994175bb-d192-47f7-b162-a840eb0f9be1\") " pod="openshift-dns/node-resolver-pzt6v"
Nov 01 03:06:34.687424 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.687367    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/de5f469f-1029-4161-bf51-6ad18a3ac9e2-systemd-units\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:06:34.687424 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.687383    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de5f469f-1029-4161-bf51-6ad18a3ac9e2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:06:34.687424 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.687401    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e8f2a24b-e770-4615-a2c7-b2d065e14f77-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qnrf8\" (UID: \"e8f2a24b-e770-4615-a2c7-b2d065e14f77\") " pod="openshift-multus/multus-additional-cni-plugins-qnrf8"
Nov 01 03:06:34.687514 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.687421    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/73b27ccb-7d7a-40c1-8479-8379b4fb50ed-crio-socket\") pod \"insights-runtime-extractor-6vv92\" (UID: \"73b27ccb-7d7a-40c1-8479-8379b4fb50ed\") " pod="openshift-insights/insights-runtime-extractor-6vv92"
Nov 01 03:06:34.687514 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.687455    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-httfj\" (UniqueName: \"kubernetes.io/projected/73b27ccb-7d7a-40c1-8479-8379b4fb50ed-kube-api-access-httfj\") pod \"insights-runtime-extractor-6vv92\" (UID: \"73b27ccb-7d7a-40c1-8479-8379b4fb50ed\") " pod="openshift-insights/insights-runtime-extractor-6vv92"
Nov 01 03:06:34.687711 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.687525    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68932dfe-7e25-4f1b-b485-77c0519203db-trusted-ca-bundle\") pod \"periodic-gathering-5rxlb-bg459\" (UID: \"68932dfe-7e25-4f1b-b485-77c0519203db\") " pod="openshift-insights/periodic-gathering-5rxlb-bg459"
Nov 01 03:06:34.687711 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.687552    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0ede655d-c20d-41cc-b5f0-2db4338c1ba7-host-var-lib-kubelet\") pod \"multus-qzl9n\" (UID: \"0ede655d-c20d-41cc-b5f0-2db4338c1ba7\") " pod="openshift-multus/multus-qzl9n"
Nov 01 03:06:34.687711 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.687579    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0ede655d-c20d-41cc-b5f0-2db4338c1ba7-host-run-netns\") pod \"multus-qzl9n\" (UID: \"0ede655d-c20d-41cc-b5f0-2db4338c1ba7\") " pod="openshift-multus/multus-qzl9n"
Nov 01 03:06:34.687711 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.687604    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/11fb87e4-ef01-4209-8207-80b66ab09f5e-etc-sysconfig\") pod \"tuned-lkbk6\" (UID: \"11fb87e4-ef01-4209-8207-80b66ab09f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-lkbk6"
Nov 01 03:06:34.687711 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.687634    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bf9dac1d-d9fc-455a-be03-ea34f8838730-registration-dir\") pod \"aws-ebs-csi-driver-node-vq4dj\" (UID: \"bf9dac1d-d9fc-455a-be03-ea34f8838730\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vq4dj"
Nov 01 03:06:34.687711 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.687666    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/68932dfe-7e25-4f1b-b485-77c0519203db-archives-path\") pod \"periodic-gathering-5rxlb-bg459\" (UID: \"68932dfe-7e25-4f1b-b485-77c0519203db\") " pod="openshift-insights/periodic-gathering-5rxlb-bg459"
Nov 01 03:06:34.687878 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.687713    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/de5f469f-1029-4161-bf51-6ad18a3ac9e2-host-cni-bin\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:06:34.687878 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.687770    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/68932dfe-7e25-4f1b-b485-77c0519203db-archives-path\") pod \"periodic-gathering-5rxlb-bg459\" (UID: \"68932dfe-7e25-4f1b-b485-77c0519203db\") " pod="openshift-insights/periodic-gathering-5rxlb-bg459"
Nov 01 03:06:34.687878 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.687784    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/11fb87e4-ef01-4209-8207-80b66ab09f5e-etc-tuned\") pod \"tuned-lkbk6\" (UID: \"11fb87e4-ef01-4209-8207-80b66ab09f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-lkbk6"
Nov 01 03:06:34.687878 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.687810    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0303d6c6-c4ec-4403-af74-f4bcb5cbf2a4-dbus\") pod \"global-pull-secret-syncer-rrdzd\" (UID: \"0303d6c6-c4ec-4403-af74-f4bcb5cbf2a4\") " pod="kube-system/global-pull-secret-syncer-rrdzd"
Nov 01 03:06:34.687878 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.687843    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/11fb87e4-ef01-4209-8207-80b66ab09f5e-etc-tuned\") pod \"tuned-lkbk6\" (UID: \"11fb87e4-ef01-4209-8207-80b66ab09f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-lkbk6"
Nov 01 03:06:34.687878 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.687872    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bf9dac1d-d9fc-455a-be03-ea34f8838730-sys-fs\") pod \"aws-ebs-csi-driver-node-vq4dj\" (UID: \"bf9dac1d-d9fc-455a-be03-ea34f8838730\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vq4dj"
Nov 01 03:06:34.688048 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.687898    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnrhp\" (UniqueName: \"kubernetes.io/projected/68932dfe-7e25-4f1b-b485-77c0519203db-kube-api-access-qnrhp\") pod \"periodic-gathering-5rxlb-bg459\" (UID: \"68932dfe-7e25-4f1b-b485-77c0519203db\") " pod="openshift-insights/periodic-gathering-5rxlb-bg459"
Nov 01 03:06:34.688048 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.687919    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0ede655d-c20d-41cc-b5f0-2db4338c1ba7-multus-cni-dir\") pod \"multus-qzl9n\" (UID: \"0ede655d-c20d-41cc-b5f0-2db4338c1ba7\") " pod="openshift-multus/multus-qzl9n"
Nov 01 03:06:34.688048 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.687956    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0ede655d-c20d-41cc-b5f0-2db4338c1ba7-multus-daemon-config\") pod \"multus-qzl9n\" (UID: \"0ede655d-c20d-41cc-b5f0-2db4338c1ba7\") " pod="openshift-multus/multus-qzl9n"
Nov 01 03:06:34.688048 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.688002    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/994175bb-d192-47f7-b162-a840eb0f9be1-tmp-dir\") pod \"node-resolver-pzt6v\" (UID: \"994175bb-d192-47f7-b162-a840eb0f9be1\") " pod="openshift-dns/node-resolver-pzt6v"
Nov 01 03:06:34.688048 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.688017    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/11fb87e4-ef01-4209-8207-80b66ab09f5e-etc-sysctl-conf\") pod \"tuned-lkbk6\" (UID: \"11fb87e4-ef01-4209-8207-80b66ab09f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-lkbk6"
Nov 01 03:06:34.688048 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.688033    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/11fb87e4-ef01-4209-8207-80b66ab09f5e-etc-systemd\") pod \"tuned-lkbk6\" (UID: \"11fb87e4-ef01-4209-8207-80b66ab09f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-lkbk6"
Nov 01 03:06:34.688048 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.688049    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/11fb87e4-ef01-4209-8207-80b66ab09f5e-tmp\") pod \"tuned-lkbk6\" (UID: \"11fb87e4-ef01-4209-8207-80b66ab09f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-lkbk6"
Nov 01 03:06:34.688260 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.688073    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bf9dac1d-d9fc-455a-be03-ea34f8838730-device-dir\") pod \"aws-ebs-csi-driver-node-vq4dj\" (UID: \"bf9dac1d-d9fc-455a-be03-ea34f8838730\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vq4dj"
Nov 01 03:06:34.688260 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.688105    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/994175bb-d192-47f7-b162-a840eb0f9be1-tmp-dir\") pod \"node-resolver-pzt6v\" (UID: \"994175bb-d192-47f7-b162-a840eb0f9be1\") " pod="openshift-dns/node-resolver-pzt6v"
Nov 01 03:06:34.688260 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.688114    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/11fb87e4-ef01-4209-8207-80b66ab09f5e-tmp\") pod \"tuned-lkbk6\" (UID: \"11fb87e4-ef01-4209-8207-80b66ab09f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-lkbk6"
Nov 01 03:06:34.688260 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.688164    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/de5f469f-1029-4161-bf51-6ad18a3ac9e2-host-kubelet\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:06:34.688260 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.688191    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/de5f469f-1029-4161-bf51-6ad18a3ac9e2-host-run-netns\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:06:34.688260 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.688210    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/de5f469f-1029-4161-bf51-6ad18a3ac9e2-host-cni-netd\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:06:34.688260 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.688235    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56aed8ff-9b41-400a-90fb-8c283ce0c820-tmp-dir\") pod \"dns-default-nzbnl\" (UID: \"56aed8ff-9b41-400a-90fb-8c283ce0c820\") " pod="openshift-dns/dns-default-nzbnl"
Nov 01 03:06:34.688463 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.688267    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/11fb87e4-ef01-4209-8207-80b66ab09f5e-run\") pod \"tuned-lkbk6\" (UID: \"11fb87e4-ef01-4209-8207-80b66ab09f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-lkbk6"
Nov 01 03:06:34.688463 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.688290    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c9e89456-9acd-408f-87b5-f5fd968c82c3-iptables-alerter-script\") pod \"iptables-alerter-wfg7k\" (UID: \"c9e89456-9acd-408f-87b5-f5fd968c82c3\") " pod="openshift-network-operator/iptables-alerter-wfg7k"
Nov 01 03:06:34.688463 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.688316    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d7cef651-3047-4e64-ae7f-b0fa71fbd0fe-serviceca\") pod \"node-ca-wq27m\" (UID: \"d7cef651-3047-4e64-ae7f-b0fa71fbd0fe\") " pod="openshift-image-registry/node-ca-wq27m"
Nov 01 03:06:34.688463 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.688348    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56aed8ff-9b41-400a-90fb-8c283ce0c820-tmp-dir\") pod \"dns-default-nzbnl\" (UID: \"56aed8ff-9b41-400a-90fb-8c283ce0c820\") " pod="openshift-dns/dns-default-nzbnl"
Nov 01 03:06:34.688463 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.688381    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e8f2a24b-e770-4615-a2c7-b2d065e14f77-cnibin\") pod \"multus-additional-cni-plugins-qnrf8\" (UID: \"e8f2a24b-e770-4615-a2c7-b2d065e14f77\") " pod="openshift-multus/multus-additional-cni-plugins-qnrf8"
Nov 01 03:06:34.688463 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.688409    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0ede655d-c20d-41cc-b5f0-2db4338c1ba7-cnibin\") pod \"multus-qzl9n\" (UID: \"0ede655d-c20d-41cc-b5f0-2db4338c1ba7\") " pod="openshift-multus/multus-qzl9n"
Nov 01 03:06:34.688463 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.688444    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de5f469f-1029-4161-bf51-6ad18a3ac9e2-var-lib-openvswitch\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:06:34.688711 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.688473    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de5f469f-1029-4161-bf51-6ad18a3ac9e2-run-openvswitch\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:06:34.688711 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.688500    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4c1dc667-d774-4db6-b221-f4e7de423260-cert\") pod \"ingress-canary-h6696\" (UID: \"4c1dc667-d774-4db6-b221-f4e7de423260\") " pod="openshift-ingress-canary/ingress-canary-h6696"
Nov 01 03:06:34.688711 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.688517    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/73b27ccb-7d7a-40c1-8479-8379b4fb50ed-data-volume\") pod \"insights-runtime-extractor-6vv92\" (UID: \"73b27ccb-7d7a-40c1-8479-8379b4fb50ed\") " pod="openshift-insights/insights-runtime-extractor-6vv92"
Nov 01 03:06:34.688711 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.688534    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxqx8\" (UniqueName: \"kubernetes.io/projected/6e302b40-9c43-4822-8c8b-4d021a0590ab-kube-api-access-nxqx8\") pod \"network-check-target-mx48d\" (UID: \"6e302b40-9c43-4822-8c8b-4d021a0590ab\") " pod="openshift-network-diagnostics/network-check-target-mx48d"
Nov 01 03:06:34.688711 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.688550    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svdzl\" (UniqueName: \"kubernetes.io/projected/2779bdf3-1495-4083-84a0-58846a5f6e58-kube-api-access-svdzl\") pod \"migrator-66df5d496b-g8ql6\" (UID: \"2779bdf3-1495-4083-84a0-58846a5f6e58\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-g8ql6"
Nov 01 03:06:34.688711 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.688565    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtjjf\" (UniqueName: \"kubernetes.io/projected/994175bb-d192-47f7-b162-a840eb0f9be1-kube-api-access-xtjjf\") pod \"node-resolver-pzt6v\" (UID: \"994175bb-d192-47f7-b162-a840eb0f9be1\") " pod="openshift-dns/node-resolver-pzt6v"
Nov 01 03:06:34.688711 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.688583    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/de5f469f-1029-4161-bf51-6ad18a3ac9e2-node-log\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:06:34.688711 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.688618    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/de5f469f-1029-4161-bf51-6ad18a3ac9e2-ovnkube-config\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:06:34.688711 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.688644    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/73b27ccb-7d7a-40c1-8479-8379b4fb50ed-data-volume\") pod \"insights-runtime-extractor-6vv92\" (UID: \"73b27ccb-7d7a-40c1-8479-8379b4fb50ed\") " pod="openshift-insights/insights-runtime-extractor-6vv92"
Nov 01 03:06:34.688711 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.688697    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/11fb87e4-ef01-4209-8207-80b66ab09f5e-var-lib-kubelet\") pod \"tuned-lkbk6\" (UID: \"11fb87e4-ef01-4209-8207-80b66ab09f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-lkbk6"
Nov 01 03:06:34.689073 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.688779    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h58g6\" (UniqueName: \"kubernetes.io/projected/11fb87e4-ef01-4209-8207-80b66ab09f5e-kube-api-access-h58g6\") pod \"tuned-lkbk6\" (UID: \"11fb87e4-ef01-4209-8207-80b66ab09f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-lkbk6"
Nov 01 03:06:34.689073 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.688804    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0ede655d-c20d-41cc-b5f0-2db4338c1ba7-cni-binary-copy\") pod \"multus-qzl9n\" (UID: \"0ede655d-c20d-41cc-b5f0-2db4338c1ba7\") " pod="openshift-multus/multus-qzl9n"
Nov 01 03:06:34.689073 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.688820    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/de5f469f-1029-4161-bf51-6ad18a3ac9e2-env-overrides\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:06:34.689073 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.688834    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/11fb87e4-ef01-4209-8207-80b66ab09f5e-etc-modprobe-d\") pod \"tuned-lkbk6\" (UID: \"11fb87e4-ef01-4209-8207-80b66ab09f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-lkbk6"
Nov 01 03:06:34.689073 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.688848    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/11fb87e4-ef01-4209-8207-80b66ab09f5e-etc-sysctl-d\") pod \"tuned-lkbk6\" (UID: \"11fb87e4-ef01-4209-8207-80b66ab09f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-lkbk6"
Nov 01 03:06:34.689073 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.688890    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bf9dac1d-d9fc-455a-be03-ea34f8838730-etc-selinux\") pod \"aws-ebs-csi-driver-node-vq4dj\" (UID: \"bf9dac1d-d9fc-455a-be03-ea34f8838730\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vq4dj"
Nov 01 03:06:34.689073 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.688911    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/11fb87e4-ef01-4209-8207-80b66ab09f5e-sys\") pod \"tuned-lkbk6\" (UID: \"11fb87e4-ef01-4209-8207-80b66ab09f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-lkbk6"
Nov 01 03:06:34.689073 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.688925    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d7cef651-3047-4e64-ae7f-b0fa71fbd0fe-host\") pod \"node-ca-wq27m\" (UID: \"d7cef651-3047-4e64-ae7f-b0fa71fbd0fe\") " pod="openshift-image-registry/node-ca-wq27m"
Nov 01 03:06:34.689073 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.688939    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bf9dac1d-d9fc-455a-be03-ea34f8838730-socket-dir\") pod \"aws-ebs-csi-driver-node-vq4dj\" (UID: \"bf9dac1d-d9fc-455a-be03-ea34f8838730\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vq4dj"
Nov 01 03:06:34.689073 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.688968    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e8f2a24b-e770-4615-a2c7-b2d065e14f77-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qnrf8\" (UID: \"e8f2a24b-e770-4615-a2c7-b2d065e14f77\") " pod="openshift-multus/multus-additional-cni-plugins-qnrf8"
Nov 01 03:06:34.689073 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.688999    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e48a634f-ab31-435c-96c1-c3633c5bc061-metrics-certs\") pod \"network-metrics-daemon-dnqjp\" (UID: \"e48a634f-ab31-435c-96c1-c3633c5bc061\") " pod="openshift-multus/network-metrics-daemon-dnqjp"
Nov 01 03:06:34.689073 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.689023    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/73b27ccb-7d7a-40c1-8479-8379b4fb50ed-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6vv92\" (UID: \"73b27ccb-7d7a-40c1-8479-8379b4fb50ed\") " pod="openshift-insights/insights-runtime-extractor-6vv92"
Nov 01 03:06:34.689073 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.689041    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/bd8260bb-3811-4cdc-91d0-c6ddc693891e-agent-certs\") pod \"konnectivity-agent-fh7sf\" (UID: \"bd8260bb-3811-4cdc-91d0-c6ddc693891e\") " pod="kube-system/konnectivity-agent-fh7sf"
Nov 01 03:06:34.689395 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.689090    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0ede655d-c20d-41cc-b5f0-2db4338c1ba7-hostroot\") pod \"multus-qzl9n\" (UID: \"0ede655d-c20d-41cc-b5f0-2db4338c1ba7\") " pod="openshift-multus/multus-qzl9n"
Nov 01 03:06:34.689395 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.689151    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0ede655d-c20d-41cc-b5f0-2db4338c1ba7-host-run-multus-certs\") pod \"multus-qzl9n\" (UID: \"0ede655d-c20d-41cc-b5f0-2db4338c1ba7\") " pod="openshift-multus/multus-qzl9n"
Nov 01 03:06:34.713687 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.713669    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-g8ql6"
Nov 01 03:06:34.716028 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.716012    4400 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 03:06:34.716109 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:06:34.716060    4400 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-133-54.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-54.us-west-1.compute.internal"
Nov 01 03:06:34.719760 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.719648    4400 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 03:06:34.719760 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:06:34.719702    4400 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-133-54.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-54.us-west-1.compute.internal"
Nov 01 03:06:34.719899 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.719770    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-wgrq6\""
Nov 01 03:06:34.719899 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.719879    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 01 03:06:34.719899 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.719884    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-kbjpv\""
Nov 01 03:06:34.720107 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.720089    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-mt5gq\""
Nov 01 03:06:34.720182 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.720164    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-77n5m\""
Nov 01 03:06:34.721287 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.721273    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 01 03:06:34.721996 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.721981    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-62njk\""
Nov 01 03:06:34.722190 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.722179    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 01 03:06:34.722251 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.722236    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-6592m\""
Nov 01 03:06:34.727524 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.727507    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/bd8260bb-3811-4cdc-91d0-c6ddc693891e-konnectivity-ca\") pod \"konnectivity-agent-fh7sf\" (UID: \"bd8260bb-3811-4cdc-91d0-c6ddc693891e\") " pod="kube-system/konnectivity-agent-fh7sf"
Nov 01 03:06:34.727608 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.727558    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0303d6c6-c4ec-4403-af74-f4bcb5cbf2a4-original-pull-secret\") pod \"global-pull-secret-syncer-rrdzd\" (UID: \"0303d6c6-c4ec-4403-af74-f4bcb5cbf2a4\") " pod="kube-system/global-pull-secret-syncer-rrdzd"
Nov 01 03:06:34.729770 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.729756    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/bd8260bb-3811-4cdc-91d0-c6ddc693891e-agent-certs\") pod \"konnectivity-agent-fh7sf\" (UID: \"bd8260bb-3811-4cdc-91d0-c6ddc693891e\") " pod="kube-system/konnectivity-agent-fh7sf"
Nov 01 03:06:34.777391 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.777374    4400 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 01 03:06:34.789464 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.789444    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/de5f469f-1029-4161-bf51-6ad18a3ac9e2-host-slash\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:06:34.789527 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.789468    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/de5f469f-1029-4161-bf51-6ad18a3ac9e2-run-ovn\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:06:34.789527 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.789484    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0ede655d-c20d-41cc-b5f0-2db4338c1ba7-system-cni-dir\") pod \"multus-qzl9n\" (UID: \"0ede655d-c20d-41cc-b5f0-2db4338c1ba7\") " pod="openshift-multus/multus-qzl9n"
Nov 01 03:06:34.789527 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.789499    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/de5f469f-1029-4161-bf51-6ad18a3ac9e2-run-systemd\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:06:34.789654 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.789538    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/de5f469f-1029-4161-bf51-6ad18a3ac9e2-run-systemd\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:06:34.789654 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.789545    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/de5f469f-1029-4161-bf51-6ad18a3ac9e2-run-ovn\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:06:34.789654 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.789561    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de5f469f-1029-4161-bf51-6ad18a3ac9e2-etc-openvswitch\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:06:34.789654 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.789561    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/de5f469f-1029-4161-bf51-6ad18a3ac9e2-host-slash\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:06:34.789654 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.789577    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/11fb87e4-ef01-4209-8207-80b66ab09f5e-etc-kubernetes\") pod \"tuned-lkbk6\" (UID: \"11fb87e4-ef01-4209-8207-80b66ab09f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-lkbk6"
Nov 01 03:06:34.789654 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.789564    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0ede655d-c20d-41cc-b5f0-2db4338c1ba7-system-cni-dir\") pod \"multus-qzl9n\" (UID: \"0ede655d-c20d-41cc-b5f0-2db4338c1ba7\") " pod="openshift-multus/multus-qzl9n"
Nov 01 03:06:34.789654 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.789585    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de5f469f-1029-4161-bf51-6ad18a3ac9e2-etc-openvswitch\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:06:34.789654 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.789608    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/994175bb-d192-47f7-b162-a840eb0f9be1-hosts-file\") pod \"node-resolver-pzt6v\" (UID: \"994175bb-d192-47f7-b162-a840eb0f9be1\") " pod="openshift-dns/node-resolver-pzt6v"
Nov 01 03:06:34.789654 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.789627    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/de5f469f-1029-4161-bf51-6ad18a3ac9e2-systemd-units\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:06:34.789654 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.789633    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/11fb87e4-ef01-4209-8207-80b66ab09f5e-etc-kubernetes\") pod \"tuned-lkbk6\" (UID: \"11fb87e4-ef01-4209-8207-80b66ab09f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-lkbk6"
Nov 01 03:06:34.789654 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.789643    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de5f469f-1029-4161-bf51-6ad18a3ac9e2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:06:34.790034 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.789664    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/73b27ccb-7d7a-40c1-8479-8379b4fb50ed-crio-socket\") pod \"insights-runtime-extractor-6vv92\" (UID: \"73b27ccb-7d7a-40c1-8479-8379b4fb50ed\") " pod="openshift-insights/insights-runtime-extractor-6vv92"
Nov 01 03:06:34.790034 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.789709    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0ede655d-c20d-41cc-b5f0-2db4338c1ba7-host-var-lib-kubelet\") pod \"multus-qzl9n\" (UID: \"0ede655d-c20d-41cc-b5f0-2db4338c1ba7\") " pod="openshift-multus/multus-qzl9n"
Nov 01 03:06:34.790034 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.789716    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de5f469f-1029-4161-bf51-6ad18a3ac9e2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:06:34.790034 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.789728    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/994175bb-d192-47f7-b162-a840eb0f9be1-hosts-file\") pod \"node-resolver-pzt6v\" (UID: \"994175bb-d192-47f7-b162-a840eb0f9be1\") " pod="openshift-dns/node-resolver-pzt6v"
Nov 01 03:06:34.790034 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.789703    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/de5f469f-1029-4161-bf51-6ad18a3ac9e2-systemd-units\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:06:34.790034 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.789740    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0ede655d-c20d-41cc-b5f0-2db4338c1ba7-host-run-netns\") pod \"multus-qzl9n\" (UID: \"0ede655d-c20d-41cc-b5f0-2db4338c1ba7\") " pod="openshift-multus/multus-qzl9n"
Nov 01 03:06:34.790034 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.789764    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/11fb87e4-ef01-4209-8207-80b66ab09f5e-etc-sysconfig\") pod \"tuned-lkbk6\" (UID: \"11fb87e4-ef01-4209-8207-80b66ab09f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-lkbk6"
Nov 01 03:06:34.790034 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.789766    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0ede655d-c20d-41cc-b5f0-2db4338c1ba7-host-var-lib-kubelet\") pod \"multus-qzl9n\" (UID: \"0ede655d-c20d-41cc-b5f0-2db4338c1ba7\") " pod="openshift-multus/multus-qzl9n"
Nov 01 03:06:34.790034 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.789788    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bf9dac1d-d9fc-455a-be03-ea34f8838730-registration-dir\") pod \"aws-ebs-csi-driver-node-vq4dj\" (UID: \"bf9dac1d-d9fc-455a-be03-ea34f8838730\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vq4dj"
Nov 01 03:06:34.790034 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.789793    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0ede655d-c20d-41cc-b5f0-2db4338c1ba7-host-run-netns\") pod \"multus-qzl9n\" (UID: \"0ede655d-c20d-41cc-b5f0-2db4338c1ba7\") " pod="openshift-multus/multus-qzl9n"
Nov 01 03:06:34.790034 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.789803    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/11fb87e4-ef01-4209-8207-80b66ab09f5e-etc-sysconfig\") pod \"tuned-lkbk6\" (UID: \"11fb87e4-ef01-4209-8207-80b66ab09f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-lkbk6"
Nov 01 03:06:34.790034 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.789764    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/73b27ccb-7d7a-40c1-8479-8379b4fb50ed-crio-socket\") pod \"insights-runtime-extractor-6vv92\" (UID: \"73b27ccb-7d7a-40c1-8479-8379b4fb50ed\") " pod="openshift-insights/insights-runtime-extractor-6vv92"
Nov 01 03:06:34.790034 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.789814    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/de5f469f-1029-4161-bf51-6ad18a3ac9e2-host-cni-bin\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:06:34.790034 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.789836    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0303d6c6-c4ec-4403-af74-f4bcb5cbf2a4-dbus\") pod \"global-pull-secret-syncer-rrdzd\" (UID: \"0303d6c6-c4ec-4403-af74-f4bcb5cbf2a4\") " pod="kube-system/global-pull-secret-syncer-rrdzd"
Nov 01 03:06:34.790034 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.789843    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bf9dac1d-d9fc-455a-be03-ea34f8838730-registration-dir\") pod \"aws-ebs-csi-driver-node-vq4dj\" (UID: \"bf9dac1d-d9fc-455a-be03-ea34f8838730\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vq4dj"
Nov 01 03:06:34.790034 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.789850    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bf9dac1d-d9fc-455a-be03-ea34f8838730-sys-fs\") pod \"aws-ebs-csi-driver-node-vq4dj\" (UID: \"bf9dac1d-d9fc-455a-be03-ea34f8838730\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vq4dj"
Nov 01 03:06:34.790034 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.789865    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0ede655d-c20d-41cc-b5f0-2db4338c1ba7-multus-cni-dir\") pod \"multus-qzl9n\" (UID: \"0ede655d-c20d-41cc-b5f0-2db4338c1ba7\") " pod="openshift-multus/multus-qzl9n"
Nov 01 03:06:34.790034 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.789886    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/11fb87e4-ef01-4209-8207-80b66ab09f5e-etc-sysctl-conf\") pod \"tuned-lkbk6\" (UID: \"11fb87e4-ef01-4209-8207-80b66ab09f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-lkbk6"
Nov 01 03:06:34.790034 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.789911    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0303d6c6-c4ec-4403-af74-f4bcb5cbf2a4-dbus\") pod \"global-pull-secret-syncer-rrdzd\" (UID: \"0303d6c6-c4ec-4403-af74-f4bcb5cbf2a4\") " pod="kube-system/global-pull-secret-syncer-rrdzd"
Nov 01 03:06:34.790034 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.789912    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/11fb87e4-ef01-4209-8207-80b66ab09f5e-etc-systemd\") pod \"tuned-lkbk6\" (UID: \"11fb87e4-ef01-4209-8207-80b66ab09f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-lkbk6"
Nov 01 03:06:34.790034 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.789919    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bf9dac1d-d9fc-455a-be03-ea34f8838730-sys-fs\") pod \"aws-ebs-csi-driver-node-vq4dj\" (UID: \"bf9dac1d-d9fc-455a-be03-ea34f8838730\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vq4dj"
Nov 01 03:06:34.790034 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.789947    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/11fb87e4-ef01-4209-8207-80b66ab09f5e-etc-systemd\") pod \"tuned-lkbk6\" (UID: \"11fb87e4-ef01-4209-8207-80b66ab09f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-lkbk6"
Nov 01 03:06:34.790034 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.789944    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bf9dac1d-d9fc-455a-be03-ea34f8838730-device-dir\") pod \"aws-ebs-csi-driver-node-vq4dj\" (UID: \"bf9dac1d-d9fc-455a-be03-ea34f8838730\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vq4dj"
Nov 01 03:06:34.790034 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.789955    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/de5f469f-1029-4161-bf51-6ad18a3ac9e2-host-cni-bin\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:06:34.790034 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.789961    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0ede655d-c20d-41cc-b5f0-2db4338c1ba7-multus-cni-dir\") pod \"multus-qzl9n\" (UID: \"0ede655d-c20d-41cc-b5f0-2db4338c1ba7\") " pod="openshift-multus/multus-qzl9n"
Nov 01 03:06:34.790034 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.789972    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/de5f469f-1029-4161-bf51-6ad18a3ac9e2-host-kubelet\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:06:34.790034 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.789976    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bf9dac1d-d9fc-455a-be03-ea34f8838730-device-dir\") pod \"aws-ebs-csi-driver-node-vq4dj\" (UID: \"bf9dac1d-d9fc-455a-be03-ea34f8838730\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vq4dj"
Nov 01 03:06:34.790034 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.789990    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/de5f469f-1029-4161-bf51-6ad18a3ac9e2-host-run-netns\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:06:34.790034 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.789999    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/de5f469f-1029-4161-bf51-6ad18a3ac9e2-host-kubelet\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:06:34.790034 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.790012    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/de5f469f-1029-4161-bf51-6ad18a3ac9e2-host-cni-netd\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:06:34.790034 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.790028    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/11fb87e4-ef01-4209-8207-80b66ab09f5e-run\") pod \"tuned-lkbk6\" (UID: \"11fb87e4-ef01-4209-8207-80b66ab09f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-lkbk6"
Nov 01 03:06:34.790034 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.790042    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e8f2a24b-e770-4615-a2c7-b2d065e14f77-cnibin\") pod \"multus-additional-cni-plugins-qnrf8\" (UID: \"e8f2a24b-e770-4615-a2c7-b2d065e14f77\") " pod="openshift-multus/multus-additional-cni-plugins-qnrf8"
Nov 01 03:06:34.790034 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.790011    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/11fb87e4-ef01-4209-8207-80b66ab09f5e-etc-sysctl-conf\") pod \"tuned-lkbk6\" (UID: \"11fb87e4-ef01-4209-8207-80b66ab09f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-lkbk6"
Nov 01 03:06:34.791266 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.790068    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/de5f469f-1029-4161-bf51-6ad18a3ac9e2-host-cni-netd\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:06:34.791266 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.790084    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/de5f469f-1029-4161-bf51-6ad18a3ac9e2-host-run-netns\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:06:34.791266 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.790088    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0ede655d-c20d-41cc-b5f0-2db4338c1ba7-cnibin\") pod \"multus-qzl9n\" (UID: \"0ede655d-c20d-41cc-b5f0-2db4338c1ba7\") " pod="openshift-multus/multus-qzl9n"
Nov 01 03:06:34.791266 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.790090    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e8f2a24b-e770-4615-a2c7-b2d065e14f77-cnibin\") pod \"multus-additional-cni-plugins-qnrf8\" (UID: \"e8f2a24b-e770-4615-a2c7-b2d065e14f77\") " pod="openshift-multus/multus-additional-cni-plugins-qnrf8"
Nov 01 03:06:34.791266 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.790091    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/11fb87e4-ef01-4209-8207-80b66ab09f5e-run\") pod \"tuned-lkbk6\" (UID: \"11fb87e4-ef01-4209-8207-80b66ab09f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-lkbk6"
Nov 01 03:06:34.791266 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.790104    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de5f469f-1029-4161-bf51-6ad18a3ac9e2-var-lib-openvswitch\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:06:34.791266 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.790120    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de5f469f-1029-4161-bf51-6ad18a3ac9e2-run-openvswitch\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:06:34.791266 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.790137    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svdzl\" (UniqueName: \"kubernetes.io/projected/2779bdf3-1495-4083-84a0-58846a5f6e58-kube-api-access-svdzl\") pod \"migrator-66df5d496b-g8ql6\" (UID: \"2779bdf3-1495-4083-84a0-58846a5f6e58\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-g8ql6"
Nov 01 03:06:34.791266 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.790142    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0ede655d-c20d-41cc-b5f0-2db4338c1ba7-cnibin\") pod \"multus-qzl9n\" (UID: \"0ede655d-c20d-41cc-b5f0-2db4338c1ba7\") " pod="openshift-multus/multus-qzl9n"
Nov 01 03:06:34.791266 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.790136    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de5f469f-1029-4161-bf51-6ad18a3ac9e2-var-lib-openvswitch\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:06:34.791266 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.790155    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de5f469f-1029-4161-bf51-6ad18a3ac9e2-run-openvswitch\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:06:34.791266 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.790156    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/de5f469f-1029-4161-bf51-6ad18a3ac9e2-node-log\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:06:34.791266 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.790177    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/de5f469f-1029-4161-bf51-6ad18a3ac9e2-node-log\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:06:34.791266 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.790199    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/11fb87e4-ef01-4209-8207-80b66ab09f5e-var-lib-kubelet\") pod \"tuned-lkbk6\" (UID: \"11fb87e4-ef01-4209-8207-80b66ab09f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-lkbk6"
Nov 01 03:06:34.791266 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.790228    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/11fb87e4-ef01-4209-8207-80b66ab09f5e-etc-modprobe-d\") pod \"tuned-lkbk6\" (UID: \"11fb87e4-ef01-4209-8207-80b66ab09f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-lkbk6"
Nov 01 03:06:34.791266 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.790249    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/11fb87e4-ef01-4209-8207-80b66ab09f5e-var-lib-kubelet\") pod \"tuned-lkbk6\" (UID: \"11fb87e4-ef01-4209-8207-80b66ab09f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-lkbk6"
Nov 01 03:06:34.791266 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.790252    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/11fb87e4-ef01-4209-8207-80b66ab09f5e-etc-sysctl-d\") pod \"tuned-lkbk6\" (UID: \"11fb87e4-ef01-4209-8207-80b66ab09f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-lkbk6"
Nov 01 03:06:34.791266 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.790281    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bf9dac1d-d9fc-455a-be03-ea34f8838730-etc-selinux\") pod \"aws-ebs-csi-driver-node-vq4dj\" (UID: \"bf9dac1d-d9fc-455a-be03-ea34f8838730\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vq4dj"
Nov 01 03:06:34.791266 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.790305    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/11fb87e4-ef01-4209-8207-80b66ab09f5e-sys\") pod \"tuned-lkbk6\" (UID: \"11fb87e4-ef01-4209-8207-80b66ab09f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-lkbk6"
Nov 01 03:06:34.791266 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.790325    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/11fb87e4-ef01-4209-8207-80b66ab09f5e-etc-modprobe-d\") pod \"tuned-lkbk6\" (UID: \"11fb87e4-ef01-4209-8207-80b66ab09f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-lkbk6"
Nov 01 03:06:34.791266 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.790327    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d7cef651-3047-4e64-ae7f-b0fa71fbd0fe-host\") pod \"node-ca-wq27m\" (UID: \"d7cef651-3047-4e64-ae7f-b0fa71fbd0fe\") " pod="openshift-image-registry/node-ca-wq27m"
Nov 01 03:06:34.791266 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.790351    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/11fb87e4-ef01-4209-8207-80b66ab09f5e-etc-sysctl-d\") pod \"tuned-lkbk6\" (UID: \"11fb87e4-ef01-4209-8207-80b66ab09f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-lkbk6"
Nov 01 03:06:34.791266 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.790358    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bf9dac1d-d9fc-455a-be03-ea34f8838730-etc-selinux\") pod \"aws-ebs-csi-driver-node-vq4dj\" (UID: \"bf9dac1d-d9fc-455a-be03-ea34f8838730\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vq4dj"
Nov 01 03:06:34.791266 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.790351    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bf9dac1d-d9fc-455a-be03-ea34f8838730-socket-dir\") pod \"aws-ebs-csi-driver-node-vq4dj\" (UID: \"bf9dac1d-d9fc-455a-be03-ea34f8838730\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vq4dj"
Nov 01 03:06:34.791266 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.790383    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/11fb87e4-ef01-4209-8207-80b66ab09f5e-sys\") pod \"tuned-lkbk6\" (UID: \"11fb87e4-ef01-4209-8207-80b66ab09f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-lkbk6"
Nov 01 03:06:34.791266 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.790390    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e48a634f-ab31-435c-96c1-c3633c5bc061-metrics-certs\") pod \"network-metrics-daemon-dnqjp\" (UID: \"e48a634f-ab31-435c-96c1-c3633c5bc061\") " pod="openshift-multus/network-metrics-daemon-dnqjp"
Nov 01 03:06:34.791266 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.790391    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d7cef651-3047-4e64-ae7f-b0fa71fbd0fe-host\") pod \"node-ca-wq27m\" (UID: \"d7cef651-3047-4e64-ae7f-b0fa71fbd0fe\") " pod="openshift-image-registry/node-ca-wq27m"
Nov 01 03:06:34.791266 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.790396    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bf9dac1d-d9fc-455a-be03-ea34f8838730-socket-dir\") pod \"aws-ebs-csi-driver-node-vq4dj\" (UID: \"bf9dac1d-d9fc-455a-be03-ea34f8838730\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vq4dj"
Nov 01 03:06:34.791266 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.790422    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/73b27ccb-7d7a-40c1-8479-8379b4fb50ed-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6vv92\" (UID: \"73b27ccb-7d7a-40c1-8479-8379b4fb50ed\") " pod="openshift-insights/insights-runtime-extractor-6vv92"
Nov 01 03:06:34.791266 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.790440    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0ede655d-c20d-41cc-b5f0-2db4338c1ba7-hostroot\") pod \"multus-qzl9n\" (UID: \"0ede655d-c20d-41cc-b5f0-2db4338c1ba7\") " pod="openshift-multus/multus-qzl9n"
Nov 01 03:06:34.791266 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.790455    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0ede655d-c20d-41cc-b5f0-2db4338c1ba7-host-run-multus-certs\") pod \"multus-qzl9n\" (UID: \"0ede655d-c20d-41cc-b5f0-2db4338c1ba7\") " pod="openshift-multus/multus-qzl9n"
Nov 01 03:06:34.791266 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.790470    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/de5f469f-1029-4161-bf51-6ad18a3ac9e2-log-socket\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:06:34.791266 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.790484    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/56aed8ff-9b41-400a-90fb-8c283ce0c820-metrics-tls\") pod \"dns-default-nzbnl\" (UID: \"56aed8ff-9b41-400a-90fb-8c283ce0c820\") " pod="openshift-dns/dns-default-nzbnl"
Nov 01 03:06:34.791266 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.790489    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0ede655d-c20d-41cc-b5f0-2db4338c1ba7-hostroot\") pod \"multus-qzl9n\" (UID: \"0ede655d-c20d-41cc-b5f0-2db4338c1ba7\") " pod="openshift-multus/multus-qzl9n"
Nov 01 03:06:34.791266 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.790499    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/11fb87e4-ef01-4209-8207-80b66ab09f5e-host\") pod \"tuned-lkbk6\" (UID: \"11fb87e4-ef01-4209-8207-80b66ab09f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-lkbk6"
Nov 01 03:06:34.791266 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.790505    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0ede655d-c20d-41cc-b5f0-2db4338c1ba7-host-run-multus-certs\") pod \"multus-qzl9n\" (UID: \"0ede655d-c20d-41cc-b5f0-2db4338c1ba7\") " pod="openshift-multus/multus-qzl9n"
Nov 01 03:06:34.791266 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.790515    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e8f2a24b-e770-4615-a2c7-b2d065e14f77-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qnrf8\" (UID: \"e8f2a24b-e770-4615-a2c7-b2d065e14f77\") " pod="openshift-multus/multus-additional-cni-plugins-qnrf8"
Nov 01 03:06:34.791266 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.790525    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/11fb87e4-ef01-4209-8207-80b66ab09f5e-host\") pod \"tuned-lkbk6\" (UID: \"11fb87e4-ef01-4209-8207-80b66ab09f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-lkbk6"
Nov 01 03:06:34.791266 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.790500    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/de5f469f-1029-4161-bf51-6ad18a3ac9e2-log-socket\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:06:34.791266 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.790529    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0ede655d-c20d-41cc-b5f0-2db4338c1ba7-host-run-k8s-cni-cncf-io\") pod \"multus-qzl9n\" (UID: \"0ede655d-c20d-41cc-b5f0-2db4338c1ba7\") " pod="openshift-multus/multus-qzl9n"
Nov 01 03:06:34.791266 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.790549    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0ede655d-c20d-41cc-b5f0-2db4338c1ba7-etc-kubernetes\") pod \"multus-qzl9n\" (UID: \"0ede655d-c20d-41cc-b5f0-2db4338c1ba7\") " pod="openshift-multus/multus-qzl9n"
Nov 01 03:06:34.791266 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.790564    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e8f2a24b-e770-4615-a2c7-b2d065e14f77-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qnrf8\" (UID: \"e8f2a24b-e770-4615-a2c7-b2d065e14f77\") " pod="openshift-multus/multus-additional-cni-plugins-qnrf8"
Nov 01 03:06:34.791266 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.790563    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0ede655d-c20d-41cc-b5f0-2db4338c1ba7-host-run-k8s-cni-cncf-io\") pod \"multus-qzl9n\" (UID: \"0ede655d-c20d-41cc-b5f0-2db4338c1ba7\") " pod="openshift-multus/multus-qzl9n"
Nov 01 03:06:34.791266 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.790571    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0ede655d-c20d-41cc-b5f0-2db4338c1ba7-host-var-lib-cni-bin\") pod \"multus-qzl9n\" (UID: \"0ede655d-c20d-41cc-b5f0-2db4338c1ba7\") " pod="openshift-multus/multus-qzl9n"
Nov 01 03:06:34.791266 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.790592    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0ede655d-c20d-41cc-b5f0-2db4338c1ba7-host-var-lib-cni-bin\") pod \"multus-qzl9n\" (UID: \"0ede655d-c20d-41cc-b5f0-2db4338c1ba7\") " pod="openshift-multus/multus-qzl9n"
Nov 01 03:06:34.791266 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.790599    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0ede655d-c20d-41cc-b5f0-2db4338c1ba7-host-var-lib-cni-multus\") pod \"multus-qzl9n\" (UID: \"0ede655d-c20d-41cc-b5f0-2db4338c1ba7\") " pod="openshift-multus/multus-qzl9n"
Nov 01 03:06:34.791266 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.790623    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0ede655d-c20d-41cc-b5f0-2db4338c1ba7-host-var-lib-cni-multus\") pod \"multus-qzl9n\" (UID: \"0ede655d-c20d-41cc-b5f0-2db4338c1ba7\") " pod="openshift-multus/multus-qzl9n"
Nov 01 03:06:34.791266 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.790624    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0ede655d-c20d-41cc-b5f0-2db4338c1ba7-multus-socket-dir-parent\") pod \"multus-qzl9n\" (UID: \"0ede655d-c20d-41cc-b5f0-2db4338c1ba7\") " pod="openshift-multus/multus-qzl9n"
Nov 01 03:06:34.791266 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.790654    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0ede655d-c20d-41cc-b5f0-2db4338c1ba7-multus-conf-dir\") pod \"multus-qzl9n\" (UID: \"0ede655d-c20d-41cc-b5f0-2db4338c1ba7\") " pod="openshift-multus/multus-qzl9n"
Nov 01 03:06:34.791266 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.790666    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0ede655d-c20d-41cc-b5f0-2db4338c1ba7-multus-socket-dir-parent\") pod \"multus-qzl9n\" (UID: \"0ede655d-c20d-41cc-b5f0-2db4338c1ba7\") " pod="openshift-multus/multus-qzl9n"
Nov 01 03:06:34.791266 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.790670    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c9e89456-9acd-408f-87b5-f5fd968c82c3-host-slash\") pod \"iptables-alerter-wfg7k\" (UID: \"c9e89456-9acd-408f-87b5-f5fd968c82c3\") " pod="openshift-network-operator/iptables-alerter-wfg7k"
Nov 01 03:06:34.791266 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.790708    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c9e89456-9acd-408f-87b5-f5fd968c82c3-host-slash\") pod \"iptables-alerter-wfg7k\" (UID: \"c9e89456-9acd-408f-87b5-f5fd968c82c3\") " pod="openshift-network-operator/iptables-alerter-wfg7k"
Nov 01 03:06:34.791266 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.790624    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0ede655d-c20d-41cc-b5f0-2db4338c1ba7-etc-kubernetes\") pod \"multus-qzl9n\" (UID: \"0ede655d-c20d-41cc-b5f0-2db4338c1ba7\") " pod="openshift-multus/multus-qzl9n"
Nov 01 03:06:34.791266 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.790717    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bf9dac1d-d9fc-455a-be03-ea34f8838730-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vq4dj\" (UID: \"bf9dac1d-d9fc-455a-be03-ea34f8838730\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vq4dj"
Nov 01 03:06:34.791266 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.790742    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de5f469f-1029-4161-bf51-6ad18a3ac9e2-host-run-ovn-kubernetes\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:06:34.791266 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.790750    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0ede655d-c20d-41cc-b5f0-2db4338c1ba7-multus-conf-dir\") pod \"multus-qzl9n\" (UID: \"0ede655d-c20d-41cc-b5f0-2db4338c1ba7\") " pod="openshift-multus/multus-qzl9n"
Nov 01 03:06:34.791266 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.790768    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/11fb87e4-ef01-4209-8207-80b66ab09f5e-lib-modules\") pod \"tuned-lkbk6\" (UID: \"11fb87e4-ef01-4209-8207-80b66ab09f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-lkbk6"
Nov 01 03:06:34.791266 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.790784    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bf9dac1d-d9fc-455a-be03-ea34f8838730-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vq4dj\" (UID: \"bf9dac1d-d9fc-455a-be03-ea34f8838730\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vq4dj"
Nov 01 03:06:34.791266 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.790796    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0303d6c6-c4ec-4403-af74-f4bcb5cbf2a4-kubelet-config\") pod \"global-pull-secret-syncer-rrdzd\" (UID: \"0303d6c6-c4ec-4403-af74-f4bcb5cbf2a4\") " pod="kube-system/global-pull-secret-syncer-rrdzd"
Nov 01 03:06:34.791266 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.790809    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de5f469f-1029-4161-bf51-6ad18a3ac9e2-host-run-ovn-kubernetes\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:06:34.791266 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.790823    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e8f2a24b-e770-4615-a2c7-b2d065e14f77-system-cni-dir\") pod \"multus-additional-cni-plugins-qnrf8\" (UID: \"e8f2a24b-e770-4615-a2c7-b2d065e14f77\") " pod="openshift-multus/multus-additional-cni-plugins-qnrf8"
Nov 01 03:06:34.791266 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.790850    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/73b27ccb-7d7a-40c1-8479-8379b4fb50ed-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6vv92\" (UID: \"73b27ccb-7d7a-40c1-8479-8379b4fb50ed\") " pod="openshift-insights/insights-runtime-extractor-6vv92"
Nov 01 03:06:34.791266 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.790856    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0303d6c6-c4ec-4403-af74-f4bcb5cbf2a4-kubelet-config\") pod \"global-pull-secret-syncer-rrdzd\" (UID: \"0303d6c6-c4ec-4403-af74-f4bcb5cbf2a4\") " pod="kube-system/global-pull-secret-syncer-rrdzd"
Nov 01 03:06:34.791266 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.790898    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/11fb87e4-ef01-4209-8207-80b66ab09f5e-lib-modules\") pod \"tuned-lkbk6\" (UID: \"11fb87e4-ef01-4209-8207-80b66ab09f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-lkbk6"
Nov 01 03:06:34.791266 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.790912    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e8f2a24b-e770-4615-a2c7-b2d065e14f77-system-cni-dir\") pod \"multus-additional-cni-plugins-qnrf8\" (UID: \"e8f2a24b-e770-4615-a2c7-b2d065e14f77\") " pod="openshift-multus/multus-additional-cni-plugins-qnrf8"
Nov 01 03:06:34.818765 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.818709    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 01 03:06:34.818765 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.818755    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 01 03:06:34.818955 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.818788    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 01 03:06:34.818955 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.818825    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 01 03:06:34.818955 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.818839    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 01 03:06:34.818955 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.818855    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 01 03:06:34.818955 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.818873    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 01 03:06:34.818955 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.818878    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 01 03:06:34.818955 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.818899    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 01 03:06:34.818955 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.818945    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 01 03:06:34.819417 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.819037    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 01 03:06:34.819417 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.819060    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 01 03:06:34.819417 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.819086    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 01 03:06:34.819417 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.819091    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 01 03:06:34.819417 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.819089    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/de5f469f-1029-4161-bf51-6ad18a3ac9e2-env-overrides\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:06:34.819417 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.819106    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 01 03:06:34.819417 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.819102    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 01 03:06:34.819417 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.819124    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 01 03:06:34.819417 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.819134    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 01 03:06:34.819417 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.819135    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 01 03:06:34.819417 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.819153    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 01 03:06:34.819417 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.819158    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 01 03:06:34.819417 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.819198    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 01 03:06:34.819417 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.819211    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 01 03:06:34.819417 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.819221    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 01 03:06:34.819417 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.819242    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 01 03:06:34.819417 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.819249    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 01 03:06:34.819417 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.819263    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 01 03:06:34.819417 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.819264    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 01 03:06:34.819417 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.819271    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 01 03:06:34.819417 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.819299    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 01 03:06:34.819417 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.819354    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 01 03:06:34.819417 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.819374    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 01 03:06:34.820230 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.819608    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 01 03:06:34.820230 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.819746    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t86zn\" (UniqueName: \"kubernetes.io/projected/d7cef651-3047-4e64-ae7f-b0fa71fbd0fe-kube-api-access-t86zn\") pod \"node-ca-wq27m\" (UID: \"d7cef651-3047-4e64-ae7f-b0fa71fbd0fe\") " pod="openshift-image-registry/node-ca-wq27m"
Nov 01 03:06:34.820230 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.819939    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 01 03:06:34.820374 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.820299    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtjjf\" (UniqueName: \"kubernetes.io/projected/994175bb-d192-47f7-b162-a840eb0f9be1-kube-api-access-xtjjf\") pod \"node-resolver-pzt6v\" (UID: \"994175bb-d192-47f7-b162-a840eb0f9be1\") " pod="openshift-dns/node-resolver-pzt6v"
Nov 01 03:06:34.820374 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.820303    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 01 03:06:34.820532 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.820516    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 01 03:06:34.820578 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:06:34.820570    4400 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 03:06:34.820650 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:06:34.820640    4400 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56aed8ff-9b41-400a-90fb-8c283ce0c820-metrics-tls podName:56aed8ff-9b41-400a-90fb-8c283ce0c820 nodeName:}" failed. No retries permitted until 2026-11-01 03:06:35.320621557 +0000 UTC m=+1.835267419 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/56aed8ff-9b41-400a-90fb-8c283ce0c820-metrics-tls") pod "dns-default-nzbnl" (UID: "56aed8ff-9b41-400a-90fb-8c283ce0c820") : secret "dns-default-metrics-tls" not found
Nov 01 03:06:34.820650 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.820641    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-74fv2\""
Nov 01 03:06:34.820795 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.820776    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-fdfjz\""
Nov 01 03:06:34.821158 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.821139    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckfr8\" (UniqueName: \"kubernetes.io/projected/de5f469f-1029-4161-bf51-6ad18a3ac9e2-kube-api-access-ckfr8\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:06:34.821205 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:06:34.821191    4400 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 01 03:06:34.821277 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:06:34.821266    4400 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/73b27ccb-7d7a-40c1-8479-8379b4fb50ed-kube-rbac-proxy-cm podName:73b27ccb-7d7a-40c1-8479-8379b4fb50ed nodeName:}" failed. No retries permitted until 2026-11-01 03:06:35.321249651 +0000 UTC m=+1.835895511 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/73b27ccb-7d7a-40c1-8479-8379b4fb50ed-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6vv92" (UID: "73b27ccb-7d7a-40c1-8479-8379b4fb50ed") : configmap "kube-rbac-proxy" not found
Nov 01 03:06:34.822019 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.822006    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2459t\" (UniqueName: \"kubernetes.io/projected/4c1dc667-d774-4db6-b221-f4e7de423260-kube-api-access-2459t\") pod \"ingress-canary-h6696\" (UID: \"4c1dc667-d774-4db6-b221-f4e7de423260\") " pod="openshift-ingress-canary/ingress-canary-h6696"
Nov 01 03:06:34.823157 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.823140    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnrhp\" (UniqueName: \"kubernetes.io/projected/68932dfe-7e25-4f1b-b485-77c0519203db-kube-api-access-qnrhp\") pod \"periodic-gathering-5rxlb-bg459\" (UID: \"68932dfe-7e25-4f1b-b485-77c0519203db\") " pod="openshift-insights/periodic-gathering-5rxlb-bg459"
Nov 01 03:06:34.825260 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.825245    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h58g6\" (UniqueName: \"kubernetes.io/projected/11fb87e4-ef01-4209-8207-80b66ab09f5e-kube-api-access-h58g6\") pod \"tuned-lkbk6\" (UID: \"11fb87e4-ef01-4209-8207-80b66ab09f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-lkbk6"
Nov 01 03:06:34.825388 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.825368    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9wst\" (UniqueName: \"kubernetes.io/projected/56aed8ff-9b41-400a-90fb-8c283ce0c820-kube-api-access-r9wst\") pod \"dns-default-nzbnl\" (UID: \"56aed8ff-9b41-400a-90fb-8c283ce0c820\") " pod="openshift-dns/dns-default-nzbnl"
Nov 01 03:06:34.825496 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.825472    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-hrmfk\""
Nov 01 03:06:34.825551 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.825536    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-httfj\" (UniqueName: \"kubernetes.io/projected/73b27ccb-7d7a-40c1-8479-8379b4fb50ed-kube-api-access-httfj\") pod \"insights-runtime-extractor-6vv92\" (UID: \"73b27ccb-7d7a-40c1-8479-8379b4fb50ed\") " pod="openshift-insights/insights-runtime-extractor-6vv92"
Nov 01 03:06:34.825600 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.825558    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-j68cz\""
Nov 01 03:06:34.825654 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.825575    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gljgq\" (UniqueName: \"kubernetes.io/projected/c9e89456-9acd-408f-87b5-f5fd968c82c3-kube-api-access-gljgq\") pod \"iptables-alerter-wfg7k\" (UID: \"c9e89456-9acd-408f-87b5-f5fd968c82c3\") " pod="openshift-network-operator/iptables-alerter-wfg7k"
Nov 01 03:06:34.825654 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.825606    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-kkkrb\""
Nov 01 03:06:34.825654 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.825628    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 01 03:06:34.825654 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.825631    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-z9gxp\""
Nov 01 03:06:34.825654 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.825652    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 01 03:06:34.825861 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.825606    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxqx8\" (UniqueName: \"kubernetes.io/projected/6e302b40-9c43-4822-8c8b-4d021a0590ab-kube-api-access-nxqx8\") pod \"network-check-target-mx48d\" (UID: \"6e302b40-9c43-4822-8c8b-4d021a0590ab\") " pod="openshift-network-diagnostics/network-check-target-mx48d"
Nov 01 03:06:34.825861 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.825710    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 01 03:06:34.825861 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.825750    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-ksxvj\""
Nov 01 03:06:34.825966 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.825953    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 01 03:06:34.827107 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.827084    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68932dfe-7e25-4f1b-b485-77c0519203db-service-ca-bundle\") pod \"periodic-gathering-5rxlb-bg459\" (UID: \"68932dfe-7e25-4f1b-b485-77c0519203db\") " pod="openshift-insights/periodic-gathering-5rxlb-bg459"
Nov 01 03:06:34.827107 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.827098    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/de5f469f-1029-4161-bf51-6ad18a3ac9e2-ovn-node-metrics-cert\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:06:34.827258 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.827227    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/56aed8ff-9b41-400a-90fb-8c283ce0c820-config-volume\") pod \"dns-default-nzbnl\" (UID: \"56aed8ff-9b41-400a-90fb-8c283ce0c820\") " pod="openshift-dns/dns-default-nzbnl"
Nov 01 03:06:34.827308 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.827277    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/de5f469f-1029-4161-bf51-6ad18a3ac9e2-ovnkube-script-lib\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:06:34.827370 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.827332    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-bt25w\""
Nov 01 03:06:34.827370 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.827358    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-dg4zg\""
Nov 01 03:06:34.827464 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.827397    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85r94\" (UniqueName: \"kubernetes.io/projected/e8f2a24b-e770-4615-a2c7-b2d065e14f77-kube-api-access-85r94\") pod \"multus-additional-cni-plugins-qnrf8\" (UID: \"e8f2a24b-e770-4615-a2c7-b2d065e14f77\") " pod="openshift-multus/multus-additional-cni-plugins-qnrf8"
Nov 01 03:06:34.827464 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.827456    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e8f2a24b-e770-4615-a2c7-b2d065e14f77-cni-binary-copy\") pod \"multus-additional-cni-plugins-qnrf8\" (UID: \"e8f2a24b-e770-4615-a2c7-b2d065e14f77\") " pod="openshift-multus/multus-additional-cni-plugins-qnrf8"
Nov 01 03:06:34.828475 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.828447    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0ede655d-c20d-41cc-b5f0-2db4338c1ba7-multus-daemon-config\") pod \"multus-qzl9n\" (UID: \"0ede655d-c20d-41cc-b5f0-2db4338c1ba7\") " pod="openshift-multus/multus-qzl9n"
Nov 01 03:06:34.828475 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.828449    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e8f2a24b-e770-4615-a2c7-b2d065e14f77-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qnrf8\" (UID: \"e8f2a24b-e770-4615-a2c7-b2d065e14f77\") " pod="openshift-multus/multus-additional-cni-plugins-qnrf8"
Nov 01 03:06:34.828574 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.828484    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d7cef651-3047-4e64-ae7f-b0fa71fbd0fe-serviceca\") pod \"node-ca-wq27m\" (UID: \"d7cef651-3047-4e64-ae7f-b0fa71fbd0fe\") " pod="openshift-image-registry/node-ca-wq27m"
Nov 01 03:06:34.828574 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.828543    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c9e89456-9acd-408f-87b5-f5fd968c82c3-iptables-alerter-script\") pod \"iptables-alerter-wfg7k\" (UID: \"c9e89456-9acd-408f-87b5-f5fd968c82c3\") " pod="openshift-network-operator/iptables-alerter-wfg7k"
Nov 01 03:06:34.828838 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.828820    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slnpd\" (UniqueName: \"kubernetes.io/projected/0ede655d-c20d-41cc-b5f0-2db4338c1ba7-kube-api-access-slnpd\") pod \"multus-qzl9n\" (UID: \"0ede655d-c20d-41cc-b5f0-2db4338c1ba7\") " pod="openshift-multus/multus-qzl9n"
Nov 01 03:06:34.828875 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.828827    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2krw\" (UniqueName: \"kubernetes.io/projected/e48a634f-ab31-435c-96c1-c3633c5bc061-kube-api-access-b2krw\") pod \"network-metrics-daemon-dnqjp\" (UID: \"e48a634f-ab31-435c-96c1-c3633c5bc061\") " pod="openshift-multus/network-metrics-daemon-dnqjp"
Nov 01 03:06:34.828965 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.828951    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4c1dc667-d774-4db6-b221-f4e7de423260-cert\") pod \"ingress-canary-h6696\" (UID: \"4c1dc667-d774-4db6-b221-f4e7de423260\") " pod="openshift-ingress-canary/ingress-canary-h6696"
Nov 01 03:06:34.829035 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.828964    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/de5f469f-1029-4161-bf51-6ad18a3ac9e2-ovnkube-config\") pod \"ovnkube-node-bcqk9\" (UID: \"de5f469f-1029-4161-bf51-6ad18a3ac9e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:06:34.829084 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.829060    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0ede655d-c20d-41cc-b5f0-2db4338c1ba7-cni-binary-copy\") pod \"multus-qzl9n\" (UID: \"0ede655d-c20d-41cc-b5f0-2db4338c1ba7\") " pod="openshift-multus/multus-qzl9n"
Nov 01 03:06:34.829217 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.829202    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e8f2a24b-e770-4615-a2c7-b2d065e14f77-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qnrf8\" (UID: \"e8f2a24b-e770-4615-a2c7-b2d065e14f77\") " pod="openshift-multus/multus-additional-cni-plugins-qnrf8"
Nov 01 03:06:34.829508 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.829489    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhdnz\" (UniqueName: \"kubernetes.io/projected/bf9dac1d-d9fc-455a-be03-ea34f8838730-kube-api-access-jhdnz\") pod \"aws-ebs-csi-driver-node-vq4dj\" (UID: \"bf9dac1d-d9fc-455a-be03-ea34f8838730\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vq4dj"
Nov 01 03:06:34.830630 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:06:34.830617    4400 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 01 03:06:34.830707 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:06:34.830659    4400 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e48a634f-ab31-435c-96c1-c3633c5bc061-metrics-certs podName:e48a634f-ab31-435c-96c1-c3633c5bc061 nodeName:}" failed. No retries permitted until 2026-11-01 03:06:35.330648327 +0000 UTC m=+1.845294201 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e48a634f-ab31-435c-96c1-c3633c5bc061-metrics-certs") pod "network-metrics-daemon-dnqjp" (UID: "e48a634f-ab31-435c-96c1-c3633c5bc061") : secret "metrics-daemon-secret" not found
Nov 01 03:06:34.830782 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:06:34.830715    4400 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 03:06:34.830782 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:06:34.830764    4400 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73b27ccb-7d7a-40c1-8479-8379b4fb50ed-insights-runtime-extractor-tls podName:73b27ccb-7d7a-40c1-8479-8379b4fb50ed nodeName:}" failed. No retries permitted until 2026-11-01 03:06:35.330748709 +0000 UTC m=+1.845394565 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/73b27ccb-7d7a-40c1-8479-8379b4fb50ed-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6vv92" (UID: "73b27ccb-7d7a-40c1-8479-8379b4fb50ed") : secret "insights-runtime-extractor-tls" not found
Nov 01 03:06:34.868888 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.868862    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 01 03:06:34.868958 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.868883    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-4hc7l\""
Nov 01 03:06:34.868958 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.868898    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-vzcml\""
Nov 01 03:06:34.869962 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.869934    4400 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 01 03:06:34.872062 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.872045    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svdzl\" (UniqueName: \"kubernetes.io/projected/2779bdf3-1495-4083-84a0-58846a5f6e58-kube-api-access-svdzl\") pod \"migrator-66df5d496b-g8ql6\" (UID: \"2779bdf3-1495-4083-84a0-58846a5f6e58\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-g8ql6"
Nov 01 03:06:34.878391 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.878374    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68932dfe-7e25-4f1b-b485-77c0519203db-trusted-ca-bundle\") pod \"periodic-gathering-5rxlb-bg459\" (UID: \"68932dfe-7e25-4f1b-b485-77c0519203db\") " pod="openshift-insights/periodic-gathering-5rxlb-bg459"
Nov 01 03:06:34.971839 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:34.971825    4400 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 03:06:35.035966 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:35.035948    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-g8ql6"
Nov 01 03:06:35.395074 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:35.394986    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e48a634f-ab31-435c-96c1-c3633c5bc061-metrics-certs\") pod \"network-metrics-daemon-dnqjp\" (UID: \"e48a634f-ab31-435c-96c1-c3633c5bc061\") " pod="openshift-multus/network-metrics-daemon-dnqjp"
Nov 01 03:06:35.395074 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:35.395027    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/73b27ccb-7d7a-40c1-8479-8379b4fb50ed-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6vv92\" (UID: \"73b27ccb-7d7a-40c1-8479-8379b4fb50ed\") " pod="openshift-insights/insights-runtime-extractor-6vv92"
Nov 01 03:06:35.395074 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:35.395057    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/56aed8ff-9b41-400a-90fb-8c283ce0c820-metrics-tls\") pod \"dns-default-nzbnl\" (UID: \"56aed8ff-9b41-400a-90fb-8c283ce0c820\") " pod="openshift-dns/dns-default-nzbnl"
Nov 01 03:06:35.395327 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:06:35.395129    4400 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 03:06:35.395327 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:06:35.395145    4400 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 03:06:35.395327 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:35.395155    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/73b27ccb-7d7a-40c1-8479-8379b4fb50ed-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6vv92\" (UID: \"73b27ccb-7d7a-40c1-8479-8379b4fb50ed\") " pod="openshift-insights/insights-runtime-extractor-6vv92"
Nov 01 03:06:35.395327 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:06:35.395133    4400 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 01 03:06:35.395327 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:06:35.395189    4400 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73b27ccb-7d7a-40c1-8479-8379b4fb50ed-insights-runtime-extractor-tls podName:73b27ccb-7d7a-40c1-8479-8379b4fb50ed nodeName:}" failed. No retries permitted until 2026-11-01 03:06:36.395174385 +0000 UTC m=+2.909820225 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/73b27ccb-7d7a-40c1-8479-8379b4fb50ed-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6vv92" (UID: "73b27ccb-7d7a-40c1-8479-8379b4fb50ed") : secret "insights-runtime-extractor-tls" not found
Nov 01 03:06:35.395327 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:06:35.395205    4400 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 01 03:06:35.395327 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:06:35.395224    4400 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56aed8ff-9b41-400a-90fb-8c283ce0c820-metrics-tls podName:56aed8ff-9b41-400a-90fb-8c283ce0c820 nodeName:}" failed. No retries permitted until 2026-11-01 03:06:36.395218429 +0000 UTC m=+2.909864270 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/56aed8ff-9b41-400a-90fb-8c283ce0c820-metrics-tls") pod "dns-default-nzbnl" (UID: "56aed8ff-9b41-400a-90fb-8c283ce0c820") : secret "dns-default-metrics-tls" not found
Nov 01 03:06:35.395327 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:06:35.395235    4400 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e48a634f-ab31-435c-96c1-c3633c5bc061-metrics-certs podName:e48a634f-ab31-435c-96c1-c3633c5bc061 nodeName:}" failed. No retries permitted until 2026-11-01 03:06:36.395229646 +0000 UTC m=+2.909875487 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e48a634f-ab31-435c-96c1-c3633c5bc061-metrics-certs") pod "network-metrics-daemon-dnqjp" (UID: "e48a634f-ab31-435c-96c1-c3633c5bc061") : secret "metrics-daemon-secret" not found
Nov 01 03:06:35.395327 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:06:35.395254    4400 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/73b27ccb-7d7a-40c1-8479-8379b4fb50ed-kube-rbac-proxy-cm podName:73b27ccb-7d7a-40c1-8479-8379b4fb50ed nodeName:}" failed. No retries permitted until 2026-11-01 03:06:36.395244138 +0000 UTC m=+2.909889980 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/73b27ccb-7d7a-40c1-8479-8379b4fb50ed-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6vv92" (UID: "73b27ccb-7d7a-40c1-8479-8379b4fb50ed") : configmap "kube-rbac-proxy" not found
Nov 01 03:06:35.429224 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:35.429199    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-g8ql6"]
Nov 01 03:06:35.432668 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:35.432637    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2779bdf3_1495_4083_84a0_58846a5f6e58.slice/crio-9ea9aa15b775073208f5f6dfcb5d76e7ad4342ec843bf68cff1b068706be7d28 WatchSource:0}: Error finding container 9ea9aa15b775073208f5f6dfcb5d76e7ad4342ec843bf68cff1b068706be7d28: Status 404 returned error can't find the container with id 9ea9aa15b775073208f5f6dfcb5d76e7ad4342ec843bf68cff1b068706be7d28
Nov 01 03:06:35.654904 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:35.654835    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-g8ql6" event={"ID":"2779bdf3-1495-4083-84a0-58846a5f6e58","Type":"ContainerStarted","Data":"9ea9aa15b775073208f5f6dfcb5d76e7ad4342ec843bf68cff1b068706be7d28"}
Nov 01 03:06:36.311003 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:36.310943    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:06:36.402703 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:36.402647    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e48a634f-ab31-435c-96c1-c3633c5bc061-metrics-certs\") pod \"network-metrics-daemon-dnqjp\" (UID: \"e48a634f-ab31-435c-96c1-c3633c5bc061\") " pod="openshift-multus/network-metrics-daemon-dnqjp"
Nov 01 03:06:36.402912 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:36.402713    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/73b27ccb-7d7a-40c1-8479-8379b4fb50ed-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6vv92\" (UID: \"73b27ccb-7d7a-40c1-8479-8379b4fb50ed\") " pod="openshift-insights/insights-runtime-extractor-6vv92"
Nov 01 03:06:36.402912 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:36.402745    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/56aed8ff-9b41-400a-90fb-8c283ce0c820-metrics-tls\") pod \"dns-default-nzbnl\" (UID: \"56aed8ff-9b41-400a-90fb-8c283ce0c820\") " pod="openshift-dns/dns-default-nzbnl"
Nov 01 03:06:36.402912 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:36.402812    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/73b27ccb-7d7a-40c1-8479-8379b4fb50ed-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6vv92\" (UID: \"73b27ccb-7d7a-40c1-8479-8379b4fb50ed\") " pod="openshift-insights/insights-runtime-extractor-6vv92"
Nov 01 03:06:36.402912 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:06:36.402896    4400 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 03:06:36.402912 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:06:36.402906    4400 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 03:06:36.403202 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:06:36.402979    4400 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73b27ccb-7d7a-40c1-8479-8379b4fb50ed-insights-runtime-extractor-tls podName:73b27ccb-7d7a-40c1-8479-8379b4fb50ed nodeName:}" failed. No retries permitted until 2026-11-01 03:06:38.402943954 +0000 UTC m=+4.917589808 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/73b27ccb-7d7a-40c1-8479-8379b4fb50ed-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6vv92" (UID: "73b27ccb-7d7a-40c1-8479-8379b4fb50ed") : secret "insights-runtime-extractor-tls" not found
Nov 01 03:06:36.403202 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:06:36.403002    4400 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56aed8ff-9b41-400a-90fb-8c283ce0c820-metrics-tls podName:56aed8ff-9b41-400a-90fb-8c283ce0c820 nodeName:}" failed. No retries permitted until 2026-11-01 03:06:38.402990381 +0000 UTC m=+4.917636224 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/56aed8ff-9b41-400a-90fb-8c283ce0c820-metrics-tls") pod "dns-default-nzbnl" (UID: "56aed8ff-9b41-400a-90fb-8c283ce0c820") : secret "dns-default-metrics-tls" not found
Nov 01 03:06:36.403400 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:36.403378    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/73b27ccb-7d7a-40c1-8479-8379b4fb50ed-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6vv92\" (UID: \"73b27ccb-7d7a-40c1-8479-8379b4fb50ed\") " pod="openshift-insights/insights-runtime-extractor-6vv92"
Nov 01 03:06:36.403547 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:06:36.403448    4400 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 01 03:06:36.403547 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:06:36.403504    4400 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e48a634f-ab31-435c-96c1-c3633c5bc061-metrics-certs podName:e48a634f-ab31-435c-96c1-c3633c5bc061 nodeName:}" failed. No retries permitted until 2026-11-01 03:06:38.403491155 +0000 UTC m=+4.918136997 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e48a634f-ab31-435c-96c1-c3633c5bc061-metrics-certs") pod "network-metrics-daemon-dnqjp" (UID: "e48a634f-ab31-435c-96c1-c3633c5bc061") : secret "metrics-daemon-secret" not found
Nov 01 03:06:37.660698 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:37.660664    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-g8ql6" event={"ID":"2779bdf3-1495-4083-84a0-58846a5f6e58","Type":"ContainerStarted","Data":"c035a4fb47cb73c0d7d47e8dca3d0c94370bf68c4c12237f9298f2ff199ef477"}
Nov 01 03:06:37.662192 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:37.662170    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-5rxlb-bg459" event={"ID":"68932dfe-7e25-4f1b-b485-77c0519203db","Type":"ContainerStarted","Data":"385c530c8bbd675abad47bac2c40e047c2996e0cc08c224c6ce011411a6853c0"}
Nov 01 03:06:37.754913 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:37.754881    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-mx48d"
Nov 01 03:06:37.777615 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:37.777536    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-g8ql6" podStartSLOduration=2.980485301 podStartE2EDuration="4.777524533s" totalImagesPullingTime="1.797039232s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:06:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:06:37.777415169 +0000 UTC m=+4.292061029" watchObservedRunningTime="2026-11-01 03:06:37.777524533 +0000 UTC m=+4.292170390"
Nov 01 03:06:38.416543 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:38.415743    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e48a634f-ab31-435c-96c1-c3633c5bc061-metrics-certs\") pod \"network-metrics-daemon-dnqjp\" (UID: \"e48a634f-ab31-435c-96c1-c3633c5bc061\") " pod="openshift-multus/network-metrics-daemon-dnqjp"
Nov 01 03:06:38.416543 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:38.415980    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/73b27ccb-7d7a-40c1-8479-8379b4fb50ed-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6vv92\" (UID: \"73b27ccb-7d7a-40c1-8479-8379b4fb50ed\") " pod="openshift-insights/insights-runtime-extractor-6vv92"
Nov 01 03:06:38.416543 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:06:38.415905    4400 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 01 03:06:38.416543 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:06:38.416099    4400 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e48a634f-ab31-435c-96c1-c3633c5bc061-metrics-certs podName:e48a634f-ab31-435c-96c1-c3633c5bc061 nodeName:}" failed. No retries permitted until 2026-11-01 03:06:42.416056972 +0000 UTC m=+8.930702830 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e48a634f-ab31-435c-96c1-c3633c5bc061-metrics-certs") pod "network-metrics-daemon-dnqjp" (UID: "e48a634f-ab31-435c-96c1-c3633c5bc061") : secret "metrics-daemon-secret" not found
Nov 01 03:06:38.416543 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:06:38.416278    4400 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 03:06:38.416543 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:06:38.416352    4400 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73b27ccb-7d7a-40c1-8479-8379b4fb50ed-insights-runtime-extractor-tls podName:73b27ccb-7d7a-40c1-8479-8379b4fb50ed nodeName:}" failed. No retries permitted until 2026-11-01 03:06:42.416327989 +0000 UTC m=+8.930973845 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/73b27ccb-7d7a-40c1-8479-8379b4fb50ed-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6vv92" (UID: "73b27ccb-7d7a-40c1-8479-8379b4fb50ed") : secret "insights-runtime-extractor-tls" not found
Nov 01 03:06:38.416916 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:38.416592    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/56aed8ff-9b41-400a-90fb-8c283ce0c820-metrics-tls\") pod \"dns-default-nzbnl\" (UID: \"56aed8ff-9b41-400a-90fb-8c283ce0c820\") " pod="openshift-dns/dns-default-nzbnl"
Nov 01 03:06:38.416916 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:06:38.416738    4400 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 03:06:38.416916 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:06:38.416783    4400 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56aed8ff-9b41-400a-90fb-8c283ce0c820-metrics-tls podName:56aed8ff-9b41-400a-90fb-8c283ce0c820 nodeName:}" failed. No retries permitted until 2026-11-01 03:06:42.416769635 +0000 UTC m=+8.931415490 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/56aed8ff-9b41-400a-90fb-8c283ce0c820-metrics-tls") pod "dns-default-nzbnl" (UID: "56aed8ff-9b41-400a-90fb-8c283ce0c820") : secret "dns-default-metrics-tls" not found
Nov 01 03:06:39.899826 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:39.899797    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:06:39.900240 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:39.899958    4400 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 01 03:06:41.222913 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:41.222883    4400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-pzt6v_994175bb-d192-47f7-b162-a840eb0f9be1/dns-node-resolver/0.log"
Nov 01 03:06:41.456890 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:41.456862    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bcqk9"
Nov 01 03:06:42.223076 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:42.223051    4400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-wq27m_d7cef651-3047-4e64-ae7f-b0fa71fbd0fe/node-ca/0.log"
Nov 01 03:06:42.440882 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:42.440852    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e48a634f-ab31-435c-96c1-c3633c5bc061-metrics-certs\") pod \"network-metrics-daemon-dnqjp\" (UID: \"e48a634f-ab31-435c-96c1-c3633c5bc061\") " pod="openshift-multus/network-metrics-daemon-dnqjp"
Nov 01 03:06:42.440882 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:42.440882    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/73b27ccb-7d7a-40c1-8479-8379b4fb50ed-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6vv92\" (UID: \"73b27ccb-7d7a-40c1-8479-8379b4fb50ed\") " pod="openshift-insights/insights-runtime-extractor-6vv92"
Nov 01 03:06:42.441057 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:42.440902    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/56aed8ff-9b41-400a-90fb-8c283ce0c820-metrics-tls\") pod \"dns-default-nzbnl\" (UID: \"56aed8ff-9b41-400a-90fb-8c283ce0c820\") " pod="openshift-dns/dns-default-nzbnl"
Nov 01 03:06:42.441057 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:06:42.440997    4400 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 01 03:06:42.441057 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:06:42.441056    4400 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e48a634f-ab31-435c-96c1-c3633c5bc061-metrics-certs podName:e48a634f-ab31-435c-96c1-c3633c5bc061 nodeName:}" failed. No retries permitted until 2026-11-01 03:06:50.441043382 +0000 UTC m=+16.955689237 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e48a634f-ab31-435c-96c1-c3633c5bc061-metrics-certs") pod "network-metrics-daemon-dnqjp" (UID: "e48a634f-ab31-435c-96c1-c3633c5bc061") : secret "metrics-daemon-secret" not found
Nov 01 03:06:42.441167 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:06:42.441002    4400 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 03:06:42.441167 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:06:42.441148    4400 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73b27ccb-7d7a-40c1-8479-8379b4fb50ed-insights-runtime-extractor-tls podName:73b27ccb-7d7a-40c1-8479-8379b4fb50ed nodeName:}" failed. No retries permitted until 2026-11-01 03:06:50.441129892 +0000 UTC m=+16.955775735 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/73b27ccb-7d7a-40c1-8479-8379b4fb50ed-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6vv92" (UID: "73b27ccb-7d7a-40c1-8479-8379b4fb50ed") : secret "insights-runtime-extractor-tls" not found
Nov 01 03:06:42.443280 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:42.443257    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/56aed8ff-9b41-400a-90fb-8c283ce0c820-metrics-tls\") pod \"dns-default-nzbnl\" (UID: \"56aed8ff-9b41-400a-90fb-8c283ce0c820\") " pod="openshift-dns/dns-default-nzbnl"
Nov 01 03:06:42.471395 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:42.471369    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nzbnl"
Nov 01 03:06:42.621810 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:42.621785    4400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-h6696_4c1dc667-d774-4db6-b221-f4e7de423260/serve-healthcheck-canary/0.log"
Nov 01 03:06:42.840819 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:42.840793    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-nzbnl"]
Nov 01 03:06:42.843531 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:42.843499    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56aed8ff_9b41_400a_90fb_8c283ce0c820.slice/crio-25caa4cf033e4af862e1950328e7826d5e0020bd44176250193feacb3fd9700c WatchSource:0}: Error finding container 25caa4cf033e4af862e1950328e7826d5e0020bd44176250193feacb3fd9700c: Status 404 returned error can't find the container with id 25caa4cf033e4af862e1950328e7826d5e0020bd44176250193feacb3fd9700c
Nov 01 03:06:43.221787 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:43.221731    4400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-g8ql6_2779bdf3-1495-4083-84a0-58846a5f6e58/migrator/0.log"
Nov 01 03:06:43.675841 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:43.675809    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nzbnl" event={"ID":"56aed8ff-9b41-400a-90fb-8c283ce0c820","Type":"ContainerStarted","Data":"25caa4cf033e4af862e1950328e7826d5e0020bd44176250193feacb3fd9700c"}
Nov 01 03:06:44.055224 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:44.055163    4400 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-54.us-west-1.compute.internal" event="NodeReady"
Nov 01 03:06:45.685423 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:45.685389    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nzbnl" event={"ID":"56aed8ff-9b41-400a-90fb-8c283ce0c820","Type":"ContainerStarted","Data":"54dfbdc58ba2999accf65e5ff3e0651e3363fb65a1790cdee79e91a5cfa0a753"}
Nov 01 03:06:45.685811 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:45.685462    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nzbnl" event={"ID":"56aed8ff-9b41-400a-90fb-8c283ce0c820","Type":"ContainerStarted","Data":"336c2c590f0765334381da4bdc89ae40a2d3bcba2fb8d34e11d755d92ebfd4c0"}
Nov 01 03:06:45.685811 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:45.685534    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-nzbnl"
Nov 01 03:06:47.558677 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:47.558652    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-fh7sf"
Nov 01 03:06:50.499697 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:50.499657    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e48a634f-ab31-435c-96c1-c3633c5bc061-metrics-certs\") pod \"network-metrics-daemon-dnqjp\" (UID: \"e48a634f-ab31-435c-96c1-c3633c5bc061\") " pod="openshift-multus/network-metrics-daemon-dnqjp"
Nov 01 03:06:50.500069 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:50.499715    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/73b27ccb-7d7a-40c1-8479-8379b4fb50ed-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6vv92\" (UID: \"73b27ccb-7d7a-40c1-8479-8379b4fb50ed\") " pod="openshift-insights/insights-runtime-extractor-6vv92"
Nov 01 03:06:50.502183 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:50.502162    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/73b27ccb-7d7a-40c1-8479-8379b4fb50ed-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6vv92\" (UID: \"73b27ccb-7d7a-40c1-8479-8379b4fb50ed\") " pod="openshift-insights/insights-runtime-extractor-6vv92"
Nov 01 03:06:50.502260 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:50.502162    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e48a634f-ab31-435c-96c1-c3633c5bc061-metrics-certs\") pod \"network-metrics-daemon-dnqjp\" (UID: \"e48a634f-ab31-435c-96c1-c3633c5bc061\") " pod="openshift-multus/network-metrics-daemon-dnqjp"
Nov 01 03:06:50.566456 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:50.566435    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dnqjp"
Nov 01 03:06:50.571519 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:50.571499    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6vv92"
Nov 01 03:06:50.936455 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:50.936414    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dnqjp"]
Nov 01 03:06:50.974036 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:50.974009    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode48a634f_ab31_435c_96c1_c3633c5bc061.slice/crio-23e4e4a35610ad5f49b631c86254f875b25bfe63dad0f1cc837f73f034afb381 WatchSource:0}: Error finding container 23e4e4a35610ad5f49b631c86254f875b25bfe63dad0f1cc837f73f034afb381: Status 404 returned error can't find the container with id 23e4e4a35610ad5f49b631c86254f875b25bfe63dad0f1cc837f73f034afb381
Nov 01 03:06:50.987263 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:50.987246    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6vv92"]
Nov 01 03:06:50.989277 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:06:50.989255    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73b27ccb_7d7a_40c1_8479_8379b4fb50ed.slice/crio-1615a7dd4f5359de9129e9a21ffb06d77ad625ad542d3c4514cb28cfa4025453 WatchSource:0}: Error finding container 1615a7dd4f5359de9129e9a21ffb06d77ad625ad542d3c4514cb28cfa4025453: Status 404 returned error can't find the container with id 1615a7dd4f5359de9129e9a21ffb06d77ad625ad542d3c4514cb28cfa4025453
Nov 01 03:06:51.707827 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:51.707799    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6vv92" event={"ID":"73b27ccb-7d7a-40c1-8479-8379b4fb50ed","Type":"ContainerStarted","Data":"bf85ff1c3d59b2889a53591bf1cfaf76bdaa2093d56cb7de1adfe535292c91fb"}
Nov 01 03:06:51.708197 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:51.707858    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6vv92" event={"ID":"73b27ccb-7d7a-40c1-8479-8379b4fb50ed","Type":"ContainerStarted","Data":"1615a7dd4f5359de9129e9a21ffb06d77ad625ad542d3c4514cb28cfa4025453"}
Nov 01 03:06:51.708872 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:51.708844    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dnqjp" event={"ID":"e48a634f-ab31-435c-96c1-c3633c5bc061","Type":"ContainerStarted","Data":"23e4e4a35610ad5f49b631c86254f875b25bfe63dad0f1cc837f73f034afb381"}
Nov 01 03:06:52.716287 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:52.716256    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6vv92" event={"ID":"73b27ccb-7d7a-40c1-8479-8379b4fb50ed","Type":"ContainerStarted","Data":"3b77ec3152f7b000dea91f79df03b7f3b5de06714243fecdf57480f8dd27b8e0"}
Nov 01 03:06:52.718402 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:52.718381    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dnqjp" event={"ID":"e48a634f-ab31-435c-96c1-c3633c5bc061","Type":"ContainerStarted","Data":"dd8f1c56c648315154dcc4f88344e70ef6e26f936e5449c8c13290c7730d65f2"}
Nov 01 03:06:52.718502 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:52.718420    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dnqjp" event={"ID":"e48a634f-ab31-435c-96c1-c3633c5bc061","Type":"ContainerStarted","Data":"318248b5bf6c219b8d0d98733528938e90f0d2bb41d119601458bae90408e801"}
Nov 01 03:06:54.727247 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:54.727170    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6vv92" event={"ID":"73b27ccb-7d7a-40c1-8479-8379b4fb50ed","Type":"ContainerStarted","Data":"3b4e088a9fd49e3848c9c44e8a9067b5bd9c62117a25e8d912eef57b4c560a59"}
Nov 01 03:06:55.690705 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:06:55.690663    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-nzbnl"
Nov 01 03:07:01.728050 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:07:01.728012    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:07:07.757235 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:07.757207    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-mx48d"
Nov 01 03:07:08.713181 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:08.713147    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-ssff4"]
Nov 01 03:07:08.735430 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:08.735408    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-ssff4"
Nov 01 03:07:08.776508 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:08.776484    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-ssff4"]
Nov 01 03:07:08.814854 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:08.814834    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cb9062e4-c82c-473d-9243-aea4fa1fa169-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-ssff4\" (UID: \"cb9062e4-c82c-473d-9243-aea4fa1fa169\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ssff4"
Nov 01 03:07:08.814955 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:08.814867    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/cb9062e4-c82c-473d-9243-aea4fa1fa169-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-ssff4\" (UID: \"cb9062e4-c82c-473d-9243-aea4fa1fa169\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ssff4"
Nov 01 03:07:08.814955 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:08.814886    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkwcn\" (UniqueName: \"kubernetes.io/projected/cb9062e4-c82c-473d-9243-aea4fa1fa169-kube-api-access-bkwcn\") pod \"prometheus-operator-9cd89b74c-ssff4\" (UID: \"cb9062e4-c82c-473d-9243-aea4fa1fa169\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ssff4"
Nov 01 03:07:08.814955 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:08.814908    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cb9062e4-c82c-473d-9243-aea4fa1fa169-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-ssff4\" (UID: \"cb9062e4-c82c-473d-9243-aea4fa1fa169\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ssff4"
Nov 01 03:07:08.826757 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:08.826737    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Nov 01 03:07:08.826830 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:08.826781    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-9twgg\""
Nov 01 03:07:08.840296 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:08.840279    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Nov 01 03:07:08.840512 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:08.840494    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 01 03:07:08.877669 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:08.877650    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 01 03:07:08.877757 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:08.877739    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 01 03:07:08.915647 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:08.915629    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/cb9062e4-c82c-473d-9243-aea4fa1fa169-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-ssff4\" (UID: \"cb9062e4-c82c-473d-9243-aea4fa1fa169\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ssff4"
Nov 01 03:07:08.915762 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:08.915656    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkwcn\" (UniqueName: \"kubernetes.io/projected/cb9062e4-c82c-473d-9243-aea4fa1fa169-kube-api-access-bkwcn\") pod \"prometheus-operator-9cd89b74c-ssff4\" (UID: \"cb9062e4-c82c-473d-9243-aea4fa1fa169\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ssff4"
Nov 01 03:07:08.915762 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:08.915679    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cb9062e4-c82c-473d-9243-aea4fa1fa169-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-ssff4\" (UID: \"cb9062e4-c82c-473d-9243-aea4fa1fa169\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ssff4"
Nov 01 03:07:08.915762 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:08.915746    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cb9062e4-c82c-473d-9243-aea4fa1fa169-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-ssff4\" (UID: \"cb9062e4-c82c-473d-9243-aea4fa1fa169\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ssff4"
Nov 01 03:07:08.916354 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:08.916336    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cb9062e4-c82c-473d-9243-aea4fa1fa169-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-ssff4\" (UID: \"cb9062e4-c82c-473d-9243-aea4fa1fa169\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ssff4"
Nov 01 03:07:08.918140 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:08.918119    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/cb9062e4-c82c-473d-9243-aea4fa1fa169-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-ssff4\" (UID: \"cb9062e4-c82c-473d-9243-aea4fa1fa169\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ssff4"
Nov 01 03:07:08.918212 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:08.918122    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cb9062e4-c82c-473d-9243-aea4fa1fa169-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-ssff4\" (UID: \"cb9062e4-c82c-473d-9243-aea4fa1fa169\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ssff4"
Nov 01 03:07:08.978627 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:08.978580    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkwcn\" (UniqueName: \"kubernetes.io/projected/cb9062e4-c82c-473d-9243-aea4fa1fa169-kube-api-access-bkwcn\") pod \"prometheus-operator-9cd89b74c-ssff4\" (UID: \"cb9062e4-c82c-473d-9243-aea4fa1fa169\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ssff4"
Nov 01 03:07:09.073972 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:09.073950    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-ssff4"
Nov 01 03:07:09.458521 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:09.458500    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-ssff4"]
Nov 01 03:07:09.459984 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:07:09.459962    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb9062e4_c82c_473d_9243_aea4fa1fa169.slice/crio-613d7096ee2fdf9763fb6dc640ffd7455562cef250d4af8020e70b270932ceab WatchSource:0}: Error finding container 613d7096ee2fdf9763fb6dc640ffd7455562cef250d4af8020e70b270932ceab: Status 404 returned error can't find the container with id 613d7096ee2fdf9763fb6dc640ffd7455562cef250d4af8020e70b270932ceab
Nov 01 03:07:09.775789 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:09.775706    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-ssff4" event={"ID":"cb9062e4-c82c-473d-9243-aea4fa1fa169","Type":"ContainerStarted","Data":"613d7096ee2fdf9763fb6dc640ffd7455562cef250d4af8020e70b270932ceab"}
Nov 01 03:07:11.783506 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:11.783472    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-ssff4" event={"ID":"cb9062e4-c82c-473d-9243-aea4fa1fa169","Type":"ContainerStarted","Data":"1d23fd34db2f9d3b8524448dab4a0e195ec8970565449e0ce79c376b69cc45b0"}
Nov 01 03:07:11.783984 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:11.783521    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-ssff4" event={"ID":"cb9062e4-c82c-473d-9243-aea4fa1fa169","Type":"ContainerStarted","Data":"c2d2d4a493692468990c162e833c88172a3dc099b8fac50d9ba27cbe89015ace"}
Nov 01 03:07:11.911450 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:11.911407    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-ssff4" podStartSLOduration=2.136370891 podStartE2EDuration="3.911394702s" totalImagesPullingTime="1.775023811s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:07:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:07:11.911130676 +0000 UTC m=+38.425776541" watchObservedRunningTime="2026-11-01 03:07:11.911394702 +0000 UTC m=+38.426040565"
Nov 01 03:07:14.386854 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:14.386781    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-6cf7cdf566-2f2jh"]
Nov 01 03:07:14.388415 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:14.388400    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cf7cdf566-2f2jh"
Nov 01 03:07:14.447560 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:14.447537    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cd688da6-42b3-49ed-96ba-1ae5c6aa3933-console-serving-cert\") pod \"console-6cf7cdf566-2f2jh\" (UID: \"cd688da6-42b3-49ed-96ba-1ae5c6aa3933\") " pod="openshift-console/console-6cf7cdf566-2f2jh"
Nov 01 03:07:14.447675 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:14.447564    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cd688da6-42b3-49ed-96ba-1ae5c6aa3933-console-config\") pod \"console-6cf7cdf566-2f2jh\" (UID: \"cd688da6-42b3-49ed-96ba-1ae5c6aa3933\") " pod="openshift-console/console-6cf7cdf566-2f2jh"
Nov 01 03:07:14.447675 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:14.447581    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cd688da6-42b3-49ed-96ba-1ae5c6aa3933-oauth-serving-cert\") pod \"console-6cf7cdf566-2f2jh\" (UID: \"cd688da6-42b3-49ed-96ba-1ae5c6aa3933\") " pod="openshift-console/console-6cf7cdf566-2f2jh"
Nov 01 03:07:14.447675 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:14.447606    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxqmf\" (UniqueName: \"kubernetes.io/projected/cd688da6-42b3-49ed-96ba-1ae5c6aa3933-kube-api-access-cxqmf\") pod \"console-6cf7cdf566-2f2jh\" (UID: \"cd688da6-42b3-49ed-96ba-1ae5c6aa3933\") " pod="openshift-console/console-6cf7cdf566-2f2jh"
Nov 01 03:07:14.447675 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:14.447656    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cd688da6-42b3-49ed-96ba-1ae5c6aa3933-service-ca\") pod \"console-6cf7cdf566-2f2jh\" (UID: \"cd688da6-42b3-49ed-96ba-1ae5c6aa3933\") " pod="openshift-console/console-6cf7cdf566-2f2jh"
Nov 01 03:07:14.447827 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:14.447789    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cd688da6-42b3-49ed-96ba-1ae5c6aa3933-tmp\") pod \"console-6cf7cdf566-2f2jh\" (UID: \"cd688da6-42b3-49ed-96ba-1ae5c6aa3933\") " pod="openshift-console/console-6cf7cdf566-2f2jh"
Nov 01 03:07:14.447827 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:14.447814    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cd688da6-42b3-49ed-96ba-1ae5c6aa3933-console-oauth-config\") pod \"console-6cf7cdf566-2f2jh\" (UID: \"cd688da6-42b3-49ed-96ba-1ae5c6aa3933\") " pod="openshift-console/console-6cf7cdf566-2f2jh"
Nov 01 03:07:14.495080 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:14.495054    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Nov 01 03:07:14.495080 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:14.495055    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Nov 01 03:07:14.495225 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:14.495087    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Nov 01 03:07:14.495225 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:14.495089    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Nov 01 03:07:14.495225 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:14.495171    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-lthkf\""
Nov 01 03:07:14.496081 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:14.496066    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6cf7cdf566-2f2jh"]
Nov 01 03:07:14.496311 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:14.496299    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Nov 01 03:07:14.496343 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:14.496303    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Nov 01 03:07:14.546084 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:14.546062    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Nov 01 03:07:14.548311 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:14.548291    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxqmf\" (UniqueName: \"kubernetes.io/projected/cd688da6-42b3-49ed-96ba-1ae5c6aa3933-kube-api-access-cxqmf\") pod \"console-6cf7cdf566-2f2jh\" (UID: \"cd688da6-42b3-49ed-96ba-1ae5c6aa3933\") " pod="openshift-console/console-6cf7cdf566-2f2jh"
Nov 01 03:07:14.548359 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:14.548324    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cd688da6-42b3-49ed-96ba-1ae5c6aa3933-service-ca\") pod \"console-6cf7cdf566-2f2jh\" (UID: \"cd688da6-42b3-49ed-96ba-1ae5c6aa3933\") " pod="openshift-console/console-6cf7cdf566-2f2jh"
Nov 01 03:07:14.548402 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:14.548360    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cd688da6-42b3-49ed-96ba-1ae5c6aa3933-tmp\") pod \"console-6cf7cdf566-2f2jh\" (UID: \"cd688da6-42b3-49ed-96ba-1ae5c6aa3933\") " pod="openshift-console/console-6cf7cdf566-2f2jh"
Nov 01 03:07:14.548484 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:14.548469    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cd688da6-42b3-49ed-96ba-1ae5c6aa3933-console-oauth-config\") pod \"console-6cf7cdf566-2f2jh\" (UID: \"cd688da6-42b3-49ed-96ba-1ae5c6aa3933\") " pod="openshift-console/console-6cf7cdf566-2f2jh"
Nov 01 03:07:14.548536 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:14.548520    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cd688da6-42b3-49ed-96ba-1ae5c6aa3933-console-serving-cert\") pod \"console-6cf7cdf566-2f2jh\" (UID: \"cd688da6-42b3-49ed-96ba-1ae5c6aa3933\") " pod="openshift-console/console-6cf7cdf566-2f2jh"
Nov 01 03:07:14.548575 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:14.548550    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cd688da6-42b3-49ed-96ba-1ae5c6aa3933-console-config\") pod \"console-6cf7cdf566-2f2jh\" (UID: \"cd688da6-42b3-49ed-96ba-1ae5c6aa3933\") " pod="openshift-console/console-6cf7cdf566-2f2jh"
Nov 01 03:07:14.548609 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:14.548572    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cd688da6-42b3-49ed-96ba-1ae5c6aa3933-oauth-serving-cert\") pod \"console-6cf7cdf566-2f2jh\" (UID: \"cd688da6-42b3-49ed-96ba-1ae5c6aa3933\") " pod="openshift-console/console-6cf7cdf566-2f2jh"
Nov 01 03:07:14.548812 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:14.548793    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cd688da6-42b3-49ed-96ba-1ae5c6aa3933-tmp\") pod \"console-6cf7cdf566-2f2jh\" (UID: \"cd688da6-42b3-49ed-96ba-1ae5c6aa3933\") " pod="openshift-console/console-6cf7cdf566-2f2jh"
Nov 01 03:07:14.549162 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:14.549141    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cd688da6-42b3-49ed-96ba-1ae5c6aa3933-console-config\") pod \"console-6cf7cdf566-2f2jh\" (UID: \"cd688da6-42b3-49ed-96ba-1ae5c6aa3933\") " pod="openshift-console/console-6cf7cdf566-2f2jh"
Nov 01 03:07:14.549242 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:14.549147    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cd688da6-42b3-49ed-96ba-1ae5c6aa3933-service-ca\") pod \"console-6cf7cdf566-2f2jh\" (UID: \"cd688da6-42b3-49ed-96ba-1ae5c6aa3933\") " pod="openshift-console/console-6cf7cdf566-2f2jh"
Nov 01 03:07:14.549242 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:14.549213    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cd688da6-42b3-49ed-96ba-1ae5c6aa3933-oauth-serving-cert\") pod \"console-6cf7cdf566-2f2jh\" (UID: \"cd688da6-42b3-49ed-96ba-1ae5c6aa3933\") " pod="openshift-console/console-6cf7cdf566-2f2jh"
Nov 01 03:07:14.551102 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:14.551082    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cd688da6-42b3-49ed-96ba-1ae5c6aa3933-console-oauth-config\") pod \"console-6cf7cdf566-2f2jh\" (UID: \"cd688da6-42b3-49ed-96ba-1ae5c6aa3933\") " pod="openshift-console/console-6cf7cdf566-2f2jh"
Nov 01 03:07:14.551180 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:14.551141    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cd688da6-42b3-49ed-96ba-1ae5c6aa3933-console-serving-cert\") pod \"console-6cf7cdf566-2f2jh\" (UID: \"cd688da6-42b3-49ed-96ba-1ae5c6aa3933\") " pod="openshift-console/console-6cf7cdf566-2f2jh"
Nov 01 03:07:14.614970 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:14.614939    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxqmf\" (UniqueName: \"kubernetes.io/projected/cd688da6-42b3-49ed-96ba-1ae5c6aa3933-kube-api-access-cxqmf\") pod \"console-6cf7cdf566-2f2jh\" (UID: \"cd688da6-42b3-49ed-96ba-1ae5c6aa3933\") " pod="openshift-console/console-6cf7cdf566-2f2jh"
Nov 01 03:07:14.700566 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:14.700507    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cf7cdf566-2f2jh"
Nov 01 03:07:15.086862 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:15.086835    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6cf7cdf566-2f2jh"]
Nov 01 03:07:15.088163 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:07:15.088140    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd688da6_42b3_49ed_96ba_1ae5c6aa3933.slice/crio-a6ffc2e5a950bd407b8d9036068673a9af8e144b1de9e98670382e89a4cb07e4 WatchSource:0}: Error finding container a6ffc2e5a950bd407b8d9036068673a9af8e144b1de9e98670382e89a4cb07e4: Status 404 returned error can't find the container with id a6ffc2e5a950bd407b8d9036068673a9af8e144b1de9e98670382e89a4cb07e4
Nov 01 03:07:15.444463 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:15.444397    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-6nx7f"]
Nov 01 03:07:15.446922 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:15.446905    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6nx7f"
Nov 01 03:07:15.513190 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:15.513166    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-6nx7f"]
Nov 01 03:07:15.519583 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:15.519556    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Nov 01 03:07:15.519817 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:15.519801    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Nov 01 03:07:15.519880 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:15.519846    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-l2wm6\""
Nov 01 03:07:15.554884 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:15.554862    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtdjw\" (UniqueName: \"kubernetes.io/projected/74fb2a00-d669-4759-a1bf-6af8425e1f71-kube-api-access-jtdjw\") pod \"openshift-state-metrics-7bd7489957-6nx7f\" (UID: \"74fb2a00-d669-4759-a1bf-6af8425e1f71\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6nx7f"
Nov 01 03:07:15.554986 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:15.554899    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/74fb2a00-d669-4759-a1bf-6af8425e1f71-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-6nx7f\" (UID: \"74fb2a00-d669-4759-a1bf-6af8425e1f71\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6nx7f"
Nov 01 03:07:15.554986 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:15.554926    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/74fb2a00-d669-4759-a1bf-6af8425e1f71-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-6nx7f\" (UID: \"74fb2a00-d669-4759-a1bf-6af8425e1f71\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6nx7f"
Nov 01 03:07:15.555060 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:15.555016    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/74fb2a00-d669-4759-a1bf-6af8425e1f71-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-6nx7f\" (UID: \"74fb2a00-d669-4759-a1bf-6af8425e1f71\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6nx7f"
Nov 01 03:07:15.611256 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:15.611230    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-ncr5j"]
Nov 01 03:07:15.613404 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:15.613388    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ncr5j"
Nov 01 03:07:15.626992 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:15.626971    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-dw9pj"]
Nov 01 03:07:15.628732 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:15.628718    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-dw9pj"
Nov 01 03:07:15.656269 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:15.656244    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtdjw\" (UniqueName: \"kubernetes.io/projected/74fb2a00-d669-4759-a1bf-6af8425e1f71-kube-api-access-jtdjw\") pod \"openshift-state-metrics-7bd7489957-6nx7f\" (UID: \"74fb2a00-d669-4759-a1bf-6af8425e1f71\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6nx7f"
Nov 01 03:07:15.656359 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:15.656279    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/74fb2a00-d669-4759-a1bf-6af8425e1f71-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-6nx7f\" (UID: \"74fb2a00-d669-4759-a1bf-6af8425e1f71\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6nx7f"
Nov 01 03:07:15.656359 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:15.656304    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/74fb2a00-d669-4759-a1bf-6af8425e1f71-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-6nx7f\" (UID: \"74fb2a00-d669-4759-a1bf-6af8425e1f71\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6nx7f"
Nov 01 03:07:15.656498 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:15.656473    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/74fb2a00-d669-4759-a1bf-6af8425e1f71-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-6nx7f\" (UID: \"74fb2a00-d669-4759-a1bf-6af8425e1f71\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6nx7f"
Nov 01 03:07:15.657055 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:15.657034    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/74fb2a00-d669-4759-a1bf-6af8425e1f71-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-6nx7f\" (UID: \"74fb2a00-d669-4759-a1bf-6af8425e1f71\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6nx7f"
Nov 01 03:07:15.658780 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:15.658754    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/74fb2a00-d669-4759-a1bf-6af8425e1f71-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-6nx7f\" (UID: \"74fb2a00-d669-4759-a1bf-6af8425e1f71\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6nx7f"
Nov 01 03:07:15.658925 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:15.658904    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/74fb2a00-d669-4759-a1bf-6af8425e1f71-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-6nx7f\" (UID: \"74fb2a00-d669-4759-a1bf-6af8425e1f71\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6nx7f"
Nov 01 03:07:15.677156 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:15.677139    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Nov 01 03:07:15.682924 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:15.682902    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-ncr5j"]
Nov 01 03:07:15.718114 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:15.718067    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtdjw\" (UniqueName: \"kubernetes.io/projected/74fb2a00-d669-4759-a1bf-6af8425e1f71-kube-api-access-jtdjw\") pod \"openshift-state-metrics-7bd7489957-6nx7f\" (UID: \"74fb2a00-d669-4759-a1bf-6af8425e1f71\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6nx7f"
Nov 01 03:07:15.724247 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:15.724229    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Nov 01 03:07:15.724401 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:15.724387    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-wvqkc\""
Nov 01 03:07:15.724632 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:15.724615    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 01 03:07:15.724792 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:15.724780    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Nov 01 03:07:15.726064 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:15.726050    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 01 03:07:15.726112 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:15.726051    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-jfdhs\""
Nov 01 03:07:15.757103 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:15.757080    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d7f2c2ca-b2a2-4a31-987a-e7049fa74f2d-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-ncr5j\" (UID: \"d7f2c2ca-b2a2-4a31-987a-e7049fa74f2d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ncr5j"
Nov 01 03:07:15.757184 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:15.757112    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d7f2c2ca-b2a2-4a31-987a-e7049fa74f2d-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-ncr5j\" (UID: \"d7f2c2ca-b2a2-4a31-987a-e7049fa74f2d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ncr5j"
Nov 01 03:07:15.757184 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:15.757140    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d7f2c2ca-b2a2-4a31-987a-e7049fa74f2d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-ncr5j\" (UID: \"d7f2c2ca-b2a2-4a31-987a-e7049fa74f2d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ncr5j"
Nov 01 03:07:15.757255 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:15.757187    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4b8bb646-3c9e-4885-abe6-8dec55c060a0-metrics-client-ca\") pod \"node-exporter-dw9pj\" (UID: \"4b8bb646-3c9e-4885-abe6-8dec55c060a0\") " pod="openshift-monitoring/node-exporter-dw9pj"
Nov 01 03:07:15.757255 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:15.757239    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4b8bb646-3c9e-4885-abe6-8dec55c060a0-node-exporter-wtmp\") pod \"node-exporter-dw9pj\" (UID: \"4b8bb646-3c9e-4885-abe6-8dec55c060a0\") " pod="openshift-monitoring/node-exporter-dw9pj"
Nov 01 03:07:15.757313 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:15.757282    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4b8bb646-3c9e-4885-abe6-8dec55c060a0-node-exporter-accelerators-collector-config\") pod \"node-exporter-dw9pj\" (UID: \"4b8bb646-3c9e-4885-abe6-8dec55c060a0\") " pod="openshift-monitoring/node-exporter-dw9pj"
Nov 01 03:07:15.757368 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:15.757352    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4b8bb646-3c9e-4885-abe6-8dec55c060a0-node-exporter-tls\") pod \"node-exporter-dw9pj\" (UID: \"4b8bb646-3c9e-4885-abe6-8dec55c060a0\") " pod="openshift-monitoring/node-exporter-dw9pj"
Nov 01 03:07:15.757403 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:15.757385    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72qm9\" (UniqueName: \"kubernetes.io/projected/4b8bb646-3c9e-4885-abe6-8dec55c060a0-kube-api-access-72qm9\") pod \"node-exporter-dw9pj\" (UID: \"4b8bb646-3c9e-4885-abe6-8dec55c060a0\") " pod="openshift-monitoring/node-exporter-dw9pj"
Nov 01 03:07:15.757434 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:15.757401    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k89ch\" (UniqueName: \"kubernetes.io/projected/d7f2c2ca-b2a2-4a31-987a-e7049fa74f2d-kube-api-access-k89ch\") pod \"kube-state-metrics-5dc8d7d89b-ncr5j\" (UID: \"d7f2c2ca-b2a2-4a31-987a-e7049fa74f2d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ncr5j"
Nov 01 03:07:15.757434 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:15.757419    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/d7f2c2ca-b2a2-4a31-987a-e7049fa74f2d-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-ncr5j\" (UID: \"d7f2c2ca-b2a2-4a31-987a-e7049fa74f2d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ncr5j"
Nov 01 03:07:15.757493 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:15.757439    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4b8bb646-3c9e-4885-abe6-8dec55c060a0-node-exporter-textfile\") pod \"node-exporter-dw9pj\" (UID: \"4b8bb646-3c9e-4885-abe6-8dec55c060a0\") " pod="openshift-monitoring/node-exporter-dw9pj"
Nov 01 03:07:15.757525 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:15.757496    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/d7f2c2ca-b2a2-4a31-987a-e7049fa74f2d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-ncr5j\" (UID: \"d7f2c2ca-b2a2-4a31-987a-e7049fa74f2d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ncr5j"
Nov 01 03:07:15.757561 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:15.757542    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4b8bb646-3c9e-4885-abe6-8dec55c060a0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dw9pj\" (UID: \"4b8bb646-3c9e-4885-abe6-8dec55c060a0\") " pod="openshift-monitoring/node-exporter-dw9pj"
Nov 01 03:07:15.757589 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:15.757560    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4b8bb646-3c9e-4885-abe6-8dec55c060a0-sys\") pod \"node-exporter-dw9pj\" (UID: \"4b8bb646-3c9e-4885-abe6-8dec55c060a0\") " pod="openshift-monitoring/node-exporter-dw9pj"
Nov 01 03:07:15.757589 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:15.757575    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4b8bb646-3c9e-4885-abe6-8dec55c060a0-root\") pod \"node-exporter-dw9pj\" (UID: \"4b8bb646-3c9e-4885-abe6-8dec55c060a0\") " pod="openshift-monitoring/node-exporter-dw9pj"
Nov 01 03:07:15.758707 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:15.758674    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 01 03:07:15.759763 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:15.759742    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6nx7f"
Nov 01 03:07:15.796899 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:15.796865    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cf7cdf566-2f2jh" event={"ID":"cd688da6-42b3-49ed-96ba-1ae5c6aa3933","Type":"ContainerStarted","Data":"a6ffc2e5a950bd407b8d9036068673a9af8e144b1de9e98670382e89a4cb07e4"}
Nov 01 03:07:15.858910 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:15.858875    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d7f2c2ca-b2a2-4a31-987a-e7049fa74f2d-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-ncr5j\" (UID: \"d7f2c2ca-b2a2-4a31-987a-e7049fa74f2d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ncr5j"
Nov 01 03:07:15.858910 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:15.858912    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d7f2c2ca-b2a2-4a31-987a-e7049fa74f2d-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-ncr5j\" (UID: \"d7f2c2ca-b2a2-4a31-987a-e7049fa74f2d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ncr5j"
Nov 01 03:07:15.859115 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:15.858945    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d7f2c2ca-b2a2-4a31-987a-e7049fa74f2d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-ncr5j\" (UID: \"d7f2c2ca-b2a2-4a31-987a-e7049fa74f2d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ncr5j"
Nov 01 03:07:15.859115 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:15.858974    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4b8bb646-3c9e-4885-abe6-8dec55c060a0-metrics-client-ca\") pod \"node-exporter-dw9pj\" (UID: \"4b8bb646-3c9e-4885-abe6-8dec55c060a0\") " pod="openshift-monitoring/node-exporter-dw9pj"
Nov 01 03:07:15.859115 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:15.859000    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4b8bb646-3c9e-4885-abe6-8dec55c060a0-node-exporter-wtmp\") pod \"node-exporter-dw9pj\" (UID: \"4b8bb646-3c9e-4885-abe6-8dec55c060a0\") " pod="openshift-monitoring/node-exporter-dw9pj"
Nov 01 03:07:15.859115 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:15.859022    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4b8bb646-3c9e-4885-abe6-8dec55c060a0-node-exporter-accelerators-collector-config\") pod \"node-exporter-dw9pj\" (UID: \"4b8bb646-3c9e-4885-abe6-8dec55c060a0\") " pod="openshift-monitoring/node-exporter-dw9pj"
Nov 01 03:07:15.859115 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:15.859061    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4b8bb646-3c9e-4885-abe6-8dec55c060a0-node-exporter-tls\") pod \"node-exporter-dw9pj\" (UID: \"4b8bb646-3c9e-4885-abe6-8dec55c060a0\") " pod="openshift-monitoring/node-exporter-dw9pj"
Nov 01 03:07:15.859115 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:15.859093    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72qm9\" (UniqueName: \"kubernetes.io/projected/4b8bb646-3c9e-4885-abe6-8dec55c060a0-kube-api-access-72qm9\") pod \"node-exporter-dw9pj\" (UID: \"4b8bb646-3c9e-4885-abe6-8dec55c060a0\") " pod="openshift-monitoring/node-exporter-dw9pj"
Nov 01 03:07:15.859115 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:15.859114    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k89ch\" (UniqueName: \"kubernetes.io/projected/d7f2c2ca-b2a2-4a31-987a-e7049fa74f2d-kube-api-access-k89ch\") pod \"kube-state-metrics-5dc8d7d89b-ncr5j\" (UID: \"d7f2c2ca-b2a2-4a31-987a-e7049fa74f2d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ncr5j"
Nov 01 03:07:15.859450 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:15.859139    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/d7f2c2ca-b2a2-4a31-987a-e7049fa74f2d-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-ncr5j\" (UID: \"d7f2c2ca-b2a2-4a31-987a-e7049fa74f2d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ncr5j"
Nov 01 03:07:15.859450 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:15.859162    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4b8bb646-3c9e-4885-abe6-8dec55c060a0-node-exporter-textfile\") pod \"node-exporter-dw9pj\" (UID: \"4b8bb646-3c9e-4885-abe6-8dec55c060a0\") " pod="openshift-monitoring/node-exporter-dw9pj"
Nov 01 03:07:15.859450 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:15.859190    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/d7f2c2ca-b2a2-4a31-987a-e7049fa74f2d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-ncr5j\" (UID: \"d7f2c2ca-b2a2-4a31-987a-e7049fa74f2d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ncr5j"
Nov 01 03:07:15.859450 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:15.859234    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4b8bb646-3c9e-4885-abe6-8dec55c060a0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dw9pj\" (UID: \"4b8bb646-3c9e-4885-abe6-8dec55c060a0\") " pod="openshift-monitoring/node-exporter-dw9pj"
Nov 01 03:07:15.859450 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:15.859262    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4b8bb646-3c9e-4885-abe6-8dec55c060a0-sys\") pod \"node-exporter-dw9pj\" (UID: \"4b8bb646-3c9e-4885-abe6-8dec55c060a0\") " pod="openshift-monitoring/node-exporter-dw9pj"
Nov 01 03:07:15.859450 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:15.859259    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4b8bb646-3c9e-4885-abe6-8dec55c060a0-node-exporter-wtmp\") pod \"node-exporter-dw9pj\" (UID: \"4b8bb646-3c9e-4885-abe6-8dec55c060a0\") " pod="openshift-monitoring/node-exporter-dw9pj"
Nov 01 03:07:15.859450 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:15.859290    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4b8bb646-3c9e-4885-abe6-8dec55c060a0-root\") pod \"node-exporter-dw9pj\" (UID: \"4b8bb646-3c9e-4885-abe6-8dec55c060a0\") " pod="openshift-monitoring/node-exporter-dw9pj"
Nov 01 03:07:15.859450 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:15.859390    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4b8bb646-3c9e-4885-abe6-8dec55c060a0-root\") pod \"node-exporter-dw9pj\" (UID: \"4b8bb646-3c9e-4885-abe6-8dec55c060a0\") " pod="openshift-monitoring/node-exporter-dw9pj"
Nov 01 03:07:15.859845 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:15.859752    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d7f2c2ca-b2a2-4a31-987a-e7049fa74f2d-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-ncr5j\" (UID: \"d7f2c2ca-b2a2-4a31-987a-e7049fa74f2d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ncr5j"
Nov 01 03:07:15.859845 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:15.859752    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4b8bb646-3c9e-4885-abe6-8dec55c060a0-metrics-client-ca\") pod \"node-exporter-dw9pj\" (UID: \"4b8bb646-3c9e-4885-abe6-8dec55c060a0\") " pod="openshift-monitoring/node-exporter-dw9pj"
Nov 01 03:07:15.860218 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:15.859985    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4b8bb646-3c9e-4885-abe6-8dec55c060a0-sys\") pod \"node-exporter-dw9pj\" (UID: \"4b8bb646-3c9e-4885-abe6-8dec55c060a0\") " pod="openshift-monitoring/node-exporter-dw9pj"
Nov 01 03:07:15.860218 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:15.860072    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/d7f2c2ca-b2a2-4a31-987a-e7049fa74f2d-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-ncr5j\" (UID: \"d7f2c2ca-b2a2-4a31-987a-e7049fa74f2d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ncr5j"
Nov 01 03:07:15.860218 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:15.860153    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4b8bb646-3c9e-4885-abe6-8dec55c060a0-node-exporter-textfile\") pod \"node-exporter-dw9pj\" (UID: \"4b8bb646-3c9e-4885-abe6-8dec55c060a0\") " pod="openshift-monitoring/node-exporter-dw9pj"
Nov 01 03:07:15.860431 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:15.860330    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4b8bb646-3c9e-4885-abe6-8dec55c060a0-node-exporter-accelerators-collector-config\") pod \"node-exporter-dw9pj\" (UID: \"4b8bb646-3c9e-4885-abe6-8dec55c060a0\") " pod="openshift-monitoring/node-exporter-dw9pj"
Nov 01 03:07:15.860827 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:15.860800    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/d7f2c2ca-b2a2-4a31-987a-e7049fa74f2d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-ncr5j\" (UID: \"d7f2c2ca-b2a2-4a31-987a-e7049fa74f2d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ncr5j"
Nov 01 03:07:15.861668 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:15.861642    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d7f2c2ca-b2a2-4a31-987a-e7049fa74f2d-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-ncr5j\" (UID: \"d7f2c2ca-b2a2-4a31-987a-e7049fa74f2d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ncr5j"
Nov 01 03:07:15.861668 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:15.861659    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d7f2c2ca-b2a2-4a31-987a-e7049fa74f2d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-ncr5j\" (UID: \"d7f2c2ca-b2a2-4a31-987a-e7049fa74f2d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ncr5j"
Nov 01 03:07:15.861952 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:15.861933    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4b8bb646-3c9e-4885-abe6-8dec55c060a0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dw9pj\" (UID: \"4b8bb646-3c9e-4885-abe6-8dec55c060a0\") " pod="openshift-monitoring/node-exporter-dw9pj"
Nov 01 03:07:15.862012 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:15.861937    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4b8bb646-3c9e-4885-abe6-8dec55c060a0-node-exporter-tls\") pod \"node-exporter-dw9pj\" (UID: \"4b8bb646-3c9e-4885-abe6-8dec55c060a0\") " pod="openshift-monitoring/node-exporter-dw9pj"
Nov 01 03:07:15.921676 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:15.921652    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k89ch\" (UniqueName: \"kubernetes.io/projected/d7f2c2ca-b2a2-4a31-987a-e7049fa74f2d-kube-api-access-k89ch\") pod \"kube-state-metrics-5dc8d7d89b-ncr5j\" (UID: \"d7f2c2ca-b2a2-4a31-987a-e7049fa74f2d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ncr5j"
Nov 01 03:07:15.921874 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:15.921853    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72qm9\" (UniqueName: \"kubernetes.io/projected/4b8bb646-3c9e-4885-abe6-8dec55c060a0-kube-api-access-72qm9\") pod \"node-exporter-dw9pj\" (UID: \"4b8bb646-3c9e-4885-abe6-8dec55c060a0\") " pod="openshift-monitoring/node-exporter-dw9pj"
Nov 01 03:07:15.925761 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:15.925745    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ncr5j"
Nov 01 03:07:15.940002 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:15.939986    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-dw9pj"
Nov 01 03:07:15.948879 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:07:15.948853    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b8bb646_3c9e_4885_abe6_8dec55c060a0.slice/crio-3652f81423ab5aaf8c039b5cb2608dd4cd035a9cccac845d7c01ff560145e844 WatchSource:0}: Error finding container 3652f81423ab5aaf8c039b5cb2608dd4cd035a9cccac845d7c01ff560145e844: Status 404 returned error can't find the container with id 3652f81423ab5aaf8c039b5cb2608dd4cd035a9cccac845d7c01ff560145e844
Nov 01 03:07:16.189645 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:16.189616    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-6nx7f"]
Nov 01 03:07:16.193398 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:07:16.193366    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74fb2a00_d669_4759_a1bf_6af8425e1f71.slice/crio-d1be2658acde40f32befa474373d5c3936250338c18c3f1e0f679f9e3079b6ed WatchSource:0}: Error finding container d1be2658acde40f32befa474373d5c3936250338c18c3f1e0f679f9e3079b6ed: Status 404 returned error can't find the container with id d1be2658acde40f32befa474373d5c3936250338c18c3f1e0f679f9e3079b6ed
Nov 01 03:07:16.300967 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:16.300938    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-ncr5j"]
Nov 01 03:07:16.307851 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:07:16.307824    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7f2c2ca_b2a2_4a31_987a_e7049fa74f2d.slice/crio-80c74a36c14c26ecc59b153dbf7b3fe3d1e70633fe49f4ffb59e5d8930193f46 WatchSource:0}: Error finding container 80c74a36c14c26ecc59b153dbf7b3fe3d1e70633fe49f4ffb59e5d8930193f46: Status 404 returned error can't find the container with id 80c74a36c14c26ecc59b153dbf7b3fe3d1e70633fe49f4ffb59e5d8930193f46
Nov 01 03:07:16.801897 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:16.801814    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ncr5j" event={"ID":"d7f2c2ca-b2a2-4a31-987a-e7049fa74f2d","Type":"ContainerStarted","Data":"80c74a36c14c26ecc59b153dbf7b3fe3d1e70633fe49f4ffb59e5d8930193f46"}
Nov 01 03:07:16.804067 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:16.804040    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6nx7f" event={"ID":"74fb2a00-d669-4759-a1bf-6af8425e1f71","Type":"ContainerStarted","Data":"9893ec69a17dc9cff645768460cd9ce88fc0eff562d092789c6ed2ae61d926fa"}
Nov 01 03:07:16.804184 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:16.804100    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6nx7f" event={"ID":"74fb2a00-d669-4759-a1bf-6af8425e1f71","Type":"ContainerStarted","Data":"5a3a701dd631191f2aae4a74436d5f01f8453a701c315e34cd0dda2ee7f6be91"}
Nov 01 03:07:16.804184 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:16.804130    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6nx7f" event={"ID":"74fb2a00-d669-4759-a1bf-6af8425e1f71","Type":"ContainerStarted","Data":"d1be2658acde40f32befa474373d5c3936250338c18c3f1e0f679f9e3079b6ed"}
Nov 01 03:07:16.805286 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:16.805266    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dw9pj" event={"ID":"4b8bb646-3c9e-4885-abe6-8dec55c060a0","Type":"ContainerStarted","Data":"3652f81423ab5aaf8c039b5cb2608dd4cd035a9cccac845d7c01ff560145e844"}
Nov 01 03:07:17.119950 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:17.119919    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5d488f58b6-s5k7j"]
Nov 01 03:07:17.122022 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:17.121998    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d488f58b6-s5k7j"
Nov 01 03:07:17.227276 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:17.227249    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Nov 01 03:07:17.228384 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:17.228358    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d488f58b6-s5k7j"]
Nov 01 03:07:17.271527 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:17.271497    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b08de522-96f8-42a8-a88f-0502b999d404-service-ca\") pod \"console-5d488f58b6-s5k7j\" (UID: \"b08de522-96f8-42a8-a88f-0502b999d404\") " pod="openshift-console/console-5d488f58b6-s5k7j"
Nov 01 03:07:17.271694 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:17.271532    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b08de522-96f8-42a8-a88f-0502b999d404-oauth-serving-cert\") pod \"console-5d488f58b6-s5k7j\" (UID: \"b08de522-96f8-42a8-a88f-0502b999d404\") " pod="openshift-console/console-5d488f58b6-s5k7j"
Nov 01 03:07:17.271694 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:17.271572    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b08de522-96f8-42a8-a88f-0502b999d404-console-config\") pod \"console-5d488f58b6-s5k7j\" (UID: \"b08de522-96f8-42a8-a88f-0502b999d404\") " pod="openshift-console/console-5d488f58b6-s5k7j"
Nov 01 03:07:17.271816 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:17.271732    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b08de522-96f8-42a8-a88f-0502b999d404-console-serving-cert\") pod \"console-5d488f58b6-s5k7j\" (UID: \"b08de522-96f8-42a8-a88f-0502b999d404\") " pod="openshift-console/console-5d488f58b6-s5k7j"
Nov 01 03:07:17.271816 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:17.271781    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b08de522-96f8-42a8-a88f-0502b999d404-console-oauth-config\") pod \"console-5d488f58b6-s5k7j\" (UID: \"b08de522-96f8-42a8-a88f-0502b999d404\") " pod="openshift-console/console-5d488f58b6-s5k7j"
Nov 01 03:07:17.271887 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:17.271869    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b08de522-96f8-42a8-a88f-0502b999d404-tmp\") pod \"console-5d488f58b6-s5k7j\" (UID: \"b08de522-96f8-42a8-a88f-0502b999d404\") " pod="openshift-console/console-5d488f58b6-s5k7j"
Nov 01 03:07:17.271953 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:17.271936    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b08de522-96f8-42a8-a88f-0502b999d404-trusted-ca-bundle\") pod \"console-5d488f58b6-s5k7j\" (UID: \"b08de522-96f8-42a8-a88f-0502b999d404\") " pod="openshift-console/console-5d488f58b6-s5k7j"
Nov 01 03:07:17.271989 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:17.271970    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbt72\" (UniqueName: \"kubernetes.io/projected/b08de522-96f8-42a8-a88f-0502b999d404-kube-api-access-fbt72\") pod \"console-5d488f58b6-s5k7j\" (UID: \"b08de522-96f8-42a8-a88f-0502b999d404\") " pod="openshift-console/console-5d488f58b6-s5k7j"
Nov 01 03:07:17.372427 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:17.372348    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b08de522-96f8-42a8-a88f-0502b999d404-console-config\") pod \"console-5d488f58b6-s5k7j\" (UID: \"b08de522-96f8-42a8-a88f-0502b999d404\") " pod="openshift-console/console-5d488f58b6-s5k7j"
Nov 01 03:07:17.372427 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:17.372399    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b08de522-96f8-42a8-a88f-0502b999d404-console-serving-cert\") pod \"console-5d488f58b6-s5k7j\" (UID: \"b08de522-96f8-42a8-a88f-0502b999d404\") " pod="openshift-console/console-5d488f58b6-s5k7j"
Nov 01 03:07:17.372641 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:17.372436    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b08de522-96f8-42a8-a88f-0502b999d404-console-oauth-config\") pod \"console-5d488f58b6-s5k7j\" (UID: \"b08de522-96f8-42a8-a88f-0502b999d404\") " pod="openshift-console/console-5d488f58b6-s5k7j"
Nov 01 03:07:17.372641 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:17.372466    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b08de522-96f8-42a8-a88f-0502b999d404-tmp\") pod \"console-5d488f58b6-s5k7j\" (UID: \"b08de522-96f8-42a8-a88f-0502b999d404\") " pod="openshift-console/console-5d488f58b6-s5k7j"
Nov 01 03:07:17.372641 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:17.372518    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b08de522-96f8-42a8-a88f-0502b999d404-trusted-ca-bundle\") pod \"console-5d488f58b6-s5k7j\" (UID: \"b08de522-96f8-42a8-a88f-0502b999d404\") " pod="openshift-console/console-5d488f58b6-s5k7j"
Nov 01 03:07:17.372641 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:17.372543    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbt72\" (UniqueName: \"kubernetes.io/projected/b08de522-96f8-42a8-a88f-0502b999d404-kube-api-access-fbt72\") pod \"console-5d488f58b6-s5k7j\" (UID: \"b08de522-96f8-42a8-a88f-0502b999d404\") " pod="openshift-console/console-5d488f58b6-s5k7j"
Nov 01 03:07:17.372641 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:17.372592    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b08de522-96f8-42a8-a88f-0502b999d404-service-ca\") pod \"console-5d488f58b6-s5k7j\" (UID: \"b08de522-96f8-42a8-a88f-0502b999d404\") " pod="openshift-console/console-5d488f58b6-s5k7j"
Nov 01 03:07:17.372641 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:17.372615    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b08de522-96f8-42a8-a88f-0502b999d404-oauth-serving-cert\") pod \"console-5d488f58b6-s5k7j\" (UID: \"b08de522-96f8-42a8-a88f-0502b999d404\") " pod="openshift-console/console-5d488f58b6-s5k7j"
Nov 01 03:07:17.373211 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:17.373189    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b08de522-96f8-42a8-a88f-0502b999d404-console-config\") pod \"console-5d488f58b6-s5k7j\" (UID: \"b08de522-96f8-42a8-a88f-0502b999d404\") " pod="openshift-console/console-5d488f58b6-s5k7j"
Nov 01 03:07:17.373312 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:17.373244    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b08de522-96f8-42a8-a88f-0502b999d404-oauth-serving-cert\") pod \"console-5d488f58b6-s5k7j\" (UID: \"b08de522-96f8-42a8-a88f-0502b999d404\") " pod="openshift-console/console-5d488f58b6-s5k7j"
Nov 01 03:07:17.373490 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:17.373472    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b08de522-96f8-42a8-a88f-0502b999d404-tmp\") pod \"console-5d488f58b6-s5k7j\" (UID: \"b08de522-96f8-42a8-a88f-0502b999d404\") " pod="openshift-console/console-5d488f58b6-s5k7j"
Nov 01 03:07:17.374097 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:17.374072    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b08de522-96f8-42a8-a88f-0502b999d404-trusted-ca-bundle\") pod \"console-5d488f58b6-s5k7j\" (UID: \"b08de522-96f8-42a8-a88f-0502b999d404\") " pod="openshift-console/console-5d488f58b6-s5k7j"
Nov 01 03:07:17.374587 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:17.374514    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b08de522-96f8-42a8-a88f-0502b999d404-service-ca\") pod \"console-5d488f58b6-s5k7j\" (UID: \"b08de522-96f8-42a8-a88f-0502b999d404\") " pod="openshift-console/console-5d488f58b6-s5k7j"
Nov 01 03:07:17.380501 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:17.380469    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b08de522-96f8-42a8-a88f-0502b999d404-console-oauth-config\") pod \"console-5d488f58b6-s5k7j\" (UID: \"b08de522-96f8-42a8-a88f-0502b999d404\") " pod="openshift-console/console-5d488f58b6-s5k7j"
Nov 01 03:07:17.381368 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:17.381344    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b08de522-96f8-42a8-a88f-0502b999d404-console-serving-cert\") pod \"console-5d488f58b6-s5k7j\" (UID: \"b08de522-96f8-42a8-a88f-0502b999d404\") " pod="openshift-console/console-5d488f58b6-s5k7j"
Nov 01 03:07:17.442511 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:17.442479    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbt72\" (UniqueName: \"kubernetes.io/projected/b08de522-96f8-42a8-a88f-0502b999d404-kube-api-access-fbt72\") pod \"console-5d488f58b6-s5k7j\" (UID: \"b08de522-96f8-42a8-a88f-0502b999d404\") " pod="openshift-console/console-5d488f58b6-s5k7j"
Nov 01 03:07:17.737052 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:17.737019    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d488f58b6-s5k7j"
Nov 01 03:07:17.817178 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:17.816886    4400 generic.go:356] "Generic (PLEG): container finished" podID="4b8bb646-3c9e-4885-abe6-8dec55c060a0" containerID="04a4be3f0ce5daf1a842416c05f001088ab43acc42dc6e5e5312056247d1fa08" exitCode=0
Nov 01 03:07:17.817178 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:17.816977    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dw9pj" event={"ID":"4b8bb646-3c9e-4885-abe6-8dec55c060a0","Type":"ContainerDied","Data":"04a4be3f0ce5daf1a842416c05f001088ab43acc42dc6e5e5312056247d1fa08"}
Nov 01 03:07:17.860720 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:17.859134    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 01 03:07:17.863443 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:17.862984    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 01 03:07:17.920657 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:17.920636    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 01 03:07:17.932184 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:17.932114    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 01 03:07:17.958125 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:17.958100    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 01 03:07:17.958381 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:17.958363    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 01 03:07:17.958470 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:17.958414    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 01 03:07:17.958470 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:17.958449    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 01 03:07:17.958583 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:17.958512    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 01 03:07:17.958640 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:17.958602    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 01 03:07:17.958718 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:17.958666    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 01 03:07:17.958844 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:17.958824    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 01 03:07:17.958982 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:17.958943    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-q4rx2\""
Nov 01 03:07:17.978361 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:17.978332    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/95482def-35e0-4879-a721-d1ce6d251061-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"95482def-35e0-4879-a721-d1ce6d251061\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 03:07:17.978468 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:17.978383    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lns96\" (UniqueName: \"kubernetes.io/projected/95482def-35e0-4879-a721-d1ce6d251061-kube-api-access-lns96\") pod \"alertmanager-main-0\" (UID: \"95482def-35e0-4879-a721-d1ce6d251061\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 03:07:17.978468 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:17.978415    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/95482def-35e0-4879-a721-d1ce6d251061-config-out\") pod \"alertmanager-main-0\" (UID: \"95482def-35e0-4879-a721-d1ce6d251061\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 03:07:17.978468 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:17.978452    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/95482def-35e0-4879-a721-d1ce6d251061-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"95482def-35e0-4879-a721-d1ce6d251061\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 03:07:17.978629 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:17.978490    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/95482def-35e0-4879-a721-d1ce6d251061-tls-assets\") pod \"alertmanager-main-0\" (UID: \"95482def-35e0-4879-a721-d1ce6d251061\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 03:07:17.978629 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:17.978529    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/95482def-35e0-4879-a721-d1ce6d251061-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"95482def-35e0-4879-a721-d1ce6d251061\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 03:07:17.978629 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:17.978565    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/95482def-35e0-4879-a721-d1ce6d251061-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"95482def-35e0-4879-a721-d1ce6d251061\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 03:07:17.978629 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:17.978595    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/95482def-35e0-4879-a721-d1ce6d251061-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"95482def-35e0-4879-a721-d1ce6d251061\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 03:07:17.978629 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:17.978622    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/95482def-35e0-4879-a721-d1ce6d251061-config-volume\") pod \"alertmanager-main-0\" (UID: \"95482def-35e0-4879-a721-d1ce6d251061\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 03:07:17.978889 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:17.978648    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95482def-35e0-4879-a721-d1ce6d251061-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"95482def-35e0-4879-a721-d1ce6d251061\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 03:07:17.978889 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:17.978674    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/95482def-35e0-4879-a721-d1ce6d251061-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"95482def-35e0-4879-a721-d1ce6d251061\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 03:07:17.978889 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:17.978727    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/95482def-35e0-4879-a721-d1ce6d251061-web-config\") pod \"alertmanager-main-0\" (UID: \"95482def-35e0-4879-a721-d1ce6d251061\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 03:07:17.978889 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:17.978765    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/95482def-35e0-4879-a721-d1ce6d251061-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"95482def-35e0-4879-a721-d1ce6d251061\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 03:07:18.079850 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:18.079812    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/95482def-35e0-4879-a721-d1ce6d251061-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"95482def-35e0-4879-a721-d1ce6d251061\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 03:07:18.080043 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:18.079872    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/95482def-35e0-4879-a721-d1ce6d251061-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"95482def-35e0-4879-a721-d1ce6d251061\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 03:07:18.080043 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:18.079906    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/95482def-35e0-4879-a721-d1ce6d251061-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"95482def-35e0-4879-a721-d1ce6d251061\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 03:07:18.080043 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:18.079940    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/95482def-35e0-4879-a721-d1ce6d251061-config-volume\") pod \"alertmanager-main-0\" (UID: \"95482def-35e0-4879-a721-d1ce6d251061\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 03:07:18.080043 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:18.079964    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95482def-35e0-4879-a721-d1ce6d251061-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"95482def-35e0-4879-a721-d1ce6d251061\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 03:07:18.080043 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:18.079993    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/95482def-35e0-4879-a721-d1ce6d251061-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"95482def-35e0-4879-a721-d1ce6d251061\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 03:07:18.080043 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:18.080019    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/95482def-35e0-4879-a721-d1ce6d251061-web-config\") pod \"alertmanager-main-0\" (UID: \"95482def-35e0-4879-a721-d1ce6d251061\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 03:07:18.080333 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:18.080052    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/95482def-35e0-4879-a721-d1ce6d251061-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"95482def-35e0-4879-a721-d1ce6d251061\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 03:07:18.080333 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:18.080102    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/95482def-35e0-4879-a721-d1ce6d251061-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"95482def-35e0-4879-a721-d1ce6d251061\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 03:07:18.080333 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:18.080134    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lns96\" (UniqueName: \"kubernetes.io/projected/95482def-35e0-4879-a721-d1ce6d251061-kube-api-access-lns96\") pod \"alertmanager-main-0\" (UID: \"95482def-35e0-4879-a721-d1ce6d251061\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 03:07:18.080333 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:18.080159    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/95482def-35e0-4879-a721-d1ce6d251061-config-out\") pod \"alertmanager-main-0\" (UID: \"95482def-35e0-4879-a721-d1ce6d251061\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 03:07:18.080333 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:18.080196    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/95482def-35e0-4879-a721-d1ce6d251061-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"95482def-35e0-4879-a721-d1ce6d251061\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 03:07:18.080333 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:18.080236    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/95482def-35e0-4879-a721-d1ce6d251061-tls-assets\") pod \"alertmanager-main-0\" (UID: \"95482def-35e0-4879-a721-d1ce6d251061\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 03:07:18.083481 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:18.081280    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/95482def-35e0-4879-a721-d1ce6d251061-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"95482def-35e0-4879-a721-d1ce6d251061\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 03:07:18.083481 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:18.081813    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/95482def-35e0-4879-a721-d1ce6d251061-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"95482def-35e0-4879-a721-d1ce6d251061\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 03:07:18.083481 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:18.083100    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95482def-35e0-4879-a721-d1ce6d251061-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"95482def-35e0-4879-a721-d1ce6d251061\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 03:07:18.083716 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:18.083557    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/95482def-35e0-4879-a721-d1ce6d251061-tls-assets\") pod \"alertmanager-main-0\" (UID: \"95482def-35e0-4879-a721-d1ce6d251061\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 03:07:18.084094 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:18.084013    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/95482def-35e0-4879-a721-d1ce6d251061-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"95482def-35e0-4879-a721-d1ce6d251061\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 03:07:18.088838 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:18.088182    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/95482def-35e0-4879-a721-d1ce6d251061-config-out\") pod \"alertmanager-main-0\" (UID: \"95482def-35e0-4879-a721-d1ce6d251061\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 03:07:18.088838 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:18.088330    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/95482def-35e0-4879-a721-d1ce6d251061-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"95482def-35e0-4879-a721-d1ce6d251061\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 03:07:18.088838 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:18.088752    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/95482def-35e0-4879-a721-d1ce6d251061-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"95482def-35e0-4879-a721-d1ce6d251061\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 03:07:18.088838 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:18.088791    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/95482def-35e0-4879-a721-d1ce6d251061-config-volume\") pod \"alertmanager-main-0\" (UID: \"95482def-35e0-4879-a721-d1ce6d251061\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 03:07:18.090391 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:18.090082    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/95482def-35e0-4879-a721-d1ce6d251061-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"95482def-35e0-4879-a721-d1ce6d251061\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 03:07:18.090391 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:18.090097    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/95482def-35e0-4879-a721-d1ce6d251061-web-config\") pod \"alertmanager-main-0\" (UID: \"95482def-35e0-4879-a721-d1ce6d251061\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 03:07:18.090391 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:18.090351    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/95482def-35e0-4879-a721-d1ce6d251061-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"95482def-35e0-4879-a721-d1ce6d251061\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 03:07:18.149480 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:18.149411    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lns96\" (UniqueName: \"kubernetes.io/projected/95482def-35e0-4879-a721-d1ce6d251061-kube-api-access-lns96\") pod \"alertmanager-main-0\" (UID: \"95482def-35e0-4879-a721-d1ce6d251061\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 03:07:18.190892 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:18.190863    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d488f58b6-s5k7j"]
Nov 01 03:07:18.199082 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:18.199061    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 01 03:07:18.216062 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:07:18.216032    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb08de522_96f8_42a8_a88f_0502b999d404.slice/crio-c76e5d12ac5a06d322affe91d31b9e3edc3dd94096fd72fb554fce811ecfa753 WatchSource:0}: Error finding container c76e5d12ac5a06d322affe91d31b9e3edc3dd94096fd72fb554fce811ecfa753: Status 404 returned error can't find the container with id c76e5d12ac5a06d322affe91d31b9e3edc3dd94096fd72fb554fce811ecfa753
Nov 01 03:07:18.262984 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:18.262959    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-59d787964f-6q8gp"]
Nov 01 03:07:18.266821 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:18.266801    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-59d787964f-6q8gp"
Nov 01 03:07:18.339705 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:18.339661    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-59d787964f-6q8gp"]
Nov 01 03:07:18.360641 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:18.360612    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Nov 01 03:07:18.360869 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:18.360848    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Nov 01 03:07:18.361180 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:18.361159    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Nov 01 03:07:18.361324 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:18.361246    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Nov 01 03:07:18.361324 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:18.361186    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Nov 01 03:07:18.361459 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:18.361348    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-cc5nc\""
Nov 01 03:07:18.361459 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:18.361414    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-6n3h8suca3nms\""
Nov 01 03:07:18.382562 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:18.382534    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/570761ed-076a-4c28-a14b-46471838f93c-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-59d787964f-6q8gp\" (UID: \"570761ed-076a-4c28-a14b-46471838f93c\") " pod="openshift-monitoring/thanos-querier-59d787964f-6q8gp"
Nov 01 03:07:18.382717 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:18.382588    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/570761ed-076a-4c28-a14b-46471838f93c-secret-grpc-tls\") pod \"thanos-querier-59d787964f-6q8gp\" (UID: \"570761ed-076a-4c28-a14b-46471838f93c\") " pod="openshift-monitoring/thanos-querier-59d787964f-6q8gp"
Nov 01 03:07:18.382910 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:18.382788    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/570761ed-076a-4c28-a14b-46471838f93c-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-59d787964f-6q8gp\" (UID: \"570761ed-076a-4c28-a14b-46471838f93c\") " pod="openshift-monitoring/thanos-querier-59d787964f-6q8gp"
Nov 01 03:07:18.383008 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:18.382990    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/570761ed-076a-4c28-a14b-46471838f93c-tmp\") pod \"thanos-querier-59d787964f-6q8gp\" (UID: \"570761ed-076a-4c28-a14b-46471838f93c\") " pod="openshift-monitoring/thanos-querier-59d787964f-6q8gp"
Nov 01 03:07:18.383280 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:18.383047    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/570761ed-076a-4c28-a14b-46471838f93c-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-59d787964f-6q8gp\" (UID: \"570761ed-076a-4c28-a14b-46471838f93c\") " pod="openshift-monitoring/thanos-querier-59d787964f-6q8gp"
Nov 01 03:07:18.383440 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:18.383302    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/570761ed-076a-4c28-a14b-46471838f93c-metrics-client-ca\") pod \"thanos-querier-59d787964f-6q8gp\" (UID: \"570761ed-076a-4c28-a14b-46471838f93c\") " pod="openshift-monitoring/thanos-querier-59d787964f-6q8gp"
Nov 01 03:07:18.383488 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:18.383459    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/570761ed-076a-4c28-a14b-46471838f93c-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-59d787964f-6q8gp\" (UID: \"570761ed-076a-4c28-a14b-46471838f93c\") " pod="openshift-monitoring/thanos-querier-59d787964f-6q8gp"
Nov 01 03:07:18.383648 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:18.383631    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsm6w\" (UniqueName: \"kubernetes.io/projected/570761ed-076a-4c28-a14b-46471838f93c-kube-api-access-qsm6w\") pod \"thanos-querier-59d787964f-6q8gp\" (UID: \"570761ed-076a-4c28-a14b-46471838f93c\") " pod="openshift-monitoring/thanos-querier-59d787964f-6q8gp"
Nov 01 03:07:18.389777 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:18.389748    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/570761ed-076a-4c28-a14b-46471838f93c-secret-thanos-querier-tls\") pod \"thanos-querier-59d787964f-6q8gp\" (UID: \"570761ed-076a-4c28-a14b-46471838f93c\") " pod="openshift-monitoring/thanos-querier-59d787964f-6q8gp"
Nov 01 03:07:18.491562 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:18.491479    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsm6w\" (UniqueName: \"kubernetes.io/projected/570761ed-076a-4c28-a14b-46471838f93c-kube-api-access-qsm6w\") pod \"thanos-querier-59d787964f-6q8gp\" (UID: \"570761ed-076a-4c28-a14b-46471838f93c\") " pod="openshift-monitoring/thanos-querier-59d787964f-6q8gp"
Nov 01 03:07:18.491562 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:18.491541    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/570761ed-076a-4c28-a14b-46471838f93c-secret-thanos-querier-tls\") pod \"thanos-querier-59d787964f-6q8gp\" (UID: \"570761ed-076a-4c28-a14b-46471838f93c\") " pod="openshift-monitoring/thanos-querier-59d787964f-6q8gp"
Nov 01 03:07:18.491906 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:18.491596    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/570761ed-076a-4c28-a14b-46471838f93c-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-59d787964f-6q8gp\" (UID: \"570761ed-076a-4c28-a14b-46471838f93c\") " pod="openshift-monitoring/thanos-querier-59d787964f-6q8gp"
Nov 01 03:07:18.491906 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:18.491645    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/570761ed-076a-4c28-a14b-46471838f93c-secret-grpc-tls\") pod \"thanos-querier-59d787964f-6q8gp\" (UID: \"570761ed-076a-4c28-a14b-46471838f93c\") " pod="openshift-monitoring/thanos-querier-59d787964f-6q8gp"
Nov 01 03:07:18.491906 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:18.491709    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/570761ed-076a-4c28-a14b-46471838f93c-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-59d787964f-6q8gp\" (UID: \"570761ed-076a-4c28-a14b-46471838f93c\") " pod="openshift-monitoring/thanos-querier-59d787964f-6q8gp"
Nov 01 03:07:18.491906 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:18.491735    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/570761ed-076a-4c28-a14b-46471838f93c-tmp\") pod \"thanos-querier-59d787964f-6q8gp\" (UID: \"570761ed-076a-4c28-a14b-46471838f93c\") " pod="openshift-monitoring/thanos-querier-59d787964f-6q8gp"
Nov 01 03:07:18.491906 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:18.491761    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/570761ed-076a-4c28-a14b-46471838f93c-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-59d787964f-6q8gp\" (UID: \"570761ed-076a-4c28-a14b-46471838f93c\") " pod="openshift-monitoring/thanos-querier-59d787964f-6q8gp"
Nov 01 03:07:18.491906 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:18.491778    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/570761ed-076a-4c28-a14b-46471838f93c-metrics-client-ca\") pod \"thanos-querier-59d787964f-6q8gp\" (UID: \"570761ed-076a-4c28-a14b-46471838f93c\") " pod="openshift-monitoring/thanos-querier-59d787964f-6q8gp"
Nov 01 03:07:18.491906 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:18.491803    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/570761ed-076a-4c28-a14b-46471838f93c-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-59d787964f-6q8gp\" (UID: \"570761ed-076a-4c28-a14b-46471838f93c\") " pod="openshift-monitoring/thanos-querier-59d787964f-6q8gp"
Nov 01 03:07:18.494117 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:18.493488    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/570761ed-076a-4c28-a14b-46471838f93c-tmp\") pod \"thanos-querier-59d787964f-6q8gp\" (UID: \"570761ed-076a-4c28-a14b-46471838f93c\") " pod="openshift-monitoring/thanos-querier-59d787964f-6q8gp"
Nov 01 03:07:18.495625 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:18.495326    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/570761ed-076a-4c28-a14b-46471838f93c-metrics-client-ca\") pod \"thanos-querier-59d787964f-6q8gp\" (UID: \"570761ed-076a-4c28-a14b-46471838f93c\") " pod="openshift-monitoring/thanos-querier-59d787964f-6q8gp"
Nov 01 03:07:18.496407 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:18.496275    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/570761ed-076a-4c28-a14b-46471838f93c-secret-thanos-querier-tls\") pod \"thanos-querier-59d787964f-6q8gp\" (UID: \"570761ed-076a-4c28-a14b-46471838f93c\") " pod="openshift-monitoring/thanos-querier-59d787964f-6q8gp"
Nov 01 03:07:18.496407 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:18.496356    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/570761ed-076a-4c28-a14b-46471838f93c-secret-grpc-tls\") pod \"thanos-querier-59d787964f-6q8gp\" (UID: \"570761ed-076a-4c28-a14b-46471838f93c\") " pod="openshift-monitoring/thanos-querier-59d787964f-6q8gp"
Nov 01 03:07:18.498831 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:18.496954    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/570761ed-076a-4c28-a14b-46471838f93c-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-59d787964f-6q8gp\" (UID: \"570761ed-076a-4c28-a14b-46471838f93c\") " pod="openshift-monitoring/thanos-querier-59d787964f-6q8gp"
Nov 01 03:07:18.498831 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:18.498353    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/570761ed-076a-4c28-a14b-46471838f93c-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-59d787964f-6q8gp\" (UID: \"570761ed-076a-4c28-a14b-46471838f93c\") " pod="openshift-monitoring/thanos-querier-59d787964f-6q8gp"
Nov 01 03:07:18.498831 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:18.498791    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/570761ed-076a-4c28-a14b-46471838f93c-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-59d787964f-6q8gp\" (UID: \"570761ed-076a-4c28-a14b-46471838f93c\") " pod="openshift-monitoring/thanos-querier-59d787964f-6q8gp"
Nov 01 03:07:18.499482 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:18.499442    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/570761ed-076a-4c28-a14b-46471838f93c-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-59d787964f-6q8gp\" (UID: \"570761ed-076a-4c28-a14b-46471838f93c\") " pod="openshift-monitoring/thanos-querier-59d787964f-6q8gp"
Nov 01 03:07:18.557005 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:18.556968    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsm6w\" (UniqueName: \"kubernetes.io/projected/570761ed-076a-4c28-a14b-46471838f93c-kube-api-access-qsm6w\") pod \"thanos-querier-59d787964f-6q8gp\" (UID: \"570761ed-076a-4c28-a14b-46471838f93c\") " pod="openshift-monitoring/thanos-querier-59d787964f-6q8gp"
Nov 01 03:07:18.587648 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:18.587626    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-59d787964f-6q8gp"
Nov 01 03:07:18.600459 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:18.600359    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 01 03:07:18.602811 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:07:18.602786    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95482def_35e0_4879_a721_d1ce6d251061.slice/crio-1f8ddae7f495423fe3dcca6e59d3084609854e69a4da9d4d144c053d691e69bc WatchSource:0}: Error finding container 1f8ddae7f495423fe3dcca6e59d3084609854e69a4da9d4d144c053d691e69bc: Status 404 returned error can't find the container with id 1f8ddae7f495423fe3dcca6e59d3084609854e69a4da9d4d144c053d691e69bc
Nov 01 03:07:18.823206 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:18.823176    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dw9pj" event={"ID":"4b8bb646-3c9e-4885-abe6-8dec55c060a0","Type":"ContainerStarted","Data":"e92a68a95400f466ae5c90f369ad2d63288c7250ddb62255b6a62571db43fade"}
Nov 01 03:07:18.823616 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:18.823240    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dw9pj" event={"ID":"4b8bb646-3c9e-4885-abe6-8dec55c060a0","Type":"ContainerStarted","Data":"3dce4a3a2b30b25c19edb86dcd3c629b69dd41bd580ca705f10161af3982e941"}
Nov 01 03:07:18.825858 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:18.825834    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ncr5j" event={"ID":"d7f2c2ca-b2a2-4a31-987a-e7049fa74f2d","Type":"ContainerStarted","Data":"e553d5887bb87b77f2f97f67755f0fd6022d7ca7dbbfe453fbdd48a4243d9886"}
Nov 01 03:07:18.825963 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:18.825888    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ncr5j" event={"ID":"d7f2c2ca-b2a2-4a31-987a-e7049fa74f2d","Type":"ContainerStarted","Data":"adedada6f0283656e8e4682c4485c2ae3b61e0b9bed06fc56b85b1f92d5986a4"}
Nov 01 03:07:18.827360 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:18.827301    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"95482def-35e0-4879-a721-d1ce6d251061","Type":"ContainerStarted","Data":"1f8ddae7f495423fe3dcca6e59d3084609854e69a4da9d4d144c053d691e69bc"}
Nov 01 03:07:18.828901 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:18.828854    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d488f58b6-s5k7j" event={"ID":"b08de522-96f8-42a8-a88f-0502b999d404","Type":"ContainerStarted","Data":"c76e5d12ac5a06d322affe91d31b9e3edc3dd94096fd72fb554fce811ecfa753"}
Nov 01 03:07:18.831307 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:18.831283    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6nx7f" event={"ID":"74fb2a00-d669-4759-a1bf-6af8425e1f71","Type":"ContainerStarted","Data":"163702ac35d29932fc12b1d47a67f867738defc6f966ad2e632c672ce9831c25"}
Nov 01 03:07:18.950351 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:18.950310    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-dw9pj" podStartSLOduration=2.240840692 podStartE2EDuration="3.950297837s" totalImagesPullingTime="1.61824588s" totalInitContainerRuntime="91.211265ms" isStatefulPod=true podCreationTimestamp="2026-11-01 03:07:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:07:18.949975789 +0000 UTC m=+45.464621663" watchObservedRunningTime="2026-11-01 03:07:18.950297837 +0000 UTC m=+45.464943694"
Nov 01 03:07:18.979874 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:18.979824    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-59d787964f-6q8gp"]
Nov 01 03:07:18.981996 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:07:18.981971    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod570761ed_076a_4c28_a14b_46471838f93c.slice/crio-9be3279e153fa0c18d2ef258a47dcc00f3d9c5d5a8d2c0a275bf9df1fae356ad WatchSource:0}: Error finding container 9be3279e153fa0c18d2ef258a47dcc00f3d9c5d5a8d2c0a275bf9df1fae356ad: Status 404 returned error can't find the container with id 9be3279e153fa0c18d2ef258a47dcc00f3d9c5d5a8d2c0a275bf9df1fae356ad
Nov 01 03:07:19.072291 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:19.072243    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6nx7f" podStartSLOduration=2.203209444 podStartE2EDuration="4.072228834s" totalImagesPullingTime="1.86901939s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:07:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:07:19.072046968 +0000 UTC m=+45.586692828" watchObservedRunningTime="2026-11-01 03:07:19.072228834 +0000 UTC m=+45.586874693"
Nov 01 03:07:19.837122 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:19.837083    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ncr5j" event={"ID":"d7f2c2ca-b2a2-4a31-987a-e7049fa74f2d","Type":"ContainerStarted","Data":"0e9dcd1e564475f04807f845c59899bd0c2420693700ebd429d73915cd30e561"}
Nov 01 03:07:19.838710 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:19.838657    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d488f58b6-s5k7j" event={"ID":"b08de522-96f8-42a8-a88f-0502b999d404","Type":"ContainerStarted","Data":"0fa21df1a169c99bd5bf3b928282d4876038c6366a34f47b3be5dced1bed1aac"}
Nov 01 03:07:19.839920 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:19.839891    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-59d787964f-6q8gp" event={"ID":"570761ed-076a-4c28-a14b-46471838f93c","Type":"ContainerStarted","Data":"9be3279e153fa0c18d2ef258a47dcc00f3d9c5d5a8d2c0a275bf9df1fae356ad"}
Nov 01 03:07:19.841438 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:19.841415    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cf7cdf566-2f2jh" event={"ID":"cd688da6-42b3-49ed-96ba-1ae5c6aa3933","Type":"ContainerStarted","Data":"7abea0024cf4002e9d661a828471a1b6005ba027e81743261e0085e1dbf830b8"}
Nov 01 03:07:20.109532 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:20.109436    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-6cf7cdf566-2f2jh" podStartSLOduration=2.281700304 podStartE2EDuration="6.109423034s" totalImagesPullingTime="3.82772273s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:07:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:07:20.109289143 +0000 UTC m=+46.623935005" watchObservedRunningTime="2026-11-01 03:07:20.109423034 +0000 UTC m=+46.624068894"
Nov 01 03:07:20.109673 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:20.109532    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ncr5j" podStartSLOduration=2.83718738 podStartE2EDuration="5.109526735s" totalImagesPullingTime="2.272339355s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:07:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:07:19.967856859 +0000 UTC m=+46.482502721" watchObservedRunningTime="2026-11-01 03:07:20.109526735 +0000 UTC m=+46.624172598"
Nov 01 03:07:20.209558 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:20.209527    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-l5ng6"]
Nov 01 03:07:20.213802 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:20.213779    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-l5ng6"
Nov 01 03:07:20.239434 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:20.239385    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5d488f58b6-s5k7j" podStartSLOduration=2.444403462 podStartE2EDuration="3.239366411s" totalImagesPullingTime="794.962949ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:07:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:07:20.238457894 +0000 UTC m=+46.753103756" watchObservedRunningTime="2026-11-01 03:07:20.239366411 +0000 UTC m=+46.754012275"
Nov 01 03:07:20.285798 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:20.284567    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-l5ng6"]
Nov 01 03:07:20.303181 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:20.303155    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-8x5tb\""
Nov 01 03:07:20.303315 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:20.303165    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Nov 01 03:07:20.307465 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:20.307440    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/32b3b49b-3371-4455-a4c5-a4152e66cc7d-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-l5ng6\" (UID: \"32b3b49b-3371-4455-a4c5-a4152e66cc7d\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-l5ng6"
Nov 01 03:07:20.307566 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:20.307529    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdv56\" (UniqueName: \"kubernetes.io/projected/32b3b49b-3371-4455-a4c5-a4152e66cc7d-kube-api-access-mdv56\") pod \"monitoring-plugin-77f4f76594-l5ng6\" (UID: \"32b3b49b-3371-4455-a4c5-a4152e66cc7d\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-l5ng6"
Nov 01 03:07:20.408526 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:20.408437    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdv56\" (UniqueName: \"kubernetes.io/projected/32b3b49b-3371-4455-a4c5-a4152e66cc7d-kube-api-access-mdv56\") pod \"monitoring-plugin-77f4f76594-l5ng6\" (UID: \"32b3b49b-3371-4455-a4c5-a4152e66cc7d\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-l5ng6"
Nov 01 03:07:20.408526 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:20.408493    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/32b3b49b-3371-4455-a4c5-a4152e66cc7d-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-l5ng6\" (UID: \"32b3b49b-3371-4455-a4c5-a4152e66cc7d\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-l5ng6"
Nov 01 03:07:20.411584 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:20.411557    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/32b3b49b-3371-4455-a4c5-a4152e66cc7d-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-l5ng6\" (UID: \"32b3b49b-3371-4455-a4c5-a4152e66cc7d\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-l5ng6"
Nov 01 03:07:20.473511 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:20.473485    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdv56\" (UniqueName: \"kubernetes.io/projected/32b3b49b-3371-4455-a4c5-a4152e66cc7d-kube-api-access-mdv56\") pod \"monitoring-plugin-77f4f76594-l5ng6\" (UID: \"32b3b49b-3371-4455-a4c5-a4152e66cc7d\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-l5ng6"
Nov 01 03:07:20.527988 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:20.527741    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-l5ng6"
Nov 01 03:07:20.527988 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:20.527970    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5845695f9b-7vfc4"]
Nov 01 03:07:20.531299 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:20.531283    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5845695f9b-7vfc4"
Nov 01 03:07:20.587515 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:20.587489    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-vt5z7\""
Nov 01 03:07:20.587634 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:20.587591    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Nov 01 03:07:20.587942 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:20.587891    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Nov 01 03:07:20.587942 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:20.587900    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-9lebbd0n9ilvf\""
Nov 01 03:07:20.588085 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:20.587952    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Nov 01 03:07:20.588085 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:20.587987    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Nov 01 03:07:20.600899 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:20.600879    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5845695f9b-7vfc4"]
Nov 01 03:07:20.610208 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:20.610177    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/147c6bba-08a4-4956-9854-c93fa11bbd15-metrics-server-audit-profiles\") pod \"metrics-server-5845695f9b-7vfc4\" (UID: \"147c6bba-08a4-4956-9854-c93fa11bbd15\") " pod="openshift-monitoring/metrics-server-5845695f9b-7vfc4"
Nov 01 03:07:20.610283 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:20.610232    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/147c6bba-08a4-4956-9854-c93fa11bbd15-secret-metrics-server-client-certs\") pod \"metrics-server-5845695f9b-7vfc4\" (UID: \"147c6bba-08a4-4956-9854-c93fa11bbd15\") " pod="openshift-monitoring/metrics-server-5845695f9b-7vfc4"
Nov 01 03:07:20.610335 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:20.610302    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/147c6bba-08a4-4956-9854-c93fa11bbd15-secret-metrics-server-tls\") pod \"metrics-server-5845695f9b-7vfc4\" (UID: \"147c6bba-08a4-4956-9854-c93fa11bbd15\") " pod="openshift-monitoring/metrics-server-5845695f9b-7vfc4"
Nov 01 03:07:20.610335 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:20.610327    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/147c6bba-08a4-4956-9854-c93fa11bbd15-client-ca-bundle\") pod \"metrics-server-5845695f9b-7vfc4\" (UID: \"147c6bba-08a4-4956-9854-c93fa11bbd15\") " pod="openshift-monitoring/metrics-server-5845695f9b-7vfc4"
Nov 01 03:07:20.610437 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:20.610344    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/147c6bba-08a4-4956-9854-c93fa11bbd15-audit-log\") pod \"metrics-server-5845695f9b-7vfc4\" (UID: \"147c6bba-08a4-4956-9854-c93fa11bbd15\") " pod="openshift-monitoring/metrics-server-5845695f9b-7vfc4"
Nov 01 03:07:20.610437 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:20.610369    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/147c6bba-08a4-4956-9854-c93fa11bbd15-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5845695f9b-7vfc4\" (UID: \"147c6bba-08a4-4956-9854-c93fa11bbd15\") " pod="openshift-monitoring/metrics-server-5845695f9b-7vfc4"
Nov 01 03:07:20.610437 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:20.610407    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxk8s\" (UniqueName: \"kubernetes.io/projected/147c6bba-08a4-4956-9854-c93fa11bbd15-kube-api-access-hxk8s\") pod \"metrics-server-5845695f9b-7vfc4\" (UID: \"147c6bba-08a4-4956-9854-c93fa11bbd15\") " pod="openshift-monitoring/metrics-server-5845695f9b-7vfc4"
Nov 01 03:07:20.711604 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:20.711539    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxk8s\" (UniqueName: \"kubernetes.io/projected/147c6bba-08a4-4956-9854-c93fa11bbd15-kube-api-access-hxk8s\") pod \"metrics-server-5845695f9b-7vfc4\" (UID: \"147c6bba-08a4-4956-9854-c93fa11bbd15\") " pod="openshift-monitoring/metrics-server-5845695f9b-7vfc4"
Nov 01 03:07:20.711736 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:20.711625    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/147c6bba-08a4-4956-9854-c93fa11bbd15-metrics-server-audit-profiles\") pod \"metrics-server-5845695f9b-7vfc4\" (UID: \"147c6bba-08a4-4956-9854-c93fa11bbd15\") " pod="openshift-monitoring/metrics-server-5845695f9b-7vfc4"
Nov 01 03:07:20.711736 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:20.711664    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/147c6bba-08a4-4956-9854-c93fa11bbd15-secret-metrics-server-client-certs\") pod \"metrics-server-5845695f9b-7vfc4\" (UID: \"147c6bba-08a4-4956-9854-c93fa11bbd15\") " pod="openshift-monitoring/metrics-server-5845695f9b-7vfc4"
Nov 01 03:07:20.711736 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:20.711715    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/147c6bba-08a4-4956-9854-c93fa11bbd15-secret-metrics-server-tls\") pod \"metrics-server-5845695f9b-7vfc4\" (UID: \"147c6bba-08a4-4956-9854-c93fa11bbd15\") " pod="openshift-monitoring/metrics-server-5845695f9b-7vfc4"
Nov 01 03:07:20.711919 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:20.711751    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/147c6bba-08a4-4956-9854-c93fa11bbd15-client-ca-bundle\") pod \"metrics-server-5845695f9b-7vfc4\" (UID: \"147c6bba-08a4-4956-9854-c93fa11bbd15\") " pod="openshift-monitoring/metrics-server-5845695f9b-7vfc4"
Nov 01 03:07:20.711919 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:20.711775    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/147c6bba-08a4-4956-9854-c93fa11bbd15-audit-log\") pod \"metrics-server-5845695f9b-7vfc4\" (UID: \"147c6bba-08a4-4956-9854-c93fa11bbd15\") " pod="openshift-monitoring/metrics-server-5845695f9b-7vfc4"
Nov 01 03:07:20.711919 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:20.711808    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/147c6bba-08a4-4956-9854-c93fa11bbd15-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5845695f9b-7vfc4\" (UID: \"147c6bba-08a4-4956-9854-c93fa11bbd15\") " pod="openshift-monitoring/metrics-server-5845695f9b-7vfc4"
Nov 01 03:07:20.712377 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:20.712348    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/147c6bba-08a4-4956-9854-c93fa11bbd15-audit-log\") pod \"metrics-server-5845695f9b-7vfc4\" (UID: \"147c6bba-08a4-4956-9854-c93fa11bbd15\") " pod="openshift-monitoring/metrics-server-5845695f9b-7vfc4"
Nov 01 03:07:20.712514 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:20.712445    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/147c6bba-08a4-4956-9854-c93fa11bbd15-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5845695f9b-7vfc4\" (UID: \"147c6bba-08a4-4956-9854-c93fa11bbd15\") " pod="openshift-monitoring/metrics-server-5845695f9b-7vfc4"
Nov 01 03:07:20.712782 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:20.712759    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/147c6bba-08a4-4956-9854-c93fa11bbd15-metrics-server-audit-profiles\") pod \"metrics-server-5845695f9b-7vfc4\" (UID: \"147c6bba-08a4-4956-9854-c93fa11bbd15\") " pod="openshift-monitoring/metrics-server-5845695f9b-7vfc4"
Nov 01 03:07:20.714476 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:20.714450    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/147c6bba-08a4-4956-9854-c93fa11bbd15-client-ca-bundle\") pod \"metrics-server-5845695f9b-7vfc4\" (UID: \"147c6bba-08a4-4956-9854-c93fa11bbd15\") " pod="openshift-monitoring/metrics-server-5845695f9b-7vfc4"
Nov 01 03:07:20.714698 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:20.714665    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/147c6bba-08a4-4956-9854-c93fa11bbd15-secret-metrics-server-client-certs\") pod \"metrics-server-5845695f9b-7vfc4\" (UID: \"147c6bba-08a4-4956-9854-c93fa11bbd15\") " pod="openshift-monitoring/metrics-server-5845695f9b-7vfc4"
Nov 01 03:07:20.714850 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:20.714835    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/147c6bba-08a4-4956-9854-c93fa11bbd15-secret-metrics-server-tls\") pod \"metrics-server-5845695f9b-7vfc4\" (UID: \"147c6bba-08a4-4956-9854-c93fa11bbd15\") " pod="openshift-monitoring/metrics-server-5845695f9b-7vfc4"
Nov 01 03:07:20.774256 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:20.774228    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxk8s\" (UniqueName: \"kubernetes.io/projected/147c6bba-08a4-4956-9854-c93fa11bbd15-kube-api-access-hxk8s\") pod \"metrics-server-5845695f9b-7vfc4\" (UID: \"147c6bba-08a4-4956-9854-c93fa11bbd15\") " pod="openshift-monitoring/metrics-server-5845695f9b-7vfc4"
Nov 01 03:07:20.847475 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:20.847439    4400 generic.go:356] "Generic (PLEG): container finished" podID="95482def-35e0-4879-a721-d1ce6d251061" containerID="1901de61e1fa20eff47f04545ffe0dd59b2ed19fffd393d9bf527bb2899f3dba" exitCode=0
Nov 01 03:07:20.847912 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:20.847624    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"95482def-35e0-4879-a721-d1ce6d251061","Type":"ContainerDied","Data":"1901de61e1fa20eff47f04545ffe0dd59b2ed19fffd393d9bf527bb2899f3dba"}
Nov 01 03:07:20.875160 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:20.875139    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5845695f9b-7vfc4"
Nov 01 03:07:20.911886 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:20.911857    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-l5ng6"]
Nov 01 03:07:20.915962 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:07:20.915939    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32b3b49b_3371_4455_a4c5_a4152e66cc7d.slice/crio-a592431ae626a6aeed686dead5dfd343b4cd605326b527cb963209616ce3050b WatchSource:0}: Error finding container a592431ae626a6aeed686dead5dfd343b4cd605326b527cb963209616ce3050b: Status 404 returned error can't find the container with id a592431ae626a6aeed686dead5dfd343b4cd605326b527cb963209616ce3050b
Nov 01 03:07:21.271843 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:21.271810    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5845695f9b-7vfc4"]
Nov 01 03:07:21.276560 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:07:21.276530    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod147c6bba_08a4_4956_9854_c93fa11bbd15.slice/crio-81a812dea474771c4ce7ebc4c50109e4d9cd4ab3d954b711911d993c65d5a94f WatchSource:0}: Error finding container 81a812dea474771c4ce7ebc4c50109e4d9cd4ab3d954b711911d993c65d5a94f: Status 404 returned error can't find the container with id 81a812dea474771c4ce7ebc4c50109e4d9cd4ab3d954b711911d993c65d5a94f
Nov 01 03:07:21.311824 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:21.311794    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-6cc696c965-lpr8d"]
Nov 01 03:07:21.316790 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:21.316769    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6cc696c965-lpr8d"
Nov 01 03:07:21.376816 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:21.376781    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Nov 01 03:07:21.376934 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:21.376918    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Nov 01 03:07:21.377086 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:21.377070    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Nov 01 03:07:21.377953 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:21.377935    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-9qbf8\""
Nov 01 03:07:21.378052 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:21.377992    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Nov 01 03:07:21.379902 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:21.379881    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Nov 01 03:07:21.380040 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:21.380022    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Nov 01 03:07:21.392127 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:21.392107    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6cc696c965-lpr8d"]
Nov 01 03:07:21.418802 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:21.418777    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a18ff3a3-2339-4b95-8239-6bea17756751-serving-certs-ca-bundle\") pod \"telemeter-client-6cc696c965-lpr8d\" (UID: \"a18ff3a3-2339-4b95-8239-6bea17756751\") " pod="openshift-monitoring/telemeter-client-6cc696c965-lpr8d"
Nov 01 03:07:21.418907 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:21.418815    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a18ff3a3-2339-4b95-8239-6bea17756751-secret-telemeter-client\") pod \"telemeter-client-6cc696c965-lpr8d\" (UID: \"a18ff3a3-2339-4b95-8239-6bea17756751\") " pod="openshift-monitoring/telemeter-client-6cc696c965-lpr8d"
Nov 01 03:07:21.418907 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:21.418842    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a18ff3a3-2339-4b95-8239-6bea17756751-telemeter-client-tls\") pod \"telemeter-client-6cc696c965-lpr8d\" (UID: \"a18ff3a3-2339-4b95-8239-6bea17756751\") " pod="openshift-monitoring/telemeter-client-6cc696c965-lpr8d"
Nov 01 03:07:21.418994 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:21.418920    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a18ff3a3-2339-4b95-8239-6bea17756751-federate-client-tls\") pod \"telemeter-client-6cc696c965-lpr8d\" (UID: \"a18ff3a3-2339-4b95-8239-6bea17756751\") " pod="openshift-monitoring/telemeter-client-6cc696c965-lpr8d"
Nov 01 03:07:21.419045 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:21.419028    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a18ff3a3-2339-4b95-8239-6bea17756751-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6cc696c965-lpr8d\" (UID: \"a18ff3a3-2339-4b95-8239-6bea17756751\") " pod="openshift-monitoring/telemeter-client-6cc696c965-lpr8d"
Nov 01 03:07:21.419094 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:21.419046    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a18ff3a3-2339-4b95-8239-6bea17756751-metrics-client-ca\") pod \"telemeter-client-6cc696c965-lpr8d\" (UID: \"a18ff3a3-2339-4b95-8239-6bea17756751\") " pod="openshift-monitoring/telemeter-client-6cc696c965-lpr8d"
Nov 01 03:07:21.419094 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:21.419072    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdccg\" (UniqueName: \"kubernetes.io/projected/a18ff3a3-2339-4b95-8239-6bea17756751-kube-api-access-pdccg\") pod \"telemeter-client-6cc696c965-lpr8d\" (UID: \"a18ff3a3-2339-4b95-8239-6bea17756751\") " pod="openshift-monitoring/telemeter-client-6cc696c965-lpr8d"
Nov 01 03:07:21.419248 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:21.419224    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a18ff3a3-2339-4b95-8239-6bea17756751-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6cc696c965-lpr8d\" (UID: \"a18ff3a3-2339-4b95-8239-6bea17756751\") " pod="openshift-monitoring/telemeter-client-6cc696c965-lpr8d"
Nov 01 03:07:21.520216 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:21.520181    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a18ff3a3-2339-4b95-8239-6bea17756751-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6cc696c965-lpr8d\" (UID: \"a18ff3a3-2339-4b95-8239-6bea17756751\") " pod="openshift-monitoring/telemeter-client-6cc696c965-lpr8d"
Nov 01 03:07:21.520387 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:21.520272    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a18ff3a3-2339-4b95-8239-6bea17756751-serving-certs-ca-bundle\") pod \"telemeter-client-6cc696c965-lpr8d\" (UID: \"a18ff3a3-2339-4b95-8239-6bea17756751\") " pod="openshift-monitoring/telemeter-client-6cc696c965-lpr8d"
Nov 01 03:07:21.520387 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:21.520297    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a18ff3a3-2339-4b95-8239-6bea17756751-secret-telemeter-client\") pod \"telemeter-client-6cc696c965-lpr8d\" (UID: \"a18ff3a3-2339-4b95-8239-6bea17756751\") " pod="openshift-monitoring/telemeter-client-6cc696c965-lpr8d"
Nov 01 03:07:21.520387 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:21.520320    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a18ff3a3-2339-4b95-8239-6bea17756751-telemeter-client-tls\") pod \"telemeter-client-6cc696c965-lpr8d\" (UID: \"a18ff3a3-2339-4b95-8239-6bea17756751\") " pod="openshift-monitoring/telemeter-client-6cc696c965-lpr8d"
Nov 01 03:07:21.520387 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:21.520347    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a18ff3a3-2339-4b95-8239-6bea17756751-federate-client-tls\") pod \"telemeter-client-6cc696c965-lpr8d\" (UID: \"a18ff3a3-2339-4b95-8239-6bea17756751\") " pod="openshift-monitoring/telemeter-client-6cc696c965-lpr8d"
Nov 01 03:07:21.520600 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:21.520398    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a18ff3a3-2339-4b95-8239-6bea17756751-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6cc696c965-lpr8d\" (UID: \"a18ff3a3-2339-4b95-8239-6bea17756751\") " pod="openshift-monitoring/telemeter-client-6cc696c965-lpr8d"
Nov 01 03:07:21.520600 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:21.520422    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a18ff3a3-2339-4b95-8239-6bea17756751-metrics-client-ca\") pod \"telemeter-client-6cc696c965-lpr8d\" (UID: \"a18ff3a3-2339-4b95-8239-6bea17756751\") " pod="openshift-monitoring/telemeter-client-6cc696c965-lpr8d"
Nov 01 03:07:21.520600 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:21.520462    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdccg\" (UniqueName: \"kubernetes.io/projected/a18ff3a3-2339-4b95-8239-6bea17756751-kube-api-access-pdccg\") pod \"telemeter-client-6cc696c965-lpr8d\" (UID: \"a18ff3a3-2339-4b95-8239-6bea17756751\") " pod="openshift-monitoring/telemeter-client-6cc696c965-lpr8d"
Nov 01 03:07:21.521085 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:21.521057    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a18ff3a3-2339-4b95-8239-6bea17756751-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6cc696c965-lpr8d\" (UID: \"a18ff3a3-2339-4b95-8239-6bea17756751\") " pod="openshift-monitoring/telemeter-client-6cc696c965-lpr8d"
Nov 01 03:07:21.522076 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:21.521430    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a18ff3a3-2339-4b95-8239-6bea17756751-serving-certs-ca-bundle\") pod \"telemeter-client-6cc696c965-lpr8d\" (UID: \"a18ff3a3-2339-4b95-8239-6bea17756751\") " pod="openshift-monitoring/telemeter-client-6cc696c965-lpr8d"
Nov 01 03:07:21.522457 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:21.522431    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a18ff3a3-2339-4b95-8239-6bea17756751-metrics-client-ca\") pod \"telemeter-client-6cc696c965-lpr8d\" (UID: \"a18ff3a3-2339-4b95-8239-6bea17756751\") " pod="openshift-monitoring/telemeter-client-6cc696c965-lpr8d"
Nov 01 03:07:21.523939 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:21.523911    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a18ff3a3-2339-4b95-8239-6bea17756751-federate-client-tls\") pod \"telemeter-client-6cc696c965-lpr8d\" (UID: \"a18ff3a3-2339-4b95-8239-6bea17756751\") " pod="openshift-monitoring/telemeter-client-6cc696c965-lpr8d"
Nov 01 03:07:21.524162 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:21.524119    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a18ff3a3-2339-4b95-8239-6bea17756751-secret-telemeter-client\") pod \"telemeter-client-6cc696c965-lpr8d\" (UID: \"a18ff3a3-2339-4b95-8239-6bea17756751\") " pod="openshift-monitoring/telemeter-client-6cc696c965-lpr8d"
Nov 01 03:07:21.524393 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:21.524368    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a18ff3a3-2339-4b95-8239-6bea17756751-telemeter-client-tls\") pod \"telemeter-client-6cc696c965-lpr8d\" (UID: \"a18ff3a3-2339-4b95-8239-6bea17756751\") " pod="openshift-monitoring/telemeter-client-6cc696c965-lpr8d"
Nov 01 03:07:21.524393 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:21.524382    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a18ff3a3-2339-4b95-8239-6bea17756751-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6cc696c965-lpr8d\" (UID: \"a18ff3a3-2339-4b95-8239-6bea17756751\") " pod="openshift-monitoring/telemeter-client-6cc696c965-lpr8d"
Nov 01 03:07:21.588054 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:21.588029    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdccg\" (UniqueName: \"kubernetes.io/projected/a18ff3a3-2339-4b95-8239-6bea17756751-kube-api-access-pdccg\") pod \"telemeter-client-6cc696c965-lpr8d\" (UID: \"a18ff3a3-2339-4b95-8239-6bea17756751\") " pod="openshift-monitoring/telemeter-client-6cc696c965-lpr8d"
Nov 01 03:07:21.634881 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:21.634859    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6cc696c965-lpr8d"
Nov 01 03:07:21.857295 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:21.857266    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-l5ng6" event={"ID":"32b3b49b-3371-4455-a4c5-a4152e66cc7d","Type":"ContainerStarted","Data":"a592431ae626a6aeed686dead5dfd343b4cd605326b527cb963209616ce3050b"}
Nov 01 03:07:21.858572 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:21.858547    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5845695f9b-7vfc4" event={"ID":"147c6bba-08a4-4956-9854-c93fa11bbd15","Type":"ContainerStarted","Data":"81a812dea474771c4ce7ebc4c50109e4d9cd4ab3d954b711911d993c65d5a94f"}
Nov 01 03:07:22.077316 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:22.077286    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6cc696c965-lpr8d"]
Nov 01 03:07:22.081900 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:07:22.081858    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda18ff3a3_2339_4b95_8239_6bea17756751.slice/crio-52532281f8588b70ed0981a90b23e8a2acaf9ff37b23bf43725bf69a1f9a94c1 WatchSource:0}: Error finding container 52532281f8588b70ed0981a90b23e8a2acaf9ff37b23bf43725bf69a1f9a94c1: Status 404 returned error can't find the container with id 52532281f8588b70ed0981a90b23e8a2acaf9ff37b23bf43725bf69a1f9a94c1
Nov 01 03:07:22.865445 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:22.865327    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6cc696c965-lpr8d" event={"ID":"a18ff3a3-2339-4b95-8239-6bea17756751","Type":"ContainerStarted","Data":"52532281f8588b70ed0981a90b23e8a2acaf9ff37b23bf43725bf69a1f9a94c1"}
Nov 01 03:07:22.868800 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:22.868764    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-59d787964f-6q8gp" event={"ID":"570761ed-076a-4c28-a14b-46471838f93c","Type":"ContainerStarted","Data":"82f01b8030b1d69e663116c326ad5629a1e83a543c2c84978b2324451091af6f"}
Nov 01 03:07:22.869039 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:22.868843    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-59d787964f-6q8gp" event={"ID":"570761ed-076a-4c28-a14b-46471838f93c","Type":"ContainerStarted","Data":"ea84538dfa9ca590b0b33b9ee932d99b65232607a6d2b83044b6d45addddb811"}
Nov 01 03:07:22.869039 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:22.868888    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-59d787964f-6q8gp" event={"ID":"570761ed-076a-4c28-a14b-46471838f93c","Type":"ContainerStarted","Data":"6a56ed4d7cd89a131c3593b10a558ce42a3ff08dd07d449251aa3a4ef91b2e09"}
Nov 01 03:07:23.876653 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:23.876622    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5845695f9b-7vfc4" event={"ID":"147c6bba-08a4-4956-9854-c93fa11bbd15","Type":"ContainerStarted","Data":"cdb783686fcb06d7c469a5a1f2e8fed374c3e58c9663e1f6e0f98268e37f1125"}
Nov 01 03:07:23.882613 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:23.882582    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-l5ng6" event={"ID":"32b3b49b-3371-4455-a4c5-a4152e66cc7d","Type":"ContainerStarted","Data":"32e909ecf2cbfb7f6269d3bc66694ecd952f9eb7fc094aef3451eb21c20276a4"}
Nov 01 03:07:23.883420 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:23.883365    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-l5ng6"
Nov 01 03:07:23.886606 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:23.886586    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-59d787964f-6q8gp" event={"ID":"570761ed-076a-4c28-a14b-46471838f93c","Type":"ContainerStarted","Data":"944b729c8a124df8a2615bc72a94a849629d8c6d9261b67b1f416690f3201749"}
Nov 01 03:07:23.889292 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:23.889233    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-l5ng6"
Nov 01 03:07:23.891501 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:23.891475    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"95482def-35e0-4879-a721-d1ce6d251061","Type":"ContainerStarted","Data":"2459da8a6b1d6965c7ca185ef8e6fcd0b8ece1d4d537ff3faaf04376ef1cd12d"}
Nov 01 03:07:23.891591 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:23.891544    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"95482def-35e0-4879-a721-d1ce6d251061","Type":"ContainerStarted","Data":"44f2d215c8d28a22ac3b18aba136454d6c518c33952c0ba3ce905e0ca4a52a99"}
Nov 01 03:07:23.891591 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:23.891585    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"95482def-35e0-4879-a721-d1ce6d251061","Type":"ContainerStarted","Data":"c5ee8cef5bccf833843ae19213ca37a09919502ccbc56af21cd5324fd51373c8"}
Nov 01 03:07:23.891740 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:23.891624    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"95482def-35e0-4879-a721-d1ce6d251061","Type":"ContainerStarted","Data":"5c5adee9bbcfd85f53a6d76b11100ec7ef4b40c433b3bdd4cc2ed7f299d987c1"}
Nov 01 03:07:23.891740 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:23.891662    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"95482def-35e0-4879-a721-d1ce6d251061","Type":"ContainerStarted","Data":"960d7c3027418725815287ced9709538984dc0ba28a6a826382c9a227041f7df"}
Nov 01 03:07:24.000560 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:24.000514    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5845695f9b-7vfc4" podStartSLOduration=2.024494617 podStartE2EDuration="4.000491677s" totalImagesPullingTime="1.97599706s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:07:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:07:23.999988572 +0000 UTC m=+50.514634436" watchObservedRunningTime="2026-11-01 03:07:24.000491677 +0000 UTC m=+50.515137543"
Nov 01 03:07:24.124480 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:24.124423    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-l5ng6" podStartSLOduration=2.023995075 podStartE2EDuration="4.124404716s" totalImagesPullingTime="2.100409641s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:07:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:07:24.123111206 +0000 UTC m=+50.637757070" watchObservedRunningTime="2026-11-01 03:07:24.124404716 +0000 UTC m=+50.639050577"
Nov 01 03:07:24.701471 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:24.701423    4400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6cf7cdf566-2f2jh"
Nov 01 03:07:24.701597 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:24.701502    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6cf7cdf566-2f2jh"
Nov 01 03:07:24.706390 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:24.706368    4400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6cf7cdf566-2f2jh"
Nov 01 03:07:24.898476 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:24.898442    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-59d787964f-6q8gp" event={"ID":"570761ed-076a-4c28-a14b-46471838f93c","Type":"ContainerStarted","Data":"bd729306440acbf0ace5d757aca5e450821c23d78f000bfdf202597e7857d94e"}
Nov 01 03:07:24.898954 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:24.898517    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-59d787964f-6q8gp" event={"ID":"570761ed-076a-4c28-a14b-46471838f93c","Type":"ContainerStarted","Data":"43cc15ac51a8b5338f97b1315deb0700c4ac2b87e50b05e7c71f0a85fca02104"}
Nov 01 03:07:24.898954 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:24.898648    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-59d787964f-6q8gp"
Nov 01 03:07:24.902439 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:24.902328    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"95482def-35e0-4879-a721-d1ce6d251061","Type":"ContainerStarted","Data":"87fe732655cd942b05e12ea5580443ce47acccbe7bc19e848e2b0f234722259f"}
Nov 01 03:07:24.904738 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:24.904711    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6cc696c965-lpr8d" event={"ID":"a18ff3a3-2339-4b95-8239-6bea17756751","Type":"ContainerStarted","Data":"630474a35a7b2b0cf7856aae18d99b095086d906c06ec65198dae104e5050c50"}
Nov 01 03:07:24.904872 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:24.904759    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6cc696c965-lpr8d" event={"ID":"a18ff3a3-2339-4b95-8239-6bea17756751","Type":"ContainerStarted","Data":"c940b7b76a95d32f19161c8cd490e6157f5f263ab4bcd77eb6e968809dc2821a"}
Nov 01 03:07:24.904872 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:24.904785    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6cc696c965-lpr8d" event={"ID":"a18ff3a3-2339-4b95-8239-6bea17756751","Type":"ContainerStarted","Data":"e2aac86697a3898eeb1b326642fec484435b86bbf95489e06c673f7799a09924"}
Nov 01 03:07:24.909574 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:24.909554    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6cf7cdf566-2f2jh"
Nov 01 03:07:25.054791 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:25.054652    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-59d787964f-6q8gp" podStartSLOduration=2.497431913 podStartE2EDuration="7.054634942s" totalImagesPullingTime="4.557203029s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:07:18 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:07:25.053805923 +0000 UTC m=+51.568451786" watchObservedRunningTime="2026-11-01 03:07:25.054634942 +0000 UTC m=+51.569280836"
Nov 01 03:07:25.181581 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:25.181520    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-6cc696c965-lpr8d" podStartSLOduration=1.692343838 podStartE2EDuration="4.181505958s" totalImagesPullingTime="2.48916212s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:07:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:07:25.180346174 +0000 UTC m=+51.694992038" watchObservedRunningTime="2026-11-01 03:07:25.181505958 +0000 UTC m=+51.696151821"
Nov 01 03:07:25.443767 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:25.443711    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.335338803 podStartE2EDuration="8.443678381s" totalImagesPullingTime="5.10362065s" totalInitContainerRuntime="4.718928ms" isStatefulPod=false podCreationTimestamp="2026-11-01 03:07:17 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:07:25.438344975 +0000 UTC m=+51.952990842" watchObservedRunningTime="2026-11-01 03:07:25.443678381 +0000 UTC m=+51.958324260"
Nov 01 03:07:25.975145 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:25.975114    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d488f58b6-s5k7j"]
Nov 01 03:07:26.068626 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:26.068599    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-c5558cd55-7tq5f"]
Nov 01 03:07:26.073994 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:26.073978    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c5558cd55-7tq5f"
Nov 01 03:07:26.136234 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:26.136204    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c5558cd55-7tq5f"]
Nov 01 03:07:26.163928 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:26.163900    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c2a5f25-e674-4b6d-95b8-b898412e095e-tmp\") pod \"console-c5558cd55-7tq5f\" (UID: \"9c2a5f25-e674-4b6d-95b8-b898412e095e\") " pod="openshift-console/console-c5558cd55-7tq5f"
Nov 01 03:07:26.164023 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:26.163932    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9c2a5f25-e674-4b6d-95b8-b898412e095e-console-oauth-config\") pod \"console-c5558cd55-7tq5f\" (UID: \"9c2a5f25-e674-4b6d-95b8-b898412e095e\") " pod="openshift-console/console-c5558cd55-7tq5f"
Nov 01 03:07:26.164023 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:26.163965    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9c2a5f25-e674-4b6d-95b8-b898412e095e-console-config\") pod \"console-c5558cd55-7tq5f\" (UID: \"9c2a5f25-e674-4b6d-95b8-b898412e095e\") " pod="openshift-console/console-c5558cd55-7tq5f"
Nov 01 03:07:26.164023 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:26.163984    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbq98\" (UniqueName: \"kubernetes.io/projected/9c2a5f25-e674-4b6d-95b8-b898412e095e-kube-api-access-sbq98\") pod \"console-c5558cd55-7tq5f\" (UID: \"9c2a5f25-e674-4b6d-95b8-b898412e095e\") " pod="openshift-console/console-c5558cd55-7tq5f"
Nov 01 03:07:26.164130 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:26.164088    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9c2a5f25-e674-4b6d-95b8-b898412e095e-service-ca\") pod \"console-c5558cd55-7tq5f\" (UID: \"9c2a5f25-e674-4b6d-95b8-b898412e095e\") " pod="openshift-console/console-c5558cd55-7tq5f"
Nov 01 03:07:26.164130 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:26.164109    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9c2a5f25-e674-4b6d-95b8-b898412e095e-oauth-serving-cert\") pod \"console-c5558cd55-7tq5f\" (UID: \"9c2a5f25-e674-4b6d-95b8-b898412e095e\") " pod="openshift-console/console-c5558cd55-7tq5f"
Nov 01 03:07:26.164197 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:26.164178    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c2a5f25-e674-4b6d-95b8-b898412e095e-trusted-ca-bundle\") pod \"console-c5558cd55-7tq5f\" (UID: \"9c2a5f25-e674-4b6d-95b8-b898412e095e\") " pod="openshift-console/console-c5558cd55-7tq5f"
Nov 01 03:07:26.164248 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:26.164237    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9c2a5f25-e674-4b6d-95b8-b898412e095e-console-serving-cert\") pod \"console-c5558cd55-7tq5f\" (UID: \"9c2a5f25-e674-4b6d-95b8-b898412e095e\") " pod="openshift-console/console-c5558cd55-7tq5f"
Nov 01 03:07:26.264913 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:26.264838    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9c2a5f25-e674-4b6d-95b8-b898412e095e-console-config\") pod \"console-c5558cd55-7tq5f\" (UID: \"9c2a5f25-e674-4b6d-95b8-b898412e095e\") " pod="openshift-console/console-c5558cd55-7tq5f"
Nov 01 03:07:26.264913 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:26.264878    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbq98\" (UniqueName: \"kubernetes.io/projected/9c2a5f25-e674-4b6d-95b8-b898412e095e-kube-api-access-sbq98\") pod \"console-c5558cd55-7tq5f\" (UID: \"9c2a5f25-e674-4b6d-95b8-b898412e095e\") " pod="openshift-console/console-c5558cd55-7tq5f"
Nov 01 03:07:26.265108 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:26.264997    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9c2a5f25-e674-4b6d-95b8-b898412e095e-service-ca\") pod \"console-c5558cd55-7tq5f\" (UID: \"9c2a5f25-e674-4b6d-95b8-b898412e095e\") " pod="openshift-console/console-c5558cd55-7tq5f"
Nov 01 03:07:26.265108 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:26.265023    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9c2a5f25-e674-4b6d-95b8-b898412e095e-oauth-serving-cert\") pod \"console-c5558cd55-7tq5f\" (UID: \"9c2a5f25-e674-4b6d-95b8-b898412e095e\") " pod="openshift-console/console-c5558cd55-7tq5f"
Nov 01 03:07:26.265108 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:26.265063    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c2a5f25-e674-4b6d-95b8-b898412e095e-trusted-ca-bundle\") pod \"console-c5558cd55-7tq5f\" (UID: \"9c2a5f25-e674-4b6d-95b8-b898412e095e\") " pod="openshift-console/console-c5558cd55-7tq5f"
Nov 01 03:07:26.265108 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:26.265100    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9c2a5f25-e674-4b6d-95b8-b898412e095e-console-serving-cert\") pod \"console-c5558cd55-7tq5f\" (UID: \"9c2a5f25-e674-4b6d-95b8-b898412e095e\") " pod="openshift-console/console-c5558cd55-7tq5f"
Nov 01 03:07:26.265307 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:26.265123    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c2a5f25-e674-4b6d-95b8-b898412e095e-tmp\") pod \"console-c5558cd55-7tq5f\" (UID: \"9c2a5f25-e674-4b6d-95b8-b898412e095e\") " pod="openshift-console/console-c5558cd55-7tq5f"
Nov 01 03:07:26.265307 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:26.265146    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9c2a5f25-e674-4b6d-95b8-b898412e095e-console-oauth-config\") pod \"console-c5558cd55-7tq5f\" (UID: \"9c2a5f25-e674-4b6d-95b8-b898412e095e\") " pod="openshift-console/console-c5558cd55-7tq5f"
Nov 01 03:07:26.265521 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:26.265494    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c2a5f25-e674-4b6d-95b8-b898412e095e-tmp\") pod \"console-c5558cd55-7tq5f\" (UID: \"9c2a5f25-e674-4b6d-95b8-b898412e095e\") " pod="openshift-console/console-c5558cd55-7tq5f"
Nov 01 03:07:26.265714 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:26.265667    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9c2a5f25-e674-4b6d-95b8-b898412e095e-console-config\") pod \"console-c5558cd55-7tq5f\" (UID: \"9c2a5f25-e674-4b6d-95b8-b898412e095e\") " pod="openshift-console/console-c5558cd55-7tq5f"
Nov 01 03:07:26.265805 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:26.265747    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9c2a5f25-e674-4b6d-95b8-b898412e095e-oauth-serving-cert\") pod \"console-c5558cd55-7tq5f\" (UID: \"9c2a5f25-e674-4b6d-95b8-b898412e095e\") " pod="openshift-console/console-c5558cd55-7tq5f"
Nov 01 03:07:26.266026 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:26.265815    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9c2a5f25-e674-4b6d-95b8-b898412e095e-service-ca\") pod \"console-c5558cd55-7tq5f\" (UID: \"9c2a5f25-e674-4b6d-95b8-b898412e095e\") " pod="openshift-console/console-c5558cd55-7tq5f"
Nov 01 03:07:26.266258 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:26.266235    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c2a5f25-e674-4b6d-95b8-b898412e095e-trusted-ca-bundle\") pod \"console-c5558cd55-7tq5f\" (UID: \"9c2a5f25-e674-4b6d-95b8-b898412e095e\") " pod="openshift-console/console-c5558cd55-7tq5f"
Nov 01 03:07:26.267770 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:26.267752    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9c2a5f25-e674-4b6d-95b8-b898412e095e-console-serving-cert\") pod \"console-c5558cd55-7tq5f\" (UID: \"9c2a5f25-e674-4b6d-95b8-b898412e095e\") " pod="openshift-console/console-c5558cd55-7tq5f"
Nov 01 03:07:26.267841 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:26.267770    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9c2a5f25-e674-4b6d-95b8-b898412e095e-console-oauth-config\") pod \"console-c5558cd55-7tq5f\" (UID: \"9c2a5f25-e674-4b6d-95b8-b898412e095e\") " pod="openshift-console/console-c5558cd55-7tq5f"
Nov 01 03:07:26.338323 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:26.338293    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbq98\" (UniqueName: \"kubernetes.io/projected/9c2a5f25-e674-4b6d-95b8-b898412e095e-kube-api-access-sbq98\") pod \"console-c5558cd55-7tq5f\" (UID: \"9c2a5f25-e674-4b6d-95b8-b898412e095e\") " pod="openshift-console/console-c5558cd55-7tq5f"
Nov 01 03:07:26.385356 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:26.385330    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c5558cd55-7tq5f"
Nov 01 03:07:26.772755 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:26.772735    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c5558cd55-7tq5f"]
Nov 01 03:07:26.774768 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:07:26.774744    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c2a5f25_e674_4b6d_95b8_b898412e095e.slice/crio-c73a2ca3eb7e4adfdfea1569d9f86e439a3a5bb8fc1b16a8d03a3f4ff49de439 WatchSource:0}: Error finding container c73a2ca3eb7e4adfdfea1569d9f86e439a3a5bb8fc1b16a8d03a3f4ff49de439: Status 404 returned error can't find the container with id c73a2ca3eb7e4adfdfea1569d9f86e439a3a5bb8fc1b16a8d03a3f4ff49de439
Nov 01 03:07:26.918717 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:26.916333    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c5558cd55-7tq5f" event={"ID":"9c2a5f25-e674-4b6d-95b8-b898412e095e","Type":"ContainerStarted","Data":"a64b8259d5ae70dd6a79f15d439f4664789f60256de8a20bb56bb824659f5b9b"}
Nov 01 03:07:26.918717 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:26.916393    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c5558cd55-7tq5f" event={"ID":"9c2a5f25-e674-4b6d-95b8-b898412e095e","Type":"ContainerStarted","Data":"c73a2ca3eb7e4adfdfea1569d9f86e439a3a5bb8fc1b16a8d03a3f4ff49de439"}
Nov 01 03:07:27.038861 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:27.038761    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-c5558cd55-7tq5f" podStartSLOduration=1.03874684 podStartE2EDuration="1.03874684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:07:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:07:27.038351247 +0000 UTC m=+53.552997109" watchObservedRunningTime="2026-11-01 03:07:27.03874684 +0000 UTC m=+53.553392700"
Nov 01 03:07:27.737776 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:27.737745    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5d488f58b6-s5k7j"
Nov 01 03:07:30.916481 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:30.916454    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-59d787964f-6q8gp"
Nov 01 03:07:31.728078 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:07:31.728041    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:07:36.386461 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:36.386429    4400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-c5558cd55-7tq5f"
Nov 01 03:07:36.386957 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:36.386478    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-c5558cd55-7tq5f"
Nov 01 03:07:36.390940 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:36.390919    4400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-c5558cd55-7tq5f"
Nov 01 03:07:36.961481 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:36.961459    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-c5558cd55-7tq5f"
Nov 01 03:07:37.134612 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:37.134584    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6cf7cdf566-2f2jh"]
Nov 01 03:07:40.875290 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:40.875255    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-5845695f9b-7vfc4"
Nov 01 03:07:40.875675 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:40.875317    4400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-5845695f9b-7vfc4"
Nov 01 03:07:50.995429 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:50.995387    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-5d488f58b6-s5k7j" podUID="b08de522-96f8-42a8-a88f-0502b999d404" containerName="console" containerID="cri-o://0fa21df1a169c99bd5bf3b928282d4876038c6366a34f47b3be5dced1bed1aac" gracePeriod=15
Nov 01 03:07:51.248901 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:51.248842    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d488f58b6-s5k7j"
Nov 01 03:07:51.363120 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:51.363088    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b08de522-96f8-42a8-a88f-0502b999d404-tmp\" (UniqueName: \"kubernetes.io/empty-dir/b08de522-96f8-42a8-a88f-0502b999d404-tmp\") pod \"b08de522-96f8-42a8-a88f-0502b999d404\" (UID: \"b08de522-96f8-42a8-a88f-0502b999d404\") "
Nov 01 03:07:51.363306 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:51.363135    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b08de522-96f8-42a8-a88f-0502b999d404-kube-api-access-fbt72\" (UniqueName: \"kubernetes.io/projected/b08de522-96f8-42a8-a88f-0502b999d404-kube-api-access-fbt72\") pod \"b08de522-96f8-42a8-a88f-0502b999d404\" (UID: \"b08de522-96f8-42a8-a88f-0502b999d404\") "
Nov 01 03:07:51.363306 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:51.363180    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b08de522-96f8-42a8-a88f-0502b999d404-service-ca\" (UniqueName: \"kubernetes.io/configmap/b08de522-96f8-42a8-a88f-0502b999d404-service-ca\") pod \"b08de522-96f8-42a8-a88f-0502b999d404\" (UID: \"b08de522-96f8-42a8-a88f-0502b999d404\") "
Nov 01 03:07:51.363306 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:51.363202    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b08de522-96f8-42a8-a88f-0502b999d404-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b08de522-96f8-42a8-a88f-0502b999d404-trusted-ca-bundle\") pod \"b08de522-96f8-42a8-a88f-0502b999d404\" (UID: \"b08de522-96f8-42a8-a88f-0502b999d404\") "
Nov 01 03:07:51.363306 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:51.363226    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b08de522-96f8-42a8-a88f-0502b999d404-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b08de522-96f8-42a8-a88f-0502b999d404-oauth-serving-cert\") pod \"b08de522-96f8-42a8-a88f-0502b999d404\" (UID: \"b08de522-96f8-42a8-a88f-0502b999d404\") "
Nov 01 03:07:51.363306 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:51.363243    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b08de522-96f8-42a8-a88f-0502b999d404-console-config\" (UniqueName: \"kubernetes.io/configmap/b08de522-96f8-42a8-a88f-0502b999d404-console-config\") pod \"b08de522-96f8-42a8-a88f-0502b999d404\" (UID: \"b08de522-96f8-42a8-a88f-0502b999d404\") "
Nov 01 03:07:51.363560 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:51.363312    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b08de522-96f8-42a8-a88f-0502b999d404-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b08de522-96f8-42a8-a88f-0502b999d404-console-serving-cert\") pod \"b08de522-96f8-42a8-a88f-0502b999d404\" (UID: \"b08de522-96f8-42a8-a88f-0502b999d404\") "
Nov 01 03:07:51.363560 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:51.363366    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b08de522-96f8-42a8-a88f-0502b999d404-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b08de522-96f8-42a8-a88f-0502b999d404-console-oauth-config\") pod \"b08de522-96f8-42a8-a88f-0502b999d404\" (UID: \"b08de522-96f8-42a8-a88f-0502b999d404\") "
Nov 01 03:07:51.363560 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:51.363463    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b08de522-96f8-42a8-a88f-0502b999d404-tmp" pod "b08de522-96f8-42a8-a88f-0502b999d404" (UID: "b08de522-96f8-42a8-a88f-0502b999d404"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:07:51.363745 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:51.363661    4400 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b08de522-96f8-42a8-a88f-0502b999d404-tmp\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:07:51.363745 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:51.363730    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b08de522-96f8-42a8-a88f-0502b999d404-service-ca" pod "b08de522-96f8-42a8-a88f-0502b999d404" (UID: "b08de522-96f8-42a8-a88f-0502b999d404"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:07:51.363890 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:51.363859    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b08de522-96f8-42a8-a88f-0502b999d404-oauth-serving-cert" pod "b08de522-96f8-42a8-a88f-0502b999d404" (UID: "b08de522-96f8-42a8-a88f-0502b999d404"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:07:51.364012 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:51.363985    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b08de522-96f8-42a8-a88f-0502b999d404-console-config" pod "b08de522-96f8-42a8-a88f-0502b999d404" (UID: "b08de522-96f8-42a8-a88f-0502b999d404"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:07:51.364081 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:51.363988    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b08de522-96f8-42a8-a88f-0502b999d404-trusted-ca-bundle" pod "b08de522-96f8-42a8-a88f-0502b999d404" (UID: "b08de522-96f8-42a8-a88f-0502b999d404"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:07:51.365970 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:51.365943    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b08de522-96f8-42a8-a88f-0502b999d404-console-serving-cert" pod "b08de522-96f8-42a8-a88f-0502b999d404" (UID: "b08de522-96f8-42a8-a88f-0502b999d404"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:07:51.366109 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:51.365973    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b08de522-96f8-42a8-a88f-0502b999d404-kube-api-access-fbt72" pod "b08de522-96f8-42a8-a88f-0502b999d404" (UID: "b08de522-96f8-42a8-a88f-0502b999d404"). InnerVolumeSpecName "kube-api-access-fbt72". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:07:51.366109 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:51.365984    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b08de522-96f8-42a8-a88f-0502b999d404-console-oauth-config" pod "b08de522-96f8-42a8-a88f-0502b999d404" (UID: "b08de522-96f8-42a8-a88f-0502b999d404"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:07:51.464445 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:51.464418    4400 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b08de522-96f8-42a8-a88f-0502b999d404-console-serving-cert\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:07:51.464445 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:51.464440    4400 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b08de522-96f8-42a8-a88f-0502b999d404-console-oauth-config\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:07:51.464445 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:51.464449    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbt72\" (UniqueName: \"kubernetes.io/projected/b08de522-96f8-42a8-a88f-0502b999d404-kube-api-access-fbt72\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:07:51.464723 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:51.464461    4400 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b08de522-96f8-42a8-a88f-0502b999d404-service-ca\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:07:51.464723 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:51.464469    4400 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b08de522-96f8-42a8-a88f-0502b999d404-trusted-ca-bundle\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:07:51.464723 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:51.464477    4400 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b08de522-96f8-42a8-a88f-0502b999d404-oauth-serving-cert\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:07:51.464723 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:51.464487    4400 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b08de522-96f8-42a8-a88f-0502b999d404-console-config\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:07:52.016180 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:52.016113    4400 generic.go:356] "Generic (PLEG): container finished" podID="b08de522-96f8-42a8-a88f-0502b999d404" containerID="0fa21df1a169c99bd5bf3b928282d4876038c6366a34f47b3be5dced1bed1aac" exitCode=0
Nov 01 03:07:52.016180 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:52.016165    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d488f58b6-s5k7j"
Nov 01 03:07:52.016700 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:52.016189    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d488f58b6-s5k7j" event={"ID":"b08de522-96f8-42a8-a88f-0502b999d404","Type":"ContainerDied","Data":"0fa21df1a169c99bd5bf3b928282d4876038c6366a34f47b3be5dced1bed1aac"}
Nov 01 03:07:52.016700 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:52.016235    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d488f58b6-s5k7j" event={"ID":"b08de522-96f8-42a8-a88f-0502b999d404","Type":"ContainerDied","Data":"c76e5d12ac5a06d322affe91d31b9e3edc3dd94096fd72fb554fce811ecfa753"}
Nov 01 03:07:52.016700 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:52.016262    4400 scope.go:122] "RemoveContainer" containerID="0fa21df1a169c99bd5bf3b928282d4876038c6366a34f47b3be5dced1bed1aac"
Nov 01 03:07:52.024341 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:52.024323    4400 scope.go:122] "RemoveContainer" containerID="0fa21df1a169c99bd5bf3b928282d4876038c6366a34f47b3be5dced1bed1aac"
Nov 01 03:07:52.024589 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:07:52.024568    4400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fa21df1a169c99bd5bf3b928282d4876038c6366a34f47b3be5dced1bed1aac\": container with ID starting with 0fa21df1a169c99bd5bf3b928282d4876038c6366a34f47b3be5dced1bed1aac not found: ID does not exist" containerID="0fa21df1a169c99bd5bf3b928282d4876038c6366a34f47b3be5dced1bed1aac"
Nov 01 03:07:52.024642 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:52.024597    4400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fa21df1a169c99bd5bf3b928282d4876038c6366a34f47b3be5dced1bed1aac"} err="failed to get container status \"0fa21df1a169c99bd5bf3b928282d4876038c6366a34f47b3be5dced1bed1aac\": rpc error: code = NotFound desc = could not find container \"0fa21df1a169c99bd5bf3b928282d4876038c6366a34f47b3be5dced1bed1aac\": container with ID starting with 0fa21df1a169c99bd5bf3b928282d4876038c6366a34f47b3be5dced1bed1aac not found: ID does not exist"
Nov 01 03:07:52.188237 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:52.188207    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d488f58b6-s5k7j"]
Nov 01 03:07:52.239015 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:52.238997    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5d488f58b6-s5k7j"]
Nov 01 03:07:53.612667 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:07:53.612637    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b08de522-96f8-42a8-a88f-0502b999d404" path="/var/lib/kubelet/pods/b08de522-96f8-42a8-a88f-0502b999d404/volumes"
Nov 01 03:08:00.880797 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:00.880769    4400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-5845695f9b-7vfc4"
Nov 01 03:08:00.884625 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:00.884601    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5845695f9b-7vfc4"
Nov 01 03:08:01.731371 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:08:01.731337    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:08:02.154423 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:02.154370    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-6cf7cdf566-2f2jh" podUID="cd688da6-42b3-49ed-96ba-1ae5c6aa3933" containerName="console" containerID="cri-o://7abea0024cf4002e9d661a828471a1b6005ba027e81743261e0085e1dbf830b8" gracePeriod=15
Nov 01 03:08:02.413849 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:02.413789    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cf7cdf566-2f2jh"
Nov 01 03:08:02.449006 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:02.448977    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd688da6-42b3-49ed-96ba-1ae5c6aa3933-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cd688da6-42b3-49ed-96ba-1ae5c6aa3933-console-serving-cert\") pod \"cd688da6-42b3-49ed-96ba-1ae5c6aa3933\" (UID: \"cd688da6-42b3-49ed-96ba-1ae5c6aa3933\") "
Nov 01 03:08:02.449245 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:02.449013    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd688da6-42b3-49ed-96ba-1ae5c6aa3933-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd688da6-42b3-49ed-96ba-1ae5c6aa3933-service-ca\") pod \"cd688da6-42b3-49ed-96ba-1ae5c6aa3933\" (UID: \"cd688da6-42b3-49ed-96ba-1ae5c6aa3933\") "
Nov 01 03:08:02.449245 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:02.449062    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd688da6-42b3-49ed-96ba-1ae5c6aa3933-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cd688da6-42b3-49ed-96ba-1ae5c6aa3933-console-oauth-config\") pod \"cd688da6-42b3-49ed-96ba-1ae5c6aa3933\" (UID: \"cd688da6-42b3-49ed-96ba-1ae5c6aa3933\") "
Nov 01 03:08:02.449245 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:02.449234    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd688da6-42b3-49ed-96ba-1ae5c6aa3933-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cd688da6-42b3-49ed-96ba-1ae5c6aa3933-oauth-serving-cert\") pod \"cd688da6-42b3-49ed-96ba-1ae5c6aa3933\" (UID: \"cd688da6-42b3-49ed-96ba-1ae5c6aa3933\") "
Nov 01 03:08:02.449408 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:02.449267    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd688da6-42b3-49ed-96ba-1ae5c6aa3933-kube-api-access-cxqmf\" (UniqueName: \"kubernetes.io/projected/cd688da6-42b3-49ed-96ba-1ae5c6aa3933-kube-api-access-cxqmf\") pod \"cd688da6-42b3-49ed-96ba-1ae5c6aa3933\" (UID: \"cd688da6-42b3-49ed-96ba-1ae5c6aa3933\") "
Nov 01 03:08:02.449408 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:02.449305    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd688da6-42b3-49ed-96ba-1ae5c6aa3933-console-config\" (UniqueName: \"kubernetes.io/configmap/cd688da6-42b3-49ed-96ba-1ae5c6aa3933-console-config\") pod \"cd688da6-42b3-49ed-96ba-1ae5c6aa3933\" (UID: \"cd688da6-42b3-49ed-96ba-1ae5c6aa3933\") "
Nov 01 03:08:02.449408 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:02.449339    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd688da6-42b3-49ed-96ba-1ae5c6aa3933-tmp\" (UniqueName: \"kubernetes.io/empty-dir/cd688da6-42b3-49ed-96ba-1ae5c6aa3933-tmp\") pod \"cd688da6-42b3-49ed-96ba-1ae5c6aa3933\" (UID: \"cd688da6-42b3-49ed-96ba-1ae5c6aa3933\") "
Nov 01 03:08:02.449564 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:02.449454    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd688da6-42b3-49ed-96ba-1ae5c6aa3933-service-ca" pod "cd688da6-42b3-49ed-96ba-1ae5c6aa3933" (UID: "cd688da6-42b3-49ed-96ba-1ae5c6aa3933"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:08:02.449759 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:02.449678    4400 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cd688da6-42b3-49ed-96ba-1ae5c6aa3933-service-ca\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:08:02.449759 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:02.449757    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd688da6-42b3-49ed-96ba-1ae5c6aa3933-tmp" pod "cd688da6-42b3-49ed-96ba-1ae5c6aa3933" (UID: "cd688da6-42b3-49ed-96ba-1ae5c6aa3933"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:08:02.449759 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:02.449769    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd688da6-42b3-49ed-96ba-1ae5c6aa3933-console-config" pod "cd688da6-42b3-49ed-96ba-1ae5c6aa3933" (UID: "cd688da6-42b3-49ed-96ba-1ae5c6aa3933"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:08:02.450089 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:02.449666    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd688da6-42b3-49ed-96ba-1ae5c6aa3933-oauth-serving-cert" pod "cd688da6-42b3-49ed-96ba-1ae5c6aa3933" (UID: "cd688da6-42b3-49ed-96ba-1ae5c6aa3933"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:08:02.451344 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:02.451316    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd688da6-42b3-49ed-96ba-1ae5c6aa3933-console-oauth-config" pod "cd688da6-42b3-49ed-96ba-1ae5c6aa3933" (UID: "cd688da6-42b3-49ed-96ba-1ae5c6aa3933"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:08:02.451442 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:02.451371    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd688da6-42b3-49ed-96ba-1ae5c6aa3933-console-serving-cert" pod "cd688da6-42b3-49ed-96ba-1ae5c6aa3933" (UID: "cd688da6-42b3-49ed-96ba-1ae5c6aa3933"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:08:02.451442 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:02.451392    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd688da6-42b3-49ed-96ba-1ae5c6aa3933-kube-api-access-cxqmf" pod "cd688da6-42b3-49ed-96ba-1ae5c6aa3933" (UID: "cd688da6-42b3-49ed-96ba-1ae5c6aa3933"). InnerVolumeSpecName "kube-api-access-cxqmf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:08:02.550672 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:02.550638    4400 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cd688da6-42b3-49ed-96ba-1ae5c6aa3933-console-oauth-config\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:08:02.550672 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:02.550661    4400 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cd688da6-42b3-49ed-96ba-1ae5c6aa3933-oauth-serving-cert\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:08:02.550672 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:02.550671    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxqmf\" (UniqueName: \"kubernetes.io/projected/cd688da6-42b3-49ed-96ba-1ae5c6aa3933-kube-api-access-cxqmf\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:08:02.550893 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:02.550706    4400 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cd688da6-42b3-49ed-96ba-1ae5c6aa3933-console-config\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:08:02.550893 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:02.550719    4400 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cd688da6-42b3-49ed-96ba-1ae5c6aa3933-tmp\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:08:02.550893 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:02.550727    4400 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cd688da6-42b3-49ed-96ba-1ae5c6aa3933-console-serving-cert\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:08:03.057774 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:03.057742    4400 generic.go:356] "Generic (PLEG): container finished" podID="cd688da6-42b3-49ed-96ba-1ae5c6aa3933" containerID="7abea0024cf4002e9d661a828471a1b6005ba027e81743261e0085e1dbf830b8" exitCode=0
Nov 01 03:08:03.058044 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:03.057792    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cf7cdf566-2f2jh"
Nov 01 03:08:03.058044 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:03.057824    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cf7cdf566-2f2jh" event={"ID":"cd688da6-42b3-49ed-96ba-1ae5c6aa3933","Type":"ContainerDied","Data":"7abea0024cf4002e9d661a828471a1b6005ba027e81743261e0085e1dbf830b8"}
Nov 01 03:08:03.058044 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:03.057875    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cf7cdf566-2f2jh" event={"ID":"cd688da6-42b3-49ed-96ba-1ae5c6aa3933","Type":"ContainerDied","Data":"a6ffc2e5a950bd407b8d9036068673a9af8e144b1de9e98670382e89a4cb07e4"}
Nov 01 03:08:03.058044 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:03.057896    4400 scope.go:122] "RemoveContainer" containerID="7abea0024cf4002e9d661a828471a1b6005ba027e81743261e0085e1dbf830b8"
Nov 01 03:08:03.068785 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:03.068768    4400 scope.go:122] "RemoveContainer" containerID="7abea0024cf4002e9d661a828471a1b6005ba027e81743261e0085e1dbf830b8"
Nov 01 03:08:03.069159 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:08:03.069132    4400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7abea0024cf4002e9d661a828471a1b6005ba027e81743261e0085e1dbf830b8\": container with ID starting with 7abea0024cf4002e9d661a828471a1b6005ba027e81743261e0085e1dbf830b8 not found: ID does not exist" containerID="7abea0024cf4002e9d661a828471a1b6005ba027e81743261e0085e1dbf830b8"
Nov 01 03:08:03.069258 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:03.069168    4400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7abea0024cf4002e9d661a828471a1b6005ba027e81743261e0085e1dbf830b8"} err="failed to get container status \"7abea0024cf4002e9d661a828471a1b6005ba027e81743261e0085e1dbf830b8\": rpc error: code = NotFound desc = could not find container \"7abea0024cf4002e9d661a828471a1b6005ba027e81743261e0085e1dbf830b8\": container with ID starting with 7abea0024cf4002e9d661a828471a1b6005ba027e81743261e0085e1dbf830b8 not found: ID does not exist"
Nov 01 03:08:03.224608 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:03.224583    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6cf7cdf566-2f2jh"]
Nov 01 03:08:03.275369 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:03.275339    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6cf7cdf566-2f2jh"]
Nov 01 03:08:03.611962 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:03.611933    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd688da6-42b3-49ed-96ba-1ae5c6aa3933" path="/var/lib/kubelet/pods/cd688da6-42b3-49ed-96ba-1ae5c6aa3933/volumes"
Nov 01 03:08:12.060381 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:12.060344    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-5rxlb-bg459"]
Nov 01 03:08:12.060887 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:12.060614    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-5rxlb-bg459" podUID="68932dfe-7e25-4f1b-b485-77c0519203db" containerName="insights-gathering" containerID="cri-o://385c530c8bbd675abad47bac2c40e047c2996e0cc08c224c6ce011411a6853c0" gracePeriod=30
Nov 01 03:08:12.316870 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:12.316819    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-5rxlb-bg459"
Nov 01 03:08:12.424520 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:12.424493    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68932dfe-7e25-4f1b-b485-77c0519203db-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68932dfe-7e25-4f1b-b485-77c0519203db-trusted-ca-bundle\") pod \"68932dfe-7e25-4f1b-b485-77c0519203db\" (UID: \"68932dfe-7e25-4f1b-b485-77c0519203db\") "
Nov 01 03:08:12.424643 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:12.424562    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68932dfe-7e25-4f1b-b485-77c0519203db-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68932dfe-7e25-4f1b-b485-77c0519203db-service-ca-bundle\") pod \"68932dfe-7e25-4f1b-b485-77c0519203db\" (UID: \"68932dfe-7e25-4f1b-b485-77c0519203db\") "
Nov 01 03:08:12.424643 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:12.424585    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68932dfe-7e25-4f1b-b485-77c0519203db-kube-api-access-qnrhp\" (UniqueName: \"kubernetes.io/projected/68932dfe-7e25-4f1b-b485-77c0519203db-kube-api-access-qnrhp\") pod \"68932dfe-7e25-4f1b-b485-77c0519203db\" (UID: \"68932dfe-7e25-4f1b-b485-77c0519203db\") "
Nov 01 03:08:12.424643 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:12.424620    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68932dfe-7e25-4f1b-b485-77c0519203db-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/68932dfe-7e25-4f1b-b485-77c0519203db-archives-path\") pod \"68932dfe-7e25-4f1b-b485-77c0519203db\" (UID: \"68932dfe-7e25-4f1b-b485-77c0519203db\") "
Nov 01 03:08:12.424978 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:12.424954    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68932dfe-7e25-4f1b-b485-77c0519203db-service-ca-bundle" pod "68932dfe-7e25-4f1b-b485-77c0519203db" (UID: "68932dfe-7e25-4f1b-b485-77c0519203db"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:08:12.425020 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:12.424975    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68932dfe-7e25-4f1b-b485-77c0519203db-archives-path" pod "68932dfe-7e25-4f1b-b485-77c0519203db" (UID: "68932dfe-7e25-4f1b-b485-77c0519203db"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:08:12.425020 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:12.424983    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68932dfe-7e25-4f1b-b485-77c0519203db-trusted-ca-bundle" pod "68932dfe-7e25-4f1b-b485-77c0519203db" (UID: "68932dfe-7e25-4f1b-b485-77c0519203db"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:08:12.426903 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:12.426874    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68932dfe-7e25-4f1b-b485-77c0519203db-kube-api-access-qnrhp" pod "68932dfe-7e25-4f1b-b485-77c0519203db" (UID: "68932dfe-7e25-4f1b-b485-77c0519203db"). InnerVolumeSpecName "kube-api-access-qnrhp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:08:12.525610 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:12.525583    4400 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68932dfe-7e25-4f1b-b485-77c0519203db-service-ca-bundle\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:08:12.525610 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:12.525608    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnrhp\" (UniqueName: \"kubernetes.io/projected/68932dfe-7e25-4f1b-b485-77c0519203db-kube-api-access-qnrhp\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:08:12.525788 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:12.525624    4400 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/68932dfe-7e25-4f1b-b485-77c0519203db-archives-path\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:08:12.525788 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:12.525637    4400 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68932dfe-7e25-4f1b-b485-77c0519203db-trusted-ca-bundle\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:08:13.095617 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:13.095578    4400 generic.go:356] "Generic (PLEG): container finished" podID="68932dfe-7e25-4f1b-b485-77c0519203db" containerID="385c530c8bbd675abad47bac2c40e047c2996e0cc08c224c6ce011411a6853c0" exitCode=2
Nov 01 03:08:13.096204 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:13.095625    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-5rxlb-bg459"
Nov 01 03:08:13.096204 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:13.095653    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-5rxlb-bg459" event={"ID":"68932dfe-7e25-4f1b-b485-77c0519203db","Type":"ContainerDied","Data":"385c530c8bbd675abad47bac2c40e047c2996e0cc08c224c6ce011411a6853c0"}
Nov 01 03:08:13.096204 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:13.095712    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-5rxlb-bg459" event={"ID":"68932dfe-7e25-4f1b-b485-77c0519203db","Type":"ContainerDied","Data":"410bd40c1eca1aaaeeea283ba9e2c2a185c749e790f96ef1fdb69feacf71c338"}
Nov 01 03:08:13.096204 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:13.095735    4400 scope.go:122] "RemoveContainer" containerID="385c530c8bbd675abad47bac2c40e047c2996e0cc08c224c6ce011411a6853c0"
Nov 01 03:08:13.104740 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:13.104724    4400 scope.go:122] "RemoveContainer" containerID="385c530c8bbd675abad47bac2c40e047c2996e0cc08c224c6ce011411a6853c0"
Nov 01 03:08:13.105017 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:08:13.104998    4400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"385c530c8bbd675abad47bac2c40e047c2996e0cc08c224c6ce011411a6853c0\": container with ID starting with 385c530c8bbd675abad47bac2c40e047c2996e0cc08c224c6ce011411a6853c0 not found: ID does not exist" containerID="385c530c8bbd675abad47bac2c40e047c2996e0cc08c224c6ce011411a6853c0"
Nov 01 03:08:13.105078 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:13.105024    4400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"385c530c8bbd675abad47bac2c40e047c2996e0cc08c224c6ce011411a6853c0"} err="failed to get container status \"385c530c8bbd675abad47bac2c40e047c2996e0cc08c224c6ce011411a6853c0\": rpc error: code = NotFound desc = could not find container \"385c530c8bbd675abad47bac2c40e047c2996e0cc08c224c6ce011411a6853c0\": container with ID starting with 385c530c8bbd675abad47bac2c40e047c2996e0cc08c224c6ce011411a6853c0 not found: ID does not exist"
Nov 01 03:08:13.265201 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:13.265176    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-5rxlb-bg459"]
Nov 01 03:08:13.315197 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:13.315176    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-5rxlb-bg459"]
Nov 01 03:08:13.612196 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:13.612172    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68932dfe-7e25-4f1b-b485-77c0519203db" path="/var/lib/kubelet/pods/68932dfe-7e25-4f1b-b485-77c0519203db/volumes"
Nov 01 03:08:31.727657 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:08:31.727619    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:08:36.555817 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:36.552367    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-879889647-wqck7"]
Nov 01 03:08:36.555817 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:36.552698    4400 cpu_manager.go:414] "containerMap: removing container" podUID="b08de522-96f8-42a8-a88f-0502b999d404" containerName="console"
Nov 01 03:08:36.555817 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:36.552712    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b08de522-96f8-42a8-a88f-0502b999d404" containerName="console"
Nov 01 03:08:36.555817 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:36.552729    4400 cpu_manager.go:414] "containerMap: removing container" podUID="68932dfe-7e25-4f1b-b485-77c0519203db" containerName="insights-gathering"
Nov 01 03:08:36.555817 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:36.552734    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68932dfe-7e25-4f1b-b485-77c0519203db" containerName="insights-gathering"
Nov 01 03:08:36.555817 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:36.552744    4400 cpu_manager.go:414] "containerMap: removing container" podUID="cd688da6-42b3-49ed-96ba-1ae5c6aa3933" containerName="console"
Nov 01 03:08:36.555817 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:36.552749    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd688da6-42b3-49ed-96ba-1ae5c6aa3933" containerName="console"
Nov 01 03:08:36.555817 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:36.552795    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="b08de522-96f8-42a8-a88f-0502b999d404" containerName="console"
Nov 01 03:08:36.555817 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:36.552805    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="68932dfe-7e25-4f1b-b485-77c0519203db" containerName="insights-gathering"
Nov 01 03:08:36.555817 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:36.552811    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd688da6-42b3-49ed-96ba-1ae5c6aa3933" containerName="console"
Nov 01 03:08:36.555817 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:36.555290    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-879889647-wqck7"
Nov 01 03:08:36.661867 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:36.661840    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-879889647-wqck7"]
Nov 01 03:08:36.694776 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:36.694754    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fe8a1210-3f73-4877-8bdf-9eb66144644e-tmp\") pod \"console-879889647-wqck7\" (UID: \"fe8a1210-3f73-4877-8bdf-9eb66144644e\") " pod="openshift-console/console-879889647-wqck7"
Nov 01 03:08:36.694891 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:36.694793    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fe8a1210-3f73-4877-8bdf-9eb66144644e-console-config\") pod \"console-879889647-wqck7\" (UID: \"fe8a1210-3f73-4877-8bdf-9eb66144644e\") " pod="openshift-console/console-879889647-wqck7"
Nov 01 03:08:36.694891 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:36.694812    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe8a1210-3f73-4877-8bdf-9eb66144644e-trusted-ca-bundle\") pod \"console-879889647-wqck7\" (UID: \"fe8a1210-3f73-4877-8bdf-9eb66144644e\") " pod="openshift-console/console-879889647-wqck7"
Nov 01 03:08:36.694965 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:36.694913    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fe8a1210-3f73-4877-8bdf-9eb66144644e-service-ca\") pod \"console-879889647-wqck7\" (UID: \"fe8a1210-3f73-4877-8bdf-9eb66144644e\") " pod="openshift-console/console-879889647-wqck7"
Nov 01 03:08:36.694965 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:36.694950    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fe8a1210-3f73-4877-8bdf-9eb66144644e-console-oauth-config\") pod \"console-879889647-wqck7\" (UID: \"fe8a1210-3f73-4877-8bdf-9eb66144644e\") " pod="openshift-console/console-879889647-wqck7"
Nov 01 03:08:36.695023 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:36.694967    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9czvb\" (UniqueName: \"kubernetes.io/projected/fe8a1210-3f73-4877-8bdf-9eb66144644e-kube-api-access-9czvb\") pod \"console-879889647-wqck7\" (UID: \"fe8a1210-3f73-4877-8bdf-9eb66144644e\") " pod="openshift-console/console-879889647-wqck7"
Nov 01 03:08:36.695023 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:36.695002    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fe8a1210-3f73-4877-8bdf-9eb66144644e-oauth-serving-cert\") pod \"console-879889647-wqck7\" (UID: \"fe8a1210-3f73-4877-8bdf-9eb66144644e\") " pod="openshift-console/console-879889647-wqck7"
Nov 01 03:08:36.695082 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:36.695026    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fe8a1210-3f73-4877-8bdf-9eb66144644e-console-serving-cert\") pod \"console-879889647-wqck7\" (UID: \"fe8a1210-3f73-4877-8bdf-9eb66144644e\") " pod="openshift-console/console-879889647-wqck7"
Nov 01 03:08:36.795426 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:36.795401    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fe8a1210-3f73-4877-8bdf-9eb66144644e-console-serving-cert\") pod \"console-879889647-wqck7\" (UID: \"fe8a1210-3f73-4877-8bdf-9eb66144644e\") " pod="openshift-console/console-879889647-wqck7"
Nov 01 03:08:36.795542 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:36.795436    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fe8a1210-3f73-4877-8bdf-9eb66144644e-tmp\") pod \"console-879889647-wqck7\" (UID: \"fe8a1210-3f73-4877-8bdf-9eb66144644e\") " pod="openshift-console/console-879889647-wqck7"
Nov 01 03:08:36.795542 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:36.795460    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fe8a1210-3f73-4877-8bdf-9eb66144644e-console-config\") pod \"console-879889647-wqck7\" (UID: \"fe8a1210-3f73-4877-8bdf-9eb66144644e\") " pod="openshift-console/console-879889647-wqck7"
Nov 01 03:08:36.795542 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:36.795478    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe8a1210-3f73-4877-8bdf-9eb66144644e-trusted-ca-bundle\") pod \"console-879889647-wqck7\" (UID: \"fe8a1210-3f73-4877-8bdf-9eb66144644e\") " pod="openshift-console/console-879889647-wqck7"
Nov 01 03:08:36.795542 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:36.795495    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fe8a1210-3f73-4877-8bdf-9eb66144644e-service-ca\") pod \"console-879889647-wqck7\" (UID: \"fe8a1210-3f73-4877-8bdf-9eb66144644e\") " pod="openshift-console/console-879889647-wqck7"
Nov 01 03:08:36.795542 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:36.795514    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fe8a1210-3f73-4877-8bdf-9eb66144644e-console-oauth-config\") pod \"console-879889647-wqck7\" (UID: \"fe8a1210-3f73-4877-8bdf-9eb66144644e\") " pod="openshift-console/console-879889647-wqck7"
Nov 01 03:08:36.795542 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:36.795535    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9czvb\" (UniqueName: \"kubernetes.io/projected/fe8a1210-3f73-4877-8bdf-9eb66144644e-kube-api-access-9czvb\") pod \"console-879889647-wqck7\" (UID: \"fe8a1210-3f73-4877-8bdf-9eb66144644e\") " pod="openshift-console/console-879889647-wqck7"
Nov 01 03:08:36.795869 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:36.795567    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fe8a1210-3f73-4877-8bdf-9eb66144644e-oauth-serving-cert\") pod \"console-879889647-wqck7\" (UID: \"fe8a1210-3f73-4877-8bdf-9eb66144644e\") " pod="openshift-console/console-879889647-wqck7"
Nov 01 03:08:36.795927 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:36.795874    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fe8a1210-3f73-4877-8bdf-9eb66144644e-tmp\") pod \"console-879889647-wqck7\" (UID: \"fe8a1210-3f73-4877-8bdf-9eb66144644e\") " pod="openshift-console/console-879889647-wqck7"
Nov 01 03:08:36.796251 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:36.796227    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fe8a1210-3f73-4877-8bdf-9eb66144644e-console-config\") pod \"console-879889647-wqck7\" (UID: \"fe8a1210-3f73-4877-8bdf-9eb66144644e\") " pod="openshift-console/console-879889647-wqck7"
Nov 01 03:08:36.796366 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:36.796345    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fe8a1210-3f73-4877-8bdf-9eb66144644e-service-ca\") pod \"console-879889647-wqck7\" (UID: \"fe8a1210-3f73-4877-8bdf-9eb66144644e\") " pod="openshift-console/console-879889647-wqck7"
Nov 01 03:08:36.796450 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:36.796433    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fe8a1210-3f73-4877-8bdf-9eb66144644e-oauth-serving-cert\") pod \"console-879889647-wqck7\" (UID: \"fe8a1210-3f73-4877-8bdf-9eb66144644e\") " pod="openshift-console/console-879889647-wqck7"
Nov 01 03:08:36.796629 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:36.796611    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe8a1210-3f73-4877-8bdf-9eb66144644e-trusted-ca-bundle\") pod \"console-879889647-wqck7\" (UID: \"fe8a1210-3f73-4877-8bdf-9eb66144644e\") " pod="openshift-console/console-879889647-wqck7"
Nov 01 03:08:36.798148 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:36.798126    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fe8a1210-3f73-4877-8bdf-9eb66144644e-console-serving-cert\") pod \"console-879889647-wqck7\" (UID: \"fe8a1210-3f73-4877-8bdf-9eb66144644e\") " pod="openshift-console/console-879889647-wqck7"
Nov 01 03:08:36.798237 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:36.798201    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fe8a1210-3f73-4877-8bdf-9eb66144644e-console-oauth-config\") pod \"console-879889647-wqck7\" (UID: \"fe8a1210-3f73-4877-8bdf-9eb66144644e\") " pod="openshift-console/console-879889647-wqck7"
Nov 01 03:08:36.857376 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:36.857347    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9czvb\" (UniqueName: \"kubernetes.io/projected/fe8a1210-3f73-4877-8bdf-9eb66144644e-kube-api-access-9czvb\") pod \"console-879889647-wqck7\" (UID: \"fe8a1210-3f73-4877-8bdf-9eb66144644e\") " pod="openshift-console/console-879889647-wqck7"
Nov 01 03:08:36.866233 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:36.866217    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-879889647-wqck7"
Nov 01 03:08:37.236144 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:37.236117    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-879889647-wqck7"]
Nov 01 03:08:37.237588 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:08:37.237562    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe8a1210_3f73_4877_8bdf_9eb66144644e.slice/crio-2e3abdd82e136b75a07afd3dc75e5fecb050d47ab0f8d8107390ed67bc379dc5 WatchSource:0}: Error finding container 2e3abdd82e136b75a07afd3dc75e5fecb050d47ab0f8d8107390ed67bc379dc5: Status 404 returned error can't find the container with id 2e3abdd82e136b75a07afd3dc75e5fecb050d47ab0f8d8107390ed67bc379dc5
Nov 01 03:08:38.186138 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:38.186097    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-879889647-wqck7" event={"ID":"fe8a1210-3f73-4877-8bdf-9eb66144644e","Type":"ContainerStarted","Data":"4e4cce73edf050acc603f8075da44eeb49d67da2b6487ba1734933e6a71a2826"}
Nov 01 03:08:38.186536 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:38.186146    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-879889647-wqck7" event={"ID":"fe8a1210-3f73-4877-8bdf-9eb66144644e","Type":"ContainerStarted","Data":"2e3abdd82e136b75a07afd3dc75e5fecb050d47ab0f8d8107390ed67bc379dc5"}
Nov 01 03:08:38.307707 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:38.307638    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-879889647-wqck7" podStartSLOduration=2.307624526 podStartE2EDuration="2.307624526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:08:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:08:38.306299712 +0000 UTC m=+124.820945587" watchObservedRunningTime="2026-11-01 03:08:38.307624526 +0000 UTC m=+124.822270389"
Nov 01 03:08:42.166963 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:42.166930    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 01 03:08:42.167351 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:42.167218    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="95482def-35e0-4879-a721-d1ce6d251061" containerName="alertmanager" containerID="cri-o://960d7c3027418725815287ced9709538984dc0ba28a6a826382c9a227041f7df" gracePeriod=120
Nov 01 03:08:42.167351 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:42.167296    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="95482def-35e0-4879-a721-d1ce6d251061" containerName="kube-rbac-proxy-metric" containerID="cri-o://2459da8a6b1d6965c7ca185ef8e6fcd0b8ece1d4d537ff3faaf04376ef1cd12d" gracePeriod=120
Nov 01 03:08:42.167495 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:42.167335    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="95482def-35e0-4879-a721-d1ce6d251061" containerName="kube-rbac-proxy" containerID="cri-o://44f2d215c8d28a22ac3b18aba136454d6c518c33952c0ba3ce905e0ca4a52a99" gracePeriod=120
Nov 01 03:08:42.167495 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:42.167358    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="95482def-35e0-4879-a721-d1ce6d251061" containerName="prom-label-proxy" containerID="cri-o://87fe732655cd942b05e12ea5580443ce47acccbe7bc19e848e2b0f234722259f" gracePeriod=120
Nov 01 03:08:42.167495 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:42.167371    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="95482def-35e0-4879-a721-d1ce6d251061" containerName="config-reloader" containerID="cri-o://5c5adee9bbcfd85f53a6d76b11100ec7ef4b40c433b3bdd4cc2ed7f299d987c1" gracePeriod=120
Nov 01 03:08:42.167495 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:42.167328    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="95482def-35e0-4879-a721-d1ce6d251061" containerName="kube-rbac-proxy-web" containerID="cri-o://c5ee8cef5bccf833843ae19213ca37a09919502ccbc56af21cd5324fd51373c8" gracePeriod=120
Nov 01 03:08:43.212431 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:43.212399    4400 generic.go:356] "Generic (PLEG): container finished" podID="95482def-35e0-4879-a721-d1ce6d251061" containerID="87fe732655cd942b05e12ea5580443ce47acccbe7bc19e848e2b0f234722259f" exitCode=0
Nov 01 03:08:43.212431 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:43.212425    4400 generic.go:356] "Generic (PLEG): container finished" podID="95482def-35e0-4879-a721-d1ce6d251061" containerID="44f2d215c8d28a22ac3b18aba136454d6c518c33952c0ba3ce905e0ca4a52a99" exitCode=0
Nov 01 03:08:43.212431 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:43.212433    4400 generic.go:356] "Generic (PLEG): container finished" podID="95482def-35e0-4879-a721-d1ce6d251061" containerID="5c5adee9bbcfd85f53a6d76b11100ec7ef4b40c433b3bdd4cc2ed7f299d987c1" exitCode=0
Nov 01 03:08:43.212431 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:43.212439    4400 generic.go:356] "Generic (PLEG): container finished" podID="95482def-35e0-4879-a721-d1ce6d251061" containerID="960d7c3027418725815287ced9709538984dc0ba28a6a826382c9a227041f7df" exitCode=0
Nov 01 03:08:43.212921 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:43.212468    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"95482def-35e0-4879-a721-d1ce6d251061","Type":"ContainerDied","Data":"87fe732655cd942b05e12ea5580443ce47acccbe7bc19e848e2b0f234722259f"}
Nov 01 03:08:43.212921 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:43.212558    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"95482def-35e0-4879-a721-d1ce6d251061","Type":"ContainerDied","Data":"44f2d215c8d28a22ac3b18aba136454d6c518c33952c0ba3ce905e0ca4a52a99"}
Nov 01 03:08:43.212921 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:43.212586    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"95482def-35e0-4879-a721-d1ce6d251061","Type":"ContainerDied","Data":"5c5adee9bbcfd85f53a6d76b11100ec7ef4b40c433b3bdd4cc2ed7f299d987c1"}
Nov 01 03:08:43.212921 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:43.212609    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"95482def-35e0-4879-a721-d1ce6d251061","Type":"ContainerDied","Data":"960d7c3027418725815287ced9709538984dc0ba28a6a826382c9a227041f7df"}
Nov 01 03:08:43.420183 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:43.420162    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 01 03:08:43.544486 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:43.544428    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95482def-35e0-4879-a721-d1ce6d251061-config-volume\" (UniqueName: \"kubernetes.io/secret/95482def-35e0-4879-a721-d1ce6d251061-config-volume\") pod \"95482def-35e0-4879-a721-d1ce6d251061\" (UID: \"95482def-35e0-4879-a721-d1ce6d251061\") "
Nov 01 03:08:43.544486 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:43.544472    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95482def-35e0-4879-a721-d1ce6d251061-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95482def-35e0-4879-a721-d1ce6d251061-alertmanager-trusted-ca-bundle\") pod \"95482def-35e0-4879-a721-d1ce6d251061\" (UID: \"95482def-35e0-4879-a721-d1ce6d251061\") "
Nov 01 03:08:43.544625 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:43.544492    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95482def-35e0-4879-a721-d1ce6d251061-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/95482def-35e0-4879-a721-d1ce6d251061-metrics-client-ca\") pod \"95482def-35e0-4879-a721-d1ce6d251061\" (UID: \"95482def-35e0-4879-a721-d1ce6d251061\") "
Nov 01 03:08:43.544625 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:43.544508    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95482def-35e0-4879-a721-d1ce6d251061-web-config\" (UniqueName: \"kubernetes.io/secret/95482def-35e0-4879-a721-d1ce6d251061-web-config\") pod \"95482def-35e0-4879-a721-d1ce6d251061\" (UID: \"95482def-35e0-4879-a721-d1ce6d251061\") "
Nov 01 03:08:43.544749 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:43.544630    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95482def-35e0-4879-a721-d1ce6d251061-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/95482def-35e0-4879-a721-d1ce6d251061-secret-alertmanager-main-tls\") pod \"95482def-35e0-4879-a721-d1ce6d251061\" (UID: \"95482def-35e0-4879-a721-d1ce6d251061\") "
Nov 01 03:08:43.544749 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:43.544704    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95482def-35e0-4879-a721-d1ce6d251061-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/95482def-35e0-4879-a721-d1ce6d251061-secret-alertmanager-kube-rbac-proxy-web\") pod \"95482def-35e0-4879-a721-d1ce6d251061\" (UID: \"95482def-35e0-4879-a721-d1ce6d251061\") "
Nov 01 03:08:43.544749 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:43.544745    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95482def-35e0-4879-a721-d1ce6d251061-kube-api-access-lns96\" (UniqueName: \"kubernetes.io/projected/95482def-35e0-4879-a721-d1ce6d251061-kube-api-access-lns96\") pod \"95482def-35e0-4879-a721-d1ce6d251061\" (UID: \"95482def-35e0-4879-a721-d1ce6d251061\") "
Nov 01 03:08:43.544908 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:43.544779    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95482def-35e0-4879-a721-d1ce6d251061-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/95482def-35e0-4879-a721-d1ce6d251061-secret-alertmanager-kube-rbac-proxy\") pod \"95482def-35e0-4879-a721-d1ce6d251061\" (UID: \"95482def-35e0-4879-a721-d1ce6d251061\") "
Nov 01 03:08:43.544908 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:43.544826    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95482def-35e0-4879-a721-d1ce6d251061-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/95482def-35e0-4879-a721-d1ce6d251061-secret-alertmanager-kube-rbac-proxy-metric\") pod \"95482def-35e0-4879-a721-d1ce6d251061\" (UID: \"95482def-35e0-4879-a721-d1ce6d251061\") "
Nov 01 03:08:43.544908 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:43.544873    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95482def-35e0-4879-a721-d1ce6d251061-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/95482def-35e0-4879-a721-d1ce6d251061-cluster-tls-config\") pod \"95482def-35e0-4879-a721-d1ce6d251061\" (UID: \"95482def-35e0-4879-a721-d1ce6d251061\") "
Nov 01 03:08:43.544908 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:43.544890    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95482def-35e0-4879-a721-d1ce6d251061-metrics-client-ca" pod "95482def-35e0-4879-a721-d1ce6d251061" (UID: "95482def-35e0-4879-a721-d1ce6d251061"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:08:43.545114 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:43.544905    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95482def-35e0-4879-a721-d1ce6d251061-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/95482def-35e0-4879-a721-d1ce6d251061-alertmanager-main-db\") pod \"95482def-35e0-4879-a721-d1ce6d251061\" (UID: \"95482def-35e0-4879-a721-d1ce6d251061\") "
Nov 01 03:08:43.545114 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:43.544913    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95482def-35e0-4879-a721-d1ce6d251061-alertmanager-trusted-ca-bundle" pod "95482def-35e0-4879-a721-d1ce6d251061" (UID: "95482def-35e0-4879-a721-d1ce6d251061"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:08:43.545114 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:43.544966    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95482def-35e0-4879-a721-d1ce6d251061-tls-assets\" (UniqueName: \"kubernetes.io/projected/95482def-35e0-4879-a721-d1ce6d251061-tls-assets\") pod \"95482def-35e0-4879-a721-d1ce6d251061\" (UID: \"95482def-35e0-4879-a721-d1ce6d251061\") "
Nov 01 03:08:43.545114 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:43.545015    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95482def-35e0-4879-a721-d1ce6d251061-config-out\" (UniqueName: \"kubernetes.io/empty-dir/95482def-35e0-4879-a721-d1ce6d251061-config-out\") pod \"95482def-35e0-4879-a721-d1ce6d251061\" (UID: \"95482def-35e0-4879-a721-d1ce6d251061\") "
Nov 01 03:08:43.545475 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:43.545335    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95482def-35e0-4879-a721-d1ce6d251061-alertmanager-main-db" pod "95482def-35e0-4879-a721-d1ce6d251061" (UID: "95482def-35e0-4879-a721-d1ce6d251061"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:08:43.545847 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:43.545575    4400 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/95482def-35e0-4879-a721-d1ce6d251061-alertmanager-main-db\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:08:43.545847 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:43.545602    4400 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95482def-35e0-4879-a721-d1ce6d251061-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:08:43.545847 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:43.545619    4400 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/95482def-35e0-4879-a721-d1ce6d251061-metrics-client-ca\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:08:43.547915 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:43.547877    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95482def-35e0-4879-a721-d1ce6d251061-config-volume" pod "95482def-35e0-4879-a721-d1ce6d251061" (UID: "95482def-35e0-4879-a721-d1ce6d251061"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:08:43.548145 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:43.548117    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95482def-35e0-4879-a721-d1ce6d251061-secret-alertmanager-kube-rbac-proxy-metric" pod "95482def-35e0-4879-a721-d1ce6d251061" (UID: "95482def-35e0-4879-a721-d1ce6d251061"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:08:43.548218 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:43.548191    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95482def-35e0-4879-a721-d1ce6d251061-secret-alertmanager-main-tls" pod "95482def-35e0-4879-a721-d1ce6d251061" (UID: "95482def-35e0-4879-a721-d1ce6d251061"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:08:43.548279 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:43.548259    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95482def-35e0-4879-a721-d1ce6d251061-config-out" pod "95482def-35e0-4879-a721-d1ce6d251061" (UID: "95482def-35e0-4879-a721-d1ce6d251061"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:08:43.548334 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:43.548295    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95482def-35e0-4879-a721-d1ce6d251061-secret-alertmanager-kube-rbac-proxy" pod "95482def-35e0-4879-a721-d1ce6d251061" (UID: "95482def-35e0-4879-a721-d1ce6d251061"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:08:43.548514 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:43.548493    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95482def-35e0-4879-a721-d1ce6d251061-tls-assets" pod "95482def-35e0-4879-a721-d1ce6d251061" (UID: "95482def-35e0-4879-a721-d1ce6d251061"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:08:43.549206 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:43.549182    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95482def-35e0-4879-a721-d1ce6d251061-kube-api-access-lns96" pod "95482def-35e0-4879-a721-d1ce6d251061" (UID: "95482def-35e0-4879-a721-d1ce6d251061"). InnerVolumeSpecName "kube-api-access-lns96". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:08:43.549645 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:43.549623    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95482def-35e0-4879-a721-d1ce6d251061-secret-alertmanager-kube-rbac-proxy-web" pod "95482def-35e0-4879-a721-d1ce6d251061" (UID: "95482def-35e0-4879-a721-d1ce6d251061"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:08:43.554187 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:43.554166    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95482def-35e0-4879-a721-d1ce6d251061-cluster-tls-config" pod "95482def-35e0-4879-a721-d1ce6d251061" (UID: "95482def-35e0-4879-a721-d1ce6d251061"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:08:43.560103 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:43.560085    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95482def-35e0-4879-a721-d1ce6d251061-web-config" pod "95482def-35e0-4879-a721-d1ce6d251061" (UID: "95482def-35e0-4879-a721-d1ce6d251061"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:08:43.646395 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:43.646369    4400 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/95482def-35e0-4879-a721-d1ce6d251061-cluster-tls-config\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:08:43.646395 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:43.646389    4400 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/95482def-35e0-4879-a721-d1ce6d251061-tls-assets\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:08:43.646500 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:43.646398    4400 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/95482def-35e0-4879-a721-d1ce6d251061-config-out\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:08:43.646500 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:43.646408    4400 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/95482def-35e0-4879-a721-d1ce6d251061-config-volume\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:08:43.646500 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:43.646416    4400 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/95482def-35e0-4879-a721-d1ce6d251061-web-config\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:08:43.646500 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:43.646426    4400 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/95482def-35e0-4879-a721-d1ce6d251061-secret-alertmanager-main-tls\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:08:43.646500 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:43.646436    4400 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/95482def-35e0-4879-a721-d1ce6d251061-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:08:43.646500 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:43.646445    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lns96\" (UniqueName: \"kubernetes.io/projected/95482def-35e0-4879-a721-d1ce6d251061-kube-api-access-lns96\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:08:43.646500 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:43.646453    4400 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/95482def-35e0-4879-a721-d1ce6d251061-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:08:43.646500 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:43.646462    4400 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/95482def-35e0-4879-a721-d1ce6d251061-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:08:44.219625 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.219541    4400 generic.go:356] "Generic (PLEG): container finished" podID="95482def-35e0-4879-a721-d1ce6d251061" containerID="2459da8a6b1d6965c7ca185ef8e6fcd0b8ece1d4d537ff3faaf04376ef1cd12d" exitCode=0
Nov 01 03:08:44.219625 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.219564    4400 generic.go:356] "Generic (PLEG): container finished" podID="95482def-35e0-4879-a721-d1ce6d251061" containerID="c5ee8cef5bccf833843ae19213ca37a09919502ccbc56af21cd5324fd51373c8" exitCode=0
Nov 01 03:08:44.219625 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.219597    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"95482def-35e0-4879-a721-d1ce6d251061","Type":"ContainerDied","Data":"2459da8a6b1d6965c7ca185ef8e6fcd0b8ece1d4d537ff3faaf04376ef1cd12d"}
Nov 01 03:08:44.219625 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.219619    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 01 03:08:44.220348 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.219665    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"95482def-35e0-4879-a721-d1ce6d251061","Type":"ContainerDied","Data":"c5ee8cef5bccf833843ae19213ca37a09919502ccbc56af21cd5324fd51373c8"}
Nov 01 03:08:44.220348 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.219725    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"95482def-35e0-4879-a721-d1ce6d251061","Type":"ContainerDied","Data":"1f8ddae7f495423fe3dcca6e59d3084609854e69a4da9d4d144c053d691e69bc"}
Nov 01 03:08:44.220348 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.219747    4400 scope.go:122] "RemoveContainer" containerID="87fe732655cd942b05e12ea5580443ce47acccbe7bc19e848e2b0f234722259f"
Nov 01 03:08:44.227585 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.227566    4400 scope.go:122] "RemoveContainer" containerID="2459da8a6b1d6965c7ca185ef8e6fcd0b8ece1d4d537ff3faaf04376ef1cd12d"
Nov 01 03:08:44.235029 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.235009    4400 scope.go:122] "RemoveContainer" containerID="44f2d215c8d28a22ac3b18aba136454d6c518c33952c0ba3ce905e0ca4a52a99"
Nov 01 03:08:44.249440 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.249419    4400 scope.go:122] "RemoveContainer" containerID="c5ee8cef5bccf833843ae19213ca37a09919502ccbc56af21cd5324fd51373c8"
Nov 01 03:08:44.256643 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.256623    4400 scope.go:122] "RemoveContainer" containerID="5c5adee9bbcfd85f53a6d76b11100ec7ef4b40c433b3bdd4cc2ed7f299d987c1"
Nov 01 03:08:44.263811 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.263793    4400 scope.go:122] "RemoveContainer" containerID="960d7c3027418725815287ced9709538984dc0ba28a6a826382c9a227041f7df"
Nov 01 03:08:44.270693 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.270661    4400 scope.go:122] "RemoveContainer" containerID="1901de61e1fa20eff47f04545ffe0dd59b2ed19fffd393d9bf527bb2899f3dba"
Nov 01 03:08:44.277695 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.277667    4400 scope.go:122] "RemoveContainer" containerID="87fe732655cd942b05e12ea5580443ce47acccbe7bc19e848e2b0f234722259f"
Nov 01 03:08:44.277978 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:08:44.277956    4400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87fe732655cd942b05e12ea5580443ce47acccbe7bc19e848e2b0f234722259f\": container with ID starting with 87fe732655cd942b05e12ea5580443ce47acccbe7bc19e848e2b0f234722259f not found: ID does not exist" containerID="87fe732655cd942b05e12ea5580443ce47acccbe7bc19e848e2b0f234722259f"
Nov 01 03:08:44.278020 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.277989    4400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87fe732655cd942b05e12ea5580443ce47acccbe7bc19e848e2b0f234722259f"} err="failed to get container status \"87fe732655cd942b05e12ea5580443ce47acccbe7bc19e848e2b0f234722259f\": rpc error: code = NotFound desc = could not find container \"87fe732655cd942b05e12ea5580443ce47acccbe7bc19e848e2b0f234722259f\": container with ID starting with 87fe732655cd942b05e12ea5580443ce47acccbe7bc19e848e2b0f234722259f not found: ID does not exist"
Nov 01 03:08:44.278020 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.278005    4400 scope.go:122] "RemoveContainer" containerID="2459da8a6b1d6965c7ca185ef8e6fcd0b8ece1d4d537ff3faaf04376ef1cd12d"
Nov 01 03:08:44.278245 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:08:44.278229    4400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2459da8a6b1d6965c7ca185ef8e6fcd0b8ece1d4d537ff3faaf04376ef1cd12d\": container with ID starting with 2459da8a6b1d6965c7ca185ef8e6fcd0b8ece1d4d537ff3faaf04376ef1cd12d not found: ID does not exist" containerID="2459da8a6b1d6965c7ca185ef8e6fcd0b8ece1d4d537ff3faaf04376ef1cd12d"
Nov 01 03:08:44.278284 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.278255    4400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2459da8a6b1d6965c7ca185ef8e6fcd0b8ece1d4d537ff3faaf04376ef1cd12d"} err="failed to get container status \"2459da8a6b1d6965c7ca185ef8e6fcd0b8ece1d4d537ff3faaf04376ef1cd12d\": rpc error: code = NotFound desc = could not find container \"2459da8a6b1d6965c7ca185ef8e6fcd0b8ece1d4d537ff3faaf04376ef1cd12d\": container with ID starting with 2459da8a6b1d6965c7ca185ef8e6fcd0b8ece1d4d537ff3faaf04376ef1cd12d not found: ID does not exist"
Nov 01 03:08:44.278284 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.278273    4400 scope.go:122] "RemoveContainer" containerID="44f2d215c8d28a22ac3b18aba136454d6c518c33952c0ba3ce905e0ca4a52a99"
Nov 01 03:08:44.278528 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:08:44.278509    4400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44f2d215c8d28a22ac3b18aba136454d6c518c33952c0ba3ce905e0ca4a52a99\": container with ID starting with 44f2d215c8d28a22ac3b18aba136454d6c518c33952c0ba3ce905e0ca4a52a99 not found: ID does not exist" containerID="44f2d215c8d28a22ac3b18aba136454d6c518c33952c0ba3ce905e0ca4a52a99"
Nov 01 03:08:44.278585 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.278535    4400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44f2d215c8d28a22ac3b18aba136454d6c518c33952c0ba3ce905e0ca4a52a99"} err="failed to get container status \"44f2d215c8d28a22ac3b18aba136454d6c518c33952c0ba3ce905e0ca4a52a99\": rpc error: code = NotFound desc = could not find container \"44f2d215c8d28a22ac3b18aba136454d6c518c33952c0ba3ce905e0ca4a52a99\": container with ID starting with 44f2d215c8d28a22ac3b18aba136454d6c518c33952c0ba3ce905e0ca4a52a99 not found: ID does not exist"
Nov 01 03:08:44.278585 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.278556    4400 scope.go:122] "RemoveContainer" containerID="c5ee8cef5bccf833843ae19213ca37a09919502ccbc56af21cd5324fd51373c8"
Nov 01 03:08:44.278812 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:08:44.278795    4400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5ee8cef5bccf833843ae19213ca37a09919502ccbc56af21cd5324fd51373c8\": container with ID starting with c5ee8cef5bccf833843ae19213ca37a09919502ccbc56af21cd5324fd51373c8 not found: ID does not exist" containerID="c5ee8cef5bccf833843ae19213ca37a09919502ccbc56af21cd5324fd51373c8"
Nov 01 03:08:44.278869 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.278817    4400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5ee8cef5bccf833843ae19213ca37a09919502ccbc56af21cd5324fd51373c8"} err="failed to get container status \"c5ee8cef5bccf833843ae19213ca37a09919502ccbc56af21cd5324fd51373c8\": rpc error: code = NotFound desc = could not find container \"c5ee8cef5bccf833843ae19213ca37a09919502ccbc56af21cd5324fd51373c8\": container with ID starting with c5ee8cef5bccf833843ae19213ca37a09919502ccbc56af21cd5324fd51373c8 not found: ID does not exist"
Nov 01 03:08:44.278869 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.278830    4400 scope.go:122] "RemoveContainer" containerID="5c5adee9bbcfd85f53a6d76b11100ec7ef4b40c433b3bdd4cc2ed7f299d987c1"
Nov 01 03:08:44.279066 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:08:44.279052    4400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c5adee9bbcfd85f53a6d76b11100ec7ef4b40c433b3bdd4cc2ed7f299d987c1\": container with ID starting with 5c5adee9bbcfd85f53a6d76b11100ec7ef4b40c433b3bdd4cc2ed7f299d987c1 not found: ID does not exist" containerID="5c5adee9bbcfd85f53a6d76b11100ec7ef4b40c433b3bdd4cc2ed7f299d987c1"
Nov 01 03:08:44.279104 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.279069    4400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c5adee9bbcfd85f53a6d76b11100ec7ef4b40c433b3bdd4cc2ed7f299d987c1"} err="failed to get container status \"5c5adee9bbcfd85f53a6d76b11100ec7ef4b40c433b3bdd4cc2ed7f299d987c1\": rpc error: code = NotFound desc = could not find container \"5c5adee9bbcfd85f53a6d76b11100ec7ef4b40c433b3bdd4cc2ed7f299d987c1\": container with ID starting with 5c5adee9bbcfd85f53a6d76b11100ec7ef4b40c433b3bdd4cc2ed7f299d987c1 not found: ID does not exist"
Nov 01 03:08:44.279104 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.279083    4400 scope.go:122] "RemoveContainer" containerID="960d7c3027418725815287ced9709538984dc0ba28a6a826382c9a227041f7df"
Nov 01 03:08:44.279292 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:08:44.279277    4400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"960d7c3027418725815287ced9709538984dc0ba28a6a826382c9a227041f7df\": container with ID starting with 960d7c3027418725815287ced9709538984dc0ba28a6a826382c9a227041f7df not found: ID does not exist" containerID="960d7c3027418725815287ced9709538984dc0ba28a6a826382c9a227041f7df"
Nov 01 03:08:44.279330 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.279296    4400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"960d7c3027418725815287ced9709538984dc0ba28a6a826382c9a227041f7df"} err="failed to get container status \"960d7c3027418725815287ced9709538984dc0ba28a6a826382c9a227041f7df\": rpc error: code = NotFound desc = could not find container \"960d7c3027418725815287ced9709538984dc0ba28a6a826382c9a227041f7df\": container with ID starting with 960d7c3027418725815287ced9709538984dc0ba28a6a826382c9a227041f7df not found: ID does not exist"
Nov 01 03:08:44.279330 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.279307    4400 scope.go:122] "RemoveContainer" containerID="1901de61e1fa20eff47f04545ffe0dd59b2ed19fffd393d9bf527bb2899f3dba"
Nov 01 03:08:44.279525 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:08:44.279507    4400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1901de61e1fa20eff47f04545ffe0dd59b2ed19fffd393d9bf527bb2899f3dba\": container with ID starting with 1901de61e1fa20eff47f04545ffe0dd59b2ed19fffd393d9bf527bb2899f3dba not found: ID does not exist" containerID="1901de61e1fa20eff47f04545ffe0dd59b2ed19fffd393d9bf527bb2899f3dba"
Nov 01 03:08:44.279575 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.279528    4400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1901de61e1fa20eff47f04545ffe0dd59b2ed19fffd393d9bf527bb2899f3dba"} err="failed to get container status \"1901de61e1fa20eff47f04545ffe0dd59b2ed19fffd393d9bf527bb2899f3dba\": rpc error: code = NotFound desc = could not find container \"1901de61e1fa20eff47f04545ffe0dd59b2ed19fffd393d9bf527bb2899f3dba\": container with ID starting with 1901de61e1fa20eff47f04545ffe0dd59b2ed19fffd393d9bf527bb2899f3dba not found: ID does not exist"
Nov 01 03:08:44.279575 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.279539    4400 scope.go:122] "RemoveContainer" containerID="87fe732655cd942b05e12ea5580443ce47acccbe7bc19e848e2b0f234722259f"
Nov 01 03:08:44.279783 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.279765    4400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87fe732655cd942b05e12ea5580443ce47acccbe7bc19e848e2b0f234722259f"} err="failed to get container status \"87fe732655cd942b05e12ea5580443ce47acccbe7bc19e848e2b0f234722259f\": rpc error: code = NotFound desc = could not find container \"87fe732655cd942b05e12ea5580443ce47acccbe7bc19e848e2b0f234722259f\": container with ID starting with 87fe732655cd942b05e12ea5580443ce47acccbe7bc19e848e2b0f234722259f not found: ID does not exist"
Nov 01 03:08:44.279826 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.279782    4400 scope.go:122] "RemoveContainer" containerID="2459da8a6b1d6965c7ca185ef8e6fcd0b8ece1d4d537ff3faaf04376ef1cd12d"
Nov 01 03:08:44.280013 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.279996    4400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2459da8a6b1d6965c7ca185ef8e6fcd0b8ece1d4d537ff3faaf04376ef1cd12d"} err="failed to get container status \"2459da8a6b1d6965c7ca185ef8e6fcd0b8ece1d4d537ff3faaf04376ef1cd12d\": rpc error: code = NotFound desc = could not find container \"2459da8a6b1d6965c7ca185ef8e6fcd0b8ece1d4d537ff3faaf04376ef1cd12d\": container with ID starting with 2459da8a6b1d6965c7ca185ef8e6fcd0b8ece1d4d537ff3faaf04376ef1cd12d not found: ID does not exist"
Nov 01 03:08:44.280061 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.280014    4400 scope.go:122] "RemoveContainer" containerID="44f2d215c8d28a22ac3b18aba136454d6c518c33952c0ba3ce905e0ca4a52a99"
Nov 01 03:08:44.280205 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.280185    4400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44f2d215c8d28a22ac3b18aba136454d6c518c33952c0ba3ce905e0ca4a52a99"} err="failed to get container status \"44f2d215c8d28a22ac3b18aba136454d6c518c33952c0ba3ce905e0ca4a52a99\": rpc error: code = NotFound desc = could not find container \"44f2d215c8d28a22ac3b18aba136454d6c518c33952c0ba3ce905e0ca4a52a99\": container with ID starting with 44f2d215c8d28a22ac3b18aba136454d6c518c33952c0ba3ce905e0ca4a52a99 not found: ID does not exist"
Nov 01 03:08:44.280249 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.280205    4400 scope.go:122] "RemoveContainer" containerID="c5ee8cef5bccf833843ae19213ca37a09919502ccbc56af21cd5324fd51373c8"
Nov 01 03:08:44.280392 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.280377    4400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5ee8cef5bccf833843ae19213ca37a09919502ccbc56af21cd5324fd51373c8"} err="failed to get container status \"c5ee8cef5bccf833843ae19213ca37a09919502ccbc56af21cd5324fd51373c8\": rpc error: code = NotFound desc = could not find container \"c5ee8cef5bccf833843ae19213ca37a09919502ccbc56af21cd5324fd51373c8\": container with ID starting with c5ee8cef5bccf833843ae19213ca37a09919502ccbc56af21cd5324fd51373c8 not found: ID does not exist"
Nov 01 03:08:44.280432 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.280392    4400 scope.go:122] "RemoveContainer" containerID="5c5adee9bbcfd85f53a6d76b11100ec7ef4b40c433b3bdd4cc2ed7f299d987c1"
Nov 01 03:08:44.280575 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.280559    4400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c5adee9bbcfd85f53a6d76b11100ec7ef4b40c433b3bdd4cc2ed7f299d987c1"} err="failed to get container status \"5c5adee9bbcfd85f53a6d76b11100ec7ef4b40c433b3bdd4cc2ed7f299d987c1\": rpc error: code = NotFound desc = could not find container \"5c5adee9bbcfd85f53a6d76b11100ec7ef4b40c433b3bdd4cc2ed7f299d987c1\": container with ID starting with 5c5adee9bbcfd85f53a6d76b11100ec7ef4b40c433b3bdd4cc2ed7f299d987c1 not found: ID does not exist"
Nov 01 03:08:44.280616 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.280575    4400 scope.go:122] "RemoveContainer" containerID="960d7c3027418725815287ced9709538984dc0ba28a6a826382c9a227041f7df"
Nov 01 03:08:44.280819 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.280801    4400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"960d7c3027418725815287ced9709538984dc0ba28a6a826382c9a227041f7df"} err="failed to get container status \"960d7c3027418725815287ced9709538984dc0ba28a6a826382c9a227041f7df\": rpc error: code = NotFound desc = could not find container \"960d7c3027418725815287ced9709538984dc0ba28a6a826382c9a227041f7df\": container with ID starting with 960d7c3027418725815287ced9709538984dc0ba28a6a826382c9a227041f7df not found: ID does not exist"
Nov 01 03:08:44.280819 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.280817    4400 scope.go:122] "RemoveContainer" containerID="1901de61e1fa20eff47f04545ffe0dd59b2ed19fffd393d9bf527bb2899f3dba"
Nov 01 03:08:44.281020 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.281005    4400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1901de61e1fa20eff47f04545ffe0dd59b2ed19fffd393d9bf527bb2899f3dba"} err="failed to get container status \"1901de61e1fa20eff47f04545ffe0dd59b2ed19fffd393d9bf527bb2899f3dba\": rpc error: code = NotFound desc = could not find container \"1901de61e1fa20eff47f04545ffe0dd59b2ed19fffd393d9bf527bb2899f3dba\": container with ID starting with 1901de61e1fa20eff47f04545ffe0dd59b2ed19fffd393d9bf527bb2899f3dba not found: ID does not exist"
Nov 01 03:08:44.440600 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.440578    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 01 03:08:44.491644 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.491596    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 01 03:08:44.492408 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.492391    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 01 03:08:44.492696 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.492669    4400 cpu_manager.go:414] "containerMap: removing container" podUID="95482def-35e0-4879-a721-d1ce6d251061" containerName="init-config-reloader"
Nov 01 03:08:44.492743 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.492697    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95482def-35e0-4879-a721-d1ce6d251061" containerName="init-config-reloader"
Nov 01 03:08:44.492743 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.492705    4400 cpu_manager.go:414] "containerMap: removing container" podUID="95482def-35e0-4879-a721-d1ce6d251061" containerName="config-reloader"
Nov 01 03:08:44.492743 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.492710    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95482def-35e0-4879-a721-d1ce6d251061" containerName="config-reloader"
Nov 01 03:08:44.492743 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.492721    4400 cpu_manager.go:414] "containerMap: removing container" podUID="95482def-35e0-4879-a721-d1ce6d251061" containerName="kube-rbac-proxy-web"
Nov 01 03:08:44.492743 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.492726    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95482def-35e0-4879-a721-d1ce6d251061" containerName="kube-rbac-proxy-web"
Nov 01 03:08:44.492743 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.492733    4400 cpu_manager.go:414] "containerMap: removing container" podUID="95482def-35e0-4879-a721-d1ce6d251061" containerName="kube-rbac-proxy"
Nov 01 03:08:44.492743 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.492737    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95482def-35e0-4879-a721-d1ce6d251061" containerName="kube-rbac-proxy"
Nov 01 03:08:44.492964 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.492746    4400 cpu_manager.go:414] "containerMap: removing container" podUID="95482def-35e0-4879-a721-d1ce6d251061" containerName="kube-rbac-proxy-metric"
Nov 01 03:08:44.492964 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.492751    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95482def-35e0-4879-a721-d1ce6d251061" containerName="kube-rbac-proxy-metric"
Nov 01 03:08:44.492964 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.492767    4400 cpu_manager.go:414] "containerMap: removing container" podUID="95482def-35e0-4879-a721-d1ce6d251061" containerName="alertmanager"
Nov 01 03:08:44.492964 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.492772    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95482def-35e0-4879-a721-d1ce6d251061" containerName="alertmanager"
Nov 01 03:08:44.492964 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.492781    4400 cpu_manager.go:414] "containerMap: removing container" podUID="95482def-35e0-4879-a721-d1ce6d251061" containerName="prom-label-proxy"
Nov 01 03:08:44.492964 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.492785    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95482def-35e0-4879-a721-d1ce6d251061" containerName="prom-label-proxy"
Nov 01 03:08:44.492964 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.492828    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="95482def-35e0-4879-a721-d1ce6d251061" containerName="kube-rbac-proxy-web"
Nov 01 03:08:44.492964 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.492834    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="95482def-35e0-4879-a721-d1ce6d251061" containerName="kube-rbac-proxy"
Nov 01 03:08:44.492964 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.492841    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="95482def-35e0-4879-a721-d1ce6d251061" containerName="kube-rbac-proxy-metric"
Nov 01 03:08:44.492964 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.492849    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="95482def-35e0-4879-a721-d1ce6d251061" containerName="alertmanager"
Nov 01 03:08:44.492964 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.492854    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="95482def-35e0-4879-a721-d1ce6d251061" containerName="prom-label-proxy"
Nov 01 03:08:44.492964 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.492860    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="95482def-35e0-4879-a721-d1ce6d251061" containerName="config-reloader"
Nov 01 03:08:44.495864 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.495850    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 01 03:08:44.549707 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.549656    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 01 03:08:44.549707 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.549673    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 01 03:08:44.549867 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.549800    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 01 03:08:44.549947 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.549932    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 01 03:08:44.550015 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.550003    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 01 03:08:44.550093 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.550077    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 01 03:08:44.550153 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.550081    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 01 03:08:44.550202 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.550192    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-q4rx2\""
Nov 01 03:08:44.550275 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.550263    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 01 03:08:44.600866 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.600840    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 01 03:08:44.653065 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.653037    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/395c33e8-a15b-42bc-b556-bf8e9b2a6d64-web-config\") pod \"alertmanager-main-0\" (UID: \"395c33e8-a15b-42bc-b556-bf8e9b2a6d64\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 03:08:44.653173 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.653079    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/395c33e8-a15b-42bc-b556-bf8e9b2a6d64-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"395c33e8-a15b-42bc-b556-bf8e9b2a6d64\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 03:08:44.653173 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.653097    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/395c33e8-a15b-42bc-b556-bf8e9b2a6d64-config-volume\") pod \"alertmanager-main-0\" (UID: \"395c33e8-a15b-42bc-b556-bf8e9b2a6d64\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 03:08:44.653173 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.653160    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/395c33e8-a15b-42bc-b556-bf8e9b2a6d64-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"395c33e8-a15b-42bc-b556-bf8e9b2a6d64\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 03:08:44.653287 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.653208    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkmtt\" (UniqueName: \"kubernetes.io/projected/395c33e8-a15b-42bc-b556-bf8e9b2a6d64-kube-api-access-dkmtt\") pod \"alertmanager-main-0\" (UID: \"395c33e8-a15b-42bc-b556-bf8e9b2a6d64\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 03:08:44.653287 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.653263    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/395c33e8-a15b-42bc-b556-bf8e9b2a6d64-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"395c33e8-a15b-42bc-b556-bf8e9b2a6d64\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 03:08:44.653367 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.653288    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/395c33e8-a15b-42bc-b556-bf8e9b2a6d64-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"395c33e8-a15b-42bc-b556-bf8e9b2a6d64\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 03:08:44.653367 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.653304    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/395c33e8-a15b-42bc-b556-bf8e9b2a6d64-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"395c33e8-a15b-42bc-b556-bf8e9b2a6d64\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 03:08:44.653442 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.653367    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/395c33e8-a15b-42bc-b556-bf8e9b2a6d64-config-out\") pod \"alertmanager-main-0\" (UID: \"395c33e8-a15b-42bc-b556-bf8e9b2a6d64\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 03:08:44.653442 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.653408    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/395c33e8-a15b-42bc-b556-bf8e9b2a6d64-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"395c33e8-a15b-42bc-b556-bf8e9b2a6d64\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 03:08:44.653442 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.653427    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/395c33e8-a15b-42bc-b556-bf8e9b2a6d64-tls-assets\") pod \"alertmanager-main-0\" (UID: \"395c33e8-a15b-42bc-b556-bf8e9b2a6d64\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 03:08:44.653538 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.653445    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/395c33e8-a15b-42bc-b556-bf8e9b2a6d64-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"395c33e8-a15b-42bc-b556-bf8e9b2a6d64\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 03:08:44.653538 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.653498    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/395c33e8-a15b-42bc-b556-bf8e9b2a6d64-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"395c33e8-a15b-42bc-b556-bf8e9b2a6d64\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 03:08:44.730116 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.730088    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 01 03:08:44.753919 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.753859    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/395c33e8-a15b-42bc-b556-bf8e9b2a6d64-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"395c33e8-a15b-42bc-b556-bf8e9b2a6d64\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 03:08:44.753919 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.753895    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/395c33e8-a15b-42bc-b556-bf8e9b2a6d64-web-config\") pod \"alertmanager-main-0\" (UID: \"395c33e8-a15b-42bc-b556-bf8e9b2a6d64\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 03:08:44.754066 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.753924    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/395c33e8-a15b-42bc-b556-bf8e9b2a6d64-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"395c33e8-a15b-42bc-b556-bf8e9b2a6d64\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 03:08:44.754066 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.753939    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/395c33e8-a15b-42bc-b556-bf8e9b2a6d64-config-volume\") pod \"alertmanager-main-0\" (UID: \"395c33e8-a15b-42bc-b556-bf8e9b2a6d64\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 03:08:44.754066 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.753961    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/395c33e8-a15b-42bc-b556-bf8e9b2a6d64-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"395c33e8-a15b-42bc-b556-bf8e9b2a6d64\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 03:08:44.754066 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.753981    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkmtt\" (UniqueName: \"kubernetes.io/projected/395c33e8-a15b-42bc-b556-bf8e9b2a6d64-kube-api-access-dkmtt\") pod \"alertmanager-main-0\" (UID: \"395c33e8-a15b-42bc-b556-bf8e9b2a6d64\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 03:08:44.754066 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.754027    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/395c33e8-a15b-42bc-b556-bf8e9b2a6d64-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"395c33e8-a15b-42bc-b556-bf8e9b2a6d64\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 03:08:44.754066 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.754065    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/395c33e8-a15b-42bc-b556-bf8e9b2a6d64-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"395c33e8-a15b-42bc-b556-bf8e9b2a6d64\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 03:08:44.754370 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.754089    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/395c33e8-a15b-42bc-b556-bf8e9b2a6d64-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"395c33e8-a15b-42bc-b556-bf8e9b2a6d64\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 03:08:44.754370 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.754113    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/395c33e8-a15b-42bc-b556-bf8e9b2a6d64-config-out\") pod \"alertmanager-main-0\" (UID: \"395c33e8-a15b-42bc-b556-bf8e9b2a6d64\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 03:08:44.754370 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.754133    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/395c33e8-a15b-42bc-b556-bf8e9b2a6d64-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"395c33e8-a15b-42bc-b556-bf8e9b2a6d64\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 03:08:44.754370 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.754151    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/395c33e8-a15b-42bc-b556-bf8e9b2a6d64-tls-assets\") pod \"alertmanager-main-0\" (UID: \"395c33e8-a15b-42bc-b556-bf8e9b2a6d64\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 03:08:44.754370 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.754171    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/395c33e8-a15b-42bc-b556-bf8e9b2a6d64-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"395c33e8-a15b-42bc-b556-bf8e9b2a6d64\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 03:08:44.754370 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.754332    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/395c33e8-a15b-42bc-b556-bf8e9b2a6d64-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"395c33e8-a15b-42bc-b556-bf8e9b2a6d64\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 03:08:44.755172 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.755141    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/395c33e8-a15b-42bc-b556-bf8e9b2a6d64-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"395c33e8-a15b-42bc-b556-bf8e9b2a6d64\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 03:08:44.755830 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.755782    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/395c33e8-a15b-42bc-b556-bf8e9b2a6d64-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"395c33e8-a15b-42bc-b556-bf8e9b2a6d64\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 03:08:44.757609 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.757268    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/395c33e8-a15b-42bc-b556-bf8e9b2a6d64-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"395c33e8-a15b-42bc-b556-bf8e9b2a6d64\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 03:08:44.757609 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.757549    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/395c33e8-a15b-42bc-b556-bf8e9b2a6d64-web-config\") pod \"alertmanager-main-0\" (UID: \"395c33e8-a15b-42bc-b556-bf8e9b2a6d64\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 03:08:44.757797 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.757737    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/395c33e8-a15b-42bc-b556-bf8e9b2a6d64-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"395c33e8-a15b-42bc-b556-bf8e9b2a6d64\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 03:08:44.757958 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.757911    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/395c33e8-a15b-42bc-b556-bf8e9b2a6d64-config-volume\") pod \"alertmanager-main-0\" (UID: \"395c33e8-a15b-42bc-b556-bf8e9b2a6d64\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 03:08:44.758064 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.758045    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/395c33e8-a15b-42bc-b556-bf8e9b2a6d64-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"395c33e8-a15b-42bc-b556-bf8e9b2a6d64\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 03:08:44.758207 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.758189    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/395c33e8-a15b-42bc-b556-bf8e9b2a6d64-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"395c33e8-a15b-42bc-b556-bf8e9b2a6d64\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 03:08:44.758265 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.758217    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/395c33e8-a15b-42bc-b556-bf8e9b2a6d64-config-out\") pod \"alertmanager-main-0\" (UID: \"395c33e8-a15b-42bc-b556-bf8e9b2a6d64\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 03:08:44.758353 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.758331    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/395c33e8-a15b-42bc-b556-bf8e9b2a6d64-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"395c33e8-a15b-42bc-b556-bf8e9b2a6d64\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 03:08:44.758906 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.758888    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/395c33e8-a15b-42bc-b556-bf8e9b2a6d64-tls-assets\") pod \"alertmanager-main-0\" (UID: \"395c33e8-a15b-42bc-b556-bf8e9b2a6d64\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 03:08:44.817099 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:44.817072    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkmtt\" (UniqueName: \"kubernetes.io/projected/395c33e8-a15b-42bc-b556-bf8e9b2a6d64-kube-api-access-dkmtt\") pod \"alertmanager-main-0\" (UID: \"395c33e8-a15b-42bc-b556-bf8e9b2a6d64\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 03:08:45.112133 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:45.112106    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 01 03:08:45.589741 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:45.589663    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 01 03:08:45.591999 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:08:45.591968    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod395c33e8_a15b_42bc_b556_bf8e9b2a6d64.slice/crio-888fee9344e43af7103e83eb579a757b7ab053a7ed4888e41922ccf6615c27ec WatchSource:0}: Error finding container 888fee9344e43af7103e83eb579a757b7ab053a7ed4888e41922ccf6615c27ec: Status 404 returned error can't find the container with id 888fee9344e43af7103e83eb579a757b7ab053a7ed4888e41922ccf6615c27ec
Nov 01 03:08:45.615294 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:45.615268    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95482def-35e0-4879-a721-d1ce6d251061" path="/var/lib/kubelet/pods/95482def-35e0-4879-a721-d1ce6d251061/volumes"
Nov 01 03:08:46.229295 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:46.229263    4400 generic.go:356] "Generic (PLEG): container finished" podID="395c33e8-a15b-42bc-b556-bf8e9b2a6d64" containerID="2e66b5a5d6dc57335513c3117b8a40b57b1e060cfa0f60081e0d8ccf39ed05d5" exitCode=0
Nov 01 03:08:46.229295 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:46.229297    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"395c33e8-a15b-42bc-b556-bf8e9b2a6d64","Type":"ContainerDied","Data":"2e66b5a5d6dc57335513c3117b8a40b57b1e060cfa0f60081e0d8ccf39ed05d5"}
Nov 01 03:08:46.229509 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:46.229349    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"395c33e8-a15b-42bc-b556-bf8e9b2a6d64","Type":"ContainerStarted","Data":"888fee9344e43af7103e83eb579a757b7ab053a7ed4888e41922ccf6615c27ec"}
Nov 01 03:08:46.867418 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:46.867384    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-879889647-wqck7"
Nov 01 03:08:46.867818 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:46.867460    4400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-879889647-wqck7"
Nov 01 03:08:46.871936 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:46.871918    4400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-879889647-wqck7"
Nov 01 03:08:47.236899 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:47.236802    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"395c33e8-a15b-42bc-b556-bf8e9b2a6d64","Type":"ContainerStarted","Data":"c43d9038677418be498ce34c962c68b85d6d51cbfaab837d610d2fd031bdf2d6"}
Nov 01 03:08:47.236899 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:47.236886    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"395c33e8-a15b-42bc-b556-bf8e9b2a6d64","Type":"ContainerStarted","Data":"998126b83f74da32b033aeb41ce11228b83261dbf714e6f038a922fbce043f72"}
Nov 01 03:08:47.237117 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:47.236923    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"395c33e8-a15b-42bc-b556-bf8e9b2a6d64","Type":"ContainerStarted","Data":"7142fca9e777c506e06c2b0d05fe4d55bd7fa9793acff4ea3d290ca8e6885a83"}
Nov 01 03:08:47.237117 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:47.236961    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"395c33e8-a15b-42bc-b556-bf8e9b2a6d64","Type":"ContainerStarted","Data":"96bac343bcf3819e73013e147499b94a61ab20c2be9a774b469985e6f4a12058"}
Nov 01 03:08:47.237117 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:47.236995    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"395c33e8-a15b-42bc-b556-bf8e9b2a6d64","Type":"ContainerStarted","Data":"f01a0550d53f7a8bc1f647fb8dc55aa705662dd5537a4e1627a4ef8894439bba"}
Nov 01 03:08:47.237117 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:47.237029    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"395c33e8-a15b-42bc-b556-bf8e9b2a6d64","Type":"ContainerStarted","Data":"35c0f7dfbd2afe74e30dc94e3c1ac7dd616990fbb05f6d4a79b9e9b0b84bbe31"}
Nov 01 03:08:47.242224 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:47.242202    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-879889647-wqck7"
Nov 01 03:08:47.370067 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:47.370021    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.359736782 podStartE2EDuration="3.37000387s" totalImagesPullingTime="0s" totalInitContainerRuntime="10.267088ms" isStatefulPod=false podCreationTimestamp="2026-11-01 03:08:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:08:47.369081931 +0000 UTC m=+133.883727794" watchObservedRunningTime="2026-11-01 03:08:47.37000387 +0000 UTC m=+133.884649733"
Nov 01 03:08:47.525127 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:08:47.525056    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-c5558cd55-7tq5f"]
Nov 01 03:09:01.729089 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:09:01.729050    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:09:12.545231 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:09:12.545168    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-c5558cd55-7tq5f" podUID="9c2a5f25-e674-4b6d-95b8-b898412e095e" containerName="console" containerID="cri-o://a64b8259d5ae70dd6a79f15d439f4664789f60256de8a20bb56bb824659f5b9b" gracePeriod=15
Nov 01 03:09:12.789885 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:09:12.789860    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c5558cd55-7tq5f"
Nov 01 03:09:12.869654 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:09:12.869629    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c2a5f25-e674-4b6d-95b8-b898412e095e-console-config\" (UniqueName: \"kubernetes.io/configmap/9c2a5f25-e674-4b6d-95b8-b898412e095e-console-config\") pod \"9c2a5f25-e674-4b6d-95b8-b898412e095e\" (UID: \"9c2a5f25-e674-4b6d-95b8-b898412e095e\") "
Nov 01 03:09:12.869803 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:09:12.869672    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c2a5f25-e674-4b6d-95b8-b898412e095e-kube-api-access-sbq98\" (UniqueName: \"kubernetes.io/projected/9c2a5f25-e674-4b6d-95b8-b898412e095e-kube-api-access-sbq98\") pod \"9c2a5f25-e674-4b6d-95b8-b898412e095e\" (UID: \"9c2a5f25-e674-4b6d-95b8-b898412e095e\") "
Nov 01 03:09:12.869803 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:09:12.869745    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c2a5f25-e674-4b6d-95b8-b898412e095e-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9c2a5f25-e674-4b6d-95b8-b898412e095e-oauth-serving-cert\") pod \"9c2a5f25-e674-4b6d-95b8-b898412e095e\" (UID: \"9c2a5f25-e674-4b6d-95b8-b898412e095e\") "
Nov 01 03:09:12.869803 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:09:12.869787    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c2a5f25-e674-4b6d-95b8-b898412e095e-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c2a5f25-e674-4b6d-95b8-b898412e095e-service-ca\") pod \"9c2a5f25-e674-4b6d-95b8-b898412e095e\" (UID: \"9c2a5f25-e674-4b6d-95b8-b898412e095e\") "
Nov 01 03:09:12.869926 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:09:12.869821    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c2a5f25-e674-4b6d-95b8-b898412e095e-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c2a5f25-e674-4b6d-95b8-b898412e095e-trusted-ca-bundle\") pod \"9c2a5f25-e674-4b6d-95b8-b898412e095e\" (UID: \"9c2a5f25-e674-4b6d-95b8-b898412e095e\") "
Nov 01 03:09:12.869926 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:09:12.869851    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c2a5f25-e674-4b6d-95b8-b898412e095e-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9c2a5f25-e674-4b6d-95b8-b898412e095e-console-serving-cert\") pod \"9c2a5f25-e674-4b6d-95b8-b898412e095e\" (UID: \"9c2a5f25-e674-4b6d-95b8-b898412e095e\") "
Nov 01 03:09:12.869926 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:09:12.869898    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c2a5f25-e674-4b6d-95b8-b898412e095e-tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c2a5f25-e674-4b6d-95b8-b898412e095e-tmp\") pod \"9c2a5f25-e674-4b6d-95b8-b898412e095e\" (UID: \"9c2a5f25-e674-4b6d-95b8-b898412e095e\") "
Nov 01 03:09:12.870078 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:09:12.869929    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c2a5f25-e674-4b6d-95b8-b898412e095e-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9c2a5f25-e674-4b6d-95b8-b898412e095e-console-oauth-config\") pod \"9c2a5f25-e674-4b6d-95b8-b898412e095e\" (UID: \"9c2a5f25-e674-4b6d-95b8-b898412e095e\") "
Nov 01 03:09:12.870078 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:09:12.870043    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c2a5f25-e674-4b6d-95b8-b898412e095e-console-config" pod "9c2a5f25-e674-4b6d-95b8-b898412e095e" (UID: "9c2a5f25-e674-4b6d-95b8-b898412e095e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:09:12.870455 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:09:12.870179    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c2a5f25-e674-4b6d-95b8-b898412e095e-oauth-serving-cert" pod "9c2a5f25-e674-4b6d-95b8-b898412e095e" (UID: "9c2a5f25-e674-4b6d-95b8-b898412e095e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:09:12.870455 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:09:12.870293    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c2a5f25-e674-4b6d-95b8-b898412e095e-trusted-ca-bundle" pod "9c2a5f25-e674-4b6d-95b8-b898412e095e" (UID: "9c2a5f25-e674-4b6d-95b8-b898412e095e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:09:12.870455 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:09:12.870285    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c2a5f25-e674-4b6d-95b8-b898412e095e-service-ca" pod "9c2a5f25-e674-4b6d-95b8-b898412e095e" (UID: "9c2a5f25-e674-4b6d-95b8-b898412e095e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:09:12.870455 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:09:12.870420    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c2a5f25-e674-4b6d-95b8-b898412e095e-tmp" pod "9c2a5f25-e674-4b6d-95b8-b898412e095e" (UID: "9c2a5f25-e674-4b6d-95b8-b898412e095e"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:09:12.870455 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:09:12.870447    4400 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9c2a5f25-e674-4b6d-95b8-b898412e095e-console-config\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:09:12.870955 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:09:12.870472    4400 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9c2a5f25-e674-4b6d-95b8-b898412e095e-oauth-serving-cert\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:09:12.872196 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:09:12.872175    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c2a5f25-e674-4b6d-95b8-b898412e095e-kube-api-access-sbq98" pod "9c2a5f25-e674-4b6d-95b8-b898412e095e" (UID: "9c2a5f25-e674-4b6d-95b8-b898412e095e"). InnerVolumeSpecName "kube-api-access-sbq98". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:09:12.872196 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:09:12.872176    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c2a5f25-e674-4b6d-95b8-b898412e095e-console-oauth-config" pod "9c2a5f25-e674-4b6d-95b8-b898412e095e" (UID: "9c2a5f25-e674-4b6d-95b8-b898412e095e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:09:12.872323 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:09:12.872180    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c2a5f25-e674-4b6d-95b8-b898412e095e-console-serving-cert" pod "9c2a5f25-e674-4b6d-95b8-b898412e095e" (UID: "9c2a5f25-e674-4b6d-95b8-b898412e095e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:09:12.971641 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:09:12.971606    4400 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9c2a5f25-e674-4b6d-95b8-b898412e095e-console-oauth-config\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:09:12.971641 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:09:12.971632    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbq98\" (UniqueName: \"kubernetes.io/projected/9c2a5f25-e674-4b6d-95b8-b898412e095e-kube-api-access-sbq98\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:09:12.971641 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:09:12.971645    4400 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9c2a5f25-e674-4b6d-95b8-b898412e095e-service-ca\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:09:12.971874 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:09:12.971653    4400 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c2a5f25-e674-4b6d-95b8-b898412e095e-trusted-ca-bundle\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:09:12.971874 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:09:12.971662    4400 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9c2a5f25-e674-4b6d-95b8-b898412e095e-console-serving-cert\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:09:12.971874 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:09:12.971671    4400 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c2a5f25-e674-4b6d-95b8-b898412e095e-tmp\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:09:13.334525 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:09:13.334492    4400 generic.go:356] "Generic (PLEG): container finished" podID="9c2a5f25-e674-4b6d-95b8-b898412e095e" containerID="a64b8259d5ae70dd6a79f15d439f4664789f60256de8a20bb56bb824659f5b9b" exitCode=0
Nov 01 03:09:13.334668 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:09:13.334541    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c5558cd55-7tq5f"
Nov 01 03:09:13.334668 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:09:13.334576    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c5558cd55-7tq5f" event={"ID":"9c2a5f25-e674-4b6d-95b8-b898412e095e","Type":"ContainerDied","Data":"a64b8259d5ae70dd6a79f15d439f4664789f60256de8a20bb56bb824659f5b9b"}
Nov 01 03:09:13.334668 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:09:13.334632    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c5558cd55-7tq5f" event={"ID":"9c2a5f25-e674-4b6d-95b8-b898412e095e","Type":"ContainerDied","Data":"c73a2ca3eb7e4adfdfea1569d9f86e439a3a5bb8fc1b16a8d03a3f4ff49de439"}
Nov 01 03:09:13.334668 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:09:13.334652    4400 scope.go:122] "RemoveContainer" containerID="a64b8259d5ae70dd6a79f15d439f4664789f60256de8a20bb56bb824659f5b9b"
Nov 01 03:09:13.343344 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:09:13.343330    4400 scope.go:122] "RemoveContainer" containerID="a64b8259d5ae70dd6a79f15d439f4664789f60256de8a20bb56bb824659f5b9b"
Nov 01 03:09:13.343610 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:09:13.343595    4400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a64b8259d5ae70dd6a79f15d439f4664789f60256de8a20bb56bb824659f5b9b\": container with ID starting with a64b8259d5ae70dd6a79f15d439f4664789f60256de8a20bb56bb824659f5b9b not found: ID does not exist" containerID="a64b8259d5ae70dd6a79f15d439f4664789f60256de8a20bb56bb824659f5b9b"
Nov 01 03:09:13.343650 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:09:13.343618    4400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a64b8259d5ae70dd6a79f15d439f4664789f60256de8a20bb56bb824659f5b9b"} err="failed to get container status \"a64b8259d5ae70dd6a79f15d439f4664789f60256de8a20bb56bb824659f5b9b\": rpc error: code = NotFound desc = could not find container \"a64b8259d5ae70dd6a79f15d439f4664789f60256de8a20bb56bb824659f5b9b\": container with ID starting with a64b8259d5ae70dd6a79f15d439f4664789f60256de8a20bb56bb824659f5b9b not found: ID does not exist"
Nov 01 03:09:13.502529 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:09:13.502499    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-c5558cd55-7tq5f"]
Nov 01 03:09:13.552996 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:09:13.552961    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-c5558cd55-7tq5f"]
Nov 01 03:09:13.611392 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:09:13.611325    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c2a5f25-e674-4b6d-95b8-b898412e095e" path="/var/lib/kubelet/pods/9c2a5f25-e674-4b6d-95b8-b898412e095e/volumes"
Nov 01 03:09:31.728164 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:09:31.728127    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:10:01.728536 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:10:01.728493    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:10:31.727407 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:10:31.727375    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:10:33.979342 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:33.979308    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cnpjc"]
Nov 01 03:10:33.979811 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:33.979612    4400 cpu_manager.go:414] "containerMap: removing container" podUID="9c2a5f25-e674-4b6d-95b8-b898412e095e" containerName="console"
Nov 01 03:10:33.979811 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:33.979622    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c2a5f25-e674-4b6d-95b8-b898412e095e" containerName="console"
Nov 01 03:10:33.979811 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:33.979721    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c2a5f25-e674-4b6d-95b8-b898412e095e" containerName="console"
Nov 01 03:10:33.982839 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:33.982818    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cnpjc"
Nov 01 03:10:34.058629 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:34.058582    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/795852c6-f698-4c60-bbf3-4751f83eccbf-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cnpjc\" (UID: \"795852c6-f698-4c60-bbf3-4751f83eccbf\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cnpjc"
Nov 01 03:10:34.058829 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:34.058667    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/795852c6-f698-4c60-bbf3-4751f83eccbf-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cnpjc\" (UID: \"795852c6-f698-4c60-bbf3-4751f83eccbf\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cnpjc"
Nov 01 03:10:34.058829 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:34.058766    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgp8m\" (UniqueName: \"kubernetes.io/projected/795852c6-f698-4c60-bbf3-4751f83eccbf-kube-api-access-mgp8m\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cnpjc\" (UID: \"795852c6-f698-4c60-bbf3-4751f83eccbf\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cnpjc"
Nov 01 03:10:34.089649 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:34.089618    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Nov 01 03:10:34.089905 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:34.089659    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Nov 01 03:10:34.089905 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:34.089817    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-gw99p\""
Nov 01 03:10:34.091336 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:34.091304    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cnpjc"]
Nov 01 03:10:34.092392 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:34.092372    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crld2r"]
Nov 01 03:10:34.096104 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:34.096085    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crld2r"
Nov 01 03:10:34.154713 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:34.154674    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-q87nw\""
Nov 01 03:10:34.160098 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:34.160081    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/795852c6-f698-4c60-bbf3-4751f83eccbf-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cnpjc\" (UID: \"795852c6-f698-4c60-bbf3-4751f83eccbf\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cnpjc"
Nov 01 03:10:34.160163 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:34.160112    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/48e4c19c-17ee-467c-a7b9-652ce87156af-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crld2r\" (UID: \"48e4c19c-17ee-467c-a7b9-652ce87156af\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crld2r"
Nov 01 03:10:34.160163 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:34.160149    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgp8m\" (UniqueName: \"kubernetes.io/projected/795852c6-f698-4c60-bbf3-4751f83eccbf-kube-api-access-mgp8m\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cnpjc\" (UID: \"795852c6-f698-4c60-bbf3-4751f83eccbf\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cnpjc"
Nov 01 03:10:34.160313 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:34.160290    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/48e4c19c-17ee-467c-a7b9-652ce87156af-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crld2r\" (UID: \"48e4c19c-17ee-467c-a7b9-652ce87156af\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crld2r"
Nov 01 03:10:34.160361 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:34.160345    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/795852c6-f698-4c60-bbf3-4751f83eccbf-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cnpjc\" (UID: \"795852c6-f698-4c60-bbf3-4751f83eccbf\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cnpjc"
Nov 01 03:10:34.160428 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:34.160409    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6t2pq\" (UniqueName: \"kubernetes.io/projected/48e4c19c-17ee-467c-a7b9-652ce87156af-kube-api-access-6t2pq\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crld2r\" (UID: \"48e4c19c-17ee-467c-a7b9-652ce87156af\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crld2r"
Nov 01 03:10:34.160472 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:34.160457    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/795852c6-f698-4c60-bbf3-4751f83eccbf-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cnpjc\" (UID: \"795852c6-f698-4c60-bbf3-4751f83eccbf\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cnpjc"
Nov 01 03:10:34.160611 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:34.160596    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/795852c6-f698-4c60-bbf3-4751f83eccbf-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cnpjc\" (UID: \"795852c6-f698-4c60-bbf3-4751f83eccbf\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cnpjc"
Nov 01 03:10:34.197631 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:34.197608    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crld2r"]
Nov 01 03:10:34.222597 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:34.222570    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgp8m\" (UniqueName: \"kubernetes.io/projected/795852c6-f698-4c60-bbf3-4751f83eccbf-kube-api-access-mgp8m\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cnpjc\" (UID: \"795852c6-f698-4c60-bbf3-4751f83eccbf\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cnpjc"
Nov 01 03:10:34.261028 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:34.260969    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/48e4c19c-17ee-467c-a7b9-652ce87156af-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crld2r\" (UID: \"48e4c19c-17ee-467c-a7b9-652ce87156af\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crld2r"
Nov 01 03:10:34.261216 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:34.261184    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6t2pq\" (UniqueName: \"kubernetes.io/projected/48e4c19c-17ee-467c-a7b9-652ce87156af-kube-api-access-6t2pq\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crld2r\" (UID: \"48e4c19c-17ee-467c-a7b9-652ce87156af\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crld2r"
Nov 01 03:10:34.261354 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:34.261253    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/48e4c19c-17ee-467c-a7b9-652ce87156af-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crld2r\" (UID: \"48e4c19c-17ee-467c-a7b9-652ce87156af\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crld2r"
Nov 01 03:10:34.261354 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:34.261331    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/48e4c19c-17ee-467c-a7b9-652ce87156af-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crld2r\" (UID: \"48e4c19c-17ee-467c-a7b9-652ce87156af\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crld2r"
Nov 01 03:10:34.261568 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:34.261552    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/48e4c19c-17ee-467c-a7b9-652ce87156af-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crld2r\" (UID: \"48e4c19c-17ee-467c-a7b9-652ce87156af\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crld2r"
Nov 01 03:10:34.298290 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:34.298268    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cnpjc"
Nov 01 03:10:34.324069 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:34.324045    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6t2pq\" (UniqueName: \"kubernetes.io/projected/48e4c19c-17ee-467c-a7b9-652ce87156af-kube-api-access-6t2pq\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crld2r\" (UID: \"48e4c19c-17ee-467c-a7b9-652ce87156af\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crld2r"
Nov 01 03:10:34.413155 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:34.413127    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crld2r"
Nov 01 03:10:34.507218 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:34.507189    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-2gxpw"]
Nov 01 03:10:34.510748 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:34.510731    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2gxpw"
Nov 01 03:10:34.564156 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:34.564131    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9110c893-fab5-4136-b5ee-f3aca8edd0a9-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-2gxpw\" (UID: \"9110c893-fab5-4136-b5ee-f3aca8edd0a9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2gxpw"
Nov 01 03:10:34.564275 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:34.564168    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9110c893-fab5-4136-b5ee-f3aca8edd0a9-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-2gxpw\" (UID: \"9110c893-fab5-4136-b5ee-f3aca8edd0a9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2gxpw"
Nov 01 03:10:34.564275 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:34.564185    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9110c893-fab5-4136-b5ee-f3aca8edd0a9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-2gxpw\" (UID: \"9110c893-fab5-4136-b5ee-f3aca8edd0a9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2gxpw"
Nov 01 03:10:34.564360 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:34.564298    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9110c893-fab5-4136-b5ee-f3aca8edd0a9-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-2gxpw\" (UID: \"9110c893-fab5-4136-b5ee-f3aca8edd0a9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2gxpw"
Nov 01 03:10:34.564360 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:34.564330    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9110c893-fab5-4136-b5ee-f3aca8edd0a9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-2gxpw\" (UID: \"9110c893-fab5-4136-b5ee-f3aca8edd0a9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2gxpw"
Nov 01 03:10:34.564461 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:34.564363    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sklft\" (UniqueName: \"kubernetes.io/projected/9110c893-fab5-4136-b5ee-f3aca8edd0a9-kube-api-access-sklft\") pod \"istiod-openshift-gateway-7876bfcb74-2gxpw\" (UID: \"9110c893-fab5-4136-b5ee-f3aca8edd0a9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2gxpw"
Nov 01 03:10:34.564461 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:34.564429    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9110c893-fab5-4136-b5ee-f3aca8edd0a9-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-2gxpw\" (UID: \"9110c893-fab5-4136-b5ee-f3aca8edd0a9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2gxpw"
Nov 01 03:10:34.565200 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:34.565179    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 01 03:10:34.565315 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:34.565247    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 01 03:10:34.565473 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:34.565461    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 01 03:10:34.577120 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:34.577075    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 01 03:10:34.577226 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:34.577116    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 01 03:10:34.577324 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:34.577306    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 01 03:10:34.577384 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:34.577332    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-8w9bn\""
Nov 01 03:10:34.580475 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:34.580454    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-2gxpw"]
Nov 01 03:10:34.665212 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:34.665182    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9110c893-fab5-4136-b5ee-f3aca8edd0a9-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-2gxpw\" (UID: \"9110c893-fab5-4136-b5ee-f3aca8edd0a9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2gxpw"
Nov 01 03:10:34.665326 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:34.665223    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9110c893-fab5-4136-b5ee-f3aca8edd0a9-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-2gxpw\" (UID: \"9110c893-fab5-4136-b5ee-f3aca8edd0a9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2gxpw"
Nov 01 03:10:34.665326 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:34.665241    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9110c893-fab5-4136-b5ee-f3aca8edd0a9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-2gxpw\" (UID: \"9110c893-fab5-4136-b5ee-f3aca8edd0a9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2gxpw"
Nov 01 03:10:34.665326 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:34.665279    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9110c893-fab5-4136-b5ee-f3aca8edd0a9-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-2gxpw\" (UID: \"9110c893-fab5-4136-b5ee-f3aca8edd0a9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2gxpw"
Nov 01 03:10:34.665326 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:34.665310    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9110c893-fab5-4136-b5ee-f3aca8edd0a9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-2gxpw\" (UID: \"9110c893-fab5-4136-b5ee-f3aca8edd0a9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2gxpw"
Nov 01 03:10:34.665528 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:34.665356    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sklft\" (UniqueName: \"kubernetes.io/projected/9110c893-fab5-4136-b5ee-f3aca8edd0a9-kube-api-access-sklft\") pod \"istiod-openshift-gateway-7876bfcb74-2gxpw\" (UID: \"9110c893-fab5-4136-b5ee-f3aca8edd0a9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2gxpw"
Nov 01 03:10:34.665528 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:34.665378    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9110c893-fab5-4136-b5ee-f3aca8edd0a9-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-2gxpw\" (UID: \"9110c893-fab5-4136-b5ee-f3aca8edd0a9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2gxpw"
Nov 01 03:10:34.665975 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:34.665952    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9110c893-fab5-4136-b5ee-f3aca8edd0a9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-2gxpw\" (UID: \"9110c893-fab5-4136-b5ee-f3aca8edd0a9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2gxpw"
Nov 01 03:10:34.667796 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:34.667760    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9110c893-fab5-4136-b5ee-f3aca8edd0a9-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-2gxpw\" (UID: \"9110c893-fab5-4136-b5ee-f3aca8edd0a9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2gxpw"
Nov 01 03:10:34.667945 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:34.667928    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9110c893-fab5-4136-b5ee-f3aca8edd0a9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-2gxpw\" (UID: \"9110c893-fab5-4136-b5ee-f3aca8edd0a9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2gxpw"
Nov 01 03:10:34.668478 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:34.668446    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9110c893-fab5-4136-b5ee-f3aca8edd0a9-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-2gxpw\" (UID: \"9110c893-fab5-4136-b5ee-f3aca8edd0a9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2gxpw"
Nov 01 03:10:34.668585 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:34.668450    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9110c893-fab5-4136-b5ee-f3aca8edd0a9-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-2gxpw\" (UID: \"9110c893-fab5-4136-b5ee-f3aca8edd0a9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2gxpw"
Nov 01 03:10:34.708162 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:34.708143    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cnpjc"]
Nov 01 03:10:34.709887 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:10:34.709866    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod795852c6_f698_4c60_bbf3_4751f83eccbf.slice/crio-fa82218587b81d26752d9d4e5860806705cf61aca17ee3a0fa1bb9f2dac2d4e5 WatchSource:0}: Error finding container fa82218587b81d26752d9d4e5860806705cf61aca17ee3a0fa1bb9f2dac2d4e5: Status 404 returned error can't find the container with id fa82218587b81d26752d9d4e5860806705cf61aca17ee3a0fa1bb9f2dac2d4e5
Nov 01 03:10:34.727282 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:34.727264    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sklft\" (UniqueName: \"kubernetes.io/projected/9110c893-fab5-4136-b5ee-f3aca8edd0a9-kube-api-access-sklft\") pod \"istiod-openshift-gateway-7876bfcb74-2gxpw\" (UID: \"9110c893-fab5-4136-b5ee-f3aca8edd0a9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2gxpw"
Nov 01 03:10:34.727366 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:34.727321    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9110c893-fab5-4136-b5ee-f3aca8edd0a9-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-2gxpw\" (UID: \"9110c893-fab5-4136-b5ee-f3aca8edd0a9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2gxpw"
Nov 01 03:10:34.787303 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:34.787286    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crld2r"]
Nov 01 03:10:34.789006 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:10:34.788973    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48e4c19c_17ee_467c_a7b9_652ce87156af.slice/crio-fa09dbec918564e81f55f689e5ccfb0fb36a5c3a7068093c0fbf27c7ebae5908 WatchSource:0}: Error finding container fa09dbec918564e81f55f689e5ccfb0fb36a5c3a7068093c0fbf27c7ebae5908: Status 404 returned error can't find the container with id fa09dbec918564e81f55f689e5ccfb0fb36a5c3a7068093c0fbf27c7ebae5908
Nov 01 03:10:34.822623 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:34.822604    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2gxpw"
Nov 01 03:10:35.197188 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:35.197167    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-2gxpw"]
Nov 01 03:10:35.198938 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:10:35.198915    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9110c893_fab5_4136_b5ee_f3aca8edd0a9.slice/crio-7cb17d9f1819f3eb248c1a9337cc44eefe802a6a7d79d92bd3499e8bc78907af WatchSource:0}: Error finding container 7cb17d9f1819f3eb248c1a9337cc44eefe802a6a7d79d92bd3499e8bc78907af: Status 404 returned error can't find the container with id 7cb17d9f1819f3eb248c1a9337cc44eefe802a6a7d79d92bd3499e8bc78907af
Nov 01 03:10:35.634410 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:35.634371    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cnpjc" event={"ID":"795852c6-f698-4c60-bbf3-4751f83eccbf","Type":"ContainerStarted","Data":"fa82218587b81d26752d9d4e5860806705cf61aca17ee3a0fa1bb9f2dac2d4e5"}
Nov 01 03:10:35.635893 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:35.635862    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crld2r" event={"ID":"48e4c19c-17ee-467c-a7b9-652ce87156af","Type":"ContainerStarted","Data":"fa09dbec918564e81f55f689e5ccfb0fb36a5c3a7068093c0fbf27c7ebae5908"}
Nov 01 03:10:35.637489 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:35.637461    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2gxpw" event={"ID":"9110c893-fab5-4136-b5ee-f3aca8edd0a9","Type":"ContainerStarted","Data":"7cb17d9f1819f3eb248c1a9337cc44eefe802a6a7d79d92bd3499e8bc78907af"}
Nov 01 03:10:40.814370 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:40.814327    4400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 03:10:40.814866 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:40.814410    4400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 03:10:41.667004 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:41.666727    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2gxpw" event={"ID":"9110c893-fab5-4136-b5ee-f3aca8edd0a9","Type":"ContainerStarted","Data":"b70506740b1a32d6bd468250d927ee61c7a6ec6f01162a43d56ce5fbf6638b8e"}
Nov 01 03:10:41.669016 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:41.668854    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2gxpw"
Nov 01 03:10:41.669263 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:41.669220    4400 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-2gxpw container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 03:10:41.669365 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:41.669310    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2gxpw" podUID="9110c893-fab5-4136-b5ee-f3aca8edd0a9" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 03:10:41.879295 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:41.879225    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2gxpw" podStartSLOduration=2.266599682 podStartE2EDuration="7.879207601s" totalImagesPullingTime="5.612607919s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:10:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:10:41.878322378 +0000 UTC m=+248.392968240" watchObservedRunningTime="2026-11-01 03:10:41.879207601 +0000 UTC m=+248.393853464"
Nov 01 03:10:42.672660 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:42.672630    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2gxpw"
Nov 01 03:10:43.677088 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:43.677052    4400 generic.go:356] "Generic (PLEG): container finished" podID="795852c6-f698-4c60-bbf3-4751f83eccbf" containerID="cd492ca1bf9a0be3efcd84682beaee145675cabe1ebb59aa86d471a1f68e5e41" exitCode=0
Nov 01 03:10:43.677506 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:43.677139    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cnpjc" event={"ID":"795852c6-f698-4c60-bbf3-4751f83eccbf","Type":"ContainerDied","Data":"cd492ca1bf9a0be3efcd84682beaee145675cabe1ebb59aa86d471a1f68e5e41"}
Nov 01 03:10:43.678832 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:43.678799    4400 generic.go:356] "Generic (PLEG): container finished" podID="48e4c19c-17ee-467c-a7b9-652ce87156af" containerID="65b0d5c99be8048f1c33ce5e33d438cbdb7ab0f9b46769de1cb14d3542c3e2b8" exitCode=0
Nov 01 03:10:43.678966 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:43.678875    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crld2r" event={"ID":"48e4c19c-17ee-467c-a7b9-652ce87156af","Type":"ContainerDied","Data":"65b0d5c99be8048f1c33ce5e33d438cbdb7ab0f9b46769de1cb14d3542c3e2b8"}
Nov 01 03:10:47.697824 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:47.697771    4400 generic.go:356] "Generic (PLEG): container finished" podID="48e4c19c-17ee-467c-a7b9-652ce87156af" containerID="7852a8f738a5c600092e56b357effd001726a4a2e1b3d9aafb3d60730bca57de" exitCode=0
Nov 01 03:10:47.698282 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:47.697853    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crld2r" event={"ID":"48e4c19c-17ee-467c-a7b9-652ce87156af","Type":"ContainerDied","Data":"7852a8f738a5c600092e56b357effd001726a4a2e1b3d9aafb3d60730bca57de"}
Nov 01 03:10:48.703564 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:48.703529    4400 generic.go:356] "Generic (PLEG): container finished" podID="795852c6-f698-4c60-bbf3-4751f83eccbf" containerID="37557b0bdc679074b42f0372ad28fe8dbf7945313b0d07203edd563eea7014d8" exitCode=0
Nov 01 03:10:48.703981 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:48.703624    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cnpjc" event={"ID":"795852c6-f698-4c60-bbf3-4751f83eccbf","Type":"ContainerDied","Data":"37557b0bdc679074b42f0372ad28fe8dbf7945313b0d07203edd563eea7014d8"}
Nov 01 03:10:57.746181 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:57.746130    4400 generic.go:356] "Generic (PLEG): container finished" podID="795852c6-f698-4c60-bbf3-4751f83eccbf" containerID="f5277b6e4635078b526506b80e2709411ad1a54f841a50e4bf24f4e5ae3e1315" exitCode=0
Nov 01 03:10:57.746550 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:57.746200    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cnpjc" event={"ID":"795852c6-f698-4c60-bbf3-4751f83eccbf","Type":"ContainerDied","Data":"f5277b6e4635078b526506b80e2709411ad1a54f841a50e4bf24f4e5ae3e1315"}
Nov 01 03:10:57.748275 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:57.748256    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crld2r" event={"ID":"48e4c19c-17ee-467c-a7b9-652ce87156af","Type":"ContainerStarted","Data":"7d1f68d39fa8710c2ab71a930247457f881a17b6527092a1b2b8f01a6a2ebd36"}
Nov 01 03:10:58.756153 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:58.756087    4400 generic.go:356] "Generic (PLEG): container finished" podID="48e4c19c-17ee-467c-a7b9-652ce87156af" containerID="7d1f68d39fa8710c2ab71a930247457f881a17b6527092a1b2b8f01a6a2ebd36" exitCode=0
Nov 01 03:10:58.756490 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:58.756142    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crld2r" event={"ID":"48e4c19c-17ee-467c-a7b9-652ce87156af","Type":"ContainerDied","Data":"7d1f68d39fa8710c2ab71a930247457f881a17b6527092a1b2b8f01a6a2ebd36"}
Nov 01 03:10:58.898005 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:58.897975    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cnpjc"
Nov 01 03:10:58.998854 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:58.998831    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/795852c6-f698-4c60-bbf3-4751f83eccbf-util\" (UniqueName: \"kubernetes.io/empty-dir/795852c6-f698-4c60-bbf3-4751f83eccbf-util\") pod \"795852c6-f698-4c60-bbf3-4751f83eccbf\" (UID: \"795852c6-f698-4c60-bbf3-4751f83eccbf\") "
Nov 01 03:10:58.998977 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:58.998862    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/795852c6-f698-4c60-bbf3-4751f83eccbf-bundle\" (UniqueName: \"kubernetes.io/empty-dir/795852c6-f698-4c60-bbf3-4751f83eccbf-bundle\") pod \"795852c6-f698-4c60-bbf3-4751f83eccbf\" (UID: \"795852c6-f698-4c60-bbf3-4751f83eccbf\") "
Nov 01 03:10:58.998977 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:58.998916    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/795852c6-f698-4c60-bbf3-4751f83eccbf-kube-api-access-mgp8m\" (UniqueName: \"kubernetes.io/projected/795852c6-f698-4c60-bbf3-4751f83eccbf-kube-api-access-mgp8m\") pod \"795852c6-f698-4c60-bbf3-4751f83eccbf\" (UID: \"795852c6-f698-4c60-bbf3-4751f83eccbf\") "
Nov 01 03:10:58.999311 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:58.999283    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/795852c6-f698-4c60-bbf3-4751f83eccbf-bundle" pod "795852c6-f698-4c60-bbf3-4751f83eccbf" (UID: "795852c6-f698-4c60-bbf3-4751f83eccbf"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:10:59.001259 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:59.001235    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/795852c6-f698-4c60-bbf3-4751f83eccbf-kube-api-access-mgp8m" pod "795852c6-f698-4c60-bbf3-4751f83eccbf" (UID: "795852c6-f698-4c60-bbf3-4751f83eccbf"). InnerVolumeSpecName "kube-api-access-mgp8m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:10:59.003211 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:59.003191    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/795852c6-f698-4c60-bbf3-4751f83eccbf-util" pod "795852c6-f698-4c60-bbf3-4751f83eccbf" (UID: "795852c6-f698-4c60-bbf3-4751f83eccbf"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:10:59.100075 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:59.100054    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgp8m\" (UniqueName: \"kubernetes.io/projected/795852c6-f698-4c60-bbf3-4751f83eccbf-kube-api-access-mgp8m\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:10:59.100075 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:59.100077    4400 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/795852c6-f698-4c60-bbf3-4751f83eccbf-util\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:10:59.100229 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:59.100086    4400 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/795852c6-f698-4c60-bbf3-4751f83eccbf-bundle\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:10:59.761925 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:59.761899    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cnpjc"
Nov 01 03:10:59.761925 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:59.761905    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cnpjc" event={"ID":"795852c6-f698-4c60-bbf3-4751f83eccbf","Type":"ContainerDied","Data":"fa82218587b81d26752d9d4e5860806705cf61aca17ee3a0fa1bb9f2dac2d4e5"}
Nov 01 03:10:59.762357 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:59.761967    4400 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa82218587b81d26752d9d4e5860806705cf61aca17ee3a0fa1bb9f2dac2d4e5"
Nov 01 03:10:59.905845 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:59.905819    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crld2r"
Nov 01 03:10:59.934132 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:59.934101    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-576db5b58f-s552l"]
Nov 01 03:10:59.934416 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:59.934404    4400 cpu_manager.go:414] "containerMap: removing container" podUID="48e4c19c-17ee-467c-a7b9-652ce87156af" containerName="pull"
Nov 01 03:10:59.934458 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:59.934417    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48e4c19c-17ee-467c-a7b9-652ce87156af" containerName="pull"
Nov 01 03:10:59.934458 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:59.934429    4400 cpu_manager.go:414] "containerMap: removing container" podUID="795852c6-f698-4c60-bbf3-4751f83eccbf" containerName="util"
Nov 01 03:10:59.934458 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:59.934435    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="795852c6-f698-4c60-bbf3-4751f83eccbf" containerName="util"
Nov 01 03:10:59.934458 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:59.934456    4400 cpu_manager.go:414] "containerMap: removing container" podUID="48e4c19c-17ee-467c-a7b9-652ce87156af" containerName="extract"
Nov 01 03:10:59.934573 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:59.934461    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48e4c19c-17ee-467c-a7b9-652ce87156af" containerName="extract"
Nov 01 03:10:59.934573 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:59.934472    4400 cpu_manager.go:414] "containerMap: removing container" podUID="48e4c19c-17ee-467c-a7b9-652ce87156af" containerName="util"
Nov 01 03:10:59.934573 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:59.934477    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48e4c19c-17ee-467c-a7b9-652ce87156af" containerName="util"
Nov 01 03:10:59.934573 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:59.934483    4400 cpu_manager.go:414] "containerMap: removing container" podUID="795852c6-f698-4c60-bbf3-4751f83eccbf" containerName="extract"
Nov 01 03:10:59.934573 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:59.934487    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="795852c6-f698-4c60-bbf3-4751f83eccbf" containerName="extract"
Nov 01 03:10:59.934573 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:59.934497    4400 cpu_manager.go:414] "containerMap: removing container" podUID="795852c6-f698-4c60-bbf3-4751f83eccbf" containerName="pull"
Nov 01 03:10:59.934573 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:59.934501    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="795852c6-f698-4c60-bbf3-4751f83eccbf" containerName="pull"
Nov 01 03:10:59.934573 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:59.934554    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="48e4c19c-17ee-467c-a7b9-652ce87156af" containerName="extract"
Nov 01 03:10:59.934573 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:59.934562    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="795852c6-f698-4c60-bbf3-4751f83eccbf" containerName="extract"
Nov 01 03:10:59.947656 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:10:59.947640    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-576db5b58f-s552l"
Nov 01 03:11:00.007731 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:11:00.007712    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48e4c19c-17ee-467c-a7b9-652ce87156af-util\" (UniqueName: \"kubernetes.io/empty-dir/48e4c19c-17ee-467c-a7b9-652ce87156af-util\") pod \"48e4c19c-17ee-467c-a7b9-652ce87156af\" (UID: \"48e4c19c-17ee-467c-a7b9-652ce87156af\") "
Nov 01 03:11:00.007838 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:11:00.007739    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48e4c19c-17ee-467c-a7b9-652ce87156af-bundle\" (UniqueName: \"kubernetes.io/empty-dir/48e4c19c-17ee-467c-a7b9-652ce87156af-bundle\") pod \"48e4c19c-17ee-467c-a7b9-652ce87156af\" (UID: \"48e4c19c-17ee-467c-a7b9-652ce87156af\") "
Nov 01 03:11:00.007838 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:11:00.007784    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48e4c19c-17ee-467c-a7b9-652ce87156af-kube-api-access-6t2pq\" (UniqueName: \"kubernetes.io/projected/48e4c19c-17ee-467c-a7b9-652ce87156af-kube-api-access-6t2pq\") pod \"48e4c19c-17ee-467c-a7b9-652ce87156af\" (UID: \"48e4c19c-17ee-467c-a7b9-652ce87156af\") "
Nov 01 03:11:00.008337 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:11:00.008313    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48e4c19c-17ee-467c-a7b9-652ce87156af-bundle" pod "48e4c19c-17ee-467c-a7b9-652ce87156af" (UID: "48e4c19c-17ee-467c-a7b9-652ce87156af"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:11:00.009877 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:11:00.009846    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48e4c19c-17ee-467c-a7b9-652ce87156af-kube-api-access-6t2pq" pod "48e4c19c-17ee-467c-a7b9-652ce87156af" (UID: "48e4c19c-17ee-467c-a7b9-652ce87156af"). InnerVolumeSpecName "kube-api-access-6t2pq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:11:00.013441 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:11:00.013412    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48e4c19c-17ee-467c-a7b9-652ce87156af-util" pod "48e4c19c-17ee-467c-a7b9-652ce87156af" (UID: "48e4c19c-17ee-467c-a7b9-652ce87156af"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:11:00.038758 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:11:00.038735    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 01 03:11:00.040406 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:11:00.040391    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 01 03:11:00.109138 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:11:00.109116    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bd228c73-be4c-442b-9c30-3d11b05db901-tmp\") pod \"opendatahub-operator-controller-manager-576db5b58f-s552l\" (UID: \"bd228c73-be4c-442b-9c30-3d11b05db901\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-576db5b58f-s552l"
Nov 01 03:11:00.109247 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:11:00.109158    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zqfr\" (UniqueName: \"kubernetes.io/projected/bd228c73-be4c-442b-9c30-3d11b05db901-kube-api-access-2zqfr\") pod \"opendatahub-operator-controller-manager-576db5b58f-s552l\" (UID: \"bd228c73-be4c-442b-9c30-3d11b05db901\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-576db5b58f-s552l"
Nov 01 03:11:00.109247 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:11:00.109183    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bd228c73-be4c-442b-9c30-3d11b05db901-webhook-cert\") pod \"opendatahub-operator-controller-manager-576db5b58f-s552l\" (UID: \"bd228c73-be4c-442b-9c30-3d11b05db901\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-576db5b58f-s552l"
Nov 01 03:11:00.109342 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:11:00.109308    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/bd228c73-be4c-442b-9c30-3d11b05db901-manifests\") pod \"opendatahub-operator-controller-manager-576db5b58f-s552l\" (UID: \"bd228c73-be4c-442b-9c30-3d11b05db901\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-576db5b58f-s552l"
Nov 01 03:11:00.109389 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:11:00.109342    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/bd228c73-be4c-442b-9c30-3d11b05db901-metrics-cert\") pod \"opendatahub-operator-controller-manager-576db5b58f-s552l\" (UID: \"bd228c73-be4c-442b-9c30-3d11b05db901\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-576db5b58f-s552l"
Nov 01 03:11:00.109389 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:11:00.109362    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/bd228c73-be4c-442b-9c30-3d11b05db901-apiservice-cert\") pod \"opendatahub-operator-controller-manager-576db5b58f-s552l\" (UID: \"bd228c73-be4c-442b-9c30-3d11b05db901\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-576db5b58f-s552l"
Nov 01 03:11:00.109465 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:11:00.109421    4400 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/48e4c19c-17ee-467c-a7b9-652ce87156af-util\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:11:00.109465 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:11:00.109432    4400 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/48e4c19c-17ee-467c-a7b9-652ce87156af-bundle\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:11:00.109465 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:11:00.109441    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6t2pq\" (UniqueName: \"kubernetes.io/projected/48e4c19c-17ee-467c-a7b9-652ce87156af-kube-api-access-6t2pq\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:11:00.132927 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:11:00.132872    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 01 03:11:00.133435 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:11:00.133417    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-445ws\""
Nov 01 03:11:00.133483 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:11:00.133455    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 01 03:11:00.133519 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:11:00.133503    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 01 03:11:00.186056 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:11:00.186026    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-576db5b58f-s552l"]
Nov 01 03:11:00.210341 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:11:00.210319    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/bd228c73-be4c-442b-9c30-3d11b05db901-manifests\") pod \"opendatahub-operator-controller-manager-576db5b58f-s552l\" (UID: \"bd228c73-be4c-442b-9c30-3d11b05db901\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-576db5b58f-s552l"
Nov 01 03:11:00.210414 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:11:00.210352    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/bd228c73-be4c-442b-9c30-3d11b05db901-metrics-cert\") pod \"opendatahub-operator-controller-manager-576db5b58f-s552l\" (UID: \"bd228c73-be4c-442b-9c30-3d11b05db901\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-576db5b58f-s552l"
Nov 01 03:11:00.210414 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:11:00.210370    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/bd228c73-be4c-442b-9c30-3d11b05db901-apiservice-cert\") pod \"opendatahub-operator-controller-manager-576db5b58f-s552l\" (UID: \"bd228c73-be4c-442b-9c30-3d11b05db901\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-576db5b58f-s552l"
Nov 01 03:11:00.210414 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:11:00.210388    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bd228c73-be4c-442b-9c30-3d11b05db901-tmp\") pod \"opendatahub-operator-controller-manager-576db5b58f-s552l\" (UID: \"bd228c73-be4c-442b-9c30-3d11b05db901\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-576db5b58f-s552l"
Nov 01 03:11:00.210576 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:11:00.210418    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zqfr\" (UniqueName: \"kubernetes.io/projected/bd228c73-be4c-442b-9c30-3d11b05db901-kube-api-access-2zqfr\") pod \"opendatahub-operator-controller-manager-576db5b58f-s552l\" (UID: \"bd228c73-be4c-442b-9c30-3d11b05db901\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-576db5b58f-s552l"
Nov 01 03:11:00.210576 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:11:00.210476    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bd228c73-be4c-442b-9c30-3d11b05db901-webhook-cert\") pod \"opendatahub-operator-controller-manager-576db5b58f-s552l\" (UID: \"bd228c73-be4c-442b-9c30-3d11b05db901\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-576db5b58f-s552l"
Nov 01 03:11:00.210743 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:11:00.210722    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/bd228c73-be4c-442b-9c30-3d11b05db901-manifests\") pod \"opendatahub-operator-controller-manager-576db5b58f-s552l\" (UID: \"bd228c73-be4c-442b-9c30-3d11b05db901\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-576db5b58f-s552l"
Nov 01 03:11:00.212834 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:11:00.212808    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bd228c73-be4c-442b-9c30-3d11b05db901-tmp\") pod \"opendatahub-operator-controller-manager-576db5b58f-s552l\" (UID: \"bd228c73-be4c-442b-9c30-3d11b05db901\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-576db5b58f-s552l"
Nov 01 03:11:00.213070 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:11:00.213049    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/bd228c73-be4c-442b-9c30-3d11b05db901-metrics-cert\") pod \"opendatahub-operator-controller-manager-576db5b58f-s552l\" (UID: \"bd228c73-be4c-442b-9c30-3d11b05db901\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-576db5b58f-s552l"
Nov 01 03:11:00.213133 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:11:00.213059    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bd228c73-be4c-442b-9c30-3d11b05db901-webhook-cert\") pod \"opendatahub-operator-controller-manager-576db5b58f-s552l\" (UID: \"bd228c73-be4c-442b-9c30-3d11b05db901\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-576db5b58f-s552l"
Nov 01 03:11:00.213189 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:11:00.213133    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/bd228c73-be4c-442b-9c30-3d11b05db901-apiservice-cert\") pod \"opendatahub-operator-controller-manager-576db5b58f-s552l\" (UID: \"bd228c73-be4c-442b-9c30-3d11b05db901\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-576db5b58f-s552l"
Nov 01 03:11:00.320132 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:11:00.320104    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zqfr\" (UniqueName: \"kubernetes.io/projected/bd228c73-be4c-442b-9c30-3d11b05db901-kube-api-access-2zqfr\") pod \"opendatahub-operator-controller-manager-576db5b58f-s552l\" (UID: \"bd228c73-be4c-442b-9c30-3d11b05db901\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-576db5b58f-s552l"
Nov 01 03:11:00.592374 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:11:00.592341    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-576db5b58f-s552l"
Nov 01 03:11:00.769165 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:11:00.769142    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crld2r"
Nov 01 03:11:00.769488 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:11:00.769145    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crld2r" event={"ID":"48e4c19c-17ee-467c-a7b9-652ce87156af","Type":"ContainerDied","Data":"fa09dbec918564e81f55f689e5ccfb0fb36a5c3a7068093c0fbf27c7ebae5908"}
Nov 01 03:11:00.769488 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:11:00.769256    4400 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa09dbec918564e81f55f689e5ccfb0fb36a5c3a7068093c0fbf27c7ebae5908"
Nov 01 03:11:01.168215 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:11:01.168188    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-576db5b58f-s552l"]
Nov 01 03:11:01.170065 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:11:01.170036    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd228c73_be4c_442b_9c30_3d11b05db901.slice/crio-dc7bd27836b15f6681e0f7c7d47b2a80be37bfe5eeacdf2ceaf68708ad940db8 WatchSource:0}: Error finding container dc7bd27836b15f6681e0f7c7d47b2a80be37bfe5eeacdf2ceaf68708ad940db8: Status 404 returned error can't find the container with id dc7bd27836b15f6681e0f7c7d47b2a80be37bfe5eeacdf2ceaf68708ad940db8
Nov 01 03:11:01.728827 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:11:01.728789    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:11:01.775083 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:11:01.775053    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-576db5b58f-s552l" event={"ID":"bd228c73-be4c-442b-9c30-3d11b05db901","Type":"ContainerStarted","Data":"dc7bd27836b15f6681e0f7c7d47b2a80be37bfe5eeacdf2ceaf68708ad940db8"}
Nov 01 03:11:04.789838 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:11:04.789807    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-576db5b58f-s552l" event={"ID":"bd228c73-be4c-442b-9c30-3d11b05db901","Type":"ContainerStarted","Data":"dbc8704b7dd0ea7156337ab418e399263fcff873682f6e64078edc794c2d0b3f"}
Nov 01 03:11:05.795365 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:11:05.795330    4400 generic.go:356] "Generic (PLEG): container finished" podID="bd228c73-be4c-442b-9c30-3d11b05db901" containerID="dbc8704b7dd0ea7156337ab418e399263fcff873682f6e64078edc794c2d0b3f" exitCode=0
Nov 01 03:11:05.795873 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:11:05.795417    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-576db5b58f-s552l" event={"ID":"bd228c73-be4c-442b-9c30-3d11b05db901","Type":"ContainerDied","Data":"dbc8704b7dd0ea7156337ab418e399263fcff873682f6e64078edc794c2d0b3f"}
Nov 01 03:11:06.801924 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:11:06.801887    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-576db5b58f-s552l" event={"ID":"bd228c73-be4c-442b-9c30-3d11b05db901","Type":"ContainerStarted","Data":"dafbc3836764406bca9cff2d793c6297a155cb9bc3be3a7438b61cf20755781c"}
Nov 01 03:11:06.802368 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:11:06.801977    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-576db5b58f-s552l"
Nov 01 03:11:06.956914 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:11:06.956818    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-576db5b58f-s552l" podStartSLOduration=3.294325567 podStartE2EDuration="7.956800542s" totalImagesPullingTime="4.209059132s" totalInitContainerRuntime="453.415843ms" isStatefulPod=false podCreationTimestamp="2026-11-01 03:10:59 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:11:06.952788631 +0000 UTC m=+273.467434486" watchObservedRunningTime="2026-11-01 03:11:06.956800542 +0000 UTC m=+273.471446409"
Nov 01 03:11:17.808848 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:11:17.808820    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-576db5b58f-s552l"
Nov 01 03:11:31.728538 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:11:31.728503    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:11:47.796344 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:11:47.796273    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7d5d9754f-zcj7p"]
Nov 01 03:11:47.798383 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:11:47.798366    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7d5d9754f-zcj7p"
Nov 01 03:11:47.885349 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:11:47.885324    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7d5d9754f-zcj7p"]
Nov 01 03:11:47.887782 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:11:47.887757    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/121dbd3a-6f50-4e38-8e72-85297471ca08-tls-certs\") pod \"kube-auth-proxy-7d5d9754f-zcj7p\" (UID: \"121dbd3a-6f50-4e38-8e72-85297471ca08\") " pod="openshift-ingress/kube-auth-proxy-7d5d9754f-zcj7p"
Nov 01 03:11:47.887925 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:11:47.887796    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/121dbd3a-6f50-4e38-8e72-85297471ca08-tmp\") pod \"kube-auth-proxy-7d5d9754f-zcj7p\" (UID: \"121dbd3a-6f50-4e38-8e72-85297471ca08\") " pod="openshift-ingress/kube-auth-proxy-7d5d9754f-zcj7p"
Nov 01 03:11:47.887925 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:11:47.887813    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfgwj\" (UniqueName: \"kubernetes.io/projected/121dbd3a-6f50-4e38-8e72-85297471ca08-kube-api-access-mfgwj\") pod \"kube-auth-proxy-7d5d9754f-zcj7p\" (UID: \"121dbd3a-6f50-4e38-8e72-85297471ca08\") " pod="openshift-ingress/kube-auth-proxy-7d5d9754f-zcj7p"
Nov 01 03:11:47.943192 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:11:47.943171    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Nov 01 03:11:47.943303 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:11:47.943249    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-b66cn\""
Nov 01 03:11:47.969763 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:11:47.969743    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Nov 01 03:11:47.989113 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:11:47.989088    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/121dbd3a-6f50-4e38-8e72-85297471ca08-tls-certs\") pod \"kube-auth-proxy-7d5d9754f-zcj7p\" (UID: \"121dbd3a-6f50-4e38-8e72-85297471ca08\") " pod="openshift-ingress/kube-auth-proxy-7d5d9754f-zcj7p"
Nov 01 03:11:47.989223 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:11:47.989124    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/121dbd3a-6f50-4e38-8e72-85297471ca08-tmp\") pod \"kube-auth-proxy-7d5d9754f-zcj7p\" (UID: \"121dbd3a-6f50-4e38-8e72-85297471ca08\") " pod="openshift-ingress/kube-auth-proxy-7d5d9754f-zcj7p"
Nov 01 03:11:47.989223 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:11:47.989141    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfgwj\" (UniqueName: \"kubernetes.io/projected/121dbd3a-6f50-4e38-8e72-85297471ca08-kube-api-access-mfgwj\") pod \"kube-auth-proxy-7d5d9754f-zcj7p\" (UID: \"121dbd3a-6f50-4e38-8e72-85297471ca08\") " pod="openshift-ingress/kube-auth-proxy-7d5d9754f-zcj7p"
Nov 01 03:11:47.991447 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:11:47.991424    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/121dbd3a-6f50-4e38-8e72-85297471ca08-tmp\") pod \"kube-auth-proxy-7d5d9754f-zcj7p\" (UID: \"121dbd3a-6f50-4e38-8e72-85297471ca08\") " pod="openshift-ingress/kube-auth-proxy-7d5d9754f-zcj7p"
Nov 01 03:11:47.991699 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:11:47.991662    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/121dbd3a-6f50-4e38-8e72-85297471ca08-tls-certs\") pod \"kube-auth-proxy-7d5d9754f-zcj7p\" (UID: \"121dbd3a-6f50-4e38-8e72-85297471ca08\") " pod="openshift-ingress/kube-auth-proxy-7d5d9754f-zcj7p"
Nov 01 03:11:48.053367 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:11:48.053307    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfgwj\" (UniqueName: \"kubernetes.io/projected/121dbd3a-6f50-4e38-8e72-85297471ca08-kube-api-access-mfgwj\") pod \"kube-auth-proxy-7d5d9754f-zcj7p\" (UID: \"121dbd3a-6f50-4e38-8e72-85297471ca08\") " pod="openshift-ingress/kube-auth-proxy-7d5d9754f-zcj7p"
Nov 01 03:11:48.108932 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:11:48.108907    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7d5d9754f-zcj7p"
Nov 01 03:11:48.486927 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:11:48.486902    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7d5d9754f-zcj7p"]
Nov 01 03:11:48.497129 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:11:48.497111    4400 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 03:11:48.976314 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:11:48.976273    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7d5d9754f-zcj7p" event={"ID":"121dbd3a-6f50-4e38-8e72-85297471ca08","Type":"ContainerStarted","Data":"699782c04d68fce6a6a9ef2e0411d9a07906fff882ad2396e0cb1ec08b0c5551"}
Nov 01 03:11:51.992181 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:11:51.992150    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7d5d9754f-zcj7p" event={"ID":"121dbd3a-6f50-4e38-8e72-85297471ca08","Type":"ContainerStarted","Data":"5eb221495b51833a5f39b081b5df58c233f1f2cd4d08462683f4236e94007529"}
Nov 01 03:11:52.111701 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:11:52.111629    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7d5d9754f-zcj7p" podStartSLOduration=2.307658684 podStartE2EDuration="5.111614799s" totalImagesPullingTime="2.803956115s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:11:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:11:52.111097923 +0000 UTC m=+318.625743789" watchObservedRunningTime="2026-11-01 03:11:52.111614799 +0000 UTC m=+318.626260666"
Nov 01 03:12:01.727849 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:12:01.727808    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:12:02.381982 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:12:02.381946    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-s2rkp"]
Nov 01 03:12:02.385000 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:12:02.384978    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-s2rkp"
Nov 01 03:12:02.392954 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:12:02.392936    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 03:12:02.395941 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:12:02.395925    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"
Nov 01 03:12:02.411769 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:12:02.411745    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5mrh\" (UniqueName: \"kubernetes.io/projected/7fbd32d3-3ead-42ac-8bae-b3a8c7d18612-kube-api-access-n5mrh\") pod \"authorino-56c6c9ddd5-s2rkp\" (UID: \"7fbd32d3-3ead-42ac-8bae-b3a8c7d18612\") " pod="kuadrant-system/authorino-56c6c9ddd5-s2rkp"
Nov 01 03:12:02.444699 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:12:02.444660    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 01 03:12:02.445262 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:12:02.445244    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-s2rkp"]
Nov 01 03:12:02.495137 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:12:02.495119    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 01 03:12:02.495703 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:12:02.495670    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-47cjs\""
Nov 01 03:12:02.497375 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:12:02.497351    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 03:12:02.512483 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:12:02.512460    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/63495340-52fe-4035-825d-50250e797b66-config-file\") pod \"limitador-limitador-7c9b87d574-mpdqs\" (UID: \"63495340-52fe-4035-825d-50250e797b66\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"
Nov 01 03:12:02.512595 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:12:02.512514    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5mrh\" (UniqueName: \"kubernetes.io/projected/7fbd32d3-3ead-42ac-8bae-b3a8c7d18612-kube-api-access-n5mrh\") pod \"authorino-56c6c9ddd5-s2rkp\" (UID: \"7fbd32d3-3ead-42ac-8bae-b3a8c7d18612\") " pod="kuadrant-system/authorino-56c6c9ddd5-s2rkp"
Nov 01 03:12:02.512595 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:12:02.512560    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p2bz\" (UniqueName: \"kubernetes.io/projected/63495340-52fe-4035-825d-50250e797b66-kube-api-access-5p2bz\") pod \"limitador-limitador-7c9b87d574-mpdqs\" (UID: \"63495340-52fe-4035-825d-50250e797b66\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"
Nov 01 03:12:02.546275 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:12:02.546246    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Nov 01 03:12:02.546390 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:12:02.546293    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-sr5bv\""
Nov 01 03:12:02.577847 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:12:02.577821    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5mrh\" (UniqueName: \"kubernetes.io/projected/7fbd32d3-3ead-42ac-8bae-b3a8c7d18612-kube-api-access-n5mrh\") pod \"authorino-56c6c9ddd5-s2rkp\" (UID: \"7fbd32d3-3ead-42ac-8bae-b3a8c7d18612\") " pod="kuadrant-system/authorino-56c6c9ddd5-s2rkp"
Nov 01 03:12:02.613485 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:12:02.613460    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p2bz\" (UniqueName: \"kubernetes.io/projected/63495340-52fe-4035-825d-50250e797b66-kube-api-access-5p2bz\") pod \"limitador-limitador-7c9b87d574-mpdqs\" (UID: \"63495340-52fe-4035-825d-50250e797b66\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"
Nov 01 03:12:02.613575 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:12:02.613536    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/63495340-52fe-4035-825d-50250e797b66-config-file\") pod \"limitador-limitador-7c9b87d574-mpdqs\" (UID: \"63495340-52fe-4035-825d-50250e797b66\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"
Nov 01 03:12:02.614134 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:12:02.614116    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/63495340-52fe-4035-825d-50250e797b66-config-file\") pod \"limitador-limitador-7c9b87d574-mpdqs\" (UID: \"63495340-52fe-4035-825d-50250e797b66\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"
Nov 01 03:12:02.675917 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:12:02.675865    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p2bz\" (UniqueName: \"kubernetes.io/projected/63495340-52fe-4035-825d-50250e797b66-kube-api-access-5p2bz\") pod \"limitador-limitador-7c9b87d574-mpdqs\" (UID: \"63495340-52fe-4035-825d-50250e797b66\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"
Nov 01 03:12:02.691241 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:12:02.691219    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 03:12:02.709145 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:12:02.709127    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-s2rkp"
Nov 01 03:12:02.717951 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:12:02.717926    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"
Nov 01 03:12:03.085606 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:12:03.085581    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-s2rkp"]
Nov 01 03:12:03.087401 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:12:03.087376    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7fbd32d3_3ead_42ac_8bae_b3a8c7d18612.slice/crio-06fff93c8e2a823a0b9a7dcf86a77aa29570a8a0b010434aa149e4a2877906c5 WatchSource:0}: Error finding container 06fff93c8e2a823a0b9a7dcf86a77aa29570a8a0b010434aa149e4a2877906c5: Status 404 returned error can't find the container with id 06fff93c8e2a823a0b9a7dcf86a77aa29570a8a0b010434aa149e4a2877906c5
Nov 01 03:12:03.136398 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:12:03.136376    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 03:12:03.137501 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:12:03.137479    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63495340_52fe_4035_825d_50250e797b66.slice/crio-4188c1867550a95c9a379c5eb449499b7ec4a36a4a8d5680b040570e27e7195e WatchSource:0}: Error finding container 4188c1867550a95c9a379c5eb449499b7ec4a36a4a8d5680b040570e27e7195e: Status 404 returned error can't find the container with id 4188c1867550a95c9a379c5eb449499b7ec4a36a4a8d5680b040570e27e7195e
Nov 01 03:12:04.044383 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:12:04.044343    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-mpdqs" event={"ID":"63495340-52fe-4035-825d-50250e797b66","Type":"ContainerStarted","Data":"4188c1867550a95c9a379c5eb449499b7ec4a36a4a8d5680b040570e27e7195e"}
Nov 01 03:12:04.045435 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:12:04.045407    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-s2rkp" event={"ID":"7fbd32d3-3ead-42ac-8bae-b3a8c7d18612","Type":"ContainerStarted","Data":"06fff93c8e2a823a0b9a7dcf86a77aa29570a8a0b010434aa149e4a2877906c5"}
Nov 01 03:12:07.064365 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:12:07.064262    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-mpdqs" event={"ID":"63495340-52fe-4035-825d-50250e797b66","Type":"ContainerStarted","Data":"2cd53361d8362893ce7c5ff7cf641591a94356ba98b874c561ce7327f1a17618"}
Nov 01 03:12:07.064880 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:12:07.064381    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"
Nov 01 03:12:07.065906 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:12:07.065880    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-s2rkp" event={"ID":"7fbd32d3-3ead-42ac-8bae-b3a8c7d18612","Type":"ContainerStarted","Data":"92ac2fa51007d34157796928f9b92b64d370969d40c9962fe6551085b013ca5c"}
Nov 01 03:12:07.199195 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:12:07.199143    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-mpdqs" podStartSLOduration=1.981768202 podStartE2EDuration="5.199127889s" totalImagesPullingTime="3.217359687s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:12:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:12:07.198213921 +0000 UTC m=+333.712859785" watchObservedRunningTime="2026-11-01 03:12:07.199127889 +0000 UTC m=+333.713773797"
Nov 01 03:12:07.320820 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:12:07.320764    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-s2rkp" podStartSLOduration=1.756892992 podStartE2EDuration="5.320749488s" totalImagesPullingTime="3.563856496s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:12:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:12:07.320042061 +0000 UTC m=+333.834687923" watchObservedRunningTime="2026-11-01 03:12:07.320749488 +0000 UTC m=+333.835395352"
Nov 01 03:12:14.936728 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:12:14.936696    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-s65d6"]
Nov 01 03:12:14.940060 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:12:14.940044    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-s65d6"
Nov 01 03:12:14.993910 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:12:14.993886    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 01 03:12:15.012317 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:12:15.012297    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6e617fe1-f094-4315-8c52-0a4d6f7fc809-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-s65d6\" (UID: \"6e617fe1-f094-4315-8c52-0a4d6f7fc809\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-s65d6"
Nov 01 03:12:15.012417 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:12:15.012351    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6e617fe1-f094-4315-8c52-0a4d6f7fc809-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-s65d6\" (UID: \"6e617fe1-f094-4315-8c52-0a4d6f7fc809\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-s65d6"
Nov 01 03:12:15.012503 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:12:15.012426    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6e617fe1-f094-4315-8c52-0a4d6f7fc809-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-s65d6\" (UID: \"6e617fe1-f094-4315-8c52-0a4d6f7fc809\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-s65d6"
Nov 01 03:12:15.012503 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:12:15.012467    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gx8d7\" (UniqueName: \"kubernetes.io/projected/6e617fe1-f094-4315-8c52-0a4d6f7fc809-kube-api-access-gx8d7\") pod \"kserve-module-controller-manager-7566cf5fc8-s65d6\" (UID: \"6e617fe1-f094-4315-8c52-0a4d6f7fc809\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-s65d6"
Nov 01 03:12:15.044771 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:12:15.044737    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 01 03:12:15.044771 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:12:15.044763    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-56pzz\""
Nov 01 03:12:15.113021 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:12:15.113001    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6e617fe1-f094-4315-8c52-0a4d6f7fc809-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-s65d6\" (UID: \"6e617fe1-f094-4315-8c52-0a4d6f7fc809\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-s65d6"
Nov 01 03:12:15.113149 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:12:15.113055    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6e617fe1-f094-4315-8c52-0a4d6f7fc809-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-s65d6\" (UID: \"6e617fe1-f094-4315-8c52-0a4d6f7fc809\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-s65d6"
Nov 01 03:12:15.113149 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:12:15.113139    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6e617fe1-f094-4315-8c52-0a4d6f7fc809-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-s65d6\" (UID: \"6e617fe1-f094-4315-8c52-0a4d6f7fc809\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-s65d6"
Nov 01 03:12:15.113270 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:12:15.113172    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gx8d7\" (UniqueName: \"kubernetes.io/projected/6e617fe1-f094-4315-8c52-0a4d6f7fc809-kube-api-access-gx8d7\") pod \"kserve-module-controller-manager-7566cf5fc8-s65d6\" (UID: \"6e617fe1-f094-4315-8c52-0a4d6f7fc809\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-s65d6"
Nov 01 03:12:15.113432 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:12:15.113411    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6e617fe1-f094-4315-8c52-0a4d6f7fc809-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-s65d6\" (UID: \"6e617fe1-f094-4315-8c52-0a4d6f7fc809\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-s65d6"
Nov 01 03:12:15.113432 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:12:15.113426    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6e617fe1-f094-4315-8c52-0a4d6f7fc809-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-s65d6\" (UID: \"6e617fe1-f094-4315-8c52-0a4d6f7fc809\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-s65d6"
Nov 01 03:12:15.147730 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:12:15.147677    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\""
Nov 01 03:12:15.149488 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:12:15.149466    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-s65d6"]
Nov 01 03:12:15.154667 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:12:15.154647    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6e617fe1-f094-4315-8c52-0a4d6f7fc809-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-s65d6\" (UID: \"6e617fe1-f094-4315-8c52-0a4d6f7fc809\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-s65d6"
Nov 01 03:12:15.175776 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:12:15.175757    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gx8d7\" (UniqueName: \"kubernetes.io/projected/6e617fe1-f094-4315-8c52-0a4d6f7fc809-kube-api-access-gx8d7\") pod \"kserve-module-controller-manager-7566cf5fc8-s65d6\" (UID: \"6e617fe1-f094-4315-8c52-0a4d6f7fc809\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-s65d6"
Nov 01 03:12:15.252809 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:12:15.252764    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-s65d6"
Nov 01 03:12:15.674274 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:12:15.674251    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-s65d6"]
Nov 01 03:12:15.675808 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:12:15.675782    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e617fe1_f094_4315_8c52_0a4d6f7fc809.slice/crio-f6fc42414b2cc4ba451a05e60f585de466b058105b00f7246d4681365aaf7538 WatchSource:0}: Error finding container f6fc42414b2cc4ba451a05e60f585de466b058105b00f7246d4681365aaf7538: Status 404 returned error can't find the container with id f6fc42414b2cc4ba451a05e60f585de466b058105b00f7246d4681365aaf7538
Nov 01 03:12:16.105963 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:12:16.105920    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-s65d6" event={"ID":"6e617fe1-f094-4315-8c52-0a4d6f7fc809","Type":"ContainerStarted","Data":"f6fc42414b2cc4ba451a05e60f585de466b058105b00f7246d4681365aaf7538"}
Nov 01 03:12:18.073188 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:12:18.073163    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"
Nov 01 03:12:18.116093 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:12:18.116067    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-s65d6" event={"ID":"6e617fe1-f094-4315-8c52-0a4d6f7fc809","Type":"ContainerStarted","Data":"a4fbcd88e90c1d304124d77fdcef6c2c0661fb2569a4e90ebba4cb1fd516ca22"}
Nov 01 03:12:18.116224 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:12:18.116195    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-s65d6"
Nov 01 03:12:18.315110 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:12:18.315049    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-s65d6" podStartSLOduration=2.543930157 podStartE2EDuration="4.315034864s" totalImagesPullingTime="1.771104707s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:12:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:12:18.313126354 +0000 UTC m=+344.827772217" watchObservedRunningTime="2026-11-01 03:12:18.315034864 +0000 UTC m=+344.829680761"
Nov 01 03:12:31.728092 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:12:31.728052    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:12:39.122408 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:12:39.122379    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-s65d6"
Nov 01 03:13:01.728867 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:13:01.728831    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:13:15.940010 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:15.939929    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lw9fr"]
Nov 01 03:13:15.943387 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:15.943371    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lw9fr"
Nov 01 03:13:16.027707 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:16.025021    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tp65\" (UniqueName: \"kubernetes.io/projected/0fca1311-9bbd-4b5d-a996-2f67742bedb3-kube-api-access-2tp65\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lw9fr\" (UID: \"0fca1311-9bbd-4b5d-a996-2f67742bedb3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lw9fr"
Nov 01 03:13:16.027707 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:16.025143    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0fca1311-9bbd-4b5d-a996-2f67742bedb3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lw9fr\" (UID: \"0fca1311-9bbd-4b5d-a996-2f67742bedb3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lw9fr"
Nov 01 03:13:16.027707 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:16.025183    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0fca1311-9bbd-4b5d-a996-2f67742bedb3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lw9fr\" (UID: \"0fca1311-9bbd-4b5d-a996-2f67742bedb3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lw9fr"
Nov 01 03:13:16.027707 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:16.025310    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fca1311-9bbd-4b5d-a996-2f67742bedb3-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lw9fr\" (UID: \"0fca1311-9bbd-4b5d-a996-2f67742bedb3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lw9fr"
Nov 01 03:13:16.027707 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:16.025349    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0fca1311-9bbd-4b5d-a996-2f67742bedb3-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lw9fr\" (UID: \"0fca1311-9bbd-4b5d-a996-2f67742bedb3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lw9fr"
Nov 01 03:13:16.027707 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:16.025457    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fca1311-9bbd-4b5d-a996-2f67742bedb3-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lw9fr\" (UID: \"0fca1311-9bbd-4b5d-a996-2f67742bedb3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lw9fr"
Nov 01 03:13:16.027707 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:16.025499    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0fca1311-9bbd-4b5d-a996-2f67742bedb3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lw9fr\" (UID: \"0fca1311-9bbd-4b5d-a996-2f67742bedb3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lw9fr"
Nov 01 03:13:16.056004 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:16.055974    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lw9fr"]
Nov 01 03:13:16.126405 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:16.126371    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fca1311-9bbd-4b5d-a996-2f67742bedb3-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lw9fr\" (UID: \"0fca1311-9bbd-4b5d-a996-2f67742bedb3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lw9fr"
Nov 01 03:13:16.126570 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:16.126414    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0fca1311-9bbd-4b5d-a996-2f67742bedb3-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lw9fr\" (UID: \"0fca1311-9bbd-4b5d-a996-2f67742bedb3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lw9fr"
Nov 01 03:13:16.126570 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:16.126443    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fca1311-9bbd-4b5d-a996-2f67742bedb3-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lw9fr\" (UID: \"0fca1311-9bbd-4b5d-a996-2f67742bedb3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lw9fr"
Nov 01 03:13:16.126570 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:16.126474    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0fca1311-9bbd-4b5d-a996-2f67742bedb3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lw9fr\" (UID: \"0fca1311-9bbd-4b5d-a996-2f67742bedb3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lw9fr"
Nov 01 03:13:16.126570 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:16.126535    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tp65\" (UniqueName: \"kubernetes.io/projected/0fca1311-9bbd-4b5d-a996-2f67742bedb3-kube-api-access-2tp65\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lw9fr\" (UID: \"0fca1311-9bbd-4b5d-a996-2f67742bedb3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lw9fr"
Nov 01 03:13:16.126765 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:16.126575    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0fca1311-9bbd-4b5d-a996-2f67742bedb3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lw9fr\" (UID: \"0fca1311-9bbd-4b5d-a996-2f67742bedb3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lw9fr"
Nov 01 03:13:16.126765 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:16.126597    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0fca1311-9bbd-4b5d-a996-2f67742bedb3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lw9fr\" (UID: \"0fca1311-9bbd-4b5d-a996-2f67742bedb3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lw9fr"
Nov 01 03:13:16.127381 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:16.127291    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0fca1311-9bbd-4b5d-a996-2f67742bedb3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lw9fr\" (UID: \"0fca1311-9bbd-4b5d-a996-2f67742bedb3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lw9fr"
Nov 01 03:13:16.129650 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:16.129620    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fca1311-9bbd-4b5d-a996-2f67742bedb3-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lw9fr\" (UID: \"0fca1311-9bbd-4b5d-a996-2f67742bedb3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lw9fr"
Nov 01 03:13:16.129786 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:16.129746    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0fca1311-9bbd-4b5d-a996-2f67742bedb3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lw9fr\" (UID: \"0fca1311-9bbd-4b5d-a996-2f67742bedb3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lw9fr"
Nov 01 03:13:16.129786 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:16.129780    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0fca1311-9bbd-4b5d-a996-2f67742bedb3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lw9fr\" (UID: \"0fca1311-9bbd-4b5d-a996-2f67742bedb3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lw9fr"
Nov 01 03:13:16.129913 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:16.129860    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0fca1311-9bbd-4b5d-a996-2f67742bedb3-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lw9fr\" (UID: \"0fca1311-9bbd-4b5d-a996-2f67742bedb3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lw9fr"
Nov 01 03:13:16.189040 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:16.189021    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fca1311-9bbd-4b5d-a996-2f67742bedb3-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lw9fr\" (UID: \"0fca1311-9bbd-4b5d-a996-2f67742bedb3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lw9fr"
Nov 01 03:13:16.189274 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:16.189257    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tp65\" (UniqueName: \"kubernetes.io/projected/0fca1311-9bbd-4b5d-a996-2f67742bedb3-kube-api-access-2tp65\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lw9fr\" (UID: \"0fca1311-9bbd-4b5d-a996-2f67742bedb3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lw9fr"
Nov 01 03:13:16.258010 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:16.257959    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lw9fr"
Nov 01 03:13:16.684479 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:16.684449    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lw9fr"]
Nov 01 03:13:16.685874 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:13:16.685846    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fca1311_9bbd_4b5d_a996_2f67742bedb3.slice/crio-ef1b74fa5836bbe855fe2440b3c290e786dbd8423808eac2fb10289ce6475b79 WatchSource:0}: Error finding container ef1b74fa5836bbe855fe2440b3c290e786dbd8423808eac2fb10289ce6475b79: Status 404 returned error can't find the container with id ef1b74fa5836bbe855fe2440b3c290e786dbd8423808eac2fb10289ce6475b79
Nov 01 03:13:16.687967 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:16.687933    4400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 03:13:16.688063 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:16.687994    4400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 03:13:17.395413 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:17.395375    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lw9fr" event={"ID":"0fca1311-9bbd-4b5d-a996-2f67742bedb3","Type":"ContainerStarted","Data":"9a384ac305809728949cd5cddd8636113c6e4162529bd7a2d8a157cd623e761d"}
Nov 01 03:13:17.395817 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:17.395435    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lw9fr" event={"ID":"0fca1311-9bbd-4b5d-a996-2f67742bedb3","Type":"ContainerStarted","Data":"ef1b74fa5836bbe855fe2440b3c290e786dbd8423808eac2fb10289ce6475b79"}
Nov 01 03:13:17.395817 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:17.395462    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lw9fr"
Nov 01 03:13:17.397316 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:17.397285    4400 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-lw9fr container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 03:13:17.397453 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:17.397350    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lw9fr" podUID="0fca1311-9bbd-4b5d-a996-2f67742bedb3" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 03:13:17.610571 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:17.610517    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lw9fr" podStartSLOduration=2.610503892 podStartE2EDuration="2.610503892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:13:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:13:17.572726801 +0000 UTC m=+404.087372668" watchObservedRunningTime="2026-11-01 03:13:17.610503892 +0000 UTC m=+404.125149755"
Nov 01 03:13:18.400546 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:18.400509    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lw9fr"
Nov 01 03:13:18.575738 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:18.575671    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-2gxpw"]
Nov 01 03:13:18.575909 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:18.575889    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2gxpw" podUID="9110c893-fab5-4136-b5ee-f3aca8edd0a9" containerName="discovery" containerID="cri-o://b70506740b1a32d6bd468250d927ee61c7a6ec6f01162a43d56ce5fbf6638b8e" gracePeriod=30
Nov 01 03:13:18.672509 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:18.672436    4400 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-2gxpw container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.26:8080/ready\": dial tcp 10.132.0.26:8080: connect: connection refused" start-of-body=
Nov 01 03:13:18.672654 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:18.672524    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2gxpw" podUID="9110c893-fab5-4136-b5ee-f3aca8edd0a9" containerName="discovery" probeResult="failure" output="Get \"http://10.132.0.26:8080/ready\": dial tcp 10.132.0.26:8080: connect: connection refused"
Nov 01 03:13:18.810888 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:18.810865    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2gxpw"
Nov 01 03:13:18.850619 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:18.850587    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9110c893-fab5-4136-b5ee-f3aca8edd0a9-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9110c893-fab5-4136-b5ee-f3aca8edd0a9-istio-kubeconfig\") pod \"9110c893-fab5-4136-b5ee-f3aca8edd0a9\" (UID: \"9110c893-fab5-4136-b5ee-f3aca8edd0a9\") "
Nov 01 03:13:18.850779 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:18.850658    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9110c893-fab5-4136-b5ee-f3aca8edd0a9-istio-token\" (UniqueName: \"kubernetes.io/projected/9110c893-fab5-4136-b5ee-f3aca8edd0a9-istio-token\") pod \"9110c893-fab5-4136-b5ee-f3aca8edd0a9\" (UID: \"9110c893-fab5-4136-b5ee-f3aca8edd0a9\") "
Nov 01 03:13:18.850779 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:18.850711    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9110c893-fab5-4136-b5ee-f3aca8edd0a9-cacerts\" (UniqueName: \"kubernetes.io/secret/9110c893-fab5-4136-b5ee-f3aca8edd0a9-cacerts\") pod \"9110c893-fab5-4136-b5ee-f3aca8edd0a9\" (UID: \"9110c893-fab5-4136-b5ee-f3aca8edd0a9\") "
Nov 01 03:13:18.850952 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:18.850780    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9110c893-fab5-4136-b5ee-f3aca8edd0a9-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9110c893-fab5-4136-b5ee-f3aca8edd0a9-istio-csr-ca-configmap\") pod \"9110c893-fab5-4136-b5ee-f3aca8edd0a9\" (UID: \"9110c893-fab5-4136-b5ee-f3aca8edd0a9\") "
Nov 01 03:13:18.850952 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:18.850811    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9110c893-fab5-4136-b5ee-f3aca8edd0a9-kube-api-access-sklft\" (UniqueName: \"kubernetes.io/projected/9110c893-fab5-4136-b5ee-f3aca8edd0a9-kube-api-access-sklft\") pod \"9110c893-fab5-4136-b5ee-f3aca8edd0a9\" (UID: \"9110c893-fab5-4136-b5ee-f3aca8edd0a9\") "
Nov 01 03:13:18.850952 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:18.850843    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9110c893-fab5-4136-b5ee-f3aca8edd0a9-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9110c893-fab5-4136-b5ee-f3aca8edd0a9-local-certs\") pod \"9110c893-fab5-4136-b5ee-f3aca8edd0a9\" (UID: \"9110c893-fab5-4136-b5ee-f3aca8edd0a9\") "
Nov 01 03:13:18.850952 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:18.850917    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9110c893-fab5-4136-b5ee-f3aca8edd0a9-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9110c893-fab5-4136-b5ee-f3aca8edd0a9-istio-csr-dns-cert\") pod \"9110c893-fab5-4136-b5ee-f3aca8edd0a9\" (UID: \"9110c893-fab5-4136-b5ee-f3aca8edd0a9\") "
Nov 01 03:13:18.851208 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:18.851176    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9110c893-fab5-4136-b5ee-f3aca8edd0a9-istio-csr-ca-configmap" pod "9110c893-fab5-4136-b5ee-f3aca8edd0a9" (UID: "9110c893-fab5-4136-b5ee-f3aca8edd0a9"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:13:18.851294 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:18.851271    4400 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9110c893-fab5-4136-b5ee-f3aca8edd0a9-istio-csr-ca-configmap\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:13:18.853182 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:18.853141    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9110c893-fab5-4136-b5ee-f3aca8edd0a9-istio-kubeconfig" pod "9110c893-fab5-4136-b5ee-f3aca8edd0a9" (UID: "9110c893-fab5-4136-b5ee-f3aca8edd0a9"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:13:18.853637 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:18.853608    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9110c893-fab5-4136-b5ee-f3aca8edd0a9-cacerts" pod "9110c893-fab5-4136-b5ee-f3aca8edd0a9" (UID: "9110c893-fab5-4136-b5ee-f3aca8edd0a9"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:13:18.853826 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:18.853638    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9110c893-fab5-4136-b5ee-f3aca8edd0a9-istio-csr-dns-cert" pod "9110c893-fab5-4136-b5ee-f3aca8edd0a9" (UID: "9110c893-fab5-4136-b5ee-f3aca8edd0a9"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:13:18.853826 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:18.853667    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9110c893-fab5-4136-b5ee-f3aca8edd0a9-local-certs" pod "9110c893-fab5-4136-b5ee-f3aca8edd0a9" (UID: "9110c893-fab5-4136-b5ee-f3aca8edd0a9"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:13:18.853826 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:18.853760    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9110c893-fab5-4136-b5ee-f3aca8edd0a9-kube-api-access-sklft" pod "9110c893-fab5-4136-b5ee-f3aca8edd0a9" (UID: "9110c893-fab5-4136-b5ee-f3aca8edd0a9"). InnerVolumeSpecName "kube-api-access-sklft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:13:18.853826 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:18.853772    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9110c893-fab5-4136-b5ee-f3aca8edd0a9-istio-token" pod "9110c893-fab5-4136-b5ee-f3aca8edd0a9" (UID: "9110c893-fab5-4136-b5ee-f3aca8edd0a9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:13:18.952051 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:18.951986    4400 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9110c893-fab5-4136-b5ee-f3aca8edd0a9-istio-kubeconfig\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:13:18.952051 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:18.952008    4400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9110c893-fab5-4136-b5ee-f3aca8edd0a9-istio-token\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:13:18.952051 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:18.952017    4400 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9110c893-fab5-4136-b5ee-f3aca8edd0a9-cacerts\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:13:18.952051 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:18.952026    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sklft\" (UniqueName: \"kubernetes.io/projected/9110c893-fab5-4136-b5ee-f3aca8edd0a9-kube-api-access-sklft\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:13:18.952051 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:18.952035    4400 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9110c893-fab5-4136-b5ee-f3aca8edd0a9-local-certs\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:13:18.952051 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:18.952044    4400 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9110c893-fab5-4136-b5ee-f3aca8edd0a9-istio-csr-dns-cert\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:13:19.405703 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:19.405643    4400 generic.go:356] "Generic (PLEG): container finished" podID="9110c893-fab5-4136-b5ee-f3aca8edd0a9" containerID="b70506740b1a32d6bd468250d927ee61c7a6ec6f01162a43d56ce5fbf6638b8e" exitCode=0
Nov 01 03:13:19.406149 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:19.405707    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2gxpw"
Nov 01 03:13:19.406149 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:19.405735    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2gxpw" event={"ID":"9110c893-fab5-4136-b5ee-f3aca8edd0a9","Type":"ContainerDied","Data":"b70506740b1a32d6bd468250d927ee61c7a6ec6f01162a43d56ce5fbf6638b8e"}
Nov 01 03:13:19.406149 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:19.405788    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2gxpw" event={"ID":"9110c893-fab5-4136-b5ee-f3aca8edd0a9","Type":"ContainerDied","Data":"7cb17d9f1819f3eb248c1a9337cc44eefe802a6a7d79d92bd3499e8bc78907af"}
Nov 01 03:13:19.406149 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:19.405814    4400 scope.go:122] "RemoveContainer" containerID="b70506740b1a32d6bd468250d927ee61c7a6ec6f01162a43d56ce5fbf6638b8e"
Nov 01 03:13:19.415987 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:19.415971    4400 scope.go:122] "RemoveContainer" containerID="b70506740b1a32d6bd468250d927ee61c7a6ec6f01162a43d56ce5fbf6638b8e"
Nov 01 03:13:19.416262 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:13:19.416241    4400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b70506740b1a32d6bd468250d927ee61c7a6ec6f01162a43d56ce5fbf6638b8e\": container with ID starting with b70506740b1a32d6bd468250d927ee61c7a6ec6f01162a43d56ce5fbf6638b8e not found: ID does not exist" containerID="b70506740b1a32d6bd468250d927ee61c7a6ec6f01162a43d56ce5fbf6638b8e"
Nov 01 03:13:19.416307 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:19.416271    4400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b70506740b1a32d6bd468250d927ee61c7a6ec6f01162a43d56ce5fbf6638b8e"} err="failed to get container status \"b70506740b1a32d6bd468250d927ee61c7a6ec6f01162a43d56ce5fbf6638b8e\": rpc error: code = NotFound desc = could not find container \"b70506740b1a32d6bd468250d927ee61c7a6ec6f01162a43d56ce5fbf6638b8e\": container with ID starting with b70506740b1a32d6bd468250d927ee61c7a6ec6f01162a43d56ce5fbf6638b8e not found: ID does not exist"
Nov 01 03:13:19.576119 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:19.576085    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-2gxpw"]
Nov 01 03:13:19.627221 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:19.627193    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-2gxpw"]
Nov 01 03:13:21.613351 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:21.613322    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9110c893-fab5-4136-b5ee-f3aca8edd0a9" path="/var/lib/kubelet/pods/9110c893-fab5-4136-b5ee-f3aca8edd0a9/volumes"
Nov 01 03:13:21.931378 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:21.931311    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-sz4xt"]
Nov 01 03:13:21.931660 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:21.931649    4400 cpu_manager.go:414] "containerMap: removing container" podUID="9110c893-fab5-4136-b5ee-f3aca8edd0a9" containerName="discovery"
Nov 01 03:13:21.931735 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:21.931662    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9110c893-fab5-4136-b5ee-f3aca8edd0a9" containerName="discovery"
Nov 01 03:13:21.931785 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:21.931774    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="9110c893-fab5-4136-b5ee-f3aca8edd0a9" containerName="discovery"
Nov 01 03:13:21.935985 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:21.935969    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-sz4xt"
Nov 01 03:13:21.978890 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:21.978861    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2vq9\" (UniqueName: \"kubernetes.io/projected/ab0fbe1a-6400-45cc-9a94-83a4b65049a4-kube-api-access-l2vq9\") pod \"model-serving-api-75bd89fdfc-sz4xt\" (UID: \"ab0fbe1a-6400-45cc-9a94-83a4b65049a4\") " pod="opendatahub/model-serving-api-75bd89fdfc-sz4xt"
Nov 01 03:13:21.979015 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:21.978916    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab0fbe1a-6400-45cc-9a94-83a4b65049a4-tls-certs\") pod \"model-serving-api-75bd89fdfc-sz4xt\" (UID: \"ab0fbe1a-6400-45cc-9a94-83a4b65049a4\") " pod="opendatahub/model-serving-api-75bd89fdfc-sz4xt"
Nov 01 03:13:22.042203 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:22.042176    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Nov 01 03:13:22.042361 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:22.042250    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-bmvnm\""
Nov 01 03:13:22.043867 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:22.043847    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-sz4xt"]
Nov 01 03:13:22.045159 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:22.045137    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-qh87r"]
Nov 01 03:13:22.048558 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:22.048531    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-qh87r"
Nov 01 03:13:22.080010 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:22.079989    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjqrd\" (UniqueName: \"kubernetes.io/projected/50cf97f8-3174-4880-a97a-5da66a3ccf7a-kube-api-access-rjqrd\") pod \"odh-model-controller-769875db56-qh87r\" (UID: \"50cf97f8-3174-4880-a97a-5da66a3ccf7a\") " pod="opendatahub/odh-model-controller-769875db56-qh87r"
Nov 01 03:13:22.080112 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:22.080026    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2vq9\" (UniqueName: \"kubernetes.io/projected/ab0fbe1a-6400-45cc-9a94-83a4b65049a4-kube-api-access-l2vq9\") pod \"model-serving-api-75bd89fdfc-sz4xt\" (UID: \"ab0fbe1a-6400-45cc-9a94-83a4b65049a4\") " pod="opendatahub/model-serving-api-75bd89fdfc-sz4xt"
Nov 01 03:13:22.080112 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:22.080061    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab0fbe1a-6400-45cc-9a94-83a4b65049a4-tls-certs\") pod \"model-serving-api-75bd89fdfc-sz4xt\" (UID: \"ab0fbe1a-6400-45cc-9a94-83a4b65049a4\") " pod="opendatahub/model-serving-api-75bd89fdfc-sz4xt"
Nov 01 03:13:22.080218 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:22.080127    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/50cf97f8-3174-4880-a97a-5da66a3ccf7a-cert\") pod \"odh-model-controller-769875db56-qh87r\" (UID: \"50cf97f8-3174-4880-a97a-5da66a3ccf7a\") " pod="opendatahub/odh-model-controller-769875db56-qh87r"
Nov 01 03:13:22.080218 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:22.080196    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50cf97f8-3174-4880-a97a-5da66a3ccf7a-metrics-certs\") pod \"odh-model-controller-769875db56-qh87r\" (UID: \"50cf97f8-3174-4880-a97a-5da66a3ccf7a\") " pod="opendatahub/odh-model-controller-769875db56-qh87r"
Nov 01 03:13:22.082428 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:22.082411    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab0fbe1a-6400-45cc-9a94-83a4b65049a4-tls-certs\") pod \"model-serving-api-75bd89fdfc-sz4xt\" (UID: \"ab0fbe1a-6400-45cc-9a94-83a4b65049a4\") " pod="opendatahub/model-serving-api-75bd89fdfc-sz4xt"
Nov 01 03:13:22.096161 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:22.096139    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-qh87r"]
Nov 01 03:13:22.103613 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:22.103592    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Nov 01 03:13:22.103722 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:22.103617    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\""
Nov 01 03:13:22.103722 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:22.103652    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-ct5hh\""
Nov 01 03:13:22.144245 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:22.144223    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2vq9\" (UniqueName: \"kubernetes.io/projected/ab0fbe1a-6400-45cc-9a94-83a4b65049a4-kube-api-access-l2vq9\") pod \"model-serving-api-75bd89fdfc-sz4xt\" (UID: \"ab0fbe1a-6400-45cc-9a94-83a4b65049a4\") " pod="opendatahub/model-serving-api-75bd89fdfc-sz4xt"
Nov 01 03:13:22.181040 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:22.181017    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/50cf97f8-3174-4880-a97a-5da66a3ccf7a-cert\") pod \"odh-model-controller-769875db56-qh87r\" (UID: \"50cf97f8-3174-4880-a97a-5da66a3ccf7a\") " pod="opendatahub/odh-model-controller-769875db56-qh87r"
Nov 01 03:13:22.181127 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:22.181049    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50cf97f8-3174-4880-a97a-5da66a3ccf7a-metrics-certs\") pod \"odh-model-controller-769875db56-qh87r\" (UID: \"50cf97f8-3174-4880-a97a-5da66a3ccf7a\") " pod="opendatahub/odh-model-controller-769875db56-qh87r"
Nov 01 03:13:22.181127 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:22.181101    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjqrd\" (UniqueName: \"kubernetes.io/projected/50cf97f8-3174-4880-a97a-5da66a3ccf7a-kube-api-access-rjqrd\") pod \"odh-model-controller-769875db56-qh87r\" (UID: \"50cf97f8-3174-4880-a97a-5da66a3ccf7a\") " pod="opendatahub/odh-model-controller-769875db56-qh87r"
Nov 01 03:13:22.183455 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:22.183395    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/50cf97f8-3174-4880-a97a-5da66a3ccf7a-cert\") pod \"odh-model-controller-769875db56-qh87r\" (UID: \"50cf97f8-3174-4880-a97a-5da66a3ccf7a\") " pod="opendatahub/odh-model-controller-769875db56-qh87r"
Nov 01 03:13:22.183455 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:22.183434    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50cf97f8-3174-4880-a97a-5da66a3ccf7a-metrics-certs\") pod \"odh-model-controller-769875db56-qh87r\" (UID: \"50cf97f8-3174-4880-a97a-5da66a3ccf7a\") " pod="opendatahub/odh-model-controller-769875db56-qh87r"
Nov 01 03:13:22.243155 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:22.243128    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjqrd\" (UniqueName: \"kubernetes.io/projected/50cf97f8-3174-4880-a97a-5da66a3ccf7a-kube-api-access-rjqrd\") pod \"odh-model-controller-769875db56-qh87r\" (UID: \"50cf97f8-3174-4880-a97a-5da66a3ccf7a\") " pod="opendatahub/odh-model-controller-769875db56-qh87r"
Nov 01 03:13:22.248944 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:22.248923    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-sz4xt"
Nov 01 03:13:22.361865 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:22.361835    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-qh87r"
Nov 01 03:13:22.631547 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:22.627866    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-sz4xt"]
Nov 01 03:13:22.633236 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:13:22.633208    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab0fbe1a_6400_45cc_9a94_83a4b65049a4.slice/crio-763e75db16e22392b927916c300478a9e254b541af14d765a82400839fbafa9c WatchSource:0}: Error finding container 763e75db16e22392b927916c300478a9e254b541af14d765a82400839fbafa9c: Status 404 returned error can't find the container with id 763e75db16e22392b927916c300478a9e254b541af14d765a82400839fbafa9c
Nov 01 03:13:22.741461 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:22.741440    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-qh87r"]
Nov 01 03:13:22.743803 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:13:22.743779    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50cf97f8_3174_4880_a97a_5da66a3ccf7a.slice/crio-4f9a50f8430fecaa0330329d5ca669c192859058f35e476ab8415d119ec77192 WatchSource:0}: Error finding container 4f9a50f8430fecaa0330329d5ca669c192859058f35e476ab8415d119ec77192: Status 404 returned error can't find the container with id 4f9a50f8430fecaa0330329d5ca669c192859058f35e476ab8415d119ec77192
Nov 01 03:13:23.426546 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:23.426515    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-qh87r" event={"ID":"50cf97f8-3174-4880-a97a-5da66a3ccf7a","Type":"ContainerStarted","Data":"4f9a50f8430fecaa0330329d5ca669c192859058f35e476ab8415d119ec77192"}
Nov 01 03:13:23.427626 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:23.427599    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-sz4xt" event={"ID":"ab0fbe1a-6400-45cc-9a94-83a4b65049a4","Type":"ContainerStarted","Data":"763e75db16e22392b927916c300478a9e254b541af14d765a82400839fbafa9c"}
Nov 01 03:13:25.441244 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:25.441212    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-sz4xt" event={"ID":"ab0fbe1a-6400-45cc-9a94-83a4b65049a4","Type":"ContainerStarted","Data":"ca426523ce064d702a2aabd48eaad80e6df64fc8a615307d7a4534309c5cc8aa"}
Nov 01 03:13:25.441644 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:25.441300    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-sz4xt"
Nov 01 03:13:25.561551 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:25.561465    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-sz4xt" podStartSLOduration=1.964831008 podStartE2EDuration="4.561451515s" totalImagesPullingTime="2.596620507s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:13:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:13:25.561191208 +0000 UTC m=+412.075837067" watchObservedRunningTime="2026-11-01 03:13:25.561451515 +0000 UTC m=+412.076097377"
Nov 01 03:13:26.446568 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:26.446477    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-qh87r" event={"ID":"50cf97f8-3174-4880-a97a-5da66a3ccf7a","Type":"ContainerStarted","Data":"be6399ae75efc89a5b1185e1184c0deaca0f369db6f273addb7a16ebe3cee88a"}
Nov 01 03:13:26.567901 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:26.567853    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-qh87r" podStartSLOduration=2.1919630740000002 podStartE2EDuration="5.567839103s" totalImagesPullingTime="3.375876029s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:13:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:13:26.566459362 +0000 UTC m=+413.081105224" watchObservedRunningTime="2026-11-01 03:13:26.567839103 +0000 UTC m=+413.082484965"
Nov 01 03:13:27.450948 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:27.450921    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-qh87r"
Nov 01 03:13:31.729705 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:13:31.729652    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:13:32.599517 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:32.599484    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 03:13:35.383641 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:35.383605    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7557fccb47-9l4f5"]
Nov 01 03:13:35.388586 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:35.388569    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7557fccb47-9l4f5"
Nov 01 03:13:35.500135 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:35.500097    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b2fe3be-c056-4ee9-a6bc-257f85273a05-metrics-certs\") pod \"maas-api-7557fccb47-9l4f5\" (UID: \"7b2fe3be-c056-4ee9-a6bc-257f85273a05\") " pod="odh-ai-gateway-infra/maas-api-7557fccb47-9l4f5"
Nov 01 03:13:35.500318 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:35.500164    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b2fe3be-c056-4ee9-a6bc-257f85273a05-maas-api-tls\") pod \"maas-api-7557fccb47-9l4f5\" (UID: \"7b2fe3be-c056-4ee9-a6bc-257f85273a05\") " pod="odh-ai-gateway-infra/maas-api-7557fccb47-9l4f5"
Nov 01 03:13:35.500318 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:35.500201    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b2fe3be-c056-4ee9-a6bc-257f85273a05-openshift-service-ca\") pod \"maas-api-7557fccb47-9l4f5\" (UID: \"7b2fe3be-c056-4ee9-a6bc-257f85273a05\") " pod="odh-ai-gateway-infra/maas-api-7557fccb47-9l4f5"
Nov 01 03:13:35.500318 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:35.500292    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhkpv\" (UniqueName: \"kubernetes.io/projected/7b2fe3be-c056-4ee9-a6bc-257f85273a05-kube-api-access-mhkpv\") pod \"maas-api-7557fccb47-9l4f5\" (UID: \"7b2fe3be-c056-4ee9-a6bc-257f85273a05\") " pod="odh-ai-gateway-infra/maas-api-7557fccb47-9l4f5"
Nov 01 03:13:35.502643 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:35.502624    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 03:13:35.502736 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:35.502658    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 01 03:13:35.502736 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:35.502705    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 03:13:35.502857 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:35.502771    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Nov 01 03:13:35.502950 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:35.502935    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-zx9nh\""
Nov 01 03:13:35.504572 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:35.504545    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7557fccb47-9l4f5"]
Nov 01 03:13:35.601300 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:35.601272    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b2fe3be-c056-4ee9-a6bc-257f85273a05-metrics-certs\") pod \"maas-api-7557fccb47-9l4f5\" (UID: \"7b2fe3be-c056-4ee9-a6bc-257f85273a05\") " pod="odh-ai-gateway-infra/maas-api-7557fccb47-9l4f5"
Nov 01 03:13:35.601463 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:35.601313    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b2fe3be-c056-4ee9-a6bc-257f85273a05-maas-api-tls\") pod \"maas-api-7557fccb47-9l4f5\" (UID: \"7b2fe3be-c056-4ee9-a6bc-257f85273a05\") " pod="odh-ai-gateway-infra/maas-api-7557fccb47-9l4f5"
Nov 01 03:13:35.601463 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:35.601330    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b2fe3be-c056-4ee9-a6bc-257f85273a05-openshift-service-ca\") pod \"maas-api-7557fccb47-9l4f5\" (UID: \"7b2fe3be-c056-4ee9-a6bc-257f85273a05\") " pod="odh-ai-gateway-infra/maas-api-7557fccb47-9l4f5"
Nov 01 03:13:35.601463 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:35.601366    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhkpv\" (UniqueName: \"kubernetes.io/projected/7b2fe3be-c056-4ee9-a6bc-257f85273a05-kube-api-access-mhkpv\") pod \"maas-api-7557fccb47-9l4f5\" (UID: \"7b2fe3be-c056-4ee9-a6bc-257f85273a05\") " pod="odh-ai-gateway-infra/maas-api-7557fccb47-9l4f5"
Nov 01 03:13:35.602070 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:35.602048    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b2fe3be-c056-4ee9-a6bc-257f85273a05-openshift-service-ca\") pod \"maas-api-7557fccb47-9l4f5\" (UID: \"7b2fe3be-c056-4ee9-a6bc-257f85273a05\") " pod="odh-ai-gateway-infra/maas-api-7557fccb47-9l4f5"
Nov 01 03:13:35.603992 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:35.603967    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b2fe3be-c056-4ee9-a6bc-257f85273a05-maas-api-tls\") pod \"maas-api-7557fccb47-9l4f5\" (UID: \"7b2fe3be-c056-4ee9-a6bc-257f85273a05\") " pod="odh-ai-gateway-infra/maas-api-7557fccb47-9l4f5"
Nov 01 03:13:35.604080 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:35.603982    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b2fe3be-c056-4ee9-a6bc-257f85273a05-metrics-certs\") pod \"maas-api-7557fccb47-9l4f5\" (UID: \"7b2fe3be-c056-4ee9-a6bc-257f85273a05\") " pod="odh-ai-gateway-infra/maas-api-7557fccb47-9l4f5"
Nov 01 03:13:35.718745 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:35.718658    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhkpv\" (UniqueName: \"kubernetes.io/projected/7b2fe3be-c056-4ee9-a6bc-257f85273a05-kube-api-access-mhkpv\") pod \"maas-api-7557fccb47-9l4f5\" (UID: \"7b2fe3be-c056-4ee9-a6bc-257f85273a05\") " pod="odh-ai-gateway-infra/maas-api-7557fccb47-9l4f5"
Nov 01 03:13:36.001369 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:36.001274    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7557fccb47-9l4f5"
Nov 01 03:13:36.139996 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:36.139964    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6bd866bf75-9b6xv"]
Nov 01 03:13:36.144361 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:36.144346    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6bd866bf75-9b6xv"
Nov 01 03:13:36.205962 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:36.205936    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Nov 01 03:13:36.206108 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:36.206053    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6bd866bf75-9b6xv"]
Nov 01 03:13:36.207309 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:36.207285    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/94f2b7e0-6308-4e0a-9637-958276c5c3aa-tls-cert\") pod \"authorino-6bd866bf75-9b6xv\" (UID: \"94f2b7e0-6308-4e0a-9637-958276c5c3aa\") " pod="kuadrant-system/authorino-6bd866bf75-9b6xv"
Nov 01 03:13:36.207477 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:36.207457    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vf8gv\" (UniqueName: \"kubernetes.io/projected/94f2b7e0-6308-4e0a-9637-958276c5c3aa-kube-api-access-vf8gv\") pod \"authorino-6bd866bf75-9b6xv\" (UID: \"94f2b7e0-6308-4e0a-9637-958276c5c3aa\") " pod="kuadrant-system/authorino-6bd866bf75-9b6xv"
Nov 01 03:13:36.307889 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:36.307821    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf8gv\" (UniqueName: \"kubernetes.io/projected/94f2b7e0-6308-4e0a-9637-958276c5c3aa-kube-api-access-vf8gv\") pod \"authorino-6bd866bf75-9b6xv\" (UID: \"94f2b7e0-6308-4e0a-9637-958276c5c3aa\") " pod="kuadrant-system/authorino-6bd866bf75-9b6xv"
Nov 01 03:13:36.307889 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:36.307863    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/94f2b7e0-6308-4e0a-9637-958276c5c3aa-tls-cert\") pod \"authorino-6bd866bf75-9b6xv\" (UID: \"94f2b7e0-6308-4e0a-9637-958276c5c3aa\") " pod="kuadrant-system/authorino-6bd866bf75-9b6xv"
Nov 01 03:13:36.310212 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:36.310190    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/94f2b7e0-6308-4e0a-9637-958276c5c3aa-tls-cert\") pod \"authorino-6bd866bf75-9b6xv\" (UID: \"94f2b7e0-6308-4e0a-9637-958276c5c3aa\") " pod="kuadrant-system/authorino-6bd866bf75-9b6xv"
Nov 01 03:13:36.370435 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:36.370407    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vf8gv\" (UniqueName: \"kubernetes.io/projected/94f2b7e0-6308-4e0a-9637-958276c5c3aa-kube-api-access-vf8gv\") pod \"authorino-6bd866bf75-9b6xv\" (UID: \"94f2b7e0-6308-4e0a-9637-958276c5c3aa\") " pod="kuadrant-system/authorino-6bd866bf75-9b6xv"
Nov 01 03:13:36.435292 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:36.435269    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7557fccb47-9l4f5"]
Nov 01 03:13:36.436921 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:13:36.436896    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b2fe3be_c056_4ee9_a6bc_257f85273a05.slice/crio-a342f8228017e8b1fb03988a792d2744d6aa448abef593cd36f75a3420ffa991 WatchSource:0}: Error finding container a342f8228017e8b1fb03988a792d2744d6aa448abef593cd36f75a3420ffa991: Status 404 returned error can't find the container with id a342f8228017e8b1fb03988a792d2744d6aa448abef593cd36f75a3420ffa991
Nov 01 03:13:36.450399 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:36.450380    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-sz4xt"
Nov 01 03:13:36.456829 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:36.456814    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6bd866bf75-9b6xv"
Nov 01 03:13:36.493676 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:36.493643    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7557fccb47-9l4f5" event={"ID":"7b2fe3be-c056-4ee9-a6bc-257f85273a05","Type":"ContainerStarted","Data":"a342f8228017e8b1fb03988a792d2744d6aa448abef593cd36f75a3420ffa991"}
Nov 01 03:13:36.829587 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:36.829554    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6bd866bf75-9b6xv"]
Nov 01 03:13:36.831473 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:13:36.831447    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod94f2b7e0_6308_4e0a_9637_958276c5c3aa.slice/crio-9b26fa17f2f1b9301881b2bb46ee23ec8acde3fd1de649ebf7818db729372a0a WatchSource:0}: Error finding container 9b26fa17f2f1b9301881b2bb46ee23ec8acde3fd1de649ebf7818db729372a0a: Status 404 returned error can't find the container with id 9b26fa17f2f1b9301881b2bb46ee23ec8acde3fd1de649ebf7818db729372a0a
Nov 01 03:13:37.501570 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:37.501533    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6bd866bf75-9b6xv" event={"ID":"94f2b7e0-6308-4e0a-9637-958276c5c3aa","Type":"ContainerStarted","Data":"9b26fa17f2f1b9301881b2bb46ee23ec8acde3fd1de649ebf7818db729372a0a"}
Nov 01 03:13:38.457467 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:38.457378    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-qh87r"
Nov 01 03:13:38.509122 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:38.509072    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6bd866bf75-9b6xv" event={"ID":"94f2b7e0-6308-4e0a-9637-958276c5c3aa","Type":"ContainerStarted","Data":"ac3a53cc60a14651b71d7294eed489124f07c749afaad588821475b223b85bd5"}
Nov 01 03:13:38.702892 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:38.702835    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6bd866bf75-9b6xv" podStartSLOduration=1.45583179 podStartE2EDuration="2.70281652s" totalImagesPullingTime="1.24698473s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:13:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:13:38.702089498 +0000 UTC m=+425.216735363" watchObservedRunningTime="2026-11-01 03:13:38.70281652 +0000 UTC m=+425.217462386"
Nov 01 03:13:38.791741 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:38.791657    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-s2rkp"]
Nov 01 03:13:38.791890 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:38.791868    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-s2rkp" podUID="7fbd32d3-3ead-42ac-8bae-b3a8c7d18612" containerName="authorino" containerID="cri-o://92ac2fa51007d34157796928f9b92b64d370969d40c9962fe6551085b013ca5c" gracePeriod=30
Nov 01 03:13:39.083972 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:39.083945    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-s2rkp"
Nov 01 03:13:39.133401 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:39.133366    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fbd32d3-3ead-42ac-8bae-b3a8c7d18612-kube-api-access-n5mrh\" (UniqueName: \"kubernetes.io/projected/7fbd32d3-3ead-42ac-8bae-b3a8c7d18612-kube-api-access-n5mrh\") pod \"7fbd32d3-3ead-42ac-8bae-b3a8c7d18612\" (UID: \"7fbd32d3-3ead-42ac-8bae-b3a8c7d18612\") "
Nov 01 03:13:39.136100 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:39.136071    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fbd32d3-3ead-42ac-8bae-b3a8c7d18612-kube-api-access-n5mrh" pod "7fbd32d3-3ead-42ac-8bae-b3a8c7d18612" (UID: "7fbd32d3-3ead-42ac-8bae-b3a8c7d18612"). InnerVolumeSpecName "kube-api-access-n5mrh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:13:39.234984 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:39.234937    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5mrh\" (UniqueName: \"kubernetes.io/projected/7fbd32d3-3ead-42ac-8bae-b3a8c7d18612-kube-api-access-n5mrh\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:13:39.515612 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:39.515584    4400 generic.go:356] "Generic (PLEG): container finished" podID="7fbd32d3-3ead-42ac-8bae-b3a8c7d18612" containerID="92ac2fa51007d34157796928f9b92b64d370969d40c9962fe6551085b013ca5c" exitCode=0
Nov 01 03:13:39.516057 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:39.515632    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-s2rkp"
Nov 01 03:13:39.516057 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:39.515669    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-s2rkp" event={"ID":"7fbd32d3-3ead-42ac-8bae-b3a8c7d18612","Type":"ContainerDied","Data":"92ac2fa51007d34157796928f9b92b64d370969d40c9962fe6551085b013ca5c"}
Nov 01 03:13:39.516057 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:39.515737    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-s2rkp" event={"ID":"7fbd32d3-3ead-42ac-8bae-b3a8c7d18612","Type":"ContainerDied","Data":"06fff93c8e2a823a0b9a7dcf86a77aa29570a8a0b010434aa149e4a2877906c5"}
Nov 01 03:13:39.516057 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:39.515760    4400 scope.go:122] "RemoveContainer" containerID="92ac2fa51007d34157796928f9b92b64d370969d40c9962fe6551085b013ca5c"
Nov 01 03:13:39.535879 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:39.535862    4400 scope.go:122] "RemoveContainer" containerID="92ac2fa51007d34157796928f9b92b64d370969d40c9962fe6551085b013ca5c"
Nov 01 03:13:39.537077 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:13:39.537051    4400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92ac2fa51007d34157796928f9b92b64d370969d40c9962fe6551085b013ca5c\": container with ID starting with 92ac2fa51007d34157796928f9b92b64d370969d40c9962fe6551085b013ca5c not found: ID does not exist" containerID="92ac2fa51007d34157796928f9b92b64d370969d40c9962fe6551085b013ca5c"
Nov 01 03:13:39.537195 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:39.537089    4400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92ac2fa51007d34157796928f9b92b64d370969d40c9962fe6551085b013ca5c"} err="failed to get container status \"92ac2fa51007d34157796928f9b92b64d370969d40c9962fe6551085b013ca5c\": rpc error: code = NotFound desc = could not find container \"92ac2fa51007d34157796928f9b92b64d370969d40c9962fe6551085b013ca5c\": container with ID starting with 92ac2fa51007d34157796928f9b92b64d370969d40c9962fe6551085b013ca5c not found: ID does not exist"
Nov 01 03:13:39.692500 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:39.692469    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-s2rkp"]
Nov 01 03:13:39.698487 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:39.698467    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-s2rkp"]
Nov 01 03:13:40.522203 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:40.522165    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7557fccb47-9l4f5" event={"ID":"7b2fe3be-c056-4ee9-a6bc-257f85273a05","Type":"ContainerStarted","Data":"c9470635d0c5a2a048de89756a521ccbb7de85eb81547ddddd6daebf8a68a9fa"}
Nov 01 03:13:40.522638 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:40.522335    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7557fccb47-9l4f5"
Nov 01 03:13:40.645532 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:40.645480    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7557fccb47-9l4f5" podStartSLOduration=2.602660754 podStartE2EDuration="5.645466916s" totalImagesPullingTime="3.042806162s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:13:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:13:40.644242678 +0000 UTC m=+427.158888541" watchObservedRunningTime="2026-11-01 03:13:40.645466916 +0000 UTC m=+427.160112779"
Nov 01 03:13:41.612229 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:41.612197    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fbd32d3-3ead-42ac-8bae-b3a8c7d18612" path="/var/lib/kubelet/pods/7fbd32d3-3ead-42ac-8bae-b3a8c7d18612/volumes"
Nov 01 03:13:46.533797 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:13:46.533764    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7557fccb47-9l4f5"
Nov 01 03:14:01.729483 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:14:01.729447    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:14:31.728394 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:14:31.728353    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:15:01.729251 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:15:01.729170    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:15:31.728586 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:15:31.728534    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:15:39.314986 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:15:39.314952    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 03:15:41.613026 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:15:41.612988    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hzp9k"]
Nov 01 03:15:41.613516 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:15:41.613498    4400 cpu_manager.go:414] "containerMap: removing container" podUID="7fbd32d3-3ead-42ac-8bae-b3a8c7d18612" containerName="authorino"
Nov 01 03:15:41.613590 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:15:41.613520    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fbd32d3-3ead-42ac-8bae-b3a8c7d18612" containerName="authorino"
Nov 01 03:15:41.613673 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:15:41.613661    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fbd32d3-3ead-42ac-8bae-b3a8c7d18612" containerName="authorino"
Nov 01 03:15:41.615700 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:15:41.615669    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hzp9k"
Nov 01 03:15:41.698589 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:15:41.698561    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdde4990-3c7e-4c5f-b374-2dd5d4fb9ff4-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hzp9k\" (UID: \"cdde4990-3c7e-4c5f-b374-2dd5d4fb9ff4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hzp9k"
Nov 01 03:15:41.698757 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:15:41.698599    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdde4990-3c7e-4c5f-b374-2dd5d4fb9ff4-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hzp9k\" (UID: \"cdde4990-3c7e-4c5f-b374-2dd5d4fb9ff4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hzp9k"
Nov 01 03:15:41.698757 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:15:41.698674    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdde4990-3c7e-4c5f-b374-2dd5d4fb9ff4-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hzp9k\" (UID: \"cdde4990-3c7e-4c5f-b374-2dd5d4fb9ff4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hzp9k"
Nov 01 03:15:41.698757 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:15:41.698737    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpvzm\" (UniqueName: \"kubernetes.io/projected/cdde4990-3c7e-4c5f-b374-2dd5d4fb9ff4-kube-api-access-kpvzm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hzp9k\" (UID: \"cdde4990-3c7e-4c5f-b374-2dd5d4fb9ff4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hzp9k"
Nov 01 03:15:41.698889 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:15:41.698810    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdde4990-3c7e-4c5f-b374-2dd5d4fb9ff4-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hzp9k\" (UID: \"cdde4990-3c7e-4c5f-b374-2dd5d4fb9ff4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hzp9k"
Nov 01 03:15:41.698889 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:15:41.698850    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdde4990-3c7e-4c5f-b374-2dd5d4fb9ff4-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hzp9k\" (UID: \"cdde4990-3c7e-4c5f-b374-2dd5d4fb9ff4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hzp9k"
Nov 01 03:15:41.722497 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:15:41.722471    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Nov 01 03:15:41.722645 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:15:41.722534    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Nov 01 03:15:41.722645 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:15:41.722639    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-5tqmw\""
Nov 01 03:15:41.722761 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:15:41.722647    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Nov 01 03:15:41.724361 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:15:41.724339    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hzp9k"]
Nov 01 03:15:41.800086 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:15:41.800057    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdde4990-3c7e-4c5f-b374-2dd5d4fb9ff4-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hzp9k\" (UID: \"cdde4990-3c7e-4c5f-b374-2dd5d4fb9ff4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hzp9k"
Nov 01 03:15:41.800260 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:15:41.800101    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpvzm\" (UniqueName: \"kubernetes.io/projected/cdde4990-3c7e-4c5f-b374-2dd5d4fb9ff4-kube-api-access-kpvzm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hzp9k\" (UID: \"cdde4990-3c7e-4c5f-b374-2dd5d4fb9ff4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hzp9k"
Nov 01 03:15:41.800260 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:15:41.800135    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdde4990-3c7e-4c5f-b374-2dd5d4fb9ff4-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hzp9k\" (UID: \"cdde4990-3c7e-4c5f-b374-2dd5d4fb9ff4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hzp9k"
Nov 01 03:15:41.800260 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:15:41.800160    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdde4990-3c7e-4c5f-b374-2dd5d4fb9ff4-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hzp9k\" (UID: \"cdde4990-3c7e-4c5f-b374-2dd5d4fb9ff4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hzp9k"
Nov 01 03:15:41.800260 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:15:41.800213    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdde4990-3c7e-4c5f-b374-2dd5d4fb9ff4-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hzp9k\" (UID: \"cdde4990-3c7e-4c5f-b374-2dd5d4fb9ff4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hzp9k"
Nov 01 03:15:41.800522 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:15:41.800338    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdde4990-3c7e-4c5f-b374-2dd5d4fb9ff4-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hzp9k\" (UID: \"cdde4990-3c7e-4c5f-b374-2dd5d4fb9ff4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hzp9k"
Nov 01 03:15:41.800591 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:15:41.800527    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdde4990-3c7e-4c5f-b374-2dd5d4fb9ff4-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hzp9k\" (UID: \"cdde4990-3c7e-4c5f-b374-2dd5d4fb9ff4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hzp9k"
Nov 01 03:15:41.800650 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:15:41.800593    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdde4990-3c7e-4c5f-b374-2dd5d4fb9ff4-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hzp9k\" (UID: \"cdde4990-3c7e-4c5f-b374-2dd5d4fb9ff4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hzp9k"
Nov 01 03:15:41.800650 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:15:41.800623    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdde4990-3c7e-4c5f-b374-2dd5d4fb9ff4-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hzp9k\" (UID: \"cdde4990-3c7e-4c5f-b374-2dd5d4fb9ff4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hzp9k"
Nov 01 03:15:41.802611 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:15:41.802589    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdde4990-3c7e-4c5f-b374-2dd5d4fb9ff4-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hzp9k\" (UID: \"cdde4990-3c7e-4c5f-b374-2dd5d4fb9ff4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hzp9k"
Nov 01 03:15:41.803088 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:15:41.803068    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdde4990-3c7e-4c5f-b374-2dd5d4fb9ff4-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hzp9k\" (UID: \"cdde4990-3c7e-4c5f-b374-2dd5d4fb9ff4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hzp9k"
Nov 01 03:15:41.870982 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:15:41.870919    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpvzm\" (UniqueName: \"kubernetes.io/projected/cdde4990-3c7e-4c5f-b374-2dd5d4fb9ff4-kube-api-access-kpvzm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hzp9k\" (UID: \"cdde4990-3c7e-4c5f-b374-2dd5d4fb9ff4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hzp9k"
Nov 01 03:15:41.928105 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:15:41.928042    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hzp9k"
Nov 01 03:15:42.302924 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:15:42.302899    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hzp9k"]
Nov 01 03:15:42.304240 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:15:42.304207    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdde4990_3c7e_4c5f_b374_2dd5d4fb9ff4.slice/crio-57a605c2f9f6b3ac6569b385d8c1965bef302b24dd245830c8a5cab12edea9a6 WatchSource:0}: Error finding container 57a605c2f9f6b3ac6569b385d8c1965bef302b24dd245830c8a5cab12edea9a6: Status 404 returned error can't find the container with id 57a605c2f9f6b3ac6569b385d8c1965bef302b24dd245830c8a5cab12edea9a6
Nov 01 03:15:43.065100 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:15:43.065060    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hzp9k" event={"ID":"cdde4990-3c7e-4c5f-b374-2dd5d4fb9ff4","Type":"ContainerStarted","Data":"57a605c2f9f6b3ac6569b385d8c1965bef302b24dd245830c8a5cab12edea9a6"}
Nov 01 03:15:45.078812 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:15:45.078760    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hzp9k" event={"ID":"cdde4990-3c7e-4c5f-b374-2dd5d4fb9ff4","Type":"ContainerStarted","Data":"ac0533be3e2a4a273882bef2e45d4c691ded83c2bea38391bb0bbe92c7172878"}
Nov 01 03:15:45.211057 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:15:45.211002    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hzp9k" podStartSLOduration=2.405016246 podStartE2EDuration="4.210987959s" totalImagesPullingTime="1.805971713s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:15:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:15:45.209887514 +0000 UTC m=+551.724533374" watchObservedRunningTime="2026-11-01 03:15:45.210987959 +0000 UTC m=+551.725633821"
Nov 01 03:15:45.929140 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:15:45.929107    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hzp9k"
Nov 01 03:15:49.227107 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:15:49.227077    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-j6ptb"]
Nov 01 03:15:49.230613 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:15:49.230593    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-j6ptb"
Nov 01 03:15:49.276465 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:15:49.276424    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94e47bc1-fd92-4865-a6b7-b7ab4617b348-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-j6ptb\" (UID: \"94e47bc1-fd92-4865-a6b7-b7ab4617b348\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-j6ptb"
Nov 01 03:15:49.276640 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:15:49.276516    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94e47bc1-fd92-4865-a6b7-b7ab4617b348-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-j6ptb\" (UID: \"94e47bc1-fd92-4865-a6b7-b7ab4617b348\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-j6ptb"
Nov 01 03:15:49.276640 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:15:49.276544    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94e47bc1-fd92-4865-a6b7-b7ab4617b348-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-j6ptb\" (UID: \"94e47bc1-fd92-4865-a6b7-b7ab4617b348\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-j6ptb"
Nov 01 03:15:49.276640 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:15:49.276588    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgr5j\" (UniqueName: \"kubernetes.io/projected/94e47bc1-fd92-4865-a6b7-b7ab4617b348-kube-api-access-vgr5j\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-j6ptb\" (UID: \"94e47bc1-fd92-4865-a6b7-b7ab4617b348\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-j6ptb"
Nov 01 03:15:49.276814 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:15:49.276653    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94e47bc1-fd92-4865-a6b7-b7ab4617b348-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-j6ptb\" (UID: \"94e47bc1-fd92-4865-a6b7-b7ab4617b348\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-j6ptb"
Nov 01 03:15:49.276814 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:15:49.276758    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94e47bc1-fd92-4865-a6b7-b7ab4617b348-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-j6ptb\" (UID: \"94e47bc1-fd92-4865-a6b7-b7ab4617b348\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-j6ptb"
Nov 01 03:15:49.335213 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:15:49.335181    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Nov 01 03:15:49.336958 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:15:49.336939    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-j6ptb"]
Nov 01 03:15:49.377165 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:15:49.377140    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94e47bc1-fd92-4865-a6b7-b7ab4617b348-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-j6ptb\" (UID: \"94e47bc1-fd92-4865-a6b7-b7ab4617b348\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-j6ptb"
Nov 01 03:15:49.377314 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:15:49.377188    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94e47bc1-fd92-4865-a6b7-b7ab4617b348-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-j6ptb\" (UID: \"94e47bc1-fd92-4865-a6b7-b7ab4617b348\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-j6ptb"
Nov 01 03:15:49.377314 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:15:49.377219    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94e47bc1-fd92-4865-a6b7-b7ab4617b348-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-j6ptb\" (UID: \"94e47bc1-fd92-4865-a6b7-b7ab4617b348\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-j6ptb"
Nov 01 03:15:49.377314 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:15:49.377241    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94e47bc1-fd92-4865-a6b7-b7ab4617b348-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-j6ptb\" (UID: \"94e47bc1-fd92-4865-a6b7-b7ab4617b348\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-j6ptb"
Nov 01 03:15:49.377314 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:15:49.377265    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgr5j\" (UniqueName: \"kubernetes.io/projected/94e47bc1-fd92-4865-a6b7-b7ab4617b348-kube-api-access-vgr5j\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-j6ptb\" (UID: \"94e47bc1-fd92-4865-a6b7-b7ab4617b348\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-j6ptb"
Nov 01 03:15:49.377532 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:15:49.377316    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94e47bc1-fd92-4865-a6b7-b7ab4617b348-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-j6ptb\" (UID: \"94e47bc1-fd92-4865-a6b7-b7ab4617b348\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-j6ptb"
Nov 01 03:15:49.377627 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:15:49.377604    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94e47bc1-fd92-4865-a6b7-b7ab4617b348-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-j6ptb\" (UID: \"94e47bc1-fd92-4865-a6b7-b7ab4617b348\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-j6ptb"
Nov 01 03:15:49.377720 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:15:49.377698    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94e47bc1-fd92-4865-a6b7-b7ab4617b348-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-j6ptb\" (UID: \"94e47bc1-fd92-4865-a6b7-b7ab4617b348\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-j6ptb"
Nov 01 03:15:49.377792 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:15:49.377740    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94e47bc1-fd92-4865-a6b7-b7ab4617b348-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-j6ptb\" (UID: \"94e47bc1-fd92-4865-a6b7-b7ab4617b348\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-j6ptb"
Nov 01 03:15:49.379749 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:15:49.379725    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94e47bc1-fd92-4865-a6b7-b7ab4617b348-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-j6ptb\" (UID: \"94e47bc1-fd92-4865-a6b7-b7ab4617b348\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-j6ptb"
Nov 01 03:15:49.379857 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:15:49.379772    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94e47bc1-fd92-4865-a6b7-b7ab4617b348-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-j6ptb\" (UID: \"94e47bc1-fd92-4865-a6b7-b7ab4617b348\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-j6ptb"
Nov 01 03:15:49.439724 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:15:49.439701    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgr5j\" (UniqueName: \"kubernetes.io/projected/94e47bc1-fd92-4865-a6b7-b7ab4617b348-kube-api-access-vgr5j\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-j6ptb\" (UID: \"94e47bc1-fd92-4865-a6b7-b7ab4617b348\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-j6ptb"
Nov 01 03:15:49.543653 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:15:49.543585    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-j6ptb"
Nov 01 03:15:49.919376 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:15:49.919354    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-j6ptb"]
Nov 01 03:15:49.921358 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:15:49.921329    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94e47bc1_fd92_4865_a6b7_b7ab4617b348.slice/crio-6a6cff343a41b36e261666603610409db30031300baa84cb168fa991ec97773a WatchSource:0}: Error finding container 6a6cff343a41b36e261666603610409db30031300baa84cb168fa991ec97773a: Status 404 returned error can't find the container with id 6a6cff343a41b36e261666603610409db30031300baa84cb168fa991ec97773a
Nov 01 03:15:50.102602 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:15:50.102573    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-j6ptb" event={"ID":"94e47bc1-fd92-4865-a6b7-b7ab4617b348","Type":"ContainerStarted","Data":"6a6cff343a41b36e261666603610409db30031300baa84cb168fa991ec97773a"}
Nov 01 03:15:51.108619 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:15:51.108586    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-j6ptb" event={"ID":"94e47bc1-fd92-4865-a6b7-b7ab4617b348","Type":"ContainerStarted","Data":"983dc0d346a2bb5af815415c5548703c6e38d7739d4154e4c225803ae28bff22"}
Nov 01 03:15:51.229494 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:15:51.229439    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-j6ptb" podStartSLOduration=1.5172036979999999 podStartE2EDuration="2.229425329s" totalImagesPullingTime="712.221631ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:15:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:15:51.228401091 +0000 UTC m=+557.743046988" watchObservedRunningTime="2026-11-01 03:15:51.229425329 +0000 UTC m=+557.744071192"
Nov 01 03:15:51.543875 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:15:51.543796    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-j6ptb"
Nov 01 03:15:51.913528 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:15:51.913499    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 03:15:51.929066 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:15:51.929021    4400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hzp9k"
Nov 01 03:15:51.942989 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:15:51.942960    4400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hzp9k"
Nov 01 03:15:52.126358 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:15:52.126328    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hzp9k"
Nov 01 03:15:59.544630 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:15:59.544596    4400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-j6ptb"
Nov 01 03:15:59.561231 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:15:59.561202    4400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-j6ptb"
Nov 01 03:16:00.162570 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:00.162543    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-j6ptb"
Nov 01 03:16:01.450210 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:01.450175    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-2g58t"]
Nov 01 03:16:01.453659 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:01.453644    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2g58t"
Nov 01 03:16:01.507523 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:01.507494    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Nov 01 03:16:01.560259 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:01.560232    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-2g58t"]
Nov 01 03:16:01.581982 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:01.581954    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zp65\" (UniqueName: \"kubernetes.io/projected/a15e6f20-9998-40fb-9474-ad3d5af53823-kube-api-access-6zp65\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-2g58t\" (UID: \"a15e6f20-9998-40fb-9474-ad3d5af53823\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2g58t"
Nov 01 03:16:01.582132 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:01.582015    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a15e6f20-9998-40fb-9474-ad3d5af53823-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-2g58t\" (UID: \"a15e6f20-9998-40fb-9474-ad3d5af53823\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2g58t"
Nov 01 03:16:01.582132 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:01.582117    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a15e6f20-9998-40fb-9474-ad3d5af53823-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-2g58t\" (UID: \"a15e6f20-9998-40fb-9474-ad3d5af53823\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2g58t"
Nov 01 03:16:01.582251 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:01.582225    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a15e6f20-9998-40fb-9474-ad3d5af53823-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-2g58t\" (UID: \"a15e6f20-9998-40fb-9474-ad3d5af53823\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2g58t"
Nov 01 03:16:01.582309 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:01.582284    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a15e6f20-9998-40fb-9474-ad3d5af53823-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-2g58t\" (UID: \"a15e6f20-9998-40fb-9474-ad3d5af53823\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2g58t"
Nov 01 03:16:01.582347 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:01.582328    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a15e6f20-9998-40fb-9474-ad3d5af53823-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-2g58t\" (UID: \"a15e6f20-9998-40fb-9474-ad3d5af53823\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2g58t"
Nov 01 03:16:01.683662 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:01.683628    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a15e6f20-9998-40fb-9474-ad3d5af53823-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-2g58t\" (UID: \"a15e6f20-9998-40fb-9474-ad3d5af53823\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2g58t"
Nov 01 03:16:01.683858 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:01.683732    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a15e6f20-9998-40fb-9474-ad3d5af53823-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-2g58t\" (UID: \"a15e6f20-9998-40fb-9474-ad3d5af53823\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2g58t"
Nov 01 03:16:01.683858 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:01.683773    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a15e6f20-9998-40fb-9474-ad3d5af53823-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-2g58t\" (UID: \"a15e6f20-9998-40fb-9474-ad3d5af53823\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2g58t"
Nov 01 03:16:01.683858 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:01.683791    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zp65\" (UniqueName: \"kubernetes.io/projected/a15e6f20-9998-40fb-9474-ad3d5af53823-kube-api-access-6zp65\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-2g58t\" (UID: \"a15e6f20-9998-40fb-9474-ad3d5af53823\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2g58t"
Nov 01 03:16:01.684043 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:01.683894    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a15e6f20-9998-40fb-9474-ad3d5af53823-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-2g58t\" (UID: \"a15e6f20-9998-40fb-9474-ad3d5af53823\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2g58t"
Nov 01 03:16:01.684043 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:01.683933    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a15e6f20-9998-40fb-9474-ad3d5af53823-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-2g58t\" (UID: \"a15e6f20-9998-40fb-9474-ad3d5af53823\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2g58t"
Nov 01 03:16:01.684143 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:01.684043    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a15e6f20-9998-40fb-9474-ad3d5af53823-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-2g58t\" (UID: \"a15e6f20-9998-40fb-9474-ad3d5af53823\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2g58t"
Nov 01 03:16:01.684143 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:01.684128    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a15e6f20-9998-40fb-9474-ad3d5af53823-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-2g58t\" (UID: \"a15e6f20-9998-40fb-9474-ad3d5af53823\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2g58t"
Nov 01 03:16:01.684392 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:01.684367    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a15e6f20-9998-40fb-9474-ad3d5af53823-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-2g58t\" (UID: \"a15e6f20-9998-40fb-9474-ad3d5af53823\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2g58t"
Nov 01 03:16:01.686250 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:01.686230    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a15e6f20-9998-40fb-9474-ad3d5af53823-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-2g58t\" (UID: \"a15e6f20-9998-40fb-9474-ad3d5af53823\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2g58t"
Nov 01 03:16:01.686492 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:01.686474    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a15e6f20-9998-40fb-9474-ad3d5af53823-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-2g58t\" (UID: \"a15e6f20-9998-40fb-9474-ad3d5af53823\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2g58t"
Nov 01 03:16:01.728994 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:16:01.728912    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:16:01.746725 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:01.746702    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zp65\" (UniqueName: \"kubernetes.io/projected/a15e6f20-9998-40fb-9474-ad3d5af53823-kube-api-access-6zp65\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-2g58t\" (UID: \"a15e6f20-9998-40fb-9474-ad3d5af53823\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2g58t"
Nov 01 03:16:01.767284 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:01.767259    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2g58t"
Nov 01 03:16:02.066869 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:02.066776    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 03:16:02.182105 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:02.182082    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-2g58t"]
Nov 01 03:16:02.183462 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:16:02.183432    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda15e6f20_9998_40fb_9474_ad3d5af53823.slice/crio-325845ca8b23990071c7f00f6567fa09266ddc522853db9e7209471958247b81 WatchSource:0}: Error finding container 325845ca8b23990071c7f00f6567fa09266ddc522853db9e7209471958247b81: Status 404 returned error can't find the container with id 325845ca8b23990071c7f00f6567fa09266ddc522853db9e7209471958247b81
Nov 01 03:16:03.167893 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:03.167807    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2g58t" event={"ID":"a15e6f20-9998-40fb-9474-ad3d5af53823","Type":"ContainerStarted","Data":"410ccf1666a36710aa32cf6d1a9d024d07fb870e25edf0c4db592f9305062d5b"}
Nov 01 03:16:03.167893 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:03.167870    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2g58t" event={"ID":"a15e6f20-9998-40fb-9474-ad3d5af53823","Type":"ContainerStarted","Data":"325845ca8b23990071c7f00f6567fa09266ddc522853db9e7209471958247b81"}
Nov 01 03:16:03.287814 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:03.287773    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2g58t" podStartSLOduration=1.589696137 podStartE2EDuration="2.287760295s" totalImagesPullingTime="698.064158ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:16:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:16:03.287747804 +0000 UTC m=+569.802393661" watchObservedRunningTime="2026-11-01 03:16:03.287760295 +0000 UTC m=+569.802406198"
Nov 01 03:16:03.768093 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:03.768060    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2g58t"
Nov 01 03:16:05.104311 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:05.104279    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88th2j"]
Nov 01 03:16:05.107962 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:05.107940    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88th2j"
Nov 01 03:16:05.212903 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:05.212873    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Nov 01 03:16:05.214969 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:05.214946    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88th2j"]
Nov 01 03:16:05.215285 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:05.215262    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/feec6c6c-1074-485e-88fe-f27a3b1606bb-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88th2j\" (UID: \"feec6c6c-1074-485e-88fe-f27a3b1606bb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88th2j"
Nov 01 03:16:05.215402 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:05.215386    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/feec6c6c-1074-485e-88fe-f27a3b1606bb-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88th2j\" (UID: \"feec6c6c-1074-485e-88fe-f27a3b1606bb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88th2j"
Nov 01 03:16:05.215502 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:05.215487    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/feec6c6c-1074-485e-88fe-f27a3b1606bb-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88th2j\" (UID: \"feec6c6c-1074-485e-88fe-f27a3b1606bb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88th2j"
Nov 01 03:16:05.215578 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:05.215529    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/feec6c6c-1074-485e-88fe-f27a3b1606bb-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88th2j\" (UID: \"feec6c6c-1074-485e-88fe-f27a3b1606bb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88th2j"
Nov 01 03:16:05.215635 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:05.215580    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/feec6c6c-1074-485e-88fe-f27a3b1606bb-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88th2j\" (UID: \"feec6c6c-1074-485e-88fe-f27a3b1606bb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88th2j"
Nov 01 03:16:05.215721 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:05.215704    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klj5j\" (UniqueName: \"kubernetes.io/projected/feec6c6c-1074-485e-88fe-f27a3b1606bb-kube-api-access-klj5j\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88th2j\" (UID: \"feec6c6c-1074-485e-88fe-f27a3b1606bb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88th2j"
Nov 01 03:16:05.316235 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:05.316202    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klj5j\" (UniqueName: \"kubernetes.io/projected/feec6c6c-1074-485e-88fe-f27a3b1606bb-kube-api-access-klj5j\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88th2j\" (UID: \"feec6c6c-1074-485e-88fe-f27a3b1606bb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88th2j"
Nov 01 03:16:05.316422 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:05.316251    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/feec6c6c-1074-485e-88fe-f27a3b1606bb-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88th2j\" (UID: \"feec6c6c-1074-485e-88fe-f27a3b1606bb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88th2j"
Nov 01 03:16:05.316422 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:05.316363    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/feec6c6c-1074-485e-88fe-f27a3b1606bb-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88th2j\" (UID: \"feec6c6c-1074-485e-88fe-f27a3b1606bb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88th2j"
Nov 01 03:16:05.316422 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:05.316410    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/feec6c6c-1074-485e-88fe-f27a3b1606bb-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88th2j\" (UID: \"feec6c6c-1074-485e-88fe-f27a3b1606bb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88th2j"
Nov 01 03:16:05.316600 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:05.316441    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/feec6c6c-1074-485e-88fe-f27a3b1606bb-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88th2j\" (UID: \"feec6c6c-1074-485e-88fe-f27a3b1606bb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88th2j"
Nov 01 03:16:05.316600 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:05.316489    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/feec6c6c-1074-485e-88fe-f27a3b1606bb-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88th2j\" (UID: \"feec6c6c-1074-485e-88fe-f27a3b1606bb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88th2j"
Nov 01 03:16:05.316978 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:05.316951    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/feec6c6c-1074-485e-88fe-f27a3b1606bb-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88th2j\" (UID: \"feec6c6c-1074-485e-88fe-f27a3b1606bb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88th2j"
Nov 01 03:16:05.317152 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:05.317130    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/feec6c6c-1074-485e-88fe-f27a3b1606bb-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88th2j\" (UID: \"feec6c6c-1074-485e-88fe-f27a3b1606bb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88th2j"
Nov 01 03:16:05.317506 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:05.317486    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/feec6c6c-1074-485e-88fe-f27a3b1606bb-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88th2j\" (UID: \"feec6c6c-1074-485e-88fe-f27a3b1606bb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88th2j"
Nov 01 03:16:05.320141 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:05.320121    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/feec6c6c-1074-485e-88fe-f27a3b1606bb-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88th2j\" (UID: \"feec6c6c-1074-485e-88fe-f27a3b1606bb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88th2j"
Nov 01 03:16:05.322041 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:05.322020    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/feec6c6c-1074-485e-88fe-f27a3b1606bb-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88th2j\" (UID: \"feec6c6c-1074-485e-88fe-f27a3b1606bb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88th2j"
Nov 01 03:16:05.384238 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:05.384129    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klj5j\" (UniqueName: \"kubernetes.io/projected/feec6c6c-1074-485e-88fe-f27a3b1606bb-kube-api-access-klj5j\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88th2j\" (UID: \"feec6c6c-1074-485e-88fe-f27a3b1606bb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88th2j"
Nov 01 03:16:05.421858 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:05.421821    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88th2j"
Nov 01 03:16:05.802729 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:05.802705    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88th2j"]
Nov 01 03:16:05.803955 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:16:05.803928    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfeec6c6c_1074_485e_88fe_f27a3b1606bb.slice/crio-9114f2ec244d886b8460384fdfef8f3382d72725e96a5b3e0ee727a197d8ae20 WatchSource:0}: Error finding container 9114f2ec244d886b8460384fdfef8f3382d72725e96a5b3e0ee727a197d8ae20: Status 404 returned error can't find the container with id 9114f2ec244d886b8460384fdfef8f3382d72725e96a5b3e0ee727a197d8ae20
Nov 01 03:16:06.183173 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:06.183142    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88th2j" event={"ID":"feec6c6c-1074-485e-88fe-f27a3b1606bb","Type":"ContainerStarted","Data":"9114f2ec244d886b8460384fdfef8f3382d72725e96a5b3e0ee727a197d8ae20"}
Nov 01 03:16:07.189543 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:07.189508    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88th2j" event={"ID":"feec6c6c-1074-485e-88fe-f27a3b1606bb","Type":"ContainerStarted","Data":"c3b98ad011412e98009b54c920140c67d6b3cd5f2ab30de0903ccde6512a8609"}
Nov 01 03:16:07.310853 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:07.310806    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88th2j" podStartSLOduration=1.608887626 podStartE2EDuration="2.310789436s" totalImagesPullingTime="701.90181ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:16:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:16:07.30977244 +0000 UTC m=+573.824418303" watchObservedRunningTime="2026-11-01 03:16:07.310789436 +0000 UTC m=+573.825435299"
Nov 01 03:16:07.422550 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:07.422519    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88th2j"
Nov 01 03:16:09.721237 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:09.721205    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 03:16:11.768211 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:11.768183    4400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2g58t"
Nov 01 03:16:11.780698 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:11.780666    4400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2g58t"
Nov 01 03:16:12.224503 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:12.224478    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2g58t"
Nov 01 03:16:15.422089 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:15.422004    4400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88th2j"
Nov 01 03:16:15.434477 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:15.434454    4400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88th2j"
Nov 01 03:16:15.756032 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:15.755953    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vttd9"]
Nov 01 03:16:15.759618 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:15.759598    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vttd9"
Nov 01 03:16:15.813589 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:15.813565    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-v7nmz\""
Nov 01 03:16:15.821509 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:15.821487    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vttd9"]
Nov 01 03:16:15.913484 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:15.913459    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca8197ec-0240-48b6-ae32-c5ad205f9d2d-istio-envoy\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-vttd9\" (UID: \"ca8197ec-0240-48b6-ae32-c5ad205f9d2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vttd9"
Nov 01 03:16:15.913484 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:15.913490    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca8197ec-0240-48b6-ae32-c5ad205f9d2d-credential-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-vttd9\" (UID: \"ca8197ec-0240-48b6-ae32-c5ad205f9d2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vttd9"
Nov 01 03:16:15.913710 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:15.913523    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca8197ec-0240-48b6-ae32-c5ad205f9d2d-workload-certs\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-vttd9\" (UID: \"ca8197ec-0240-48b6-ae32-c5ad205f9d2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vttd9"
Nov 01 03:16:15.913710 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:15.913622    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn6x2\" (UniqueName: \"kubernetes.io/projected/ca8197ec-0240-48b6-ae32-c5ad205f9d2d-kube-api-access-bn6x2\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-vttd9\" (UID: \"ca8197ec-0240-48b6-ae32-c5ad205f9d2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vttd9"
Nov 01 03:16:15.913710 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:15.913642    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca8197ec-0240-48b6-ae32-c5ad205f9d2d-istio-token\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-vttd9\" (UID: \"ca8197ec-0240-48b6-ae32-c5ad205f9d2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vttd9"
Nov 01 03:16:15.913710 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:15.913701    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca8197ec-0240-48b6-ae32-c5ad205f9d2d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-vttd9\" (UID: \"ca8197ec-0240-48b6-ae32-c5ad205f9d2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vttd9"
Nov 01 03:16:15.913850 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:15.913794    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca8197ec-0240-48b6-ae32-c5ad205f9d2d-workload-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-vttd9\" (UID: \"ca8197ec-0240-48b6-ae32-c5ad205f9d2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vttd9"
Nov 01 03:16:15.913850 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:15.913820    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca8197ec-0240-48b6-ae32-c5ad205f9d2d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-vttd9\" (UID: \"ca8197ec-0240-48b6-ae32-c5ad205f9d2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vttd9"
Nov 01 03:16:15.913850 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:15.913845    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca8197ec-0240-48b6-ae32-c5ad205f9d2d-istio-data\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-vttd9\" (UID: \"ca8197ec-0240-48b6-ae32-c5ad205f9d2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vttd9"
Nov 01 03:16:16.014862 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:16.014801    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca8197ec-0240-48b6-ae32-c5ad205f9d2d-istio-envoy\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-vttd9\" (UID: \"ca8197ec-0240-48b6-ae32-c5ad205f9d2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vttd9"
Nov 01 03:16:16.014862 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:16.014835    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca8197ec-0240-48b6-ae32-c5ad205f9d2d-credential-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-vttd9\" (UID: \"ca8197ec-0240-48b6-ae32-c5ad205f9d2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vttd9"
Nov 01 03:16:16.015025 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:16.014877    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca8197ec-0240-48b6-ae32-c5ad205f9d2d-workload-certs\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-vttd9\" (UID: \"ca8197ec-0240-48b6-ae32-c5ad205f9d2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vttd9"
Nov 01 03:16:16.015025 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:16.014904    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn6x2\" (UniqueName: \"kubernetes.io/projected/ca8197ec-0240-48b6-ae32-c5ad205f9d2d-kube-api-access-bn6x2\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-vttd9\" (UID: \"ca8197ec-0240-48b6-ae32-c5ad205f9d2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vttd9"
Nov 01 03:16:16.015025 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:16.014935    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca8197ec-0240-48b6-ae32-c5ad205f9d2d-istio-token\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-vttd9\" (UID: \"ca8197ec-0240-48b6-ae32-c5ad205f9d2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vttd9"
Nov 01 03:16:16.015025 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:16.014987    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca8197ec-0240-48b6-ae32-c5ad205f9d2d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-vttd9\" (UID: \"ca8197ec-0240-48b6-ae32-c5ad205f9d2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vttd9"
Nov 01 03:16:16.015228 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:16.015033    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca8197ec-0240-48b6-ae32-c5ad205f9d2d-workload-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-vttd9\" (UID: \"ca8197ec-0240-48b6-ae32-c5ad205f9d2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vttd9"
Nov 01 03:16:16.015228 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:16.015057    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca8197ec-0240-48b6-ae32-c5ad205f9d2d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-vttd9\" (UID: \"ca8197ec-0240-48b6-ae32-c5ad205f9d2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vttd9"
Nov 01 03:16:16.015228 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:16.015096    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca8197ec-0240-48b6-ae32-c5ad205f9d2d-istio-data\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-vttd9\" (UID: \"ca8197ec-0240-48b6-ae32-c5ad205f9d2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vttd9"
Nov 01 03:16:16.015379 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:16.015251    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca8197ec-0240-48b6-ae32-c5ad205f9d2d-credential-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-vttd9\" (UID: \"ca8197ec-0240-48b6-ae32-c5ad205f9d2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vttd9"
Nov 01 03:16:16.015439 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:16.015384    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca8197ec-0240-48b6-ae32-c5ad205f9d2d-workload-certs\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-vttd9\" (UID: \"ca8197ec-0240-48b6-ae32-c5ad205f9d2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vttd9"
Nov 01 03:16:16.015509 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:16.015465    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca8197ec-0240-48b6-ae32-c5ad205f9d2d-workload-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-vttd9\" (UID: \"ca8197ec-0240-48b6-ae32-c5ad205f9d2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vttd9"
Nov 01 03:16:16.015568 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:16.015519    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca8197ec-0240-48b6-ae32-c5ad205f9d2d-istio-data\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-vttd9\" (UID: \"ca8197ec-0240-48b6-ae32-c5ad205f9d2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vttd9"
Nov 01 03:16:16.016262 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:16.016237    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca8197ec-0240-48b6-ae32-c5ad205f9d2d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-vttd9\" (UID: \"ca8197ec-0240-48b6-ae32-c5ad205f9d2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vttd9"
Nov 01 03:16:16.017473 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:16.017447    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca8197ec-0240-48b6-ae32-c5ad205f9d2d-istio-envoy\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-vttd9\" (UID: \"ca8197ec-0240-48b6-ae32-c5ad205f9d2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vttd9"
Nov 01 03:16:16.017631 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:16.017614    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca8197ec-0240-48b6-ae32-c5ad205f9d2d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-vttd9\" (UID: \"ca8197ec-0240-48b6-ae32-c5ad205f9d2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vttd9"
Nov 01 03:16:16.077855 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:16.077829    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca8197ec-0240-48b6-ae32-c5ad205f9d2d-istio-token\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-vttd9\" (UID: \"ca8197ec-0240-48b6-ae32-c5ad205f9d2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vttd9"
Nov 01 03:16:16.078032 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:16.077987    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn6x2\" (UniqueName: \"kubernetes.io/projected/ca8197ec-0240-48b6-ae32-c5ad205f9d2d-kube-api-access-bn6x2\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-vttd9\" (UID: \"ca8197ec-0240-48b6-ae32-c5ad205f9d2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vttd9"
Nov 01 03:16:16.242987 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:16.242964    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88th2j"
Nov 01 03:16:16.373102 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:16.373079    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vttd9"
Nov 01 03:16:16.799835 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:16.799815    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vttd9"]
Nov 01 03:16:16.801664 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:16:16.801636    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca8197ec_0240_48b6_ae32_c5ad205f9d2d.slice/crio-1140de05843223be09dcb11dc23a1d75ee36c6f7d3e3741891e13658c46e5803 WatchSource:0}: Error finding container 1140de05843223be09dcb11dc23a1d75ee36c6f7d3e3741891e13658c46e5803: Status 404 returned error can't find the container with id 1140de05843223be09dcb11dc23a1d75ee36c6f7d3e3741891e13658c46e5803
Nov 01 03:16:17.237961 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:17.237932    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vttd9" event={"ID":"ca8197ec-0240-48b6-ae32-c5ad205f9d2d","Type":"ContainerStarted","Data":"1140de05843223be09dcb11dc23a1d75ee36c6f7d3e3741891e13658c46e5803"}
Nov 01 03:16:20.033087 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:20.033047    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 03:16:20.493499 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:20.493471    4400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 03:16:20.493592 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:20.493540    4400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 03:16:20.493592 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:20.493571    4400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 03:16:21.260445 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:21.260410    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vttd9" event={"ID":"ca8197ec-0240-48b6-ae32-c5ad205f9d2d","Type":"ContainerStarted","Data":"b637a2b17b7136b5451f57c211b80db2fad3d1c0f7fd119463e9eaca636d7de8"}
Nov 01 03:16:21.373261 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:21.373231    4400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vttd9"
Nov 01 03:16:21.374771 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:21.374747    4400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-vttd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body=
Nov 01 03:16:21.374939 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:21.374798    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vttd9" podUID="ca8197ec-0240-48b6-ae32-c5ad205f9d2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused"
Nov 01 03:16:22.373546 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:22.373514    4400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-vttd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body=
Nov 01 03:16:22.373963 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:22.373570    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vttd9" podUID="ca8197ec-0240-48b6-ae32-c5ad205f9d2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused"
Nov 01 03:16:23.373429 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:23.373400    4400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-vttd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body=
Nov 01 03:16:23.373606 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:23.373449    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vttd9" podUID="ca8197ec-0240-48b6-ae32-c5ad205f9d2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused"
Nov 01 03:16:24.374308 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:24.374273    4400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-vttd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body=
Nov 01 03:16:24.374759 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:24.374336    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vttd9" podUID="ca8197ec-0240-48b6-ae32-c5ad205f9d2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused"
Nov 01 03:16:25.373903 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:25.373870    4400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-vttd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body=
Nov 01 03:16:25.374104 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:25.373941    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vttd9" podUID="ca8197ec-0240-48b6-ae32-c5ad205f9d2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused"
Nov 01 03:16:26.374151 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:26.374116    4400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-vttd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body=
Nov 01 03:16:26.374670 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:26.374172    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vttd9" podUID="ca8197ec-0240-48b6-ae32-c5ad205f9d2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused"
Nov 01 03:16:27.374247 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:27.374205    4400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-vttd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body=
Nov 01 03:16:27.374784 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:27.374268    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vttd9" podUID="ca8197ec-0240-48b6-ae32-c5ad205f9d2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused"
Nov 01 03:16:28.664047 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:28.664019    4400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vttd9"
Nov 01 03:16:28.664496 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:28.664324    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vttd9"
Nov 01 03:16:28.665194 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:28.665174    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vttd9"
Nov 01 03:16:28.838254 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:28.838209    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vttd9" podStartSLOduration=10.149143336 podStartE2EDuration="13.838197742s" totalImagesPullingTime="3.689054406s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:16:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:16:21.48326024 +0000 UTC m=+587.997906139" watchObservedRunningTime="2026-11-01 03:16:28.838197742 +0000 UTC m=+595.352843604"
Nov 01 03:16:30.876844 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:30.876813    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vs8gs"]
Nov 01 03:16:30.880754 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:30.880734    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vs8gs"
Nov 01 03:16:30.987754 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:30.987726    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vs8gs"]
Nov 01 03:16:31.075674 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:31.075645    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c94ffb2b-3e93-4898-97e8-1de6587c38a7-istio-token\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-vs8gs\" (UID: \"c94ffb2b-3e93-4898-97e8-1de6587c38a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vs8gs"
Nov 01 03:16:31.075820 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:31.075732    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxxsr\" (UniqueName: \"kubernetes.io/projected/c94ffb2b-3e93-4898-97e8-1de6587c38a7-kube-api-access-bxxsr\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-vs8gs\" (UID: \"c94ffb2b-3e93-4898-97e8-1de6587c38a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vs8gs"
Nov 01 03:16:31.075820 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:31.075806    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c94ffb2b-3e93-4898-97e8-1de6587c38a7-credential-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-vs8gs\" (UID: \"c94ffb2b-3e93-4898-97e8-1de6587c38a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vs8gs"
Nov 01 03:16:31.075891 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:31.075832    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c94ffb2b-3e93-4898-97e8-1de6587c38a7-workload-certs\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-vs8gs\" (UID: \"c94ffb2b-3e93-4898-97e8-1de6587c38a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vs8gs"
Nov 01 03:16:31.075891 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:31.075865    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c94ffb2b-3e93-4898-97e8-1de6587c38a7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-vs8gs\" (UID: \"c94ffb2b-3e93-4898-97e8-1de6587c38a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vs8gs"
Nov 01 03:16:31.075960 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:31.075944    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c94ffb2b-3e93-4898-97e8-1de6587c38a7-istio-envoy\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-vs8gs\" (UID: \"c94ffb2b-3e93-4898-97e8-1de6587c38a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vs8gs"
Nov 01 03:16:31.075992 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:31.075981    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c94ffb2b-3e93-4898-97e8-1de6587c38a7-istio-data\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-vs8gs\" (UID: \"c94ffb2b-3e93-4898-97e8-1de6587c38a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vs8gs"
Nov 01 03:16:31.076025 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:31.076010    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c94ffb2b-3e93-4898-97e8-1de6587c38a7-workload-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-vs8gs\" (UID: \"c94ffb2b-3e93-4898-97e8-1de6587c38a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vs8gs"
Nov 01 03:16:31.076059 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:31.076031    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c94ffb2b-3e93-4898-97e8-1de6587c38a7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-vs8gs\" (UID: \"c94ffb2b-3e93-4898-97e8-1de6587c38a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vs8gs"
Nov 01 03:16:31.176530 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:31.176468    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c94ffb2b-3e93-4898-97e8-1de6587c38a7-istio-envoy\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-vs8gs\" (UID: \"c94ffb2b-3e93-4898-97e8-1de6587c38a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vs8gs"
Nov 01 03:16:31.176530 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:31.176499    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c94ffb2b-3e93-4898-97e8-1de6587c38a7-istio-data\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-vs8gs\" (UID: \"c94ffb2b-3e93-4898-97e8-1de6587c38a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vs8gs"
Nov 01 03:16:31.176530 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:31.176522    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c94ffb2b-3e93-4898-97e8-1de6587c38a7-workload-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-vs8gs\" (UID: \"c94ffb2b-3e93-4898-97e8-1de6587c38a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vs8gs"
Nov 01 03:16:31.176787 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:31.176536    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c94ffb2b-3e93-4898-97e8-1de6587c38a7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-vs8gs\" (UID: \"c94ffb2b-3e93-4898-97e8-1de6587c38a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vs8gs"
Nov 01 03:16:31.176787 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:31.176677    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c94ffb2b-3e93-4898-97e8-1de6587c38a7-istio-token\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-vs8gs\" (UID: \"c94ffb2b-3e93-4898-97e8-1de6587c38a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vs8gs"
Nov 01 03:16:31.176901 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:31.176832    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxxsr\" (UniqueName: \"kubernetes.io/projected/c94ffb2b-3e93-4898-97e8-1de6587c38a7-kube-api-access-bxxsr\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-vs8gs\" (UID: \"c94ffb2b-3e93-4898-97e8-1de6587c38a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vs8gs"
Nov 01 03:16:31.176957 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:31.176900    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c94ffb2b-3e93-4898-97e8-1de6587c38a7-credential-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-vs8gs\" (UID: \"c94ffb2b-3e93-4898-97e8-1de6587c38a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vs8gs"
Nov 01 03:16:31.176957 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:31.176928    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c94ffb2b-3e93-4898-97e8-1de6587c38a7-workload-certs\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-vs8gs\" (UID: \"c94ffb2b-3e93-4898-97e8-1de6587c38a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vs8gs"
Nov 01 03:16:31.176957 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:31.176942    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c94ffb2b-3e93-4898-97e8-1de6587c38a7-istio-data\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-vs8gs\" (UID: \"c94ffb2b-3e93-4898-97e8-1de6587c38a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vs8gs"
Nov 01 03:16:31.176957 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:31.176952    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c94ffb2b-3e93-4898-97e8-1de6587c38a7-workload-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-vs8gs\" (UID: \"c94ffb2b-3e93-4898-97e8-1de6587c38a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vs8gs"
Nov 01 03:16:31.177152 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:31.176973    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c94ffb2b-3e93-4898-97e8-1de6587c38a7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-vs8gs\" (UID: \"c94ffb2b-3e93-4898-97e8-1de6587c38a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vs8gs"
Nov 01 03:16:31.177210 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:31.177156    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c94ffb2b-3e93-4898-97e8-1de6587c38a7-workload-certs\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-vs8gs\" (UID: \"c94ffb2b-3e93-4898-97e8-1de6587c38a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vs8gs"
Nov 01 03:16:31.177293 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:31.177270    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c94ffb2b-3e93-4898-97e8-1de6587c38a7-credential-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-vs8gs\" (UID: \"c94ffb2b-3e93-4898-97e8-1de6587c38a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vs8gs"
Nov 01 03:16:31.177585 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:31.177564    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c94ffb2b-3e93-4898-97e8-1de6587c38a7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-vs8gs\" (UID: \"c94ffb2b-3e93-4898-97e8-1de6587c38a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vs8gs"
Nov 01 03:16:31.179062 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:31.179044    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c94ffb2b-3e93-4898-97e8-1de6587c38a7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-vs8gs\" (UID: \"c94ffb2b-3e93-4898-97e8-1de6587c38a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vs8gs"
Nov 01 03:16:31.179127 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:31.179069    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c94ffb2b-3e93-4898-97e8-1de6587c38a7-istio-envoy\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-vs8gs\" (UID: \"c94ffb2b-3e93-4898-97e8-1de6587c38a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vs8gs"
Nov 01 03:16:31.239188 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:31.239168    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c94ffb2b-3e93-4898-97e8-1de6587c38a7-istio-token\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-vs8gs\" (UID: \"c94ffb2b-3e93-4898-97e8-1de6587c38a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vs8gs"
Nov 01 03:16:31.239526 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:31.239506    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxxsr\" (UniqueName: \"kubernetes.io/projected/c94ffb2b-3e93-4898-97e8-1de6587c38a7-kube-api-access-bxxsr\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-vs8gs\" (UID: \"c94ffb2b-3e93-4898-97e8-1de6587c38a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vs8gs"
Nov 01 03:16:31.493751 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:31.493675    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vs8gs"
Nov 01 03:16:31.728828 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:16:31.728793    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:16:31.920040 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:31.920016    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vs8gs"]
Nov 01 03:16:31.921540 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:16:31.921510    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc94ffb2b_3e93_4898_97e8_1de6587c38a7.slice/crio-a493a4cc4bac3470c5ee404f4632607653e51f05c3e1f9b2ea0c6fda3a245441 WatchSource:0}: Error finding container a493a4cc4bac3470c5ee404f4632607653e51f05c3e1f9b2ea0c6fda3a245441: Status 404 returned error can't find the container with id a493a4cc4bac3470c5ee404f4632607653e51f05c3e1f9b2ea0c6fda3a245441
Nov 01 03:16:31.924472 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:31.924440    4400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 03:16:31.924570 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:31.924504    4400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 03:16:31.924570 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:31.924533    4400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 03:16:32.315113 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:32.315028    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vs8gs" event={"ID":"c94ffb2b-3e93-4898-97e8-1de6587c38a7","Type":"ContainerStarted","Data":"6f9527523882d778e6155135752f0e21ddccb5ac129164e1a3b9ed20a392c0c3"}
Nov 01 03:16:32.315113 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:32.315085    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vs8gs" event={"ID":"c94ffb2b-3e93-4898-97e8-1de6587c38a7","Type":"ContainerStarted","Data":"a493a4cc4bac3470c5ee404f4632607653e51f05c3e1f9b2ea0c6fda3a245441"}
Nov 01 03:16:32.492842 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:32.492789    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vs8gs" podStartSLOduration=2.49277381 podStartE2EDuration="2.49277381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:16:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:16:32.491155403 +0000 UTC m=+599.005801268" watchObservedRunningTime="2026-11-01 03:16:32.49277381 +0000 UTC m=+599.007419761"
Nov 01 03:16:32.494746 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:32.494725    4400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vs8gs"
Nov 01 03:16:32.496168 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:32.496145    4400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-vs8gs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body=
Nov 01 03:16:32.496266 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:32.496189    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vs8gs" podUID="c94ffb2b-3e93-4898-97e8-1de6587c38a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused"
Nov 01 03:16:33.494287 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:33.494251    4400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-vs8gs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body=
Nov 01 03:16:33.494699 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:33.494301    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vs8gs" podUID="c94ffb2b-3e93-4898-97e8-1de6587c38a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused"
Nov 01 03:16:33.991324 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:33.991286    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-5f6dd4f69f-zv4pl"]
Nov 01 03:16:33.995298 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:33.995276    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-zv4pl"
Nov 01 03:16:34.050122 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:34.050092    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\""
Nov 01 03:16:34.123706 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:34.104426    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5f6dd4f69f-zv4pl"]
Nov 01 03:16:34.123706 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:34.115395    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99765938-e61b-45a7-abc9-13d109dfc4e2-metrics-certs\") pod \"ai-gateway-controller-5f6dd4f69f-zv4pl\" (UID: \"99765938-e61b-45a7-abc9-13d109dfc4e2\") " pod="opendatahub/ai-gateway-controller-5f6dd4f69f-zv4pl"
Nov 01 03:16:34.123706 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:34.115506    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9mn5\" (UniqueName: \"kubernetes.io/projected/99765938-e61b-45a7-abc9-13d109dfc4e2-kube-api-access-p9mn5\") pod \"ai-gateway-controller-5f6dd4f69f-zv4pl\" (UID: \"99765938-e61b-45a7-abc9-13d109dfc4e2\") " pod="opendatahub/ai-gateway-controller-5f6dd4f69f-zv4pl"
Nov 01 03:16:34.123706 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:34.116223    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-d4chm\""
Nov 01 03:16:34.133795 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:34.133760    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Nov 01 03:16:34.222286 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:34.222248    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99765938-e61b-45a7-abc9-13d109dfc4e2-metrics-certs\") pod \"ai-gateway-controller-5f6dd4f69f-zv4pl\" (UID: \"99765938-e61b-45a7-abc9-13d109dfc4e2\") " pod="opendatahub/ai-gateway-controller-5f6dd4f69f-zv4pl"
Nov 01 03:16:34.223092 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:34.223068    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9mn5\" (UniqueName: \"kubernetes.io/projected/99765938-e61b-45a7-abc9-13d109dfc4e2-kube-api-access-p9mn5\") pod \"ai-gateway-controller-5f6dd4f69f-zv4pl\" (UID: \"99765938-e61b-45a7-abc9-13d109dfc4e2\") " pod="opendatahub/ai-gateway-controller-5f6dd4f69f-zv4pl"
Nov 01 03:16:34.236917 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:34.236896    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99765938-e61b-45a7-abc9-13d109dfc4e2-metrics-certs\") pod \"ai-gateway-controller-5f6dd4f69f-zv4pl\" (UID: \"99765938-e61b-45a7-abc9-13d109dfc4e2\") " pod="opendatahub/ai-gateway-controller-5f6dd4f69f-zv4pl"
Nov 01 03:16:34.286081 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:34.286015    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9mn5\" (UniqueName: \"kubernetes.io/projected/99765938-e61b-45a7-abc9-13d109dfc4e2-kube-api-access-p9mn5\") pod \"ai-gateway-controller-5f6dd4f69f-zv4pl\" (UID: \"99765938-e61b-45a7-abc9-13d109dfc4e2\") " pod="opendatahub/ai-gateway-controller-5f6dd4f69f-zv4pl"
Nov 01 03:16:34.340525 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:34.340495    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-zv4pl"
Nov 01 03:16:34.494988 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:34.494951    4400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-vs8gs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body=
Nov 01 03:16:34.495626 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:34.495010    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vs8gs" podUID="c94ffb2b-3e93-4898-97e8-1de6587c38a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused"
Nov 01 03:16:34.746528 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:34.746500    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5f6dd4f69f-zv4pl"]
Nov 01 03:16:34.747722 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:16:34.747655    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99765938_e61b_45a7_abc9_13d109dfc4e2.slice/crio-c4625cfc1acf7e6636e806df8a5351bb23b179bd055405f99bb994294934f236 WatchSource:0}: Error finding container c4625cfc1acf7e6636e806df8a5351bb23b179bd055405f99bb994294934f236: Status 404 returned error can't find the container with id c4625cfc1acf7e6636e806df8a5351bb23b179bd055405f99bb994294934f236
Nov 01 03:16:35.340864 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:35.340828    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-zv4pl" event={"ID":"99765938-e61b-45a7-abc9-13d109dfc4e2","Type":"ContainerStarted","Data":"c4625cfc1acf7e6636e806df8a5351bb23b179bd055405f99bb994294934f236"}
Nov 01 03:16:35.494843 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:35.494813    4400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-vs8gs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body=
Nov 01 03:16:35.495008 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:35.494865    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vs8gs" podUID="c94ffb2b-3e93-4898-97e8-1de6587c38a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused"
Nov 01 03:16:36.494268 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:36.494231    4400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-vs8gs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body=
Nov 01 03:16:36.494436 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:36.494282    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vs8gs" podUID="c94ffb2b-3e93-4898-97e8-1de6587c38a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused"
Nov 01 03:16:36.899887 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:36.899854    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6bd866bf75-9b6xv"]
Nov 01 03:16:36.900330 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:36.900025    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6bd866bf75-9b6xv" podUID="94f2b7e0-6308-4e0a-9637-958276c5c3aa" containerName="authorino" containerID="cri-o://ac3a53cc60a14651b71d7294eed489124f07c749afaad588821475b223b85bd5" gracePeriod=30
Nov 01 03:16:37.232427 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:37.232399    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6bd866bf75-9b6xv"
Nov 01 03:16:37.257237 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:37.257212    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94f2b7e0-6308-4e0a-9637-958276c5c3aa-tls-cert\" (UniqueName: \"kubernetes.io/secret/94f2b7e0-6308-4e0a-9637-958276c5c3aa-tls-cert\") pod \"94f2b7e0-6308-4e0a-9637-958276c5c3aa\" (UID: \"94f2b7e0-6308-4e0a-9637-958276c5c3aa\") "
Nov 01 03:16:37.257361 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:37.257319    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94f2b7e0-6308-4e0a-9637-958276c5c3aa-kube-api-access-vf8gv\" (UniqueName: \"kubernetes.io/projected/94f2b7e0-6308-4e0a-9637-958276c5c3aa-kube-api-access-vf8gv\") pod \"94f2b7e0-6308-4e0a-9637-958276c5c3aa\" (UID: \"94f2b7e0-6308-4e0a-9637-958276c5c3aa\") "
Nov 01 03:16:37.259406 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:37.259371    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94f2b7e0-6308-4e0a-9637-958276c5c3aa-kube-api-access-vf8gv" pod "94f2b7e0-6308-4e0a-9637-958276c5c3aa" (UID: "94f2b7e0-6308-4e0a-9637-958276c5c3aa"). InnerVolumeSpecName "kube-api-access-vf8gv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:16:37.268083 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:37.268061    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94f2b7e0-6308-4e0a-9637-958276c5c3aa-tls-cert" pod "94f2b7e0-6308-4e0a-9637-958276c5c3aa" (UID: "94f2b7e0-6308-4e0a-9637-958276c5c3aa"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:16:37.351470 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:37.351440    4400 generic.go:356] "Generic (PLEG): container finished" podID="94f2b7e0-6308-4e0a-9637-958276c5c3aa" containerID="ac3a53cc60a14651b71d7294eed489124f07c749afaad588821475b223b85bd5" exitCode=0
Nov 01 03:16:37.351618 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:37.351550    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6bd866bf75-9b6xv" event={"ID":"94f2b7e0-6308-4e0a-9637-958276c5c3aa","Type":"ContainerDied","Data":"ac3a53cc60a14651b71d7294eed489124f07c749afaad588821475b223b85bd5"}
Nov 01 03:16:37.351618 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:37.351577    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6bd866bf75-9b6xv" event={"ID":"94f2b7e0-6308-4e0a-9637-958276c5c3aa","Type":"ContainerDied","Data":"9b26fa17f2f1b9301881b2bb46ee23ec8acde3fd1de649ebf7818db729372a0a"}
Nov 01 03:16:37.351618 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:37.351593    4400 scope.go:122] "RemoveContainer" containerID="ac3a53cc60a14651b71d7294eed489124f07c749afaad588821475b223b85bd5"
Nov 01 03:16:37.351764 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:37.351754    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6bd866bf75-9b6xv"
Nov 01 03:16:37.359712 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:37.359643    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vf8gv\" (UniqueName: \"kubernetes.io/projected/94f2b7e0-6308-4e0a-9637-958276c5c3aa-kube-api-access-vf8gv\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:16:37.359712 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:37.359673    4400 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/94f2b7e0-6308-4e0a-9637-958276c5c3aa-tls-cert\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:16:37.363062 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:37.363039    4400 scope.go:122] "RemoveContainer" containerID="ac3a53cc60a14651b71d7294eed489124f07c749afaad588821475b223b85bd5"
Nov 01 03:16:37.363338 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:16:37.363318    4400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac3a53cc60a14651b71d7294eed489124f07c749afaad588821475b223b85bd5\": container with ID starting with ac3a53cc60a14651b71d7294eed489124f07c749afaad588821475b223b85bd5 not found: ID does not exist" containerID="ac3a53cc60a14651b71d7294eed489124f07c749afaad588821475b223b85bd5"
Nov 01 03:16:37.363407 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:37.363347    4400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac3a53cc60a14651b71d7294eed489124f07c749afaad588821475b223b85bd5"} err="failed to get container status \"ac3a53cc60a14651b71d7294eed489124f07c749afaad588821475b223b85bd5\": rpc error: code = NotFound desc = could not find container \"ac3a53cc60a14651b71d7294eed489124f07c749afaad588821475b223b85bd5\": container with ID starting with ac3a53cc60a14651b71d7294eed489124f07c749afaad588821475b223b85bd5 not found: ID does not exist"
Nov 01 03:16:37.494233 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:37.494161    4400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-vs8gs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body=
Nov 01 03:16:37.494233 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:37.494212    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vs8gs" podUID="c94ffb2b-3e93-4898-97e8-1de6587c38a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused"
Nov 01 03:16:37.541097 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:37.541066    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6bd866bf75-9b6xv"]
Nov 01 03:16:37.592160 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:37.592133    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6bd866bf75-9b6xv"]
Nov 01 03:16:37.630582 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:37.630554    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94f2b7e0-6308-4e0a-9637-958276c5c3aa" path="/var/lib/kubelet/pods/94f2b7e0-6308-4e0a-9637-958276c5c3aa/volumes"
Nov 01 03:16:38.494623 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:38.494591    4400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-vs8gs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body=
Nov 01 03:16:38.495025 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:38.494653    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vs8gs" podUID="c94ffb2b-3e93-4898-97e8-1de6587c38a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused"
Nov 01 03:16:40.222727 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:40.222669    4400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vs8gs"
Nov 01 03:16:40.223388 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:40.223316    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vs8gs"
Nov 01 03:16:40.224030 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:40.224012    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vs8gs"
Nov 01 03:16:41.375853 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:41.375817    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-zv4pl" event={"ID":"99765938-e61b-45a7-abc9-13d109dfc4e2","Type":"ContainerStarted","Data":"e7887c2fd850cfad3845a1aaa0a7201567f128b2691323ab53989c708d98f840"}
Nov 01 03:16:41.376258 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:41.375995    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-zv4pl"
Nov 01 03:16:41.497002 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:41.496958    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-zv4pl" podStartSLOduration=2.486415201 podStartE2EDuration="8.496945787s" totalImagesPullingTime="6.010530586s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:16:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:16:41.496521227 +0000 UTC m=+608.011167105" watchObservedRunningTime="2026-11-01 03:16:41.496945787 +0000 UTC m=+608.011591649"
Nov 01 03:16:43.876869 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:43.876838    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-z8nnz"]
Nov 01 03:16:43.877264 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:43.877251    4400 cpu_manager.go:414] "containerMap: removing container" podUID="94f2b7e0-6308-4e0a-9637-958276c5c3aa" containerName="authorino"
Nov 01 03:16:43.877304 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:43.877265    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94f2b7e0-6308-4e0a-9637-958276c5c3aa" containerName="authorino"
Nov 01 03:16:43.877352 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:43.877343    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="94f2b7e0-6308-4e0a-9637-958276c5c3aa" containerName="authorino"
Nov 01 03:16:43.880542 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:43.880527    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-z8nnz"
Nov 01 03:16:43.921427 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:43.921399    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c892w\" (UniqueName: \"kubernetes.io/projected/555aa9f6-e7e9-4e90-b3a3-1749c88e35ac-kube-api-access-c892w\") pod \"payload-pre-processing-7b98bb678c-z8nnz\" (UID: \"555aa9f6-e7e9-4e90-b3a3-1749c88e35ac\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-z8nnz"
Nov 01 03:16:43.921542 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:43.921512    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/555aa9f6-e7e9-4e90-b3a3-1749c88e35ac-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-z8nnz\" (UID: \"555aa9f6-e7e9-4e90-b3a3-1749c88e35ac\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-z8nnz"
Nov 01 03:16:43.938510 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:43.938487    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-z8nnz"]
Nov 01 03:16:43.986641 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:43.986602    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Nov 01 03:16:43.986832 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:43.986736    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-25cp9\""
Nov 01 03:16:44.022695 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:44.022662    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/555aa9f6-e7e9-4e90-b3a3-1749c88e35ac-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-z8nnz\" (UID: \"555aa9f6-e7e9-4e90-b3a3-1749c88e35ac\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-z8nnz"
Nov 01 03:16:44.022859 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:44.022787    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c892w\" (UniqueName: \"kubernetes.io/projected/555aa9f6-e7e9-4e90-b3a3-1749c88e35ac-kube-api-access-c892w\") pod \"payload-pre-processing-7b98bb678c-z8nnz\" (UID: \"555aa9f6-e7e9-4e90-b3a3-1749c88e35ac\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-z8nnz"
Nov 01 03:16:44.023564 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:44.023536    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/555aa9f6-e7e9-4e90-b3a3-1749c88e35ac-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-z8nnz\" (UID: \"555aa9f6-e7e9-4e90-b3a3-1749c88e35ac\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-z8nnz"
Nov 01 03:16:44.041154 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:44.041120    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-26mft"]
Nov 01 03:16:44.045604 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:44.045583    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-26mft"
Nov 01 03:16:44.085514 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:44.085477    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c892w\" (UniqueName: \"kubernetes.io/projected/555aa9f6-e7e9-4e90-b3a3-1749c88e35ac-kube-api-access-c892w\") pod \"payload-pre-processing-7b98bb678c-z8nnz\" (UID: \"555aa9f6-e7e9-4e90-b3a3-1749c88e35ac\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-z8nnz"
Nov 01 03:16:44.097874 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:44.097797    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-26mft"]
Nov 01 03:16:44.124231 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:44.124197    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ffcb7e5-2dfc-4b83-ab39-85bbb773329f-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-26mft\" (UID: \"3ffcb7e5-2dfc-4b83-ab39-85bbb773329f\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-26mft"
Nov 01 03:16:44.124395 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:44.124351    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cks4\" (UniqueName: \"kubernetes.io/projected/3ffcb7e5-2dfc-4b83-ab39-85bbb773329f-kube-api-access-2cks4\") pod \"payload-processing-55fbbbf9dc-26mft\" (UID: \"3ffcb7e5-2dfc-4b83-ab39-85bbb773329f\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-26mft"
Nov 01 03:16:44.193903 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:44.193824    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-z8nnz"
Nov 01 03:16:44.227017 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:44.226987    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cks4\" (UniqueName: \"kubernetes.io/projected/3ffcb7e5-2dfc-4b83-ab39-85bbb773329f-kube-api-access-2cks4\") pod \"payload-processing-55fbbbf9dc-26mft\" (UID: \"3ffcb7e5-2dfc-4b83-ab39-85bbb773329f\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-26mft"
Nov 01 03:16:44.227158 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:44.227139    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ffcb7e5-2dfc-4b83-ab39-85bbb773329f-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-26mft\" (UID: \"3ffcb7e5-2dfc-4b83-ab39-85bbb773329f\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-26mft"
Nov 01 03:16:44.228076 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:44.228050    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ffcb7e5-2dfc-4b83-ab39-85bbb773329f-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-26mft\" (UID: \"3ffcb7e5-2dfc-4b83-ab39-85bbb773329f\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-26mft"
Nov 01 03:16:44.292039 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:44.291936    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cks4\" (UniqueName: \"kubernetes.io/projected/3ffcb7e5-2dfc-4b83-ab39-85bbb773329f-kube-api-access-2cks4\") pod \"payload-processing-55fbbbf9dc-26mft\" (UID: \"3ffcb7e5-2dfc-4b83-ab39-85bbb773329f\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-26mft"
Nov 01 03:16:44.362141 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:44.362104    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-26mft"
Nov 01 03:16:44.572759 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:44.572731    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-z8nnz"]
Nov 01 03:16:44.575071 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:16:44.575046    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod555aa9f6_e7e9_4e90_b3a3_1749c88e35ac.slice/crio-9b676b4902bc076492da36750121d9e2bee8d7be816cba27184ac4bccfe6128d WatchSource:0}: Error finding container 9b676b4902bc076492da36750121d9e2bee8d7be816cba27184ac4bccfe6128d: Status 404 returned error can't find the container with id 9b676b4902bc076492da36750121d9e2bee8d7be816cba27184ac4bccfe6128d
Nov 01 03:16:44.748625 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:44.748603    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-26mft"]
Nov 01 03:16:44.750853 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:16:44.750825    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ffcb7e5_2dfc_4b83_ab39_85bbb773329f.slice/crio-6ed140e82503878acf5331f6fef693e5c4d0ab3953fca93f39fb775d0276f555 WatchSource:0}: Error finding container 6ed140e82503878acf5331f6fef693e5c4d0ab3953fca93f39fb775d0276f555: Status 404 returned error can't find the container with id 6ed140e82503878acf5331f6fef693e5c4d0ab3953fca93f39fb775d0276f555
Nov 01 03:16:45.397225 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:45.397185    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-z8nnz" event={"ID":"555aa9f6-e7e9-4e90-b3a3-1749c88e35ac","Type":"ContainerStarted","Data":"9b676b4902bc076492da36750121d9e2bee8d7be816cba27184ac4bccfe6128d"}
Nov 01 03:16:45.398359 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:45.398333    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-26mft" event={"ID":"3ffcb7e5-2dfc-4b83-ab39-85bbb773329f","Type":"ContainerStarted","Data":"6ed140e82503878acf5331f6fef693e5c4d0ab3953fca93f39fb775d0276f555"}
Nov 01 03:16:48.418087 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:48.418054    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-z8nnz" event={"ID":"555aa9f6-e7e9-4e90-b3a3-1749c88e35ac","Type":"ContainerStarted","Data":"b2427766b83b1c8010bde80ba981e74170d42d67facb287a3e534081e720b3a7"}
Nov 01 03:16:48.418512 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:48.418244    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-z8nnz"
Nov 01 03:16:48.419526 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:48.419505    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-26mft" event={"ID":"3ffcb7e5-2dfc-4b83-ab39-85bbb773329f","Type":"ContainerStarted","Data":"5832a7b39b407e760adc7487521171ff42034c4edd438e330d71153ff28a8113"}
Nov 01 03:16:48.419651 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:48.419624    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-26mft"
Nov 01 03:16:48.537624 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:48.537582    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-z8nnz" podStartSLOduration=2.117835529 podStartE2EDuration="5.53757137s" totalImagesPullingTime="3.419735841s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:16:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:16:48.535762415 +0000 UTC m=+615.050408277" watchObservedRunningTime="2026-11-01 03:16:48.53757137 +0000 UTC m=+615.052217233"
Nov 01 03:16:48.656456 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:48.656393    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-26mft" podStartSLOduration=2.445784066 podStartE2EDuration="5.656373571s" totalImagesPullingTime="3.210589505s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:16:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:16:48.654060373 +0000 UTC m=+615.168706283" watchObservedRunningTime="2026-11-01 03:16:48.656373571 +0000 UTC m=+615.171019468"
Nov 01 03:16:52.382484 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:52.382455    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-zv4pl"
Nov 01 03:16:52.742664 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:52.742580    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-67978ff98b-l5wnn"]
Nov 01 03:16:52.746801 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:52.746780    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-67978ff98b-l5wnn"
Nov 01 03:16:52.804225 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:52.804194    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-67978ff98b-l5wnn"]
Nov 01 03:16:52.815308 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:52.815286    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a86d4816-aeb1-4bbf-8b17-07a0ff269577-metrics-certs\") pod \"ai-gateway-controller-67978ff98b-l5wnn\" (UID: \"a86d4816-aeb1-4bbf-8b17-07a0ff269577\") " pod="opendatahub/ai-gateway-controller-67978ff98b-l5wnn"
Nov 01 03:16:52.815441 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:52.815413    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5g6g6\" (UniqueName: \"kubernetes.io/projected/a86d4816-aeb1-4bbf-8b17-07a0ff269577-kube-api-access-5g6g6\") pod \"ai-gateway-controller-67978ff98b-l5wnn\" (UID: \"a86d4816-aeb1-4bbf-8b17-07a0ff269577\") " pod="opendatahub/ai-gateway-controller-67978ff98b-l5wnn"
Nov 01 03:16:52.916487 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:52.916464    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5g6g6\" (UniqueName: \"kubernetes.io/projected/a86d4816-aeb1-4bbf-8b17-07a0ff269577-kube-api-access-5g6g6\") pod \"ai-gateway-controller-67978ff98b-l5wnn\" (UID: \"a86d4816-aeb1-4bbf-8b17-07a0ff269577\") " pod="opendatahub/ai-gateway-controller-67978ff98b-l5wnn"
Nov 01 03:16:52.916601 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:52.916514    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a86d4816-aeb1-4bbf-8b17-07a0ff269577-metrics-certs\") pod \"ai-gateway-controller-67978ff98b-l5wnn\" (UID: \"a86d4816-aeb1-4bbf-8b17-07a0ff269577\") " pod="opendatahub/ai-gateway-controller-67978ff98b-l5wnn"
Nov 01 03:16:52.919048 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:52.919024    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a86d4816-aeb1-4bbf-8b17-07a0ff269577-metrics-certs\") pod \"ai-gateway-controller-67978ff98b-l5wnn\" (UID: \"a86d4816-aeb1-4bbf-8b17-07a0ff269577\") " pod="opendatahub/ai-gateway-controller-67978ff98b-l5wnn"
Nov 01 03:16:52.981661 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:52.981633    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5g6g6\" (UniqueName: \"kubernetes.io/projected/a86d4816-aeb1-4bbf-8b17-07a0ff269577-kube-api-access-5g6g6\") pod \"ai-gateway-controller-67978ff98b-l5wnn\" (UID: \"a86d4816-aeb1-4bbf-8b17-07a0ff269577\") " pod="opendatahub/ai-gateway-controller-67978ff98b-l5wnn"
Nov 01 03:16:53.059151 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:53.059095    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-67978ff98b-l5wnn"
Nov 01 03:16:53.441072 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:53.439992    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-67978ff98b-l5wnn"]
Nov 01 03:16:53.444527 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:16:53.444486    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda86d4816_aeb1_4bbf_8b17_07a0ff269577.slice/crio-ccbd63e9b8e9fbfd1a91ccf5dfdd76def63ca70a1e3ed161145c512e561d8bd7 WatchSource:0}: Error finding container ccbd63e9b8e9fbfd1a91ccf5dfdd76def63ca70a1e3ed161145c512e561d8bd7: Status 404 returned error can't find the container with id ccbd63e9b8e9fbfd1a91ccf5dfdd76def63ca70a1e3ed161145c512e561d8bd7
Nov 01 03:16:53.445792 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:53.445773    4400 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 03:16:54.451764 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:54.451730    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-67978ff98b-l5wnn" event={"ID":"a86d4816-aeb1-4bbf-8b17-07a0ff269577","Type":"ContainerStarted","Data":"e206b7b33f4c87cca3c577c2cb3a118ceb587b4b840c84c240de0023d5418b33"}
Nov 01 03:16:54.452138 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:54.451779    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-67978ff98b-l5wnn" event={"ID":"a86d4816-aeb1-4bbf-8b17-07a0ff269577","Type":"ContainerStarted","Data":"ccbd63e9b8e9fbfd1a91ccf5dfdd76def63ca70a1e3ed161145c512e561d8bd7"}
Nov 01 03:16:54.452138 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:54.451881    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-67978ff98b-l5wnn"
Nov 01 03:16:54.573259 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:54.573210    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-67978ff98b-l5wnn" podStartSLOduration=1.7476078149999998 podStartE2EDuration="2.573196338s" totalImagesPullingTime="825.588523ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:16:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:16:54.572167987 +0000 UTC m=+621.086813851" watchObservedRunningTime="2026-11-01 03:16:54.573196338 +0000 UTC m=+621.087842200"
Nov 01 03:16:59.428503 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:59.428475    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-z8nnz"
Nov 01 03:16:59.429117 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:16:59.428554    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-26mft"
Nov 01 03:17:01.729786 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:17:01.729751    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:17:05.459094 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:05.459064    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-67978ff98b-l5wnn"
Nov 01 03:17:05.631500 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:05.631473    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5f6dd4f69f-zv4pl"]
Nov 01 03:17:05.631671 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:05.631652    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-zv4pl" podUID="99765938-e61b-45a7-abc9-13d109dfc4e2" containerName="manager" containerID="cri-o://e7887c2fd850cfad3845a1aaa0a7201567f128b2691323ab53989c708d98f840" gracePeriod=30
Nov 01 03:17:05.883605 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:05.883580    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-zv4pl"
Nov 01 03:17:06.030583 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:06.030520    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99765938-e61b-45a7-abc9-13d109dfc4e2-kube-api-access-p9mn5\" (UniqueName: \"kubernetes.io/projected/99765938-e61b-45a7-abc9-13d109dfc4e2-kube-api-access-p9mn5\") pod \"99765938-e61b-45a7-abc9-13d109dfc4e2\" (UID: \"99765938-e61b-45a7-abc9-13d109dfc4e2\") "
Nov 01 03:17:06.030719 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:06.030652    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99765938-e61b-45a7-abc9-13d109dfc4e2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/99765938-e61b-45a7-abc9-13d109dfc4e2-metrics-certs\") pod \"99765938-e61b-45a7-abc9-13d109dfc4e2\" (UID: \"99765938-e61b-45a7-abc9-13d109dfc4e2\") "
Nov 01 03:17:06.032771 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:06.032745    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99765938-e61b-45a7-abc9-13d109dfc4e2-kube-api-access-p9mn5" pod "99765938-e61b-45a7-abc9-13d109dfc4e2" (UID: "99765938-e61b-45a7-abc9-13d109dfc4e2"). InnerVolumeSpecName "kube-api-access-p9mn5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:17:06.032872 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:06.032862    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99765938-e61b-45a7-abc9-13d109dfc4e2-metrics-certs" pod "99765938-e61b-45a7-abc9-13d109dfc4e2" (UID: "99765938-e61b-45a7-abc9-13d109dfc4e2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:17:06.131848 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:06.131818    4400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99765938-e61b-45a7-abc9-13d109dfc4e2-metrics-certs\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:17:06.131848 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:06.131846    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9mn5\" (UniqueName: \"kubernetes.io/projected/99765938-e61b-45a7-abc9-13d109dfc4e2-kube-api-access-p9mn5\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:17:06.509694 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:06.509659    4400 generic.go:356] "Generic (PLEG): container finished" podID="99765938-e61b-45a7-abc9-13d109dfc4e2" containerID="e7887c2fd850cfad3845a1aaa0a7201567f128b2691323ab53989c708d98f840" exitCode=0
Nov 01 03:17:06.510054 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:06.509718    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-zv4pl"
Nov 01 03:17:06.510054 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:06.509748    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-zv4pl" event={"ID":"99765938-e61b-45a7-abc9-13d109dfc4e2","Type":"ContainerDied","Data":"e7887c2fd850cfad3845a1aaa0a7201567f128b2691323ab53989c708d98f840"}
Nov 01 03:17:06.510054 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:06.509802    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-zv4pl" event={"ID":"99765938-e61b-45a7-abc9-13d109dfc4e2","Type":"ContainerDied","Data":"c4625cfc1acf7e6636e806df8a5351bb23b179bd055405f99bb994294934f236"}
Nov 01 03:17:06.510054 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:06.509827    4400 scope.go:122] "RemoveContainer" containerID="e7887c2fd850cfad3845a1aaa0a7201567f128b2691323ab53989c708d98f840"
Nov 01 03:17:06.518627 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:06.518606    4400 scope.go:122] "RemoveContainer" containerID="e7887c2fd850cfad3845a1aaa0a7201567f128b2691323ab53989c708d98f840"
Nov 01 03:17:06.518957 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:17:06.518932    4400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7887c2fd850cfad3845a1aaa0a7201567f128b2691323ab53989c708d98f840\": container with ID starting with e7887c2fd850cfad3845a1aaa0a7201567f128b2691323ab53989c708d98f840 not found: ID does not exist" containerID="e7887c2fd850cfad3845a1aaa0a7201567f128b2691323ab53989c708d98f840"
Nov 01 03:17:06.519020 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:06.518968    4400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7887c2fd850cfad3845a1aaa0a7201567f128b2691323ab53989c708d98f840"} err="failed to get container status \"e7887c2fd850cfad3845a1aaa0a7201567f128b2691323ab53989c708d98f840\": rpc error: code = NotFound desc = could not find container \"e7887c2fd850cfad3845a1aaa0a7201567f128b2691323ab53989c708d98f840\": container with ID starting with e7887c2fd850cfad3845a1aaa0a7201567f128b2691323ab53989c708d98f840 not found: ID does not exist"
Nov 01 03:17:06.677985 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:06.677960    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5f6dd4f69f-zv4pl"]
Nov 01 03:17:06.729153 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:06.729122    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-5f6dd4f69f-zv4pl"]
Nov 01 03:17:07.612884 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:07.612854    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99765938-e61b-45a7-abc9-13d109dfc4e2" path="/var/lib/kubelet/pods/99765938-e61b-45a7-abc9-13d109dfc4e2/volumes"
Nov 01 03:17:16.249860 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:16.249828    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zkjpt"]
Nov 01 03:17:16.250312 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:16.250299    4400 cpu_manager.go:414] "containerMap: removing container" podUID="99765938-e61b-45a7-abc9-13d109dfc4e2" containerName="manager"
Nov 01 03:17:16.250366 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:16.250314    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99765938-e61b-45a7-abc9-13d109dfc4e2" containerName="manager"
Nov 01 03:17:16.250405 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:16.250383    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="99765938-e61b-45a7-abc9-13d109dfc4e2" containerName="manager"
Nov 01 03:17:16.256006 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:16.255986    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zkjpt"
Nov 01 03:17:16.424980 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:16.424944    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7204663c-ef51-407b-8b56-65e9e54a2d51-istio-token\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-zkjpt\" (UID: \"7204663c-ef51-407b-8b56-65e9e54a2d51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zkjpt"
Nov 01 03:17:16.425126 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:16.424986    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7204663c-ef51-407b-8b56-65e9e54a2d51-workload-certs\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-zkjpt\" (UID: \"7204663c-ef51-407b-8b56-65e9e54a2d51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zkjpt"
Nov 01 03:17:16.425126 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:16.425007    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7204663c-ef51-407b-8b56-65e9e54a2d51-istio-data\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-zkjpt\" (UID: \"7204663c-ef51-407b-8b56-65e9e54a2d51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zkjpt"
Nov 01 03:17:16.425126 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:16.425099    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7204663c-ef51-407b-8b56-65e9e54a2d51-workload-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-zkjpt\" (UID: \"7204663c-ef51-407b-8b56-65e9e54a2d51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zkjpt"
Nov 01 03:17:16.425264 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:16.425160    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7204663c-ef51-407b-8b56-65e9e54a2d51-istio-envoy\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-zkjpt\" (UID: \"7204663c-ef51-407b-8b56-65e9e54a2d51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zkjpt"
Nov 01 03:17:16.425264 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:16.425177    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7204663c-ef51-407b-8b56-65e9e54a2d51-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-zkjpt\" (UID: \"7204663c-ef51-407b-8b56-65e9e54a2d51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zkjpt"
Nov 01 03:17:16.425264 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:16.425198    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7204663c-ef51-407b-8b56-65e9e54a2d51-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-zkjpt\" (UID: \"7204663c-ef51-407b-8b56-65e9e54a2d51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zkjpt"
Nov 01 03:17:16.425370 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:16.425306    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6w8zf\" (UniqueName: \"kubernetes.io/projected/7204663c-ef51-407b-8b56-65e9e54a2d51-kube-api-access-6w8zf\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-zkjpt\" (UID: \"7204663c-ef51-407b-8b56-65e9e54a2d51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zkjpt"
Nov 01 03:17:16.425423 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:16.425395    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7204663c-ef51-407b-8b56-65e9e54a2d51-credential-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-zkjpt\" (UID: \"7204663c-ef51-407b-8b56-65e9e54a2d51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zkjpt"
Nov 01 03:17:16.466203 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:16.466180    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zkjpt"]
Nov 01 03:17:16.526676 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:16.526624    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7204663c-ef51-407b-8b56-65e9e54a2d51-istio-envoy\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-zkjpt\" (UID: \"7204663c-ef51-407b-8b56-65e9e54a2d51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zkjpt"
Nov 01 03:17:16.526676 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:16.526649    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7204663c-ef51-407b-8b56-65e9e54a2d51-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-zkjpt\" (UID: \"7204663c-ef51-407b-8b56-65e9e54a2d51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zkjpt"
Nov 01 03:17:16.526676 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:16.526671    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7204663c-ef51-407b-8b56-65e9e54a2d51-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-zkjpt\" (UID: \"7204663c-ef51-407b-8b56-65e9e54a2d51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zkjpt"
Nov 01 03:17:16.526879 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:16.526712    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6w8zf\" (UniqueName: \"kubernetes.io/projected/7204663c-ef51-407b-8b56-65e9e54a2d51-kube-api-access-6w8zf\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-zkjpt\" (UID: \"7204663c-ef51-407b-8b56-65e9e54a2d51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zkjpt"
Nov 01 03:17:16.526879 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:16.526749    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7204663c-ef51-407b-8b56-65e9e54a2d51-credential-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-zkjpt\" (UID: \"7204663c-ef51-407b-8b56-65e9e54a2d51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zkjpt"
Nov 01 03:17:16.526879 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:16.526787    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7204663c-ef51-407b-8b56-65e9e54a2d51-istio-token\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-zkjpt\" (UID: \"7204663c-ef51-407b-8b56-65e9e54a2d51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zkjpt"
Nov 01 03:17:16.526879 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:16.526806    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7204663c-ef51-407b-8b56-65e9e54a2d51-workload-certs\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-zkjpt\" (UID: \"7204663c-ef51-407b-8b56-65e9e54a2d51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zkjpt"
Nov 01 03:17:16.526879 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:16.526826    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7204663c-ef51-407b-8b56-65e9e54a2d51-istio-data\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-zkjpt\" (UID: \"7204663c-ef51-407b-8b56-65e9e54a2d51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zkjpt"
Nov 01 03:17:16.526879 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:16.526861    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7204663c-ef51-407b-8b56-65e9e54a2d51-workload-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-zkjpt\" (UID: \"7204663c-ef51-407b-8b56-65e9e54a2d51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zkjpt"
Nov 01 03:17:16.527184 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:16.527162    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7204663c-ef51-407b-8b56-65e9e54a2d51-credential-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-zkjpt\" (UID: \"7204663c-ef51-407b-8b56-65e9e54a2d51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zkjpt"
Nov 01 03:17:16.527295 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:16.527252    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7204663c-ef51-407b-8b56-65e9e54a2d51-workload-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-zkjpt\" (UID: \"7204663c-ef51-407b-8b56-65e9e54a2d51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zkjpt"
Nov 01 03:17:16.527351 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:16.527299    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7204663c-ef51-407b-8b56-65e9e54a2d51-workload-certs\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-zkjpt\" (UID: \"7204663c-ef51-407b-8b56-65e9e54a2d51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zkjpt"
Nov 01 03:17:16.527351 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:16.527329    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7204663c-ef51-407b-8b56-65e9e54a2d51-istio-data\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-zkjpt\" (UID: \"7204663c-ef51-407b-8b56-65e9e54a2d51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zkjpt"
Nov 01 03:17:16.527647 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:16.527626    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7204663c-ef51-407b-8b56-65e9e54a2d51-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-zkjpt\" (UID: \"7204663c-ef51-407b-8b56-65e9e54a2d51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zkjpt"
Nov 01 03:17:16.529136 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:16.529112    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7204663c-ef51-407b-8b56-65e9e54a2d51-istio-envoy\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-zkjpt\" (UID: \"7204663c-ef51-407b-8b56-65e9e54a2d51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zkjpt"
Nov 01 03:17:16.529543 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:16.529521    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7204663c-ef51-407b-8b56-65e9e54a2d51-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-zkjpt\" (UID: \"7204663c-ef51-407b-8b56-65e9e54a2d51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zkjpt"
Nov 01 03:17:16.589343 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:16.589320    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6w8zf\" (UniqueName: \"kubernetes.io/projected/7204663c-ef51-407b-8b56-65e9e54a2d51-kube-api-access-6w8zf\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-zkjpt\" (UID: \"7204663c-ef51-407b-8b56-65e9e54a2d51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zkjpt"
Nov 01 03:17:16.589343 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:16.589337    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7204663c-ef51-407b-8b56-65e9e54a2d51-istio-token\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-zkjpt\" (UID: \"7204663c-ef51-407b-8b56-65e9e54a2d51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zkjpt"
Nov 01 03:17:16.870072 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:16.870047    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zkjpt"
Nov 01 03:17:17.299678 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:17.299610    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zkjpt"]
Nov 01 03:17:17.335156 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:17:17.335125    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7204663c_ef51_407b_8b56_65e9e54a2d51.slice/crio-82520dd8aace8afbb3275ea57da91c6035ea83c1a127b28275ddfabd85d218c0 WatchSource:0}: Error finding container 82520dd8aace8afbb3275ea57da91c6035ea83c1a127b28275ddfabd85d218c0: Status 404 returned error can't find the container with id 82520dd8aace8afbb3275ea57da91c6035ea83c1a127b28275ddfabd85d218c0
Nov 01 03:17:17.337239 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:17.337211    4400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 03:17:17.337320 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:17.337272    4400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 03:17:17.337320 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:17.337299    4400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 03:17:17.566339 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:17.566260    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zkjpt" event={"ID":"7204663c-ef51-407b-8b56-65e9e54a2d51","Type":"ContainerStarted","Data":"d9629afe985c933aefae348e9746c5911ff7952bcaa8db84ac25ed19918f1f3a"}
Nov 01 03:17:17.566339 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:17.566314    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zkjpt" event={"ID":"7204663c-ef51-407b-8b56-65e9e54a2d51","Type":"ContainerStarted","Data":"82520dd8aace8afbb3275ea57da91c6035ea83c1a127b28275ddfabd85d218c0"}
Nov 01 03:17:17.692085 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:17.692035    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zkjpt" podStartSLOduration=1.692022125 podStartE2EDuration="1.692022125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:17:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:17:17.689920623 +0000 UTC m=+644.204566521" watchObservedRunningTime="2026-11-01 03:17:17.692022125 +0000 UTC m=+644.206667987"
Nov 01 03:17:17.870488 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:17.870455    4400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zkjpt"
Nov 01 03:17:18.872702 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:18.872657    4400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-zkjpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Nov 01 03:17:18.873112 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:18.872726    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zkjpt" podUID="7204663c-ef51-407b-8b56-65e9e54a2d51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Nov 01 03:17:19.870893 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:19.870856    4400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-zkjpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Nov 01 03:17:19.871080 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:19.870910    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zkjpt" podUID="7204663c-ef51-407b-8b56-65e9e54a2d51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Nov 01 03:17:20.871441 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:20.871409    4400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-zkjpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Nov 01 03:17:20.871866 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:20.871462    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zkjpt" podUID="7204663c-ef51-407b-8b56-65e9e54a2d51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Nov 01 03:17:21.870861 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:21.870828    4400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-zkjpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Nov 01 03:17:21.871034 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:21.870886    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zkjpt" podUID="7204663c-ef51-407b-8b56-65e9e54a2d51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Nov 01 03:17:22.871116 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:22.871084    4400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-zkjpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Nov 01 03:17:22.871510 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:22.871144    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zkjpt" podUID="7204663c-ef51-407b-8b56-65e9e54a2d51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Nov 01 03:17:23.871330 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:23.871301    4400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-zkjpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Nov 01 03:17:23.871779 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:23.871347    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zkjpt" podUID="7204663c-ef51-407b-8b56-65e9e54a2d51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Nov 01 03:17:25.561414 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:25.561380    4400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zkjpt"
Nov 01 03:17:25.561975 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:25.561841    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zkjpt"
Nov 01 03:17:25.562788 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:25.562759    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zkjpt"
Nov 01 03:17:31.731267 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:17:31.731234    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:17:44.562013 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:44.561936    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-b12199-6d8d9bb99-sjn42"]
Nov 01 03:17:44.569209 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:44.569186    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b12199-6d8d9bb99-sjn42"
Nov 01 03:17:44.641368 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:44.641302    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-b12199\""
Nov 01 03:17:44.692391 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:44.692358    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-b12199-dockercfg-zg595\""
Nov 01 03:17:44.695015 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:44.694987    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-b12199-6d8d9bb99-sjn42"]
Nov 01 03:17:44.698666 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:44.698640    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdvrl\" (UniqueName: \"kubernetes.io/projected/1b60b3a2-9941-49b1-a528-ee002c10a700-kube-api-access-jdvrl\") pod \"payload-pre-processing-e2e-worker-w1-b12199-6d8d9bb99-sjn42\" (UID: \"1b60b3a2-9941-49b1-a528-ee002c10a700\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b12199-6d8d9bb99-sjn42"
Nov 01 03:17:44.698830 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:44.698762    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b60b3a2-9941-49b1-a528-ee002c10a700-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-b12199-6d8d9bb99-sjn42\" (UID: \"1b60b3a2-9941-49b1-a528-ee002c10a700\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b12199-6d8d9bb99-sjn42"
Nov 01 03:17:44.733733 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:44.733704    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-b12199-76bc58d68f-mdsld"]
Nov 01 03:17:44.737645 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:44.737627    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-b12199-76bc58d68f-mdsld"
Nov 01 03:17:44.795744 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:44.795722    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-b12199-76bc58d68f-mdsld"]
Nov 01 03:17:44.799492 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:44.799470    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58d4db89-1d9c-46f7-a944-43b865313c8c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-b12199-76bc58d68f-mdsld\" (UID: \"58d4db89-1d9c-46f7-a944-43b865313c8c\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-b12199-76bc58d68f-mdsld"
Nov 01 03:17:44.799604 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:44.799506    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srhxs\" (UniqueName: \"kubernetes.io/projected/58d4db89-1d9c-46f7-a944-43b865313c8c-kube-api-access-srhxs\") pod \"payload-processing-e2e-worker-w1-b12199-76bc58d68f-mdsld\" (UID: \"58d4db89-1d9c-46f7-a944-43b865313c8c\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-b12199-76bc58d68f-mdsld"
Nov 01 03:17:44.799604 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:44.799536    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b60b3a2-9941-49b1-a528-ee002c10a700-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-b12199-6d8d9bb99-sjn42\" (UID: \"1b60b3a2-9941-49b1-a528-ee002c10a700\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b12199-6d8d9bb99-sjn42"
Nov 01 03:17:44.799745 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:44.799727    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdvrl\" (UniqueName: \"kubernetes.io/projected/1b60b3a2-9941-49b1-a528-ee002c10a700-kube-api-access-jdvrl\") pod \"payload-pre-processing-e2e-worker-w1-b12199-6d8d9bb99-sjn42\" (UID: \"1b60b3a2-9941-49b1-a528-ee002c10a700\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b12199-6d8d9bb99-sjn42"
Nov 01 03:17:44.800440 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:44.800417    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b60b3a2-9941-49b1-a528-ee002c10a700-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-b12199-6d8d9bb99-sjn42\" (UID: \"1b60b3a2-9941-49b1-a528-ee002c10a700\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b12199-6d8d9bb99-sjn42"
Nov 01 03:17:44.864502 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:44.864470    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdvrl\" (UniqueName: \"kubernetes.io/projected/1b60b3a2-9941-49b1-a528-ee002c10a700-kube-api-access-jdvrl\") pod \"payload-pre-processing-e2e-worker-w1-b12199-6d8d9bb99-sjn42\" (UID: \"1b60b3a2-9941-49b1-a528-ee002c10a700\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b12199-6d8d9bb99-sjn42"
Nov 01 03:17:44.881646 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:44.881623    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b12199-6d8d9bb99-sjn42"
Nov 01 03:17:44.900674 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:44.900652    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58d4db89-1d9c-46f7-a944-43b865313c8c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-b12199-76bc58d68f-mdsld\" (UID: \"58d4db89-1d9c-46f7-a944-43b865313c8c\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-b12199-76bc58d68f-mdsld"
Nov 01 03:17:44.900780 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:44.900706    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srhxs\" (UniqueName: \"kubernetes.io/projected/58d4db89-1d9c-46f7-a944-43b865313c8c-kube-api-access-srhxs\") pod \"payload-processing-e2e-worker-w1-b12199-76bc58d68f-mdsld\" (UID: \"58d4db89-1d9c-46f7-a944-43b865313c8c\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-b12199-76bc58d68f-mdsld"
Nov 01 03:17:44.901495 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:44.901474    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58d4db89-1d9c-46f7-a944-43b865313c8c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-b12199-76bc58d68f-mdsld\" (UID: \"58d4db89-1d9c-46f7-a944-43b865313c8c\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-b12199-76bc58d68f-mdsld"
Nov 01 03:17:44.968927 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:44.968847    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srhxs\" (UniqueName: \"kubernetes.io/projected/58d4db89-1d9c-46f7-a944-43b865313c8c-kube-api-access-srhxs\") pod \"payload-processing-e2e-worker-w1-b12199-76bc58d68f-mdsld\" (UID: \"58d4db89-1d9c-46f7-a944-43b865313c8c\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-b12199-76bc58d68f-mdsld"
Nov 01 03:17:45.050466 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:45.050440    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-b12199-76bc58d68f-mdsld"
Nov 01 03:17:45.269390 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:45.269363    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-b12199-6d8d9bb99-sjn42"]
Nov 01 03:17:45.269753 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:17:45.269729    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b60b3a2_9941_49b1_a528_ee002c10a700.slice/crio-2a59c4f64b5eb234f97c8288d48802f25277809962bcfa0008c1abeb3c2e9511 WatchSource:0}: Error finding container 2a59c4f64b5eb234f97c8288d48802f25277809962bcfa0008c1abeb3c2e9511: Status 404 returned error can't find the container with id 2a59c4f64b5eb234f97c8288d48802f25277809962bcfa0008c1abeb3c2e9511
Nov 01 03:17:45.426413 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:45.426391    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-b12199-76bc58d68f-mdsld"]
Nov 01 03:17:45.427607 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:17:45.427580    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58d4db89_1d9c_46f7_a944_43b865313c8c.slice/crio-bc83a7f7288c96f2c07068dca6675539b8b7015d7f38c3fa7728107622f132e3 WatchSource:0}: Error finding container bc83a7f7288c96f2c07068dca6675539b8b7015d7f38c3fa7728107622f132e3: Status 404 returned error can't find the container with id bc83a7f7288c96f2c07068dca6675539b8b7015d7f38c3fa7728107622f132e3
Nov 01 03:17:45.702923 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:45.702890    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b12199-6d8d9bb99-sjn42" event={"ID":"1b60b3a2-9941-49b1-a528-ee002c10a700","Type":"ContainerStarted","Data":"9e4abb443db248ac27ccf6bcdd3337d3950db54c58f5a1a44e72c76daaa451dd"}
Nov 01 03:17:45.703345 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:45.702950    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b12199-6d8d9bb99-sjn42" event={"ID":"1b60b3a2-9941-49b1-a528-ee002c10a700","Type":"ContainerStarted","Data":"2a59c4f64b5eb234f97c8288d48802f25277809962bcfa0008c1abeb3c2e9511"}
Nov 01 03:17:45.703345 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:45.703053    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b12199-6d8d9bb99-sjn42"
Nov 01 03:17:45.704467 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:45.704441    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-b12199-76bc58d68f-mdsld" event={"ID":"58d4db89-1d9c-46f7-a944-43b865313c8c","Type":"ContainerStarted","Data":"9e5453964fd1a097ecc7e6ef842c74c17a90885bfdedbc48fa52eeb21e1c2911"}
Nov 01 03:17:45.704613 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:45.704489    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-b12199-76bc58d68f-mdsld"
Nov 01 03:17:45.704613 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:45.704505    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-b12199-76bc58d68f-mdsld" event={"ID":"58d4db89-1d9c-46f7-a944-43b865313c8c","Type":"ContainerStarted","Data":"bc83a7f7288c96f2c07068dca6675539b8b7015d7f38c3fa7728107622f132e3"}
Nov 01 03:17:45.824992 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:45.824949    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b12199-6d8d9bb99-sjn42" podStartSLOduration=1.824932945 podStartE2EDuration="1.824932945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:17:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:17:45.824865611 +0000 UTC m=+672.339511474" watchObservedRunningTime="2026-11-01 03:17:45.824932945 +0000 UTC m=+672.339578809"
Nov 01 03:17:46.694875 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:46.694830    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-b12199-76bc58d68f-mdsld" podStartSLOduration=2.694816813 podStartE2EDuration="2.694816813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:17:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:17:45.949112614 +0000 UTC m=+672.463758478" watchObservedRunningTime="2026-11-01 03:17:46.694816813 +0000 UTC m=+673.209462702"
Nov 01 03:17:46.696664 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:46.696647    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-b12199-9c46654f4-j5wwb"]
Nov 01 03:17:46.700137 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:46.700122    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b12199-9c46654f4-j5wwb"
Nov 01 03:17:46.756646 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:46.756622    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-b12199-serving-cert\""
Nov 01 03:17:46.809844 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:46.809815    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-b12199-9c46654f4-j5wwb"]
Nov 01 03:17:46.820557 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:46.820525    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0737af1-9631-4f69-ba0d-18e6cd2765d1-maas-api-tls\") pod \"maas-api-e2e-worker-w1-b12199-9c46654f4-j5wwb\" (UID: \"a0737af1-9631-4f69-ba0d-18e6cd2765d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b12199-9c46654f4-j5wwb"
Nov 01 03:17:46.820813 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:46.820788    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxnfw\" (UniqueName: \"kubernetes.io/projected/a0737af1-9631-4f69-ba0d-18e6cd2765d1-kube-api-access-cxnfw\") pod \"maas-api-e2e-worker-w1-b12199-9c46654f4-j5wwb\" (UID: \"a0737af1-9631-4f69-ba0d-18e6cd2765d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b12199-9c46654f4-j5wwb"
Nov 01 03:17:46.821090 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:46.821068    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0737af1-9631-4f69-ba0d-18e6cd2765d1-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-b12199-9c46654f4-j5wwb\" (UID: \"a0737af1-9631-4f69-ba0d-18e6cd2765d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b12199-9c46654f4-j5wwb"
Nov 01 03:17:46.821199 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:46.821104    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0737af1-9631-4f69-ba0d-18e6cd2765d1-metrics-certs\") pod \"maas-api-e2e-worker-w1-b12199-9c46654f4-j5wwb\" (UID: \"a0737af1-9631-4f69-ba0d-18e6cd2765d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b12199-9c46654f4-j5wwb"
Nov 01 03:17:46.926031 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:46.925993    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0737af1-9631-4f69-ba0d-18e6cd2765d1-maas-api-tls\") pod \"maas-api-e2e-worker-w1-b12199-9c46654f4-j5wwb\" (UID: \"a0737af1-9631-4f69-ba0d-18e6cd2765d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b12199-9c46654f4-j5wwb"
Nov 01 03:17:46.926226 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:46.926083    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxnfw\" (UniqueName: \"kubernetes.io/projected/a0737af1-9631-4f69-ba0d-18e6cd2765d1-kube-api-access-cxnfw\") pod \"maas-api-e2e-worker-w1-b12199-9c46654f4-j5wwb\" (UID: \"a0737af1-9631-4f69-ba0d-18e6cd2765d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b12199-9c46654f4-j5wwb"
Nov 01 03:17:46.926226 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:46.926183    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0737af1-9631-4f69-ba0d-18e6cd2765d1-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-b12199-9c46654f4-j5wwb\" (UID: \"a0737af1-9631-4f69-ba0d-18e6cd2765d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b12199-9c46654f4-j5wwb"
Nov 01 03:17:46.926226 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:46.926207    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0737af1-9631-4f69-ba0d-18e6cd2765d1-metrics-certs\") pod \"maas-api-e2e-worker-w1-b12199-9c46654f4-j5wwb\" (UID: \"a0737af1-9631-4f69-ba0d-18e6cd2765d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b12199-9c46654f4-j5wwb"
Nov 01 03:17:46.927795 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:46.927769    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0737af1-9631-4f69-ba0d-18e6cd2765d1-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-b12199-9c46654f4-j5wwb\" (UID: \"a0737af1-9631-4f69-ba0d-18e6cd2765d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b12199-9c46654f4-j5wwb"
Nov 01 03:17:46.946676 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:46.944877    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0737af1-9631-4f69-ba0d-18e6cd2765d1-metrics-certs\") pod \"maas-api-e2e-worker-w1-b12199-9c46654f4-j5wwb\" (UID: \"a0737af1-9631-4f69-ba0d-18e6cd2765d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b12199-9c46654f4-j5wwb"
Nov 01 03:17:46.947983 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:46.947950    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0737af1-9631-4f69-ba0d-18e6cd2765d1-maas-api-tls\") pod \"maas-api-e2e-worker-w1-b12199-9c46654f4-j5wwb\" (UID: \"a0737af1-9631-4f69-ba0d-18e6cd2765d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b12199-9c46654f4-j5wwb"
Nov 01 03:17:46.989173 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:46.989147    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxnfw\" (UniqueName: \"kubernetes.io/projected/a0737af1-9631-4f69-ba0d-18e6cd2765d1-kube-api-access-cxnfw\") pod \"maas-api-e2e-worker-w1-b12199-9c46654f4-j5wwb\" (UID: \"a0737af1-9631-4f69-ba0d-18e6cd2765d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b12199-9c46654f4-j5wwb"
Nov 01 03:17:47.014054 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:47.014031    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b12199-9c46654f4-j5wwb"
Nov 01 03:17:47.390982 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:47.390174    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-b12199-9c46654f4-j5wwb"]
Nov 01 03:17:47.392287 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:17:47.392254    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0737af1_9631_4f69_ba0d_18e6cd2765d1.slice/crio-bc42d49ad26e64b2895f3fe62119a0ff1b8873f6dedf36fb682acc12051cafce WatchSource:0}: Error finding container bc42d49ad26e64b2895f3fe62119a0ff1b8873f6dedf36fb682acc12051cafce: Status 404 returned error can't find the container with id bc42d49ad26e64b2895f3fe62119a0ff1b8873f6dedf36fb682acc12051cafce
Nov 01 03:17:47.715739 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:47.715637    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b12199-9c46654f4-j5wwb" event={"ID":"a0737af1-9631-4f69-ba0d-18e6cd2765d1","Type":"ContainerStarted","Data":"bc42d49ad26e64b2895f3fe62119a0ff1b8873f6dedf36fb682acc12051cafce"}
Nov 01 03:17:48.723085 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:48.723008    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b12199-9c46654f4-j5wwb" event={"ID":"a0737af1-9631-4f69-ba0d-18e6cd2765d1","Type":"ContainerStarted","Data":"70edca7784a169295f94713bbab5f5dc0becbb0ba9a39106f15feb2ea987bc74"}
Nov 01 03:17:48.723495 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:48.723151    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b12199-9c46654f4-j5wwb"
Nov 01 03:17:48.845066 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:48.845011    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b12199-9c46654f4-j5wwb" podStartSLOduration=1.89357373 podStartE2EDuration="2.844997686s" totalImagesPullingTime="951.423956ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:17:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:17:48.843384873 +0000 UTC m=+675.358030738" watchObservedRunningTime="2026-11-01 03:17:48.844997686 +0000 UTC m=+675.359643549"
Nov 01 03:17:51.807090 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:51.807056    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-ded2d7-748487bb8c-9w7mw"]
Nov 01 03:17:51.811581 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:51.811563    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ded2d7-748487bb8c-9w7mw"
Nov 01 03:17:51.868065 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:51.868040    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-ded2d7-748487bb8c-9w7mw"]
Nov 01 03:17:51.882774 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:51.882736    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c1732ba-87a1-43cf-92d6-5edf0869787f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-ded2d7-748487bb8c-9w7mw\" (UID: \"7c1732ba-87a1-43cf-92d6-5edf0869787f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ded2d7-748487bb8c-9w7mw"
Nov 01 03:17:51.883068 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:51.883046    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fg2dp\" (UniqueName: \"kubernetes.io/projected/7c1732ba-87a1-43cf-92d6-5edf0869787f-kube-api-access-fg2dp\") pod \"payload-pre-processing-e2e-worker-w0-ded2d7-748487bb8c-9w7mw\" (UID: \"7c1732ba-87a1-43cf-92d6-5edf0869787f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ded2d7-748487bb8c-9w7mw"
Nov 01 03:17:51.918072 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:51.918038    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-ded2d7\""
Nov 01 03:17:51.918450 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:51.918425    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-ded2d7-dockercfg-zv8f4\""
Nov 01 03:17:51.971490 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:51.971463    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-ded2d7-54f99bddc-gjh62"]
Nov 01 03:17:51.975143 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:51.975121    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-ded2d7-54f99bddc-gjh62"
Nov 01 03:17:51.984128 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:51.984103    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c1732ba-87a1-43cf-92d6-5edf0869787f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-ded2d7-748487bb8c-9w7mw\" (UID: \"7c1732ba-87a1-43cf-92d6-5edf0869787f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ded2d7-748487bb8c-9w7mw"
Nov 01 03:17:51.984224 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:51.984152    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fg2dp\" (UniqueName: \"kubernetes.io/projected/7c1732ba-87a1-43cf-92d6-5edf0869787f-kube-api-access-fg2dp\") pod \"payload-pre-processing-e2e-worker-w0-ded2d7-748487bb8c-9w7mw\" (UID: \"7c1732ba-87a1-43cf-92d6-5edf0869787f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ded2d7-748487bb8c-9w7mw"
Nov 01 03:17:51.984791 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:51.984775    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c1732ba-87a1-43cf-92d6-5edf0869787f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-ded2d7-748487bb8c-9w7mw\" (UID: \"7c1732ba-87a1-43cf-92d6-5edf0869787f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ded2d7-748487bb8c-9w7mw"
Nov 01 03:17:52.029980 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:52.029953    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-ded2d7-54f99bddc-gjh62"]
Nov 01 03:17:52.047055 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:52.047028    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fg2dp\" (UniqueName: \"kubernetes.io/projected/7c1732ba-87a1-43cf-92d6-5edf0869787f-kube-api-access-fg2dp\") pod \"payload-pre-processing-e2e-worker-w0-ded2d7-748487bb8c-9w7mw\" (UID: \"7c1732ba-87a1-43cf-92d6-5edf0869787f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ded2d7-748487bb8c-9w7mw"
Nov 01 03:17:52.085227 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:52.085206    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4498n\" (UniqueName: \"kubernetes.io/projected/466179e3-7600-464e-9f7c-ec099bc8086d-kube-api-access-4498n\") pod \"payload-processing-e2e-worker-w0-ded2d7-54f99bddc-gjh62\" (UID: \"466179e3-7600-464e-9f7c-ec099bc8086d\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-ded2d7-54f99bddc-gjh62"
Nov 01 03:17:52.085330 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:52.085257    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/466179e3-7600-464e-9f7c-ec099bc8086d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-ded2d7-54f99bddc-gjh62\" (UID: \"466179e3-7600-464e-9f7c-ec099bc8086d\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-ded2d7-54f99bddc-gjh62"
Nov 01 03:17:52.124149 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:52.124131    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ded2d7-748487bb8c-9w7mw"
Nov 01 03:17:52.186464 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:52.186424    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/466179e3-7600-464e-9f7c-ec099bc8086d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-ded2d7-54f99bddc-gjh62\" (UID: \"466179e3-7600-464e-9f7c-ec099bc8086d\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-ded2d7-54f99bddc-gjh62"
Nov 01 03:17:52.186617 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:52.186599    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4498n\" (UniqueName: \"kubernetes.io/projected/466179e3-7600-464e-9f7c-ec099bc8086d-kube-api-access-4498n\") pod \"payload-processing-e2e-worker-w0-ded2d7-54f99bddc-gjh62\" (UID: \"466179e3-7600-464e-9f7c-ec099bc8086d\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-ded2d7-54f99bddc-gjh62"
Nov 01 03:17:52.187479 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:52.187454    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/466179e3-7600-464e-9f7c-ec099bc8086d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-ded2d7-54f99bddc-gjh62\" (UID: \"466179e3-7600-464e-9f7c-ec099bc8086d\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-ded2d7-54f99bddc-gjh62"
Nov 01 03:17:52.248955 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:52.248922    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4498n\" (UniqueName: \"kubernetes.io/projected/466179e3-7600-464e-9f7c-ec099bc8086d-kube-api-access-4498n\") pod \"payload-processing-e2e-worker-w0-ded2d7-54f99bddc-gjh62\" (UID: \"466179e3-7600-464e-9f7c-ec099bc8086d\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-ded2d7-54f99bddc-gjh62"
Nov 01 03:17:52.288946 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:52.288913    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-ded2d7-54f99bddc-gjh62"
Nov 01 03:17:52.500475 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:52.500444    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-ded2d7-748487bb8c-9w7mw"]
Nov 01 03:17:52.501625 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:17:52.501601    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c1732ba_87a1_43cf_92d6_5edf0869787f.slice/crio-affac3f5fc0e9e934c6b9e17cba4cb6ff8097f3b47b7a2fba872d56dc119799c WatchSource:0}: Error finding container affac3f5fc0e9e934c6b9e17cba4cb6ff8097f3b47b7a2fba872d56dc119799c: Status 404 returned error can't find the container with id affac3f5fc0e9e934c6b9e17cba4cb6ff8097f3b47b7a2fba872d56dc119799c
Nov 01 03:17:52.663451 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:52.663428    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-ded2d7-54f99bddc-gjh62"]
Nov 01 03:17:52.664546 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:17:52.664525    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod466179e3_7600_464e_9f7c_ec099bc8086d.slice/crio-75173a332a054796b1db6fb4afc582e064237a2eec7fb67c1e39b4863387a924 WatchSource:0}: Error finding container 75173a332a054796b1db6fb4afc582e064237a2eec7fb67c1e39b4863387a924: Status 404 returned error can't find the container with id 75173a332a054796b1db6fb4afc582e064237a2eec7fb67c1e39b4863387a924
Nov 01 03:17:52.746110 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:52.746084    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ded2d7-748487bb8c-9w7mw" event={"ID":"7c1732ba-87a1-43cf-92d6-5edf0869787f","Type":"ContainerStarted","Data":"ca7234f808bb82dd2442bdf2c2bfba1bc875cde71f5216a7f6847d34d3c5fb82"}
Nov 01 03:17:52.746225 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:52.746143    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ded2d7-748487bb8c-9w7mw" event={"ID":"7c1732ba-87a1-43cf-92d6-5edf0869787f","Type":"ContainerStarted","Data":"affac3f5fc0e9e934c6b9e17cba4cb6ff8097f3b47b7a2fba872d56dc119799c"}
Nov 01 03:17:52.746225 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:52.746178    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ded2d7-748487bb8c-9w7mw"
Nov 01 03:17:52.747761 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:52.747738    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-ded2d7-54f99bddc-gjh62" event={"ID":"466179e3-7600-464e-9f7c-ec099bc8086d","Type":"ContainerStarted","Data":"c2662c6c96fa2bb6dc157a754b41999ca03b687ca8158aee5a3832e0bce615e4"}
Nov 01 03:17:52.747848 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:52.747781    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-ded2d7-54f99bddc-gjh62" event={"ID":"466179e3-7600-464e-9f7c-ec099bc8086d","Type":"ContainerStarted","Data":"75173a332a054796b1db6fb4afc582e064237a2eec7fb67c1e39b4863387a924"}
Nov 01 03:17:52.747848 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:52.747801    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-ded2d7-54f99bddc-gjh62"
Nov 01 03:17:52.870933 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:52.870889    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ded2d7-748487bb8c-9w7mw" podStartSLOduration=1.870875454 podStartE2EDuration="1.870875454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:17:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:17:52.86882811 +0000 UTC m=+679.383473973" watchObservedRunningTime="2026-11-01 03:17:52.870875454 +0000 UTC m=+679.385521318"
Nov 01 03:17:53.001769 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:52.998479    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-ded2d7-54f99bddc-gjh62" podStartSLOduration=1.998450419 podStartE2EDuration="1.998450419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:17:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:17:52.997159262 +0000 UTC m=+679.511805142" watchObservedRunningTime="2026-11-01 03:17:52.998450419 +0000 UTC m=+679.513096286"
Nov 01 03:17:54.329100 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:54.329064    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-19faa5-67dd968c47-scbcb"]
Nov 01 03:17:54.332527 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:54.332499    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-19faa5-67dd968c47-scbcb"
Nov 01 03:17:54.388541 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:54.388518    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-19faa5\""
Nov 01 03:17:54.422063 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:54.421328    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a73d166a-a23f-4e1f-a653-4abf676d9acb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-19faa5-67dd968c47-scbcb\" (UID: \"a73d166a-a23f-4e1f-a653-4abf676d9acb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-19faa5-67dd968c47-scbcb"
Nov 01 03:17:54.422063 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:54.421568    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgzx5\" (UniqueName: \"kubernetes.io/projected/a73d166a-a23f-4e1f-a653-4abf676d9acb-kube-api-access-qgzx5\") pod \"payload-pre-processing-e2e-worker-w6-19faa5-67dd968c47-scbcb\" (UID: \"a73d166a-a23f-4e1f-a653-4abf676d9acb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-19faa5-67dd968c47-scbcb"
Nov 01 03:17:54.439056 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:54.438949    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-19faa5-dockercfg-mnctd\""
Nov 01 03:17:54.441889 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:54.441857    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-19faa5-67dd968c47-scbcb"]
Nov 01 03:17:54.491739 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:54.491710    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-19faa5-85bf5575cc-9lhg9"]
Nov 01 03:17:54.499827 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:54.497921    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-19faa5-85bf5575cc-9lhg9"
Nov 01 03:17:54.522888 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:54.522866    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgzx5\" (UniqueName: \"kubernetes.io/projected/a73d166a-a23f-4e1f-a653-4abf676d9acb-kube-api-access-qgzx5\") pod \"payload-pre-processing-e2e-worker-w6-19faa5-67dd968c47-scbcb\" (UID: \"a73d166a-a23f-4e1f-a653-4abf676d9acb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-19faa5-67dd968c47-scbcb"
Nov 01 03:17:54.523007 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:54.522912    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a73d166a-a23f-4e1f-a653-4abf676d9acb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-19faa5-67dd968c47-scbcb\" (UID: \"a73d166a-a23f-4e1f-a653-4abf676d9acb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-19faa5-67dd968c47-scbcb"
Nov 01 03:17:54.523555 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:54.523531    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a73d166a-a23f-4e1f-a653-4abf676d9acb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-19faa5-67dd968c47-scbcb\" (UID: \"a73d166a-a23f-4e1f-a653-4abf676d9acb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-19faa5-67dd968c47-scbcb"
Nov 01 03:17:54.549426 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:54.549401    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-19faa5-85bf5575cc-9lhg9"]
Nov 01 03:17:54.586224 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:54.586158    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgzx5\" (UniqueName: \"kubernetes.io/projected/a73d166a-a23f-4e1f-a653-4abf676d9acb-kube-api-access-qgzx5\") pod \"payload-pre-processing-e2e-worker-w6-19faa5-67dd968c47-scbcb\" (UID: \"a73d166a-a23f-4e1f-a653-4abf676d9acb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-19faa5-67dd968c47-scbcb"
Nov 01 03:17:54.624384 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:54.624355    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcjs2\" (UniqueName: \"kubernetes.io/projected/92f2c828-dd25-4783-ab42-3840e6417e29-kube-api-access-tcjs2\") pod \"payload-processing-e2e-worker-w6-19faa5-85bf5575cc-9lhg9\" (UID: \"92f2c828-dd25-4783-ab42-3840e6417e29\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-19faa5-85bf5575cc-9lhg9"
Nov 01 03:17:54.624522 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:54.624407    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92f2c828-dd25-4783-ab42-3840e6417e29-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-19faa5-85bf5575cc-9lhg9\" (UID: \"92f2c828-dd25-4783-ab42-3840e6417e29\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-19faa5-85bf5575cc-9lhg9"
Nov 01 03:17:54.646249 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:54.646217    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-19faa5-67dd968c47-scbcb"
Nov 01 03:17:54.725344 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:54.725314    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92f2c828-dd25-4783-ab42-3840e6417e29-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-19faa5-85bf5575cc-9lhg9\" (UID: \"92f2c828-dd25-4783-ab42-3840e6417e29\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-19faa5-85bf5575cc-9lhg9"
Nov 01 03:17:54.725698 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:54.725499    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcjs2\" (UniqueName: \"kubernetes.io/projected/92f2c828-dd25-4783-ab42-3840e6417e29-kube-api-access-tcjs2\") pod \"payload-processing-e2e-worker-w6-19faa5-85bf5575cc-9lhg9\" (UID: \"92f2c828-dd25-4783-ab42-3840e6417e29\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-19faa5-85bf5575cc-9lhg9"
Nov 01 03:17:54.726100 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:54.726081    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92f2c828-dd25-4783-ab42-3840e6417e29-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-19faa5-85bf5575cc-9lhg9\" (UID: \"92f2c828-dd25-4783-ab42-3840e6417e29\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-19faa5-85bf5575cc-9lhg9"
Nov 01 03:17:54.734065 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:54.734046    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b12199-9c46654f4-j5wwb"
Nov 01 03:17:54.788452 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:54.788427    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcjs2\" (UniqueName: \"kubernetes.io/projected/92f2c828-dd25-4783-ab42-3840e6417e29-kube-api-access-tcjs2\") pod \"payload-processing-e2e-worker-w6-19faa5-85bf5575cc-9lhg9\" (UID: \"92f2c828-dd25-4783-ab42-3840e6417e29\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-19faa5-85bf5575cc-9lhg9"
Nov 01 03:17:54.812223 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:54.812202    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-19faa5-85bf5575cc-9lhg9"
Nov 01 03:17:55.025499 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:55.025472    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-19faa5-67dd968c47-scbcb"]
Nov 01 03:17:55.025908 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:17:55.025880    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda73d166a_a23f_4e1f_a653_4abf676d9acb.slice/crio-3fb90508fb83989514f53006685fc4c0fae5f4b496ebe454a580f99947f5964b WatchSource:0}: Error finding container 3fb90508fb83989514f53006685fc4c0fae5f4b496ebe454a580f99947f5964b: Status 404 returned error can't find the container with id 3fb90508fb83989514f53006685fc4c0fae5f4b496ebe454a580f99947f5964b
Nov 01 03:17:55.100746 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:55.099732    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 03:17:55.229439 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:55.229419    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-19faa5-85bf5575cc-9lhg9"]
Nov 01 03:17:55.230588 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:17:55.230552    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92f2c828_dd25_4783_ab42_3840e6417e29.slice/crio-d5c20d86c070c9197f2d05d0cd737c967dcbb27aa8f54c07fde6cbe0e4f1e4a1 WatchSource:0}: Error finding container d5c20d86c070c9197f2d05d0cd737c967dcbb27aa8f54c07fde6cbe0e4f1e4a1: Status 404 returned error can't find the container with id d5c20d86c070c9197f2d05d0cd737c967dcbb27aa8f54c07fde6cbe0e4f1e4a1
Nov 01 03:17:55.764036 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:55.764002    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-19faa5-85bf5575cc-9lhg9" event={"ID":"92f2c828-dd25-4783-ab42-3840e6417e29","Type":"ContainerStarted","Data":"748594fad63267be4df1395de90ff1f53acd8f169ac92b6a9f5dbaf08cd01410"}
Nov 01 03:17:55.764433 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:55.764052    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-19faa5-85bf5575cc-9lhg9" event={"ID":"92f2c828-dd25-4783-ab42-3840e6417e29","Type":"ContainerStarted","Data":"d5c20d86c070c9197f2d05d0cd737c967dcbb27aa8f54c07fde6cbe0e4f1e4a1"}
Nov 01 03:17:55.764433 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:55.764104    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-19faa5-85bf5575cc-9lhg9"
Nov 01 03:17:55.765791 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:55.765767    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-19faa5-67dd968c47-scbcb" event={"ID":"a73d166a-a23f-4e1f-a653-4abf676d9acb","Type":"ContainerStarted","Data":"22bfd96ee23fec39b6a6de2dcdf4a74680c53d9865718c4ddd4a76a298e00a87"}
Nov 01 03:17:55.765899 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:55.765809    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-19faa5-67dd968c47-scbcb" event={"ID":"a73d166a-a23f-4e1f-a653-4abf676d9acb","Type":"ContainerStarted","Data":"3fb90508fb83989514f53006685fc4c0fae5f4b496ebe454a580f99947f5964b"}
Nov 01 03:17:55.765899 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:55.765849    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-19faa5-67dd968c47-scbcb"
Nov 01 03:17:55.883058 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:55.883014    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-19faa5-85bf5575cc-9lhg9" podStartSLOduration=1.8830011500000001 podStartE2EDuration="1.88300115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:17:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:17:55.882464064 +0000 UTC m=+682.397109930" watchObservedRunningTime="2026-11-01 03:17:55.88300115 +0000 UTC m=+682.397647036"
Nov 01 03:17:56.009489 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:56.004077    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-19faa5-67dd968c47-scbcb" podStartSLOduration=2.004057045 podStartE2EDuration="2.004057045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:17:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:17:56.001810758 +0000 UTC m=+682.516456623" watchObservedRunningTime="2026-11-01 03:17:56.004057045 +0000 UTC m=+682.518702910"
Nov 01 03:17:56.712042 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:56.712012    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b12199-6d8d9bb99-sjn42"
Nov 01 03:17:56.712239 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:56.712091    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-b12199-76bc58d68f-mdsld"
Nov 01 03:17:56.864652 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:56.864626    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-d7f535-8659cb78cf-wxmbp"]
Nov 01 03:17:56.868449 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:56.868431    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d7f535-8659cb78cf-wxmbp"
Nov 01 03:17:56.922424 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:56.922400    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-d7f535\""
Nov 01 03:17:56.922550 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:56.922482    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-d7f535-dockercfg-pd5jk\""
Nov 01 03:17:56.981773 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:56.963235    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-d7f535-8659cb78cf-wxmbp"]
Nov 01 03:17:56.981773 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:56.971456    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7rvh\" (UniqueName: \"kubernetes.io/projected/a6b65b4c-f4d1-49a4-a4d3-7ef365d8aac9-kube-api-access-r7rvh\") pod \"payload-pre-processing-e2e-worker-w3-d7f535-8659cb78cf-wxmbp\" (UID: \"a6b65b4c-f4d1-49a4-a4d3-7ef365d8aac9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d7f535-8659cb78cf-wxmbp"
Nov 01 03:17:56.981773 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:56.971638    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6b65b4c-f4d1-49a4-a4d3-7ef365d8aac9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-d7f535-8659cb78cf-wxmbp\" (UID: \"a6b65b4c-f4d1-49a4-a4d3-7ef365d8aac9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d7f535-8659cb78cf-wxmbp"
Nov 01 03:17:57.001160 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:57.001126    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-d7f535-67cfcb74d9-lzhln"]
Nov 01 03:17:57.005732 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:57.005675    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-d7f535-67cfcb74d9-lzhln"
Nov 01 03:17:57.067743 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:57.067716    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-d7f535-67cfcb74d9-lzhln"]
Nov 01 03:17:57.073259 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:57.073230    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6b65b4c-f4d1-49a4-a4d3-7ef365d8aac9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-d7f535-8659cb78cf-wxmbp\" (UID: \"a6b65b4c-f4d1-49a4-a4d3-7ef365d8aac9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d7f535-8659cb78cf-wxmbp"
Nov 01 03:17:57.073427 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:57.073349    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7rvh\" (UniqueName: \"kubernetes.io/projected/a6b65b4c-f4d1-49a4-a4d3-7ef365d8aac9-kube-api-access-r7rvh\") pod \"payload-pre-processing-e2e-worker-w3-d7f535-8659cb78cf-wxmbp\" (UID: \"a6b65b4c-f4d1-49a4-a4d3-7ef365d8aac9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d7f535-8659cb78cf-wxmbp"
Nov 01 03:17:57.074052 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:57.074027    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6b65b4c-f4d1-49a4-a4d3-7ef365d8aac9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-d7f535-8659cb78cf-wxmbp\" (UID: \"a6b65b4c-f4d1-49a4-a4d3-7ef365d8aac9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d7f535-8659cb78cf-wxmbp"
Nov 01 03:17:57.136258 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:57.136230    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7rvh\" (UniqueName: \"kubernetes.io/projected/a6b65b4c-f4d1-49a4-a4d3-7ef365d8aac9-kube-api-access-r7rvh\") pod \"payload-pre-processing-e2e-worker-w3-d7f535-8659cb78cf-wxmbp\" (UID: \"a6b65b4c-f4d1-49a4-a4d3-7ef365d8aac9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d7f535-8659cb78cf-wxmbp"
Nov 01 03:17:57.174081 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:57.174057    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed1ec2ab-443d-42d9-9221-20cdf668121a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-d7f535-67cfcb74d9-lzhln\" (UID: \"ed1ec2ab-443d-42d9-9221-20cdf668121a\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-d7f535-67cfcb74d9-lzhln"
Nov 01 03:17:57.174180 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:57.174108    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2n9n\" (UniqueName: \"kubernetes.io/projected/ed1ec2ab-443d-42d9-9221-20cdf668121a-kube-api-access-s2n9n\") pod \"payload-processing-e2e-worker-w3-d7f535-67cfcb74d9-lzhln\" (UID: \"ed1ec2ab-443d-42d9-9221-20cdf668121a\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-d7f535-67cfcb74d9-lzhln"
Nov 01 03:17:57.182930 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:57.182906    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d7f535-8659cb78cf-wxmbp"
Nov 01 03:17:57.194893 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:57.194871    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-19faa5-b9cb6f5d-7tr6v"]
Nov 01 03:17:57.199539 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:57.199519    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-19faa5-b9cb6f5d-7tr6v"
Nov 01 03:17:57.255038 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:57.255014    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w6-19faa5-serving-cert\""
Nov 01 03:17:57.256350 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:57.256328    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-19faa5-b9cb6f5d-7tr6v"]
Nov 01 03:17:57.275194 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:57.275167    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed1ec2ab-443d-42d9-9221-20cdf668121a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-d7f535-67cfcb74d9-lzhln\" (UID: \"ed1ec2ab-443d-42d9-9221-20cdf668121a\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-d7f535-67cfcb74d9-lzhln"
Nov 01 03:17:57.275279 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:57.275256    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2n9n\" (UniqueName: \"kubernetes.io/projected/ed1ec2ab-443d-42d9-9221-20cdf668121a-kube-api-access-s2n9n\") pod \"payload-processing-e2e-worker-w3-d7f535-67cfcb74d9-lzhln\" (UID: \"ed1ec2ab-443d-42d9-9221-20cdf668121a\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-d7f535-67cfcb74d9-lzhln"
Nov 01 03:17:57.276083 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:57.276062    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed1ec2ab-443d-42d9-9221-20cdf668121a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-d7f535-67cfcb74d9-lzhln\" (UID: \"ed1ec2ab-443d-42d9-9221-20cdf668121a\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-d7f535-67cfcb74d9-lzhln"
Nov 01 03:17:57.337967 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:57.337947    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2n9n\" (UniqueName: \"kubernetes.io/projected/ed1ec2ab-443d-42d9-9221-20cdf668121a-kube-api-access-s2n9n\") pod \"payload-processing-e2e-worker-w3-d7f535-67cfcb74d9-lzhln\" (UID: \"ed1ec2ab-443d-42d9-9221-20cdf668121a\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-d7f535-67cfcb74d9-lzhln"
Nov 01 03:17:57.376078 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:57.376057    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e73051a-861a-4bb4-b3cf-b06d447768aa-metrics-certs\") pod \"maas-api-e2e-worker-w6-19faa5-b9cb6f5d-7tr6v\" (UID: \"0e73051a-861a-4bb4-b3cf-b06d447768aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-19faa5-b9cb6f5d-7tr6v"
Nov 01 03:17:57.376175 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:57.376140    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e73051a-861a-4bb4-b3cf-b06d447768aa-maas-api-tls\") pod \"maas-api-e2e-worker-w6-19faa5-b9cb6f5d-7tr6v\" (UID: \"0e73051a-861a-4bb4-b3cf-b06d447768aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-19faa5-b9cb6f5d-7tr6v"
Nov 01 03:17:57.376227 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:57.376194    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e73051a-861a-4bb4-b3cf-b06d447768aa-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-19faa5-b9cb6f5d-7tr6v\" (UID: \"0e73051a-861a-4bb4-b3cf-b06d447768aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-19faa5-b9cb6f5d-7tr6v"
Nov 01 03:17:57.376328 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:57.376309    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fn4w9\" (UniqueName: \"kubernetes.io/projected/0e73051a-861a-4bb4-b3cf-b06d447768aa-kube-api-access-fn4w9\") pod \"maas-api-e2e-worker-w6-19faa5-b9cb6f5d-7tr6v\" (UID: \"0e73051a-861a-4bb4-b3cf-b06d447768aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-19faa5-b9cb6f5d-7tr6v"
Nov 01 03:17:57.477939 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:57.477907    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn4w9\" (UniqueName: \"kubernetes.io/projected/0e73051a-861a-4bb4-b3cf-b06d447768aa-kube-api-access-fn4w9\") pod \"maas-api-e2e-worker-w6-19faa5-b9cb6f5d-7tr6v\" (UID: \"0e73051a-861a-4bb4-b3cf-b06d447768aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-19faa5-b9cb6f5d-7tr6v"
Nov 01 03:17:57.478123 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:57.478016    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e73051a-861a-4bb4-b3cf-b06d447768aa-metrics-certs\") pod \"maas-api-e2e-worker-w6-19faa5-b9cb6f5d-7tr6v\" (UID: \"0e73051a-861a-4bb4-b3cf-b06d447768aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-19faa5-b9cb6f5d-7tr6v"
Nov 01 03:17:57.478123 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:57.478043    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e73051a-861a-4bb4-b3cf-b06d447768aa-maas-api-tls\") pod \"maas-api-e2e-worker-w6-19faa5-b9cb6f5d-7tr6v\" (UID: \"0e73051a-861a-4bb4-b3cf-b06d447768aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-19faa5-b9cb6f5d-7tr6v"
Nov 01 03:17:57.478123 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:57.478061    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e73051a-861a-4bb4-b3cf-b06d447768aa-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-19faa5-b9cb6f5d-7tr6v\" (UID: \"0e73051a-861a-4bb4-b3cf-b06d447768aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-19faa5-b9cb6f5d-7tr6v"
Nov 01 03:17:57.478264 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:17:57.478191    4400 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w6-19faa5-serving-cert: secret "maas-api-e2e-worker-w6-19faa5-serving-cert" not found
Nov 01 03:17:57.478313 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:17:57.478282    4400 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e73051a-861a-4bb4-b3cf-b06d447768aa-maas-api-tls podName:0e73051a-861a-4bb4-b3cf-b06d447768aa nodeName:}" failed. No retries permitted until 2026-11-01 03:17:57.978259815 +0000 UTC m=+684.492905657 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0e73051a-861a-4bb4-b3cf-b06d447768aa-maas-api-tls") pod "maas-api-e2e-worker-w6-19faa5-b9cb6f5d-7tr6v" (UID: "0e73051a-861a-4bb4-b3cf-b06d447768aa") : secret "maas-api-e2e-worker-w6-19faa5-serving-cert" not found
Nov 01 03:17:57.478709 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:57.478661    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e73051a-861a-4bb4-b3cf-b06d447768aa-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-19faa5-b9cb6f5d-7tr6v\" (UID: \"0e73051a-861a-4bb4-b3cf-b06d447768aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-19faa5-b9cb6f5d-7tr6v"
Nov 01 03:17:57.480801 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:57.480775    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e73051a-861a-4bb4-b3cf-b06d447768aa-metrics-certs\") pod \"maas-api-e2e-worker-w6-19faa5-b9cb6f5d-7tr6v\" (UID: \"0e73051a-861a-4bb4-b3cf-b06d447768aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-19faa5-b9cb6f5d-7tr6v"
Nov 01 03:17:57.542924 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:57.542873    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn4w9\" (UniqueName: \"kubernetes.io/projected/0e73051a-861a-4bb4-b3cf-b06d447768aa-kube-api-access-fn4w9\") pod \"maas-api-e2e-worker-w6-19faa5-b9cb6f5d-7tr6v\" (UID: \"0e73051a-861a-4bb4-b3cf-b06d447768aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-19faa5-b9cb6f5d-7tr6v"
Nov 01 03:17:57.568818 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:57.568793    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-d7f535-8659cb78cf-wxmbp"]
Nov 01 03:17:57.569935 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:17:57.569911    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6b65b4c_f4d1_49a4_a4d3_7ef365d8aac9.slice/crio-23dcab7dc5582e89866262bf403961e478accfc3ca6e967bd6646edf3e3609b3 WatchSource:0}: Error finding container 23dcab7dc5582e89866262bf403961e478accfc3ca6e967bd6646edf3e3609b3: Status 404 returned error can't find the container with id 23dcab7dc5582e89866262bf403961e478accfc3ca6e967bd6646edf3e3609b3
Nov 01 03:17:57.623536 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:57.623502    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-d7f535-67cfcb74d9-lzhln"
Nov 01 03:17:57.778186 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:57.778153    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d7f535-8659cb78cf-wxmbp" event={"ID":"a6b65b4c-f4d1-49a4-a4d3-7ef365d8aac9","Type":"ContainerStarted","Data":"d156422b113d9f662ae479f0a27ca60e20b80380fdce48654cd5bc7b2177028e"}
Nov 01 03:17:57.778326 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:57.778212    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d7f535-8659cb78cf-wxmbp" event={"ID":"a6b65b4c-f4d1-49a4-a4d3-7ef365d8aac9","Type":"ContainerStarted","Data":"23dcab7dc5582e89866262bf403961e478accfc3ca6e967bd6646edf3e3609b3"}
Nov 01 03:17:57.778326 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:57.778266    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d7f535-8659cb78cf-wxmbp"
Nov 01 03:17:57.899193 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:57.899153    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d7f535-8659cb78cf-wxmbp" podStartSLOduration=1.899139645 podStartE2EDuration="1.899139645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:17:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:17:57.896745391 +0000 UTC m=+684.411391254" watchObservedRunningTime="2026-11-01 03:17:57.899139645 +0000 UTC m=+684.413785517"
Nov 01 03:17:57.983742 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:57.983713    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e73051a-861a-4bb4-b3cf-b06d447768aa-maas-api-tls\") pod \"maas-api-e2e-worker-w6-19faa5-b9cb6f5d-7tr6v\" (UID: \"0e73051a-861a-4bb4-b3cf-b06d447768aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-19faa5-b9cb6f5d-7tr6v"
Nov 01 03:17:57.986183 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:57.986158    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e73051a-861a-4bb4-b3cf-b06d447768aa-maas-api-tls\") pod \"maas-api-e2e-worker-w6-19faa5-b9cb6f5d-7tr6v\" (UID: \"0e73051a-861a-4bb4-b3cf-b06d447768aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-19faa5-b9cb6f5d-7tr6v"
Nov 01 03:17:58.006034 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:58.006007    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-d7f535-67cfcb74d9-lzhln"]
Nov 01 03:17:58.007289 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:17:58.007260    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded1ec2ab_443d_42d9_9221_20cdf668121a.slice/crio-bcd8c50498b0bfa5306832eafe11b506ab00d647fd6b6678b7381edab1956b44 WatchSource:0}: Error finding container bcd8c50498b0bfa5306832eafe11b506ab00d647fd6b6678b7381edab1956b44: Status 404 returned error can't find the container with id bcd8c50498b0bfa5306832eafe11b506ab00d647fd6b6678b7381edab1956b44
Nov 01 03:17:58.124465 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:58.124443    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-19faa5-b9cb6f5d-7tr6v"
Nov 01 03:17:58.499455 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:58.499426    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-19faa5-b9cb6f5d-7tr6v"]
Nov 01 03:17:58.500998 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:17:58.500969    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e73051a_861a_4bb4_b3cf_b06d447768aa.slice/crio-4078ef7d628944ee04a0e7c2b1ee93a99cabaab1468d26aca36920917c01803f WatchSource:0}: Error finding container 4078ef7d628944ee04a0e7c2b1ee93a99cabaab1468d26aca36920917c01803f: Status 404 returned error can't find the container with id 4078ef7d628944ee04a0e7c2b1ee93a99cabaab1468d26aca36920917c01803f
Nov 01 03:17:58.785733 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:58.785649    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-d7f535-67cfcb74d9-lzhln" event={"ID":"ed1ec2ab-443d-42d9-9221-20cdf668121a","Type":"ContainerStarted","Data":"0600998a757bb0c1749ee1486fd4e78f1d41db1b5bb4c74e86d3797d5f22786c"}
Nov 01 03:17:58.785733 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:58.785732    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-d7f535-67cfcb74d9-lzhln" event={"ID":"ed1ec2ab-443d-42d9-9221-20cdf668121a","Type":"ContainerStarted","Data":"bcd8c50498b0bfa5306832eafe11b506ab00d647fd6b6678b7381edab1956b44"}
Nov 01 03:17:58.785885 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:58.785849    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-d7f535-67cfcb74d9-lzhln"
Nov 01 03:17:58.786935 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:58.786906    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-19faa5-b9cb6f5d-7tr6v" event={"ID":"0e73051a-861a-4bb4-b3cf-b06d447768aa","Type":"ContainerStarted","Data":"4078ef7d628944ee04a0e7c2b1ee93a99cabaab1468d26aca36920917c01803f"}
Nov 01 03:17:58.905967 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:58.905919    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-d7f535-67cfcb74d9-lzhln" podStartSLOduration=2.9059068850000003 podStartE2EDuration="2.905906885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:17:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:17:58.904051322 +0000 UTC m=+685.418697185" watchObservedRunningTime="2026-11-01 03:17:58.905906885 +0000 UTC m=+685.420552748"
Nov 01 03:17:59.794340 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:59.794307    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-19faa5-b9cb6f5d-7tr6v" event={"ID":"0e73051a-861a-4bb4-b3cf-b06d447768aa","Type":"ContainerStarted","Data":"b6422dc5f05f33558d1ddb0ed04e5726feda67fa66fd8bdd3444270d577b41b0"}
Nov 01 03:17:59.794534 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:59.794518    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-19faa5-b9cb6f5d-7tr6v"
Nov 01 03:17:59.916255 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:17:59.916201    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-19faa5-b9cb6f5d-7tr6v" podStartSLOduration=2.150278483 podStartE2EDuration="2.916183796s" totalImagesPullingTime="765.905313ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:17:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:17:59.914548932 +0000 UTC m=+686.429194795" watchObservedRunningTime="2026-11-01 03:17:59.916183796 +0000 UTC m=+686.430829660"
Nov 01 03:18:01.729676 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:18:01.729645    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:18:02.297135 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:02.297102    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-c22dd8-5ccf5b65f-tqnf8"]
Nov 01 03:18:02.301247 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:02.301228    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c22dd8-5ccf5b65f-tqnf8"
Nov 01 03:18:02.363373 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:02.363349    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-c22dd8-serving-cert\""
Nov 01 03:18:02.416777 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:02.416750    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-c22dd8-5ccf5b65f-tqnf8"]
Nov 01 03:18:02.432239 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:02.432199    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcvjs\" (UniqueName: \"kubernetes.io/projected/ac5c9c0b-3403-4f6a-b5f6-500a3df457dd-kube-api-access-zcvjs\") pod \"maas-api-e2e-worker-w5-c22dd8-5ccf5b65f-tqnf8\" (UID: \"ac5c9c0b-3403-4f6a-b5f6-500a3df457dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c22dd8-5ccf5b65f-tqnf8"
Nov 01 03:18:02.432365 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:02.432270    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac5c9c0b-3403-4f6a-b5f6-500a3df457dd-maas-api-tls\") pod \"maas-api-e2e-worker-w5-c22dd8-5ccf5b65f-tqnf8\" (UID: \"ac5c9c0b-3403-4f6a-b5f6-500a3df457dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c22dd8-5ccf5b65f-tqnf8"
Nov 01 03:18:02.432365 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:02.432347    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac5c9c0b-3403-4f6a-b5f6-500a3df457dd-metrics-certs\") pod \"maas-api-e2e-worker-w5-c22dd8-5ccf5b65f-tqnf8\" (UID: \"ac5c9c0b-3403-4f6a-b5f6-500a3df457dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c22dd8-5ccf5b65f-tqnf8"
Nov 01 03:18:02.432519 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:02.432485    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac5c9c0b-3403-4f6a-b5f6-500a3df457dd-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-c22dd8-5ccf5b65f-tqnf8\" (UID: \"ac5c9c0b-3403-4f6a-b5f6-500a3df457dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c22dd8-5ccf5b65f-tqnf8"
Nov 01 03:18:02.533744 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:02.533722    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac5c9c0b-3403-4f6a-b5f6-500a3df457dd-maas-api-tls\") pod \"maas-api-e2e-worker-w5-c22dd8-5ccf5b65f-tqnf8\" (UID: \"ac5c9c0b-3403-4f6a-b5f6-500a3df457dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c22dd8-5ccf5b65f-tqnf8"
Nov 01 03:18:02.533899 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:02.533750    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac5c9c0b-3403-4f6a-b5f6-500a3df457dd-metrics-certs\") pod \"maas-api-e2e-worker-w5-c22dd8-5ccf5b65f-tqnf8\" (UID: \"ac5c9c0b-3403-4f6a-b5f6-500a3df457dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c22dd8-5ccf5b65f-tqnf8"
Nov 01 03:18:02.533899 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:18:02.533876    4400 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w5-c22dd8-serving-cert: secret "maas-api-e2e-worker-w5-c22dd8-serving-cert" not found
Nov 01 03:18:02.533899 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:02.533885    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac5c9c0b-3403-4f6a-b5f6-500a3df457dd-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-c22dd8-5ccf5b65f-tqnf8\" (UID: \"ac5c9c0b-3403-4f6a-b5f6-500a3df457dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c22dd8-5ccf5b65f-tqnf8"
Nov 01 03:18:02.534069 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:18:02.533944    4400 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac5c9c0b-3403-4f6a-b5f6-500a3df457dd-maas-api-tls podName:ac5c9c0b-3403-4f6a-b5f6-500a3df457dd nodeName:}" failed. No retries permitted until 2026-11-01 03:18:03.03392049 +0000 UTC m=+689.548566334 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ac5c9c0b-3403-4f6a-b5f6-500a3df457dd-maas-api-tls") pod "maas-api-e2e-worker-w5-c22dd8-5ccf5b65f-tqnf8" (UID: "ac5c9c0b-3403-4f6a-b5f6-500a3df457dd") : secret "maas-api-e2e-worker-w5-c22dd8-serving-cert" not found
Nov 01 03:18:02.534069 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:02.534057    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcvjs\" (UniqueName: \"kubernetes.io/projected/ac5c9c0b-3403-4f6a-b5f6-500a3df457dd-kube-api-access-zcvjs\") pod \"maas-api-e2e-worker-w5-c22dd8-5ccf5b65f-tqnf8\" (UID: \"ac5c9c0b-3403-4f6a-b5f6-500a3df457dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c22dd8-5ccf5b65f-tqnf8"
Nov 01 03:18:02.534517 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:02.534499    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac5c9c0b-3403-4f6a-b5f6-500a3df457dd-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-c22dd8-5ccf5b65f-tqnf8\" (UID: \"ac5c9c0b-3403-4f6a-b5f6-500a3df457dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c22dd8-5ccf5b65f-tqnf8"
Nov 01 03:18:02.536243 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:02.536227    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac5c9c0b-3403-4f6a-b5f6-500a3df457dd-metrics-certs\") pod \"maas-api-e2e-worker-w5-c22dd8-5ccf5b65f-tqnf8\" (UID: \"ac5c9c0b-3403-4f6a-b5f6-500a3df457dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c22dd8-5ccf5b65f-tqnf8"
Nov 01 03:18:02.596804 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:02.596775    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcvjs\" (UniqueName: \"kubernetes.io/projected/ac5c9c0b-3403-4f6a-b5f6-500a3df457dd-kube-api-access-zcvjs\") pod \"maas-api-e2e-worker-w5-c22dd8-5ccf5b65f-tqnf8\" (UID: \"ac5c9c0b-3403-4f6a-b5f6-500a3df457dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c22dd8-5ccf5b65f-tqnf8"
Nov 01 03:18:03.041198 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:03.041119    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac5c9c0b-3403-4f6a-b5f6-500a3df457dd-maas-api-tls\") pod \"maas-api-e2e-worker-w5-c22dd8-5ccf5b65f-tqnf8\" (UID: \"ac5c9c0b-3403-4f6a-b5f6-500a3df457dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c22dd8-5ccf5b65f-tqnf8"
Nov 01 03:18:03.043639 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:03.043618    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac5c9c0b-3403-4f6a-b5f6-500a3df457dd-maas-api-tls\") pod \"maas-api-e2e-worker-w5-c22dd8-5ccf5b65f-tqnf8\" (UID: \"ac5c9c0b-3403-4f6a-b5f6-500a3df457dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c22dd8-5ccf5b65f-tqnf8"
Nov 01 03:18:03.214075 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:03.214051    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c22dd8-5ccf5b65f-tqnf8"
Nov 01 03:18:03.590031 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:03.589999    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-c22dd8-5ccf5b65f-tqnf8"]
Nov 01 03:18:03.590885 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:18:03.590848    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac5c9c0b_3403_4f6a_b5f6_500a3df457dd.slice/crio-38129c4ebbe1d3f84d9d8197dd19f8213278224ba31be0ee6801350ba49473d3 WatchSource:0}: Error finding container 38129c4ebbe1d3f84d9d8197dd19f8213278224ba31be0ee6801350ba49473d3: Status 404 returned error can't find the container with id 38129c4ebbe1d3f84d9d8197dd19f8213278224ba31be0ee6801350ba49473d3
Nov 01 03:18:03.755140 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:03.755112    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ded2d7-748487bb8c-9w7mw"
Nov 01 03:18:03.755293 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:03.755200    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-ded2d7-54f99bddc-gjh62"
Nov 01 03:18:03.819160 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:03.819134    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c22dd8-5ccf5b65f-tqnf8" event={"ID":"ac5c9c0b-3403-4f6a-b5f6-500a3df457dd","Type":"ContainerStarted","Data":"38129c4ebbe1d3f84d9d8197dd19f8213278224ba31be0ee6801350ba49473d3"}
Nov 01 03:18:04.224223 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:04.224182    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 03:18:04.827247 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:04.827212    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c22dd8-5ccf5b65f-tqnf8" event={"ID":"ac5c9c0b-3403-4f6a-b5f6-500a3df457dd","Type":"ContainerStarted","Data":"0ee4bdb00f9ad0168239112aaee64579a19f7b481cf34349a18896a1cfeaa826"}
Nov 01 03:18:04.827412 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:04.827329    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c22dd8-5ccf5b65f-tqnf8"
Nov 01 03:18:04.950500 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:04.950443    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c22dd8-5ccf5b65f-tqnf8" podStartSLOduration=2.0978358520000002 podStartE2EDuration="2.95042404s" totalImagesPullingTime="852.588188ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:18:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:18:04.947848419 +0000 UTC m=+691.462494285" watchObservedRunningTime="2026-11-01 03:18:04.95042404 +0000 UTC m=+691.465069903"
Nov 01 03:18:05.805851 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:05.805821    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-19faa5-b9cb6f5d-7tr6v"
Nov 01 03:18:06.773951 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:06.773920    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-19faa5-85bf5575cc-9lhg9"
Nov 01 03:18:06.774134 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:06.774075    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-19faa5-67dd968c47-scbcb"
Nov 01 03:18:08.790475 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:08.790444    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d7f535-8659cb78cf-wxmbp"
Nov 01 03:18:09.798349 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:09.798324    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-d7f535-67cfcb74d9-lzhln"
Nov 01 03:18:10.838963 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:10.838938    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c22dd8-5ccf5b65f-tqnf8"
Nov 01 03:18:11.568603 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:11.568571    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-c22dd8-6986b5b4cf-r9pws"]
Nov 01 03:18:11.572986 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:11.572966    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-c22dd8-6986b5b4cf-r9pws"
Nov 01 03:18:11.630762 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:11.630737    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-c22dd8\""
Nov 01 03:18:11.630929 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:11.630840    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-c22dd8-dockercfg-r9tjf\""
Nov 01 03:18:11.631562 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:11.631053    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-c22dd8-6986b5b4cf-r9pws"]
Nov 01 03:18:11.633649 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:11.633609    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c1fcbce-b87d-45df-877f-89394755bd40-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-c22dd8-6986b5b4cf-r9pws\" (UID: \"1c1fcbce-b87d-45df-877f-89394755bd40\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-c22dd8-6986b5b4cf-r9pws"
Nov 01 03:18:11.633868 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:11.633810    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbr4n\" (UniqueName: \"kubernetes.io/projected/1c1fcbce-b87d-45df-877f-89394755bd40-kube-api-access-rbr4n\") pod \"payload-processing-e2e-worker-w5-c22dd8-6986b5b4cf-r9pws\" (UID: \"1c1fcbce-b87d-45df-877f-89394755bd40\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-c22dd8-6986b5b4cf-r9pws"
Nov 01 03:18:11.735009 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:11.734986    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbr4n\" (UniqueName: \"kubernetes.io/projected/1c1fcbce-b87d-45df-877f-89394755bd40-kube-api-access-rbr4n\") pod \"payload-processing-e2e-worker-w5-c22dd8-6986b5b4cf-r9pws\" (UID: \"1c1fcbce-b87d-45df-877f-89394755bd40\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-c22dd8-6986b5b4cf-r9pws"
Nov 01 03:18:11.735117 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:11.735031    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c1fcbce-b87d-45df-877f-89394755bd40-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-c22dd8-6986b5b4cf-r9pws\" (UID: \"1c1fcbce-b87d-45df-877f-89394755bd40\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-c22dd8-6986b5b4cf-r9pws"
Nov 01 03:18:11.735737 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:11.735721    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c1fcbce-b87d-45df-877f-89394755bd40-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-c22dd8-6986b5b4cf-r9pws\" (UID: \"1c1fcbce-b87d-45df-877f-89394755bd40\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-c22dd8-6986b5b4cf-r9pws"
Nov 01 03:18:11.797087 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:11.797068    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbr4n\" (UniqueName: \"kubernetes.io/projected/1c1fcbce-b87d-45df-877f-89394755bd40-kube-api-access-rbr4n\") pod \"payload-processing-e2e-worker-w5-c22dd8-6986b5b4cf-r9pws\" (UID: \"1c1fcbce-b87d-45df-877f-89394755bd40\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-c22dd8-6986b5b4cf-r9pws"
Nov 01 03:18:11.887482 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:11.887466    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-c22dd8-6986b5b4cf-r9pws"
Nov 01 03:18:12.263857 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:12.263835    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-c22dd8-6986b5b4cf-r9pws"]
Nov 01 03:18:12.265036 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:18:12.265006    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c1fcbce_b87d_45df_877f_89394755bd40.slice/crio-fb7fa7169957f52d6da0ff53dac0aa77f00e3962a977210bca90515b07f300be WatchSource:0}: Error finding container fb7fa7169957f52d6da0ff53dac0aa77f00e3962a977210bca90515b07f300be: Status 404 returned error can't find the container with id fb7fa7169957f52d6da0ff53dac0aa77f00e3962a977210bca90515b07f300be
Nov 01 03:18:12.628033 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:12.628002    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 03:18:12.879317 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:12.879215    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-c22dd8-6986b5b4cf-r9pws" event={"ID":"1c1fcbce-b87d-45df-877f-89394755bd40","Type":"ContainerStarted","Data":"5dd44e49425b514ac9cfbaf1ab6c857adebdbe1429747db6b3d497edb31f0f0e"}
Nov 01 03:18:12.879317 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:12.879285    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-c22dd8-6986b5b4cf-r9pws" event={"ID":"1c1fcbce-b87d-45df-877f-89394755bd40","Type":"ContainerStarted","Data":"fb7fa7169957f52d6da0ff53dac0aa77f00e3962a977210bca90515b07f300be"}
Nov 01 03:18:12.879548 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:12.879348    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-c22dd8-6986b5b4cf-r9pws"
Nov 01 03:18:13.000387 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:13.000340    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-c22dd8-6986b5b4cf-r9pws" podStartSLOduration=2.000326746 podStartE2EDuration="2.000326746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:18:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:18:12.999419681 +0000 UTC m=+699.514065549" watchObservedRunningTime="2026-11-01 03:18:13.000326746 +0000 UTC m=+699.514972609"
Nov 01 03:18:18.508751 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:18.508721    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-d7f535-6d5d847766-s754n"]
Nov 01 03:18:18.513022 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:18.513006    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d7f535-6d5d847766-s754n"
Nov 01 03:18:18.568262 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:18.568191    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-d7f535-serving-cert\""
Nov 01 03:18:18.571476 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:18.571452    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-d7f535-6d5d847766-s754n"]
Nov 01 03:18:18.607359 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:18.607333    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa3e1de2-5fb8-4692-b87f-5b0cb323fe03-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-d7f535-6d5d847766-s754n\" (UID: \"aa3e1de2-5fb8-4692-b87f-5b0cb323fe03\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d7f535-6d5d847766-s754n"
Nov 01 03:18:18.607470 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:18.607378    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cm75\" (UniqueName: \"kubernetes.io/projected/aa3e1de2-5fb8-4692-b87f-5b0cb323fe03-kube-api-access-7cm75\") pod \"maas-api-e2e-worker-w3-d7f535-6d5d847766-s754n\" (UID: \"aa3e1de2-5fb8-4692-b87f-5b0cb323fe03\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d7f535-6d5d847766-s754n"
Nov 01 03:18:18.607470 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:18.607448    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa3e1de2-5fb8-4692-b87f-5b0cb323fe03-maas-api-tls\") pod \"maas-api-e2e-worker-w3-d7f535-6d5d847766-s754n\" (UID: \"aa3e1de2-5fb8-4692-b87f-5b0cb323fe03\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d7f535-6d5d847766-s754n"
Nov 01 03:18:18.607558 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:18.607538    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa3e1de2-5fb8-4692-b87f-5b0cb323fe03-metrics-certs\") pod \"maas-api-e2e-worker-w3-d7f535-6d5d847766-s754n\" (UID: \"aa3e1de2-5fb8-4692-b87f-5b0cb323fe03\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d7f535-6d5d847766-s754n"
Nov 01 03:18:18.708922 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:18.708892    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa3e1de2-5fb8-4692-b87f-5b0cb323fe03-maas-api-tls\") pod \"maas-api-e2e-worker-w3-d7f535-6d5d847766-s754n\" (UID: \"aa3e1de2-5fb8-4692-b87f-5b0cb323fe03\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d7f535-6d5d847766-s754n"
Nov 01 03:18:18.709066 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:18.708965    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa3e1de2-5fb8-4692-b87f-5b0cb323fe03-metrics-certs\") pod \"maas-api-e2e-worker-w3-d7f535-6d5d847766-s754n\" (UID: \"aa3e1de2-5fb8-4692-b87f-5b0cb323fe03\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d7f535-6d5d847766-s754n"
Nov 01 03:18:18.709066 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:18.709021    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa3e1de2-5fb8-4692-b87f-5b0cb323fe03-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-d7f535-6d5d847766-s754n\" (UID: \"aa3e1de2-5fb8-4692-b87f-5b0cb323fe03\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d7f535-6d5d847766-s754n"
Nov 01 03:18:18.709066 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:18:18.709050    4400 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-d7f535-serving-cert: secret "maas-api-e2e-worker-w3-d7f535-serving-cert" not found
Nov 01 03:18:18.709220 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:18.709087    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cm75\" (UniqueName: \"kubernetes.io/projected/aa3e1de2-5fb8-4692-b87f-5b0cb323fe03-kube-api-access-7cm75\") pod \"maas-api-e2e-worker-w3-d7f535-6d5d847766-s754n\" (UID: \"aa3e1de2-5fb8-4692-b87f-5b0cb323fe03\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d7f535-6d5d847766-s754n"
Nov 01 03:18:18.709220 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:18:18.709136    4400 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa3e1de2-5fb8-4692-b87f-5b0cb323fe03-maas-api-tls podName:aa3e1de2-5fb8-4692-b87f-5b0cb323fe03 nodeName:}" failed. No retries permitted until 2026-11-01 03:18:19.209119073 +0000 UTC m=+705.723764914 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/aa3e1de2-5fb8-4692-b87f-5b0cb323fe03-maas-api-tls") pod "maas-api-e2e-worker-w3-d7f535-6d5d847766-s754n" (UID: "aa3e1de2-5fb8-4692-b87f-5b0cb323fe03") : secret "maas-api-e2e-worker-w3-d7f535-serving-cert" not found
Nov 01 03:18:18.710069 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:18.710043    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa3e1de2-5fb8-4692-b87f-5b0cb323fe03-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-d7f535-6d5d847766-s754n\" (UID: \"aa3e1de2-5fb8-4692-b87f-5b0cb323fe03\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d7f535-6d5d847766-s754n"
Nov 01 03:18:18.711617 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:18.711594    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa3e1de2-5fb8-4692-b87f-5b0cb323fe03-metrics-certs\") pod \"maas-api-e2e-worker-w3-d7f535-6d5d847766-s754n\" (UID: \"aa3e1de2-5fb8-4692-b87f-5b0cb323fe03\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d7f535-6d5d847766-s754n"
Nov 01 03:18:18.773629 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:18.773578    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cm75\" (UniqueName: \"kubernetes.io/projected/aa3e1de2-5fb8-4692-b87f-5b0cb323fe03-kube-api-access-7cm75\") pod \"maas-api-e2e-worker-w3-d7f535-6d5d847766-s754n\" (UID: \"aa3e1de2-5fb8-4692-b87f-5b0cb323fe03\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d7f535-6d5d847766-s754n"
Nov 01 03:18:19.215104 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:19.215070    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa3e1de2-5fb8-4692-b87f-5b0cb323fe03-maas-api-tls\") pod \"maas-api-e2e-worker-w3-d7f535-6d5d847766-s754n\" (UID: \"aa3e1de2-5fb8-4692-b87f-5b0cb323fe03\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d7f535-6d5d847766-s754n"
Nov 01 03:18:19.217730 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:19.217705    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa3e1de2-5fb8-4692-b87f-5b0cb323fe03-maas-api-tls\") pod \"maas-api-e2e-worker-w3-d7f535-6d5d847766-s754n\" (UID: \"aa3e1de2-5fb8-4692-b87f-5b0cb323fe03\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d7f535-6d5d847766-s754n"
Nov 01 03:18:19.425886 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:19.425859    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d7f535-6d5d847766-s754n"
Nov 01 03:18:19.805808 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:19.805781    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-d7f535-6d5d847766-s754n"]
Nov 01 03:18:19.806435 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:18:19.806412    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa3e1de2_5fb8_4692_b87f_5b0cb323fe03.slice/crio-ef236b1156d231475c6b806f26559397c79308d74f48c1faab93e618d409eff3 WatchSource:0}: Error finding container ef236b1156d231475c6b806f26559397c79308d74f48c1faab93e618d409eff3: Status 404 returned error can't find the container with id ef236b1156d231475c6b806f26559397c79308d74f48c1faab93e618d409eff3
Nov 01 03:18:19.921765 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:19.921723    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d7f535-6d5d847766-s754n" event={"ID":"aa3e1de2-5fb8-4692-b87f-5b0cb323fe03","Type":"ContainerStarted","Data":"ef236b1156d231475c6b806f26559397c79308d74f48c1faab93e618d409eff3"}
Nov 01 03:18:20.928409 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:20.928333    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d7f535-6d5d847766-s754n" event={"ID":"aa3e1de2-5fb8-4692-b87f-5b0cb323fe03","Type":"ContainerStarted","Data":"a74b4e8bc2244072bd1de603bacd4e578d65c1332f6b0f1d90a540b0d50e3c54"}
Nov 01 03:18:20.928826 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:20.928433    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d7f535-6d5d847766-s754n"
Nov 01 03:18:21.049237 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:21.049184    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d7f535-6d5d847766-s754n" podStartSLOduration=2.284756299 podStartE2EDuration="3.049170782s" totalImagesPullingTime="764.414483ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:18:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:18:21.048243943 +0000 UTC m=+707.562889817" watchObservedRunningTime="2026-11-01 03:18:21.049170782 +0000 UTC m=+707.563816644"
Nov 01 03:18:21.904098 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:21.901409    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 03:18:23.411161 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:23.411129    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-e8c789-574898d88f-mqljq"]
Nov 01 03:18:23.416132 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:23.416115    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e8c789-574898d88f-mqljq"
Nov 01 03:18:23.457873 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:23.457838    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b31e254-c788-4169-9fc3-d286a54d50d7-maas-api-tls\") pod \"maas-api-e2e-worker-w4-e8c789-574898d88f-mqljq\" (UID: \"9b31e254-c788-4169-9fc3-d286a54d50d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e8c789-574898d88f-mqljq"
Nov 01 03:18:23.457976 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:23.457951    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b31e254-c788-4169-9fc3-d286a54d50d7-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-e8c789-574898d88f-mqljq\" (UID: \"9b31e254-c788-4169-9fc3-d286a54d50d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e8c789-574898d88f-mqljq"
Nov 01 03:18:23.458013 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:23.457993    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b31e254-c788-4169-9fc3-d286a54d50d7-metrics-certs\") pod \"maas-api-e2e-worker-w4-e8c789-574898d88f-mqljq\" (UID: \"9b31e254-c788-4169-9fc3-d286a54d50d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e8c789-574898d88f-mqljq"
Nov 01 03:18:23.458084 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:23.458069    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gk62r\" (UniqueName: \"kubernetes.io/projected/9b31e254-c788-4169-9fc3-d286a54d50d7-kube-api-access-gk62r\") pod \"maas-api-e2e-worker-w4-e8c789-574898d88f-mqljq\" (UID: \"9b31e254-c788-4169-9fc3-d286a54d50d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e8c789-574898d88f-mqljq"
Nov 01 03:18:23.470385 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:23.470360    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-e8c789-serving-cert\""
Nov 01 03:18:23.523730 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:23.523704    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-e8c789-574898d88f-mqljq"]
Nov 01 03:18:23.559232 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:23.559209    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b31e254-c788-4169-9fc3-d286a54d50d7-metrics-certs\") pod \"maas-api-e2e-worker-w4-e8c789-574898d88f-mqljq\" (UID: \"9b31e254-c788-4169-9fc3-d286a54d50d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e8c789-574898d88f-mqljq"
Nov 01 03:18:23.559342 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:23.559280    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gk62r\" (UniqueName: \"kubernetes.io/projected/9b31e254-c788-4169-9fc3-d286a54d50d7-kube-api-access-gk62r\") pod \"maas-api-e2e-worker-w4-e8c789-574898d88f-mqljq\" (UID: \"9b31e254-c788-4169-9fc3-d286a54d50d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e8c789-574898d88f-mqljq"
Nov 01 03:18:23.559385 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:23.559343    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b31e254-c788-4169-9fc3-d286a54d50d7-maas-api-tls\") pod \"maas-api-e2e-worker-w4-e8c789-574898d88f-mqljq\" (UID: \"9b31e254-c788-4169-9fc3-d286a54d50d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e8c789-574898d88f-mqljq"
Nov 01 03:18:23.559454 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:23.559439    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b31e254-c788-4169-9fc3-d286a54d50d7-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-e8c789-574898d88f-mqljq\" (UID: \"9b31e254-c788-4169-9fc3-d286a54d50d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e8c789-574898d88f-mqljq"
Nov 01 03:18:23.559505 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:18:23.559469    4400 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-e8c789-serving-cert: secret "maas-api-e2e-worker-w4-e8c789-serving-cert" not found
Nov 01 03:18:23.559570 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:18:23.559543    4400 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b31e254-c788-4169-9fc3-d286a54d50d7-maas-api-tls podName:9b31e254-c788-4169-9fc3-d286a54d50d7 nodeName:}" failed. No retries permitted until 2026-11-01 03:18:24.059523627 +0000 UTC m=+710.574169489 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9b31e254-c788-4169-9fc3-d286a54d50d7-maas-api-tls") pod "maas-api-e2e-worker-w4-e8c789-574898d88f-mqljq" (UID: "9b31e254-c788-4169-9fc3-d286a54d50d7") : secret "maas-api-e2e-worker-w4-e8c789-serving-cert" not found
Nov 01 03:18:23.560066 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:23.560041    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b31e254-c788-4169-9fc3-d286a54d50d7-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-e8c789-574898d88f-mqljq\" (UID: \"9b31e254-c788-4169-9fc3-d286a54d50d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e8c789-574898d88f-mqljq"
Nov 01 03:18:23.561729 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:23.561677    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b31e254-c788-4169-9fc3-d286a54d50d7-metrics-certs\") pod \"maas-api-e2e-worker-w4-e8c789-574898d88f-mqljq\" (UID: \"9b31e254-c788-4169-9fc3-d286a54d50d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e8c789-574898d88f-mqljq"
Nov 01 03:18:23.623027 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:23.623006    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gk62r\" (UniqueName: \"kubernetes.io/projected/9b31e254-c788-4169-9fc3-d286a54d50d7-kube-api-access-gk62r\") pod \"maas-api-e2e-worker-w4-e8c789-574898d88f-mqljq\" (UID: \"9b31e254-c788-4169-9fc3-d286a54d50d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e8c789-574898d88f-mqljq"
Nov 01 03:18:23.887809 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:23.887784    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-c22dd8-6986b5b4cf-r9pws"
Nov 01 03:18:24.065532 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:24.065499    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b31e254-c788-4169-9fc3-d286a54d50d7-maas-api-tls\") pod \"maas-api-e2e-worker-w4-e8c789-574898d88f-mqljq\" (UID: \"9b31e254-c788-4169-9fc3-d286a54d50d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e8c789-574898d88f-mqljq"
Nov 01 03:18:24.067993 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:24.067972    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b31e254-c788-4169-9fc3-d286a54d50d7-maas-api-tls\") pod \"maas-api-e2e-worker-w4-e8c789-574898d88f-mqljq\" (UID: \"9b31e254-c788-4169-9fc3-d286a54d50d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e8c789-574898d88f-mqljq"
Nov 01 03:18:24.329215 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:24.329189    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e8c789-574898d88f-mqljq"
Nov 01 03:18:24.705776 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:24.705746    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-e8c789-574898d88f-mqljq"]
Nov 01 03:18:24.706884 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:18:24.706859    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b31e254_c788_4169_9fc3_d286a54d50d7.slice/crio-bb9108d8a3f66d5d7a60322f9b7134a789079303be4ca1d39d0551675903288b WatchSource:0}: Error finding container bb9108d8a3f66d5d7a60322f9b7134a789079303be4ca1d39d0551675903288b: Status 404 returned error can't find the container with id bb9108d8a3f66d5d7a60322f9b7134a789079303be4ca1d39d0551675903288b
Nov 01 03:18:24.949480 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:24.949442    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e8c789-574898d88f-mqljq" event={"ID":"9b31e254-c788-4169-9fc3-d286a54d50d7","Type":"ContainerStarted","Data":"bb9108d8a3f66d5d7a60322f9b7134a789079303be4ca1d39d0551675903288b"}
Nov 01 03:18:25.957307 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:25.957272    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e8c789-574898d88f-mqljq" event={"ID":"9b31e254-c788-4169-9fc3-d286a54d50d7","Type":"ContainerStarted","Data":"fcc13b0315515141384a6057237f7c06f59f351c92e287002fd2060985638330"}
Nov 01 03:18:25.957734 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:25.957448    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e8c789-574898d88f-mqljq"
Nov 01 03:18:26.938549 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:26.938515    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d7f535-6d5d847766-s754n"
Nov 01 03:18:27.060141 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:27.060077    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e8c789-574898d88f-mqljq" podStartSLOduration=3.241105372 podStartE2EDuration="4.060064512s" totalImagesPullingTime="818.95914ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:18:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:18:26.083362153 +0000 UTC m=+712.598008002" watchObservedRunningTime="2026-11-01 03:18:27.060064512 +0000 UTC m=+713.574710372"
Nov 01 03:18:31.729859 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:18:31.729826    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:18:31.968310 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:31.968277    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e8c789-574898d88f-mqljq"
Nov 01 03:18:34.692244 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:34.692212    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 03:18:53.598011 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:18:53.597978    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 03:19:01.730823 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:19:01.730787    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:19:20.523063 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:19:20.522982    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 03:19:31.729787 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:19:31.729745    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:19:43.575812 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:19:43.575776    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 03:20:00.492315 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:20:00.492280    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 03:20:01.728764 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:20:01.728729    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:20:22.326435 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:20:22.326393    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 03:20:31.731829 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:20:31.731790    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:20:54.589573 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:20:54.589534    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 03:21:01.729403 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:21:01.729368    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:21:09.398946 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:21:09.398906    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-e8c789/premium-w4-e8c789-kserve-9458f8874-dtpnm"]
Nov 01 03:21:09.403016 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:21:09.402995    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e8c789/premium-w4-e8c789-kserve-9458f8874-dtpnm"
Nov 01 03:21:09.457862 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:21:09.457839    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-e8c789\"/\"premium-w4-e8c789-kserve-self-signed-certs\""
Nov 01 03:21:09.509177 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:21:09.509156    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-e8c789\"/\"openshift-service-ca.crt\""
Nov 01 03:21:09.509177 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:21:09.509158    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-e8c789\"/\"kube-root-ca.crt\""
Nov 01 03:21:09.511491 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:21:09.511466    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-e8c789/premium-w4-e8c789-kserve-9458f8874-dtpnm"]
Nov 01 03:21:09.534060 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:21:09.534034    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67bba91a-5ace-4d3f-89da-06bb9db17c78-model-cache\") pod \"premium-w4-e8c789-kserve-9458f8874-dtpnm\" (UID: \"67bba91a-5ace-4d3f-89da-06bb9db17c78\") " pod="e2e-models-e2e-worker-w4-e8c789/premium-w4-e8c789-kserve-9458f8874-dtpnm"
Nov 01 03:21:09.534163 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:21:09.534066    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67bba91a-5ace-4d3f-89da-06bb9db17c78-dshm\") pod \"premium-w4-e8c789-kserve-9458f8874-dtpnm\" (UID: \"67bba91a-5ace-4d3f-89da-06bb9db17c78\") " pod="e2e-models-e2e-worker-w4-e8c789/premium-w4-e8c789-kserve-9458f8874-dtpnm"
Nov 01 03:21:09.534236 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:21:09.534217    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zngf4\" (UniqueName: \"kubernetes.io/projected/67bba91a-5ace-4d3f-89da-06bb9db17c78-kube-api-access-zngf4\") pod \"premium-w4-e8c789-kserve-9458f8874-dtpnm\" (UID: \"67bba91a-5ace-4d3f-89da-06bb9db17c78\") " pod="e2e-models-e2e-worker-w4-e8c789/premium-w4-e8c789-kserve-9458f8874-dtpnm"
Nov 01 03:21:09.534277 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:21:09.534249    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67bba91a-5ace-4d3f-89da-06bb9db17c78-home\") pod \"premium-w4-e8c789-kserve-9458f8874-dtpnm\" (UID: \"67bba91a-5ace-4d3f-89da-06bb9db17c78\") " pod="e2e-models-e2e-worker-w4-e8c789/premium-w4-e8c789-kserve-9458f8874-dtpnm"
Nov 01 03:21:09.534277 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:21:09.534274    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67bba91a-5ace-4d3f-89da-06bb9db17c78-tmp-dir\") pod \"premium-w4-e8c789-kserve-9458f8874-dtpnm\" (UID: \"67bba91a-5ace-4d3f-89da-06bb9db17c78\") " pod="e2e-models-e2e-worker-w4-e8c789/premium-w4-e8c789-kserve-9458f8874-dtpnm"
Nov 01 03:21:09.534393 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:21:09.534375    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67bba91a-5ace-4d3f-89da-06bb9db17c78-tls-certs\") pod \"premium-w4-e8c789-kserve-9458f8874-dtpnm\" (UID: \"67bba91a-5ace-4d3f-89da-06bb9db17c78\") " pod="e2e-models-e2e-worker-w4-e8c789/premium-w4-e8c789-kserve-9458f8874-dtpnm"
Nov 01 03:21:09.558127 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:21:09.558095    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-e8c789\"/\"default-dockercfg-5fgpq\""
Nov 01 03:21:09.635823 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:21:09.635796    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zngf4\" (UniqueName: \"kubernetes.io/projected/67bba91a-5ace-4d3f-89da-06bb9db17c78-kube-api-access-zngf4\") pod \"premium-w4-e8c789-kserve-9458f8874-dtpnm\" (UID: \"67bba91a-5ace-4d3f-89da-06bb9db17c78\") " pod="e2e-models-e2e-worker-w4-e8c789/premium-w4-e8c789-kserve-9458f8874-dtpnm"
Nov 01 03:21:09.635823 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:21:09.635828    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67bba91a-5ace-4d3f-89da-06bb9db17c78-home\") pod \"premium-w4-e8c789-kserve-9458f8874-dtpnm\" (UID: \"67bba91a-5ace-4d3f-89da-06bb9db17c78\") " pod="e2e-models-e2e-worker-w4-e8c789/premium-w4-e8c789-kserve-9458f8874-dtpnm"
Nov 01 03:21:09.636033 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:21:09.635853    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67bba91a-5ace-4d3f-89da-06bb9db17c78-tmp-dir\") pod \"premium-w4-e8c789-kserve-9458f8874-dtpnm\" (UID: \"67bba91a-5ace-4d3f-89da-06bb9db17c78\") " pod="e2e-models-e2e-worker-w4-e8c789/premium-w4-e8c789-kserve-9458f8874-dtpnm"
Nov 01 03:21:09.636033 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:21:09.635891    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67bba91a-5ace-4d3f-89da-06bb9db17c78-tls-certs\") pod \"premium-w4-e8c789-kserve-9458f8874-dtpnm\" (UID: \"67bba91a-5ace-4d3f-89da-06bb9db17c78\") " pod="e2e-models-e2e-worker-w4-e8c789/premium-w4-e8c789-kserve-9458f8874-dtpnm"
Nov 01 03:21:09.636033 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:21:09.635931    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67bba91a-5ace-4d3f-89da-06bb9db17c78-model-cache\") pod \"premium-w4-e8c789-kserve-9458f8874-dtpnm\" (UID: \"67bba91a-5ace-4d3f-89da-06bb9db17c78\") " pod="e2e-models-e2e-worker-w4-e8c789/premium-w4-e8c789-kserve-9458f8874-dtpnm"
Nov 01 03:21:09.636033 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:21:09.635948    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67bba91a-5ace-4d3f-89da-06bb9db17c78-dshm\") pod \"premium-w4-e8c789-kserve-9458f8874-dtpnm\" (UID: \"67bba91a-5ace-4d3f-89da-06bb9db17c78\") " pod="e2e-models-e2e-worker-w4-e8c789/premium-w4-e8c789-kserve-9458f8874-dtpnm"
Nov 01 03:21:09.636277 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:21:09.636248    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67bba91a-5ace-4d3f-89da-06bb9db17c78-home\") pod \"premium-w4-e8c789-kserve-9458f8874-dtpnm\" (UID: \"67bba91a-5ace-4d3f-89da-06bb9db17c78\") " pod="e2e-models-e2e-worker-w4-e8c789/premium-w4-e8c789-kserve-9458f8874-dtpnm"
Nov 01 03:21:09.636331 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:21:09.636259    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67bba91a-5ace-4d3f-89da-06bb9db17c78-tmp-dir\") pod \"premium-w4-e8c789-kserve-9458f8874-dtpnm\" (UID: \"67bba91a-5ace-4d3f-89da-06bb9db17c78\") " pod="e2e-models-e2e-worker-w4-e8c789/premium-w4-e8c789-kserve-9458f8874-dtpnm"
Nov 01 03:21:09.636331 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:21:09.636302    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67bba91a-5ace-4d3f-89da-06bb9db17c78-model-cache\") pod \"premium-w4-e8c789-kserve-9458f8874-dtpnm\" (UID: \"67bba91a-5ace-4d3f-89da-06bb9db17c78\") " pod="e2e-models-e2e-worker-w4-e8c789/premium-w4-e8c789-kserve-9458f8874-dtpnm"
Nov 01 03:21:09.638199 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:21:09.638183    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67bba91a-5ace-4d3f-89da-06bb9db17c78-dshm\") pod \"premium-w4-e8c789-kserve-9458f8874-dtpnm\" (UID: \"67bba91a-5ace-4d3f-89da-06bb9db17c78\") " pod="e2e-models-e2e-worker-w4-e8c789/premium-w4-e8c789-kserve-9458f8874-dtpnm"
Nov 01 03:21:09.638569 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:21:09.638548    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67bba91a-5ace-4d3f-89da-06bb9db17c78-tls-certs\") pod \"premium-w4-e8c789-kserve-9458f8874-dtpnm\" (UID: \"67bba91a-5ace-4d3f-89da-06bb9db17c78\") " pod="e2e-models-e2e-worker-w4-e8c789/premium-w4-e8c789-kserve-9458f8874-dtpnm"
Nov 01 03:21:09.701332 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:21:09.701270    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zngf4\" (UniqueName: \"kubernetes.io/projected/67bba91a-5ace-4d3f-89da-06bb9db17c78-kube-api-access-zngf4\") pod \"premium-w4-e8c789-kserve-9458f8874-dtpnm\" (UID: \"67bba91a-5ace-4d3f-89da-06bb9db17c78\") " pod="e2e-models-e2e-worker-w4-e8c789/premium-w4-e8c789-kserve-9458f8874-dtpnm"
Nov 01 03:21:09.733670 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:21:09.733646    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e8c789/premium-w4-e8c789-kserve-9458f8874-dtpnm"
Nov 01 03:21:10.125601 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:21:10.125571    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-e8c789/premium-w4-e8c789-kserve-9458f8874-dtpnm"]
Nov 01 03:21:10.126376 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:21:10.126350    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod67bba91a_5ace_4d3f_89da_06bb9db17c78.slice/crio-63e1c9437027d5b2d1d055975caaaac31de26a288c10798d49ab1051ee8e5e7e WatchSource:0}: Error finding container 63e1c9437027d5b2d1d055975caaaac31de26a288c10798d49ab1051ee8e5e7e: Status 404 returned error can't find the container with id 63e1c9437027d5b2d1d055975caaaac31de26a288c10798d49ab1051ee8e5e7e
Nov 01 03:21:10.937114 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:21:10.937075    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e8c789/premium-w4-e8c789-kserve-9458f8874-dtpnm" event={"ID":"67bba91a-5ace-4d3f-89da-06bb9db17c78","Type":"ContainerStarted","Data":"c217396d376bc52bb8ff8290331a2a1f5b616ad67868deec88e48b0863c34a71"}
Nov 01 03:21:10.937114 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:21:10.937117    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e8c789/premium-w4-e8c789-kserve-9458f8874-dtpnm" event={"ID":"67bba91a-5ace-4d3f-89da-06bb9db17c78","Type":"ContainerStarted","Data":"63e1c9437027d5b2d1d055975caaaac31de26a288c10798d49ab1051ee8e5e7e"}
Nov 01 03:21:11.061599 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:21:11.061546    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-e8c789/premium-w4-e8c789-kserve-9458f8874-dtpnm" podStartSLOduration=2.061533198 podStartE2EDuration="2.061533198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:21:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:21:11.06010891 +0000 UTC m=+877.574754769" watchObservedRunningTime="2026-11-01 03:21:11.061533198 +0000 UTC m=+877.576179055"
Nov 01 03:21:11.734018 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:21:11.733989    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e8c789/premium-w4-e8c789-kserve-9458f8874-dtpnm"
Nov 01 03:21:19.734350 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:21:19.734320    4400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-e8c789/premium-w4-e8c789-kserve-9458f8874-dtpnm"
Nov 01 03:21:19.746237 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:21:19.746213    4400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-e8c789/premium-w4-e8c789-kserve-9458f8874-dtpnm"
Nov 01 03:21:20.000028 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:21:19.999943    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e8c789/premium-w4-e8c789-kserve-9458f8874-dtpnm"
Nov 01 03:21:20.392965 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:21:20.392935    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 03:21:22.941170 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:21:22.941142    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-57cp9"]
Nov 01 03:21:22.945173 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:21:22.945149    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-57cp9"
Nov 01 03:21:22.999472 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:21:22.999452    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-b12199-openshift-default-dockercfg-lc8zw\""
Nov 01 03:21:23.052875 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:21:23.052854    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-57cp9"]
Nov 01 03:21:23.059969 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:21:23.059945    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/543cb5b7-afc2-4042-af65-69bf92772c80-credential-socket\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-57cp9\" (UID: \"543cb5b7-afc2-4042-af65-69bf92772c80\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-57cp9"
Nov 01 03:21:23.060077 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:21:23.060051    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8wr2\" (UniqueName: \"kubernetes.io/projected/543cb5b7-afc2-4042-af65-69bf92772c80-kube-api-access-q8wr2\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-57cp9\" (UID: \"543cb5b7-afc2-4042-af65-69bf92772c80\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-57cp9"
Nov 01 03:21:23.060165 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:21:23.060146    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/543cb5b7-afc2-4042-af65-69bf92772c80-istio-token\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-57cp9\" (UID: \"543cb5b7-afc2-4042-af65-69bf92772c80\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-57cp9"
Nov 01 03:21:23.060205 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:21:23.060194    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/543cb5b7-afc2-4042-af65-69bf92772c80-workload-socket\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-57cp9\" (UID: \"543cb5b7-afc2-4042-af65-69bf92772c80\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-57cp9"
Nov 01 03:21:23.060248 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:21:23.060234    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/543cb5b7-afc2-4042-af65-69bf92772c80-workload-certs\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-57cp9\" (UID: \"543cb5b7-afc2-4042-af65-69bf92772c80\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-57cp9"
Nov 01 03:21:23.060325 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:21:23.060309    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/543cb5b7-afc2-4042-af65-69bf92772c80-istio-data\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-57cp9\" (UID: \"543cb5b7-afc2-4042-af65-69bf92772c80\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-57cp9"
Nov 01 03:21:23.060423 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:21:23.060408    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/543cb5b7-afc2-4042-af65-69bf92772c80-istio-envoy\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-57cp9\" (UID: \"543cb5b7-afc2-4042-af65-69bf92772c80\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-57cp9"
Nov 01 03:21:23.060465 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:21:23.060438    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/543cb5b7-afc2-4042-af65-69bf92772c80-istio-podinfo\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-57cp9\" (UID: \"543cb5b7-afc2-4042-af65-69bf92772c80\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-57cp9"
Nov 01 03:21:23.060502 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:21:23.060475    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/543cb5b7-afc2-4042-af65-69bf92772c80-istiod-ca-cert\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-57cp9\" (UID: \"543cb5b7-afc2-4042-af65-69bf92772c80\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-57cp9"
Nov 01 03:21:23.161056 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:21:23.161029    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8wr2\" (UniqueName: \"kubernetes.io/projected/543cb5b7-afc2-4042-af65-69bf92772c80-kube-api-access-q8wr2\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-57cp9\" (UID: \"543cb5b7-afc2-4042-af65-69bf92772c80\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-57cp9"
Nov 01 03:21:23.161200 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:21:23.161087    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/543cb5b7-afc2-4042-af65-69bf92772c80-istio-token\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-57cp9\" (UID: \"543cb5b7-afc2-4042-af65-69bf92772c80\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-57cp9"
Nov 01 03:21:23.161200 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:21:23.161114    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/543cb5b7-afc2-4042-af65-69bf92772c80-workload-socket\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-57cp9\" (UID: \"543cb5b7-afc2-4042-af65-69bf92772c80\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-57cp9"
Nov 01 03:21:23.161200 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:21:23.161140    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/543cb5b7-afc2-4042-af65-69bf92772c80-workload-certs\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-57cp9\" (UID: \"543cb5b7-afc2-4042-af65-69bf92772c80\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-57cp9"
Nov 01 03:21:23.161200 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:21:23.161170    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/543cb5b7-afc2-4042-af65-69bf92772c80-istio-data\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-57cp9\" (UID: \"543cb5b7-afc2-4042-af65-69bf92772c80\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-57cp9"
Nov 01 03:21:23.161427 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:21:23.161255    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/543cb5b7-afc2-4042-af65-69bf92772c80-istio-envoy\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-57cp9\" (UID: \"543cb5b7-afc2-4042-af65-69bf92772c80\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-57cp9"
Nov 01 03:21:23.161427 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:21:23.161280    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/543cb5b7-afc2-4042-af65-69bf92772c80-istio-podinfo\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-57cp9\" (UID: \"543cb5b7-afc2-4042-af65-69bf92772c80\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-57cp9"
Nov 01 03:21:23.161427 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:21:23.161311    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/543cb5b7-afc2-4042-af65-69bf92772c80-istiod-ca-cert\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-57cp9\" (UID: \"543cb5b7-afc2-4042-af65-69bf92772c80\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-57cp9"
Nov 01 03:21:23.161427 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:21:23.161356    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/543cb5b7-afc2-4042-af65-69bf92772c80-credential-socket\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-57cp9\" (UID: \"543cb5b7-afc2-4042-af65-69bf92772c80\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-57cp9"
Nov 01 03:21:23.161639 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:21:23.161543    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/543cb5b7-afc2-4042-af65-69bf92772c80-workload-socket\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-57cp9\" (UID: \"543cb5b7-afc2-4042-af65-69bf92772c80\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-57cp9"
Nov 01 03:21:23.161639 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:21:23.161590    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/543cb5b7-afc2-4042-af65-69bf92772c80-workload-certs\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-57cp9\" (UID: \"543cb5b7-afc2-4042-af65-69bf92772c80\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-57cp9"
Nov 01 03:21:23.161783 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:21:23.161721    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/543cb5b7-afc2-4042-af65-69bf92772c80-credential-socket\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-57cp9\" (UID: \"543cb5b7-afc2-4042-af65-69bf92772c80\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-57cp9"
Nov 01 03:21:23.161868 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:21:23.161847    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/543cb5b7-afc2-4042-af65-69bf92772c80-istio-data\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-57cp9\" (UID: \"543cb5b7-afc2-4042-af65-69bf92772c80\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-57cp9"
Nov 01 03:21:23.162274 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:21:23.162250    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/543cb5b7-afc2-4042-af65-69bf92772c80-istiod-ca-cert\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-57cp9\" (UID: \"543cb5b7-afc2-4042-af65-69bf92772c80\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-57cp9"
Nov 01 03:21:23.163701 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:21:23.163662    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/543cb5b7-afc2-4042-af65-69bf92772c80-istio-envoy\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-57cp9\" (UID: \"543cb5b7-afc2-4042-af65-69bf92772c80\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-57cp9"
Nov 01 03:21:23.164767 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:21:23.164748    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/543cb5b7-afc2-4042-af65-69bf92772c80-istio-podinfo\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-57cp9\" (UID: \"543cb5b7-afc2-4042-af65-69bf92772c80\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-57cp9"
Nov 01 03:21:23.224959 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:21:23.224906    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/543cb5b7-afc2-4042-af65-69bf92772c80-istio-token\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-57cp9\" (UID: \"543cb5b7-afc2-4042-af65-69bf92772c80\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-57cp9"
Nov 01 03:21:23.225251 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:21:23.225228    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8wr2\" (UniqueName: \"kubernetes.io/projected/543cb5b7-afc2-4042-af65-69bf92772c80-kube-api-access-q8wr2\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-57cp9\" (UID: \"543cb5b7-afc2-4042-af65-69bf92772c80\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-57cp9"
Nov 01 03:21:23.270259 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:21:23.269737    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-57cp9"
Nov 01 03:21:23.708657 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:21:23.708628    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-57cp9"]
Nov 01 03:21:23.710435 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:21:23.710407    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod543cb5b7_afc2_4042_af65_69bf92772c80.slice/crio-df171d0e3fa955992cf36d59fbf241249dae4b152b5493dca2ef8e17c451ea14 WatchSource:0}: Error finding container df171d0e3fa955992cf36d59fbf241249dae4b152b5493dca2ef8e17c451ea14: Status 404 returned error can't find the container with id df171d0e3fa955992cf36d59fbf241249dae4b152b5493dca2ef8e17c451ea14
Nov 01 03:21:23.712794 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:21:23.712765    4400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 03:21:23.712881 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:21:23.712823    4400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 03:21:23.712881 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:21:23.712852    4400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 03:21:24.012429 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:21:24.012340    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-57cp9" event={"ID":"543cb5b7-afc2-4042-af65-69bf92772c80","Type":"ContainerStarted","Data":"78f6bd63b99b9fa7a6a4d06f1f1dbbb3a0ba7d1968322da13ca109fa9bc48e90"}
Nov 01 03:21:24.012429 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:21:24.012401    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-57cp9" event={"ID":"543cb5b7-afc2-4042-af65-69bf92772c80","Type":"ContainerStarted","Data":"df171d0e3fa955992cf36d59fbf241249dae4b152b5493dca2ef8e17c451ea14"}
Nov 01 03:21:24.140493 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:21:24.140435    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-57cp9" podStartSLOduration=2.140418471 podStartE2EDuration="2.140418471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:21:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:21:24.138995641 +0000 UTC m=+890.653641517" watchObservedRunningTime="2026-11-01 03:21:24.140418471 +0000 UTC m=+890.655064334"
Nov 01 03:21:24.271647 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:21:24.271559    4400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-57cp9"
Nov 01 03:21:24.273111 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:21:24.273087    4400 patch_prober.go:29] interesting pod/e2e-worker-w1-b12199-openshift-default-774cb8c567-57cp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Nov 01 03:21:24.273241 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:21:24.273130    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-57cp9" podUID="543cb5b7-afc2-4042-af65-69bf92772c80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Nov 01 03:21:25.271433 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:21:25.271392    4400 patch_prober.go:29] interesting pod/e2e-worker-w1-b12199-openshift-default-774cb8c567-57cp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Nov 01 03:21:25.271955 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:21:25.271450    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-57cp9" podUID="543cb5b7-afc2-4042-af65-69bf92772c80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Nov 01 03:21:26.271868 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:21:26.271833    4400 patch_prober.go:29] interesting pod/e2e-worker-w1-b12199-openshift-default-774cb8c567-57cp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Nov 01 03:21:26.272263 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:21:26.271885    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-57cp9" podUID="543cb5b7-afc2-4042-af65-69bf92772c80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Nov 01 03:21:27.271407 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:21:27.271369    4400 patch_prober.go:29] interesting pod/e2e-worker-w1-b12199-openshift-default-774cb8c567-57cp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Nov 01 03:21:27.271580 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:21:27.271426    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-57cp9" podUID="543cb5b7-afc2-4042-af65-69bf92772c80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Nov 01 03:21:28.271319 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:21:28.271288    4400 patch_prober.go:29] interesting pod/e2e-worker-w1-b12199-openshift-default-774cb8c567-57cp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Nov 01 03:21:28.271646 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:21:28.271341    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-57cp9" podUID="543cb5b7-afc2-4042-af65-69bf92772c80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Nov 01 03:21:29.271435 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:21:29.271391    4400 patch_prober.go:29] interesting pod/e2e-worker-w1-b12199-openshift-default-774cb8c567-57cp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Nov 01 03:21:29.271860 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:21:29.271455    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-57cp9" podUID="543cb5b7-afc2-4042-af65-69bf92772c80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Nov 01 03:21:30.321446 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:21:30.321413    4400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-57cp9"
Nov 01 03:21:30.321906 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:21:30.321733    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-57cp9"
Nov 01 03:21:30.322746 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:21:30.322722    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-57cp9"
Nov 01 03:21:31.729655 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:21:31.729615    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:21:46.712598 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:21:46.712564    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 03:22:01.730172 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:22:01.730137    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:22:22.768520 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:22:22.768439    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 03:22:26.273697 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:22:26.273650    4400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-vs8gs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 03:22:26.274168 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:22:26.273757    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vs8gs" podUID="c94ffb2b-3e93-4898-97e8-1de6587c38a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:22:26.628708 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:22:26.624774    4400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-zkjpt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 03:22:26.628708 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:22:26.624844    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zkjpt" podUID="7204663c-ef51-407b-8b56-65e9e54a2d51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:22:31.729818 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:22:31.729782    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:23:01.729256 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:23:01.729218    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:23:15.871636 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:15.871607    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 03:23:24.566659 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:24.566625    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-tl6jm"]
Nov 01 03:23:24.570791 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:24.570765    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-tl6jm"
Nov 01 03:23:24.624343 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:24.624314    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-ded2d7-openshift-default-dockercfg-glqxh\""
Nov 01 03:23:24.662342 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:24.662304    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51a974d4-ac5e-4dc5-b5b4-2fac0f648c06-workload-socket\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-tl6jm\" (UID: \"51a974d4-ac5e-4dc5-b5b4-2fac0f648c06\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-tl6jm"
Nov 01 03:23:24.662541 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:24.662499    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51a974d4-ac5e-4dc5-b5b4-2fac0f648c06-istio-podinfo\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-tl6jm\" (UID: \"51a974d4-ac5e-4dc5-b5b4-2fac0f648c06\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-tl6jm"
Nov 01 03:23:24.662616 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:24.662596    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51a974d4-ac5e-4dc5-b5b4-2fac0f648c06-credential-socket\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-tl6jm\" (UID: \"51a974d4-ac5e-4dc5-b5b4-2fac0f648c06\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-tl6jm"
Nov 01 03:23:24.662676 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:24.662628    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51a974d4-ac5e-4dc5-b5b4-2fac0f648c06-workload-certs\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-tl6jm\" (UID: \"51a974d4-ac5e-4dc5-b5b4-2fac0f648c06\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-tl6jm"
Nov 01 03:23:24.663705 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:24.663664    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51a974d4-ac5e-4dc5-b5b4-2fac0f648c06-istio-envoy\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-tl6jm\" (UID: \"51a974d4-ac5e-4dc5-b5b4-2fac0f648c06\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-tl6jm"
Nov 01 03:23:24.663819 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:24.663722    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51a974d4-ac5e-4dc5-b5b4-2fac0f648c06-istio-data\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-tl6jm\" (UID: \"51a974d4-ac5e-4dc5-b5b4-2fac0f648c06\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-tl6jm"
Nov 01 03:23:24.663819 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:24.663808    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51a974d4-ac5e-4dc5-b5b4-2fac0f648c06-istio-token\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-tl6jm\" (UID: \"51a974d4-ac5e-4dc5-b5b4-2fac0f648c06\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-tl6jm"
Nov 01 03:23:24.663945 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:24.663828    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51a974d4-ac5e-4dc5-b5b4-2fac0f648c06-istiod-ca-cert\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-tl6jm\" (UID: \"51a974d4-ac5e-4dc5-b5b4-2fac0f648c06\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-tl6jm"
Nov 01 03:23:24.663945 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:24.663856    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvt22\" (UniqueName: \"kubernetes.io/projected/51a974d4-ac5e-4dc5-b5b4-2fac0f648c06-kube-api-access-hvt22\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-tl6jm\" (UID: \"51a974d4-ac5e-4dc5-b5b4-2fac0f648c06\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-tl6jm"
Nov 01 03:23:24.678158 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:24.678133    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-tl6jm"]
Nov 01 03:23:24.765026 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:24.764982    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51a974d4-ac5e-4dc5-b5b4-2fac0f648c06-istio-token\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-tl6jm\" (UID: \"51a974d4-ac5e-4dc5-b5b4-2fac0f648c06\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-tl6jm"
Nov 01 03:23:24.765026 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:24.765028    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51a974d4-ac5e-4dc5-b5b4-2fac0f648c06-istiod-ca-cert\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-tl6jm\" (UID: \"51a974d4-ac5e-4dc5-b5b4-2fac0f648c06\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-tl6jm"
Nov 01 03:23:24.765300 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:24.765053    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvt22\" (UniqueName: \"kubernetes.io/projected/51a974d4-ac5e-4dc5-b5b4-2fac0f648c06-kube-api-access-hvt22\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-tl6jm\" (UID: \"51a974d4-ac5e-4dc5-b5b4-2fac0f648c06\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-tl6jm"
Nov 01 03:23:24.765300 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:24.765083    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51a974d4-ac5e-4dc5-b5b4-2fac0f648c06-workload-socket\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-tl6jm\" (UID: \"51a974d4-ac5e-4dc5-b5b4-2fac0f648c06\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-tl6jm"
Nov 01 03:23:24.765300 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:24.765117    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51a974d4-ac5e-4dc5-b5b4-2fac0f648c06-istio-podinfo\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-tl6jm\" (UID: \"51a974d4-ac5e-4dc5-b5b4-2fac0f648c06\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-tl6jm"
Nov 01 03:23:24.765300 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:24.765160    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51a974d4-ac5e-4dc5-b5b4-2fac0f648c06-credential-socket\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-tl6jm\" (UID: \"51a974d4-ac5e-4dc5-b5b4-2fac0f648c06\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-tl6jm"
Nov 01 03:23:24.765497 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:24.765316    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51a974d4-ac5e-4dc5-b5b4-2fac0f648c06-workload-certs\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-tl6jm\" (UID: \"51a974d4-ac5e-4dc5-b5b4-2fac0f648c06\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-tl6jm"
Nov 01 03:23:24.765497 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:24.765395    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51a974d4-ac5e-4dc5-b5b4-2fac0f648c06-istio-envoy\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-tl6jm\" (UID: \"51a974d4-ac5e-4dc5-b5b4-2fac0f648c06\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-tl6jm"
Nov 01 03:23:24.765497 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:24.765439    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51a974d4-ac5e-4dc5-b5b4-2fac0f648c06-istio-data\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-tl6jm\" (UID: \"51a974d4-ac5e-4dc5-b5b4-2fac0f648c06\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-tl6jm"
Nov 01 03:23:24.765652 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:24.765546    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51a974d4-ac5e-4dc5-b5b4-2fac0f648c06-workload-socket\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-tl6jm\" (UID: \"51a974d4-ac5e-4dc5-b5b4-2fac0f648c06\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-tl6jm"
Nov 01 03:23:24.765652 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:24.765574    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51a974d4-ac5e-4dc5-b5b4-2fac0f648c06-credential-socket\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-tl6jm\" (UID: \"51a974d4-ac5e-4dc5-b5b4-2fac0f648c06\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-tl6jm"
Nov 01 03:23:24.765807 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:24.765648    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51a974d4-ac5e-4dc5-b5b4-2fac0f648c06-workload-certs\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-tl6jm\" (UID: \"51a974d4-ac5e-4dc5-b5b4-2fac0f648c06\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-tl6jm"
Nov 01 03:23:24.765891 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:24.765869    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51a974d4-ac5e-4dc5-b5b4-2fac0f648c06-istiod-ca-cert\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-tl6jm\" (UID: \"51a974d4-ac5e-4dc5-b5b4-2fac0f648c06\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-tl6jm"
Nov 01 03:23:24.765929 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:24.765869    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51a974d4-ac5e-4dc5-b5b4-2fac0f648c06-istio-data\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-tl6jm\" (UID: \"51a974d4-ac5e-4dc5-b5b4-2fac0f648c06\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-tl6jm"
Nov 01 03:23:24.767661 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:24.767640    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51a974d4-ac5e-4dc5-b5b4-2fac0f648c06-istio-envoy\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-tl6jm\" (UID: \"51a974d4-ac5e-4dc5-b5b4-2fac0f648c06\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-tl6jm"
Nov 01 03:23:24.767894 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:24.767876    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51a974d4-ac5e-4dc5-b5b4-2fac0f648c06-istio-podinfo\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-tl6jm\" (UID: \"51a974d4-ac5e-4dc5-b5b4-2fac0f648c06\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-tl6jm"
Nov 01 03:23:24.830970 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:24.830935    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51a974d4-ac5e-4dc5-b5b4-2fac0f648c06-istio-token\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-tl6jm\" (UID: \"51a974d4-ac5e-4dc5-b5b4-2fac0f648c06\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-tl6jm"
Nov 01 03:23:24.831110 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:24.831081    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvt22\" (UniqueName: \"kubernetes.io/projected/51a974d4-ac5e-4dc5-b5b4-2fac0f648c06-kube-api-access-hvt22\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-tl6jm\" (UID: \"51a974d4-ac5e-4dc5-b5b4-2fac0f648c06\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-tl6jm"
Nov 01 03:23:24.886458 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:24.886430    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-tl6jm"
Nov 01 03:23:25.315284 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:25.315254    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-tl6jm"]
Nov 01 03:23:25.316270 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:23:25.316236    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51a974d4_ac5e_4dc5_b5b4_2fac0f648c06.slice/crio-4f1a83ca6f15fd85c6392811686b8785943d75b997cbd8c73b7c8a7cea34a408 WatchSource:0}: Error finding container 4f1a83ca6f15fd85c6392811686b8785943d75b997cbd8c73b7c8a7cea34a408: Status 404 returned error can't find the container with id 4f1a83ca6f15fd85c6392811686b8785943d75b997cbd8c73b7c8a7cea34a408
Nov 01 03:23:25.318711 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:25.318656    4400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 03:23:25.318789 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:25.318738    4400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 03:23:25.318789 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:25.318769    4400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 03:23:25.812342 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:25.811804    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-tl6jm" event={"ID":"51a974d4-ac5e-4dc5-b5b4-2fac0f648c06","Type":"ContainerStarted","Data":"f2a4f73c0b087f7e1b0de473be68dab046bdf6875ceaf7484965722206db7cbe"}
Nov 01 03:23:25.812342 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:25.811873    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-tl6jm" event={"ID":"51a974d4-ac5e-4dc5-b5b4-2fac0f648c06","Type":"ContainerStarted","Data":"4f1a83ca6f15fd85c6392811686b8785943d75b997cbd8c73b7c8a7cea34a408"}
Nov 01 03:23:25.886956 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:25.886923    4400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-tl6jm"
Nov 01 03:23:25.936852 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:25.936802    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-tl6jm" podStartSLOduration=1.936788844 podStartE2EDuration="1.936788844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:23:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:23:25.934866065 +0000 UTC m=+1012.449511952" watchObservedRunningTime="2026-11-01 03:23:25.936788844 +0000 UTC m=+1012.451434712"
Nov 01 03:23:26.888965 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:26.888928    4400 patch_prober.go:29] interesting pod/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-tl6jm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Nov 01 03:23:26.889352 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:26.889000    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-tl6jm" podUID="51a974d4-ac5e-4dc5-b5b4-2fac0f648c06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Nov 01 03:23:27.887112 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:27.887080    4400 patch_prober.go:29] interesting pod/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-tl6jm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Nov 01 03:23:27.887279 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:27.887141    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-tl6jm" podUID="51a974d4-ac5e-4dc5-b5b4-2fac0f648c06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Nov 01 03:23:28.887210 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:28.887175    4400 patch_prober.go:29] interesting pod/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-tl6jm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Nov 01 03:23:28.887621 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:28.887237    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-tl6jm" podUID="51a974d4-ac5e-4dc5-b5b4-2fac0f648c06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Nov 01 03:23:29.887498 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:29.887463    4400 patch_prober.go:29] interesting pod/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-tl6jm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Nov 01 03:23:29.887927 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:29.887534    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-tl6jm" podUID="51a974d4-ac5e-4dc5-b5b4-2fac0f648c06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Nov 01 03:23:30.887518 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:30.887479    4400 patch_prober.go:29] interesting pod/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-tl6jm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Nov 01 03:23:30.887955 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:30.887540    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-tl6jm" podUID="51a974d4-ac5e-4dc5-b5b4-2fac0f648c06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Nov 01 03:23:31.729918 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:23:31.729885    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:23:32.226421 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:32.226392    4400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-tl6jm"
Nov 01 03:23:32.226949 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:32.226780    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-tl6jm"
Nov 01 03:23:32.227592 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:32.227573    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-tl6jm"
Nov 01 03:23:47.506196 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:47.506115    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 03:23:55.160179 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:55.160143    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-h57fm"]
Nov 01 03:23:55.164922 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:55.164907    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-h57fm"
Nov 01 03:23:55.218963 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:55.218936    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-d7f535-openshift-default-dockercfg-7hqkk\""
Nov 01 03:23:55.242765 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:55.242730    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjxq5\" (UniqueName: \"kubernetes.io/projected/c6f13d0b-fb65-44b9-b627-ec07234e87bc-kube-api-access-qjxq5\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-h57fm\" (UID: \"c6f13d0b-fb65-44b9-b627-ec07234e87bc\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-h57fm"
Nov 01 03:23:55.242896 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:55.242792    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6f13d0b-fb65-44b9-b627-ec07234e87bc-workload-certs\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-h57fm\" (UID: \"c6f13d0b-fb65-44b9-b627-ec07234e87bc\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-h57fm"
Nov 01 03:23:55.242896 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:55.242870    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6f13d0b-fb65-44b9-b627-ec07234e87bc-istio-data\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-h57fm\" (UID: \"c6f13d0b-fb65-44b9-b627-ec07234e87bc\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-h57fm"
Nov 01 03:23:55.243002 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:55.242926    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6f13d0b-fb65-44b9-b627-ec07234e87bc-istio-podinfo\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-h57fm\" (UID: \"c6f13d0b-fb65-44b9-b627-ec07234e87bc\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-h57fm"
Nov 01 03:23:55.243037 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:55.243008    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6f13d0b-fb65-44b9-b627-ec07234e87bc-istio-envoy\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-h57fm\" (UID: \"c6f13d0b-fb65-44b9-b627-ec07234e87bc\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-h57fm"
Nov 01 03:23:55.243153 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:55.243129    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6f13d0b-fb65-44b9-b627-ec07234e87bc-credential-socket\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-h57fm\" (UID: \"c6f13d0b-fb65-44b9-b627-ec07234e87bc\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-h57fm"
Nov 01 03:23:55.243257 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:55.243162    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6f13d0b-fb65-44b9-b627-ec07234e87bc-istio-token\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-h57fm\" (UID: \"c6f13d0b-fb65-44b9-b627-ec07234e87bc\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-h57fm"
Nov 01 03:23:55.243257 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:55.243192    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6f13d0b-fb65-44b9-b627-ec07234e87bc-workload-socket\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-h57fm\" (UID: \"c6f13d0b-fb65-44b9-b627-ec07234e87bc\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-h57fm"
Nov 01 03:23:55.243257 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:55.243240    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6f13d0b-fb65-44b9-b627-ec07234e87bc-istiod-ca-cert\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-h57fm\" (UID: \"c6f13d0b-fb65-44b9-b627-ec07234e87bc\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-h57fm"
Nov 01 03:23:55.272778 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:55.272749    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-h57fm"]
Nov 01 03:23:55.344424 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:55.344392    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjxq5\" (UniqueName: \"kubernetes.io/projected/c6f13d0b-fb65-44b9-b627-ec07234e87bc-kube-api-access-qjxq5\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-h57fm\" (UID: \"c6f13d0b-fb65-44b9-b627-ec07234e87bc\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-h57fm"
Nov 01 03:23:55.344656 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:55.344449    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6f13d0b-fb65-44b9-b627-ec07234e87bc-workload-certs\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-h57fm\" (UID: \"c6f13d0b-fb65-44b9-b627-ec07234e87bc\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-h57fm"
Nov 01 03:23:55.344656 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:55.344631    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6f13d0b-fb65-44b9-b627-ec07234e87bc-istio-data\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-h57fm\" (UID: \"c6f13d0b-fb65-44b9-b627-ec07234e87bc\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-h57fm"
Nov 01 03:23:55.344765 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:55.344717    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6f13d0b-fb65-44b9-b627-ec07234e87bc-istio-podinfo\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-h57fm\" (UID: \"c6f13d0b-fb65-44b9-b627-ec07234e87bc\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-h57fm"
Nov 01 03:23:55.344765 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:55.344753    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6f13d0b-fb65-44b9-b627-ec07234e87bc-istio-envoy\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-h57fm\" (UID: \"c6f13d0b-fb65-44b9-b627-ec07234e87bc\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-h57fm"
Nov 01 03:23:55.344855 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:55.344842    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6f13d0b-fb65-44b9-b627-ec07234e87bc-credential-socket\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-h57fm\" (UID: \"c6f13d0b-fb65-44b9-b627-ec07234e87bc\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-h57fm"
Nov 01 03:23:55.344897 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:55.344878    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6f13d0b-fb65-44b9-b627-ec07234e87bc-workload-certs\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-h57fm\" (UID: \"c6f13d0b-fb65-44b9-b627-ec07234e87bc\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-h57fm"
Nov 01 03:23:55.344897 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:55.344890    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6f13d0b-fb65-44b9-b627-ec07234e87bc-istio-token\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-h57fm\" (UID: \"c6f13d0b-fb65-44b9-b627-ec07234e87bc\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-h57fm"
Nov 01 03:23:55.344971 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:55.344939    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6f13d0b-fb65-44b9-b627-ec07234e87bc-istio-data\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-h57fm\" (UID: \"c6f13d0b-fb65-44b9-b627-ec07234e87bc\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-h57fm"
Nov 01 03:23:55.345007 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:55.344966    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6f13d0b-fb65-44b9-b627-ec07234e87bc-workload-socket\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-h57fm\" (UID: \"c6f13d0b-fb65-44b9-b627-ec07234e87bc\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-h57fm"
Nov 01 03:23:55.345058 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:55.345003    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6f13d0b-fb65-44b9-b627-ec07234e87bc-istiod-ca-cert\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-h57fm\" (UID: \"c6f13d0b-fb65-44b9-b627-ec07234e87bc\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-h57fm"
Nov 01 03:23:55.345245 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:55.345225    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6f13d0b-fb65-44b9-b627-ec07234e87bc-credential-socket\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-h57fm\" (UID: \"c6f13d0b-fb65-44b9-b627-ec07234e87bc\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-h57fm"
Nov 01 03:23:55.345355 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:55.345311    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6f13d0b-fb65-44b9-b627-ec07234e87bc-workload-socket\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-h57fm\" (UID: \"c6f13d0b-fb65-44b9-b627-ec07234e87bc\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-h57fm"
Nov 01 03:23:55.345662 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:55.345645    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6f13d0b-fb65-44b9-b627-ec07234e87bc-istiod-ca-cert\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-h57fm\" (UID: \"c6f13d0b-fb65-44b9-b627-ec07234e87bc\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-h57fm"
Nov 01 03:23:55.347029 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:55.347012    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6f13d0b-fb65-44b9-b627-ec07234e87bc-istio-envoy\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-h57fm\" (UID: \"c6f13d0b-fb65-44b9-b627-ec07234e87bc\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-h57fm"
Nov 01 03:23:55.347287 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:55.347273    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6f13d0b-fb65-44b9-b627-ec07234e87bc-istio-podinfo\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-h57fm\" (UID: \"c6f13d0b-fb65-44b9-b627-ec07234e87bc\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-h57fm"
Nov 01 03:23:55.407475 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:55.407427    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6f13d0b-fb65-44b9-b627-ec07234e87bc-istio-token\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-h57fm\" (UID: \"c6f13d0b-fb65-44b9-b627-ec07234e87bc\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-h57fm"
Nov 01 03:23:55.407571 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:55.407547    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjxq5\" (UniqueName: \"kubernetes.io/projected/c6f13d0b-fb65-44b9-b627-ec07234e87bc-kube-api-access-qjxq5\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-h57fm\" (UID: \"c6f13d0b-fb65-44b9-b627-ec07234e87bc\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-h57fm"
Nov 01 03:23:55.478885 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:55.478831    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-h57fm"
Nov 01 03:23:55.908345 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:55.908323    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-h57fm"]
Nov 01 03:23:55.908894 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:23:55.908867    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6f13d0b_fb65_44b9_b627_ec07234e87bc.slice/crio-d3aea43c4b74817c08cc1f35bfa69ec5376b96995b501cb3b54035c1f8bdeae4 WatchSource:0}: Error finding container d3aea43c4b74817c08cc1f35bfa69ec5376b96995b501cb3b54035c1f8bdeae4: Status 404 returned error can't find the container with id d3aea43c4b74817c08cc1f35bfa69ec5376b96995b501cb3b54035c1f8bdeae4
Nov 01 03:23:55.911261 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:55.911231    4400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 03:23:55.911352 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:55.911296    4400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 03:23:55.911352 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:55.911323    4400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 03:23:56.009384 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:56.009350    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-h57fm" event={"ID":"c6f13d0b-fb65-44b9-b627-ec07234e87bc","Type":"ContainerStarted","Data":"7b32988cce7d6ed88c8bd44b962326c2d0a18e16f1e464bf945e6bda6e83b638"}
Nov 01 03:23:56.009556 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:56.009415    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-h57fm" event={"ID":"c6f13d0b-fb65-44b9-b627-ec07234e87bc","Type":"ContainerStarted","Data":"d3aea43c4b74817c08cc1f35bfa69ec5376b96995b501cb3b54035c1f8bdeae4"}
Nov 01 03:23:56.136279 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:56.136220    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-h57fm" podStartSLOduration=1.136201042 podStartE2EDuration="1.136201042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:23:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:23:56.132716019 +0000 UTC m=+1042.647361878" watchObservedRunningTime="2026-11-01 03:23:56.136201042 +0000 UTC m=+1042.650846908"
Nov 01 03:23:56.479492 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:56.479460    4400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-h57fm"
Nov 01 03:23:56.481020 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:56.480998    4400 patch_prober.go:29] interesting pod/e2e-worker-w3-d7f535-openshift-default-856bb56cf-h57fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Nov 01 03:23:56.481081 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:56.481044    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-h57fm" podUID="c6f13d0b-fb65-44b9-b627-ec07234e87bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Nov 01 03:23:57.479403 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:57.479365    4400 patch_prober.go:29] interesting pod/e2e-worker-w3-d7f535-openshift-default-856bb56cf-h57fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Nov 01 03:23:57.479578 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:57.479418    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-h57fm" podUID="c6f13d0b-fb65-44b9-b627-ec07234e87bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Nov 01 03:23:58.479768 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:58.479728    4400 patch_prober.go:29] interesting pod/e2e-worker-w3-d7f535-openshift-default-856bb56cf-h57fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Nov 01 03:23:58.480158 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:58.479789    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-h57fm" podUID="c6f13d0b-fb65-44b9-b627-ec07234e87bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Nov 01 03:23:59.479348 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:59.479316    4400 patch_prober.go:29] interesting pod/e2e-worker-w3-d7f535-openshift-default-856bb56cf-h57fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Nov 01 03:23:59.479520 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:23:59.479367    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-h57fm" podUID="c6f13d0b-fb65-44b9-b627-ec07234e87bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Nov 01 03:24:00.479927 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:00.479893    4400 patch_prober.go:29] interesting pod/e2e-worker-w3-d7f535-openshift-default-856bb56cf-h57fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Nov 01 03:24:00.480301 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:00.479950    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-h57fm" podUID="c6f13d0b-fb65-44b9-b627-ec07234e87bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Nov 01 03:24:01.479666 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:01.479635    4400 patch_prober.go:29] interesting pod/e2e-worker-w3-d7f535-openshift-default-856bb56cf-h57fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Nov 01 03:24:01.479857 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:01.479704    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-h57fm" podUID="c6f13d0b-fb65-44b9-b627-ec07234e87bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Nov 01 03:24:01.755138 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:24:01.755058    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:24:02.689440 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:02.689390    4400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-h57fm"
Nov 01 03:24:02.690219 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:02.690194    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-h57fm"
Nov 01 03:24:02.691374 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:02.691346    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-h57fm"
Nov 01 03:24:07.204469 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:07.204434    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-c22dd8/distinct-w5-c22dd8-kserve-5c65c9bcd7-nnf7d"]
Nov 01 03:24:07.208557 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:07.208541    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-w5-c22dd8-kserve-5c65c9bcd7-nnf7d"
Nov 01 03:24:07.300817 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:07.300786    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cd440c6-7191-4fae-a806-f8728975b3cf-dshm\") pod \"distinct-w5-c22dd8-kserve-5c65c9bcd7-nnf7d\" (UID: \"9cd440c6-7191-4fae-a806-f8728975b3cf\") " pod="e2e-models-e2e-worker-w5-c22dd8/distinct-w5-c22dd8-kserve-5c65c9bcd7-nnf7d"
Nov 01 03:24:07.301008 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:07.300845    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cd440c6-7191-4fae-a806-f8728975b3cf-model-cache\") pod \"distinct-w5-c22dd8-kserve-5c65c9bcd7-nnf7d\" (UID: \"9cd440c6-7191-4fae-a806-f8728975b3cf\") " pod="e2e-models-e2e-worker-w5-c22dd8/distinct-w5-c22dd8-kserve-5c65c9bcd7-nnf7d"
Nov 01 03:24:07.301071 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:07.301003    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cd440c6-7191-4fae-a806-f8728975b3cf-tmp-dir\") pod \"distinct-w5-c22dd8-kserve-5c65c9bcd7-nnf7d\" (UID: \"9cd440c6-7191-4fae-a806-f8728975b3cf\") " pod="e2e-models-e2e-worker-w5-c22dd8/distinct-w5-c22dd8-kserve-5c65c9bcd7-nnf7d"
Nov 01 03:24:07.301071 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:07.301044    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsr5l\" (UniqueName: \"kubernetes.io/projected/9cd440c6-7191-4fae-a806-f8728975b3cf-kube-api-access-hsr5l\") pod \"distinct-w5-c22dd8-kserve-5c65c9bcd7-nnf7d\" (UID: \"9cd440c6-7191-4fae-a806-f8728975b3cf\") " pod="e2e-models-e2e-worker-w5-c22dd8/distinct-w5-c22dd8-kserve-5c65c9bcd7-nnf7d"
Nov 01 03:24:07.301071 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:07.301068    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cd440c6-7191-4fae-a806-f8728975b3cf-tls-certs\") pod \"distinct-w5-c22dd8-kserve-5c65c9bcd7-nnf7d\" (UID: \"9cd440c6-7191-4fae-a806-f8728975b3cf\") " pod="e2e-models-e2e-worker-w5-c22dd8/distinct-w5-c22dd8-kserve-5c65c9bcd7-nnf7d"
Nov 01 03:24:07.301190 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:07.301159    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9cd440c6-7191-4fae-a806-f8728975b3cf-home\") pod \"distinct-w5-c22dd8-kserve-5c65c9bcd7-nnf7d\" (UID: \"9cd440c6-7191-4fae-a806-f8728975b3cf\") " pod="e2e-models-e2e-worker-w5-c22dd8/distinct-w5-c22dd8-kserve-5c65c9bcd7-nnf7d"
Nov 01 03:24:07.313271 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:07.313241    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-c22dd8\"/\"kube-root-ca.crt\""
Nov 01 03:24:07.313390 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:07.313318    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-c22dd8\"/\"distinct-w5-c22dd8-kserve-self-signed-certs\""
Nov 01 03:24:07.313390 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:07.313343    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-c22dd8\"/\"openshift-service-ca.crt\""
Nov 01 03:24:07.316101 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:07.316080    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-c22dd8/distinct-w5-c22dd8-kserve-5c65c9bcd7-nnf7d"]
Nov 01 03:24:07.317184 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:07.317172    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-c22dd8\"/\"default-dockercfg-khjq4\""
Nov 01 03:24:07.402037 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:07.401989    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9cd440c6-7191-4fae-a806-f8728975b3cf-home\") pod \"distinct-w5-c22dd8-kserve-5c65c9bcd7-nnf7d\" (UID: \"9cd440c6-7191-4fae-a806-f8728975b3cf\") " pod="e2e-models-e2e-worker-w5-c22dd8/distinct-w5-c22dd8-kserve-5c65c9bcd7-nnf7d"
Nov 01 03:24:07.402248 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:07.402060    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cd440c6-7191-4fae-a806-f8728975b3cf-dshm\") pod \"distinct-w5-c22dd8-kserve-5c65c9bcd7-nnf7d\" (UID: \"9cd440c6-7191-4fae-a806-f8728975b3cf\") " pod="e2e-models-e2e-worker-w5-c22dd8/distinct-w5-c22dd8-kserve-5c65c9bcd7-nnf7d"
Nov 01 03:24:07.402248 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:07.402093    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cd440c6-7191-4fae-a806-f8728975b3cf-model-cache\") pod \"distinct-w5-c22dd8-kserve-5c65c9bcd7-nnf7d\" (UID: \"9cd440c6-7191-4fae-a806-f8728975b3cf\") " pod="e2e-models-e2e-worker-w5-c22dd8/distinct-w5-c22dd8-kserve-5c65c9bcd7-nnf7d"
Nov 01 03:24:07.402248 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:07.402218    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cd440c6-7191-4fae-a806-f8728975b3cf-tmp-dir\") pod \"distinct-w5-c22dd8-kserve-5c65c9bcd7-nnf7d\" (UID: \"9cd440c6-7191-4fae-a806-f8728975b3cf\") " pod="e2e-models-e2e-worker-w5-c22dd8/distinct-w5-c22dd8-kserve-5c65c9bcd7-nnf7d"
Nov 01 03:24:07.402421 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:07.402266    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsr5l\" (UniqueName: \"kubernetes.io/projected/9cd440c6-7191-4fae-a806-f8728975b3cf-kube-api-access-hsr5l\") pod \"distinct-w5-c22dd8-kserve-5c65c9bcd7-nnf7d\" (UID: \"9cd440c6-7191-4fae-a806-f8728975b3cf\") " pod="e2e-models-e2e-worker-w5-c22dd8/distinct-w5-c22dd8-kserve-5c65c9bcd7-nnf7d"
Nov 01 03:24:07.402421 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:07.402292    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cd440c6-7191-4fae-a806-f8728975b3cf-tls-certs\") pod \"distinct-w5-c22dd8-kserve-5c65c9bcd7-nnf7d\" (UID: \"9cd440c6-7191-4fae-a806-f8728975b3cf\") " pod="e2e-models-e2e-worker-w5-c22dd8/distinct-w5-c22dd8-kserve-5c65c9bcd7-nnf7d"
Nov 01 03:24:07.402528 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:07.402457    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9cd440c6-7191-4fae-a806-f8728975b3cf-home\") pod \"distinct-w5-c22dd8-kserve-5c65c9bcd7-nnf7d\" (UID: \"9cd440c6-7191-4fae-a806-f8728975b3cf\") " pod="e2e-models-e2e-worker-w5-c22dd8/distinct-w5-c22dd8-kserve-5c65c9bcd7-nnf7d"
Nov 01 03:24:07.402528 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:07.402518    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cd440c6-7191-4fae-a806-f8728975b3cf-model-cache\") pod \"distinct-w5-c22dd8-kserve-5c65c9bcd7-nnf7d\" (UID: \"9cd440c6-7191-4fae-a806-f8728975b3cf\") " pod="e2e-models-e2e-worker-w5-c22dd8/distinct-w5-c22dd8-kserve-5c65c9bcd7-nnf7d"
Nov 01 03:24:07.402609 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:07.402563    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cd440c6-7191-4fae-a806-f8728975b3cf-tmp-dir\") pod \"distinct-w5-c22dd8-kserve-5c65c9bcd7-nnf7d\" (UID: \"9cd440c6-7191-4fae-a806-f8728975b3cf\") " pod="e2e-models-e2e-worker-w5-c22dd8/distinct-w5-c22dd8-kserve-5c65c9bcd7-nnf7d"
Nov 01 03:24:07.404535 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:07.404514    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cd440c6-7191-4fae-a806-f8728975b3cf-dshm\") pod \"distinct-w5-c22dd8-kserve-5c65c9bcd7-nnf7d\" (UID: \"9cd440c6-7191-4fae-a806-f8728975b3cf\") " pod="e2e-models-e2e-worker-w5-c22dd8/distinct-w5-c22dd8-kserve-5c65c9bcd7-nnf7d"
Nov 01 03:24:07.404852 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:07.404836    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cd440c6-7191-4fae-a806-f8728975b3cf-tls-certs\") pod \"distinct-w5-c22dd8-kserve-5c65c9bcd7-nnf7d\" (UID: \"9cd440c6-7191-4fae-a806-f8728975b3cf\") " pod="e2e-models-e2e-worker-w5-c22dd8/distinct-w5-c22dd8-kserve-5c65c9bcd7-nnf7d"
Nov 01 03:24:07.472282 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:07.472208    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsr5l\" (UniqueName: \"kubernetes.io/projected/9cd440c6-7191-4fae-a806-f8728975b3cf-kube-api-access-hsr5l\") pod \"distinct-w5-c22dd8-kserve-5c65c9bcd7-nnf7d\" (UID: \"9cd440c6-7191-4fae-a806-f8728975b3cf\") " pod="e2e-models-e2e-worker-w5-c22dd8/distinct-w5-c22dd8-kserve-5c65c9bcd7-nnf7d"
Nov 01 03:24:07.521467 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:07.521435    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-w5-c22dd8-kserve-5c65c9bcd7-nnf7d"
Nov 01 03:24:07.923101 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:07.923074    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-c22dd8/distinct-w5-c22dd8-kserve-5c65c9bcd7-nnf7d"]
Nov 01 03:24:07.923915 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:24:07.923888    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9cd440c6_7191_4fae_a806_f8728975b3cf.slice/crio-b741237d2ccf3e77a3acd97fea72e0f7fc4b4a0f1066994e69d380fc543e6dd5 WatchSource:0}: Error finding container b741237d2ccf3e77a3acd97fea72e0f7fc4b4a0f1066994e69d380fc543e6dd5: Status 404 returned error can't find the container with id b741237d2ccf3e77a3acd97fea72e0f7fc4b4a0f1066994e69d380fc543e6dd5
Nov 01 03:24:08.098650 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:08.098615    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-w5-c22dd8-kserve-5c65c9bcd7-nnf7d" event={"ID":"9cd440c6-7191-4fae-a806-f8728975b3cf","Type":"ContainerStarted","Data":"ad9717c2c4f658e6682eb6e046f42a4a1a10e2fc0876e2db5a78c8aacffc45e0"}
Nov 01 03:24:08.098847 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:08.098655    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-w5-c22dd8-kserve-5c65c9bcd7-nnf7d" event={"ID":"9cd440c6-7191-4fae-a806-f8728975b3cf","Type":"ContainerStarted","Data":"b741237d2ccf3e77a3acd97fea72e0f7fc4b4a0f1066994e69d380fc543e6dd5"}
Nov 01 03:24:08.234425 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:08.234310    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-w5-c22dd8-kserve-5c65c9bcd7-nnf7d" podStartSLOduration=1.234296161 podStartE2EDuration="1.234296161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:24:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:24:08.230586512 +0000 UTC m=+1054.745232378" watchObservedRunningTime="2026-11-01 03:24:08.234296161 +0000 UTC m=+1054.748942071"
Nov 01 03:24:08.522259 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:08.522167    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-w5-c22dd8-kserve-5c65c9bcd7-nnf7d"
Nov 01 03:24:10.920036 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:10.919998    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-7pxrv"]
Nov 01 03:24:10.924279 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:10.924250    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-7pxrv"
Nov 01 03:24:10.962856 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:10.962815    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/925a7174-5bbd-4400-869a-c3b62c79199e-istio-token\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-7pxrv\" (UID: \"925a7174-5bbd-4400-869a-c3b62c79199e\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-7pxrv"
Nov 01 03:24:10.963052 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:10.962863    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/925a7174-5bbd-4400-869a-c3b62c79199e-istio-envoy\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-7pxrv\" (UID: \"925a7174-5bbd-4400-869a-c3b62c79199e\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-7pxrv"
Nov 01 03:24:10.963052 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:10.962895    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8bfx\" (UniqueName: \"kubernetes.io/projected/925a7174-5bbd-4400-869a-c3b62c79199e-kube-api-access-f8bfx\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-7pxrv\" (UID: \"925a7174-5bbd-4400-869a-c3b62c79199e\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-7pxrv"
Nov 01 03:24:10.963052 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:10.962987    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/925a7174-5bbd-4400-869a-c3b62c79199e-workload-certs\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-7pxrv\" (UID: \"925a7174-5bbd-4400-869a-c3b62c79199e\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-7pxrv"
Nov 01 03:24:10.963052 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:10.963024    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/925a7174-5bbd-4400-869a-c3b62c79199e-istio-data\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-7pxrv\" (UID: \"925a7174-5bbd-4400-869a-c3b62c79199e\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-7pxrv"
Nov 01 03:24:10.963258 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:10.963135    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/925a7174-5bbd-4400-869a-c3b62c79199e-istiod-ca-cert\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-7pxrv\" (UID: \"925a7174-5bbd-4400-869a-c3b62c79199e\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-7pxrv"
Nov 01 03:24:10.963258 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:10.963197    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/925a7174-5bbd-4400-869a-c3b62c79199e-credential-socket\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-7pxrv\" (UID: \"925a7174-5bbd-4400-869a-c3b62c79199e\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-7pxrv"
Nov 01 03:24:10.963331 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:10.963285    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/925a7174-5bbd-4400-869a-c3b62c79199e-workload-socket\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-7pxrv\" (UID: \"925a7174-5bbd-4400-869a-c3b62c79199e\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-7pxrv"
Nov 01 03:24:10.963389 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:10.963321    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/925a7174-5bbd-4400-869a-c3b62c79199e-istio-podinfo\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-7pxrv\" (UID: \"925a7174-5bbd-4400-869a-c3b62c79199e\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-7pxrv"
Nov 01 03:24:10.978036 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:10.977996    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-c8cb9a-openshift-default-dockercfg-r4cfc\""
Nov 01 03:24:11.033571 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:11.033535    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-7pxrv"]
Nov 01 03:24:11.064068 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:11.064032    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/925a7174-5bbd-4400-869a-c3b62c79199e-istio-token\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-7pxrv\" (UID: \"925a7174-5bbd-4400-869a-c3b62c79199e\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-7pxrv"
Nov 01 03:24:11.064068 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:11.064070    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/925a7174-5bbd-4400-869a-c3b62c79199e-istio-envoy\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-7pxrv\" (UID: \"925a7174-5bbd-4400-869a-c3b62c79199e\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-7pxrv"
Nov 01 03:24:11.064271 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:11.064093    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8bfx\" (UniqueName: \"kubernetes.io/projected/925a7174-5bbd-4400-869a-c3b62c79199e-kube-api-access-f8bfx\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-7pxrv\" (UID: \"925a7174-5bbd-4400-869a-c3b62c79199e\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-7pxrv"
Nov 01 03:24:11.064271 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:11.064220    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/925a7174-5bbd-4400-869a-c3b62c79199e-workload-certs\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-7pxrv\" (UID: \"925a7174-5bbd-4400-869a-c3b62c79199e\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-7pxrv"
Nov 01 03:24:11.064381 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:11.064279    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/925a7174-5bbd-4400-869a-c3b62c79199e-istio-data\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-7pxrv\" (UID: \"925a7174-5bbd-4400-869a-c3b62c79199e\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-7pxrv"
Nov 01 03:24:11.064381 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:11.064345    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/925a7174-5bbd-4400-869a-c3b62c79199e-istiod-ca-cert\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-7pxrv\" (UID: \"925a7174-5bbd-4400-869a-c3b62c79199e\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-7pxrv"
Nov 01 03:24:11.064467 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:11.064404    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/925a7174-5bbd-4400-869a-c3b62c79199e-credential-socket\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-7pxrv\" (UID: \"925a7174-5bbd-4400-869a-c3b62c79199e\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-7pxrv"
Nov 01 03:24:11.064516 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:11.064499    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/925a7174-5bbd-4400-869a-c3b62c79199e-workload-socket\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-7pxrv\" (UID: \"925a7174-5bbd-4400-869a-c3b62c79199e\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-7pxrv"
Nov 01 03:24:11.064617 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:11.064547    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/925a7174-5bbd-4400-869a-c3b62c79199e-istio-podinfo\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-7pxrv\" (UID: \"925a7174-5bbd-4400-869a-c3b62c79199e\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-7pxrv"
Nov 01 03:24:11.064730 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:11.064658    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/925a7174-5bbd-4400-869a-c3b62c79199e-istio-data\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-7pxrv\" (UID: \"925a7174-5bbd-4400-869a-c3b62c79199e\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-7pxrv"
Nov 01 03:24:11.064730 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:11.064723    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/925a7174-5bbd-4400-869a-c3b62c79199e-workload-certs\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-7pxrv\" (UID: \"925a7174-5bbd-4400-869a-c3b62c79199e\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-7pxrv"
Nov 01 03:24:11.064929 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:11.064906    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/925a7174-5bbd-4400-869a-c3b62c79199e-workload-socket\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-7pxrv\" (UID: \"925a7174-5bbd-4400-869a-c3b62c79199e\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-7pxrv"
Nov 01 03:24:11.064992 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:11.064943    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/925a7174-5bbd-4400-869a-c3b62c79199e-credential-socket\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-7pxrv\" (UID: \"925a7174-5bbd-4400-869a-c3b62c79199e\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-7pxrv"
Nov 01 03:24:11.065422 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:11.065388    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/925a7174-5bbd-4400-869a-c3b62c79199e-istiod-ca-cert\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-7pxrv\" (UID: \"925a7174-5bbd-4400-869a-c3b62c79199e\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-7pxrv"
Nov 01 03:24:11.066802 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:11.066781    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/925a7174-5bbd-4400-869a-c3b62c79199e-istio-envoy\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-7pxrv\" (UID: \"925a7174-5bbd-4400-869a-c3b62c79199e\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-7pxrv"
Nov 01 03:24:11.067290 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:11.067270    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/925a7174-5bbd-4400-869a-c3b62c79199e-istio-podinfo\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-7pxrv\" (UID: \"925a7174-5bbd-4400-869a-c3b62c79199e\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-7pxrv"
Nov 01 03:24:11.127121 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:11.127083    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/925a7174-5bbd-4400-869a-c3b62c79199e-istio-token\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-7pxrv\" (UID: \"925a7174-5bbd-4400-869a-c3b62c79199e\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-7pxrv"
Nov 01 03:24:11.127272 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:11.127150    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8bfx\" (UniqueName: \"kubernetes.io/projected/925a7174-5bbd-4400-869a-c3b62c79199e-kube-api-access-f8bfx\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-7pxrv\" (UID: \"925a7174-5bbd-4400-869a-c3b62c79199e\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-7pxrv"
Nov 01 03:24:11.239959 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:11.239874    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-7pxrv"
Nov 01 03:24:11.672867 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:11.672838    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-7pxrv"]
Nov 01 03:24:11.674282 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:24:11.674254    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod925a7174_5bbd_4400_869a_c3b62c79199e.slice/crio-5459c3bef4a99f337d1599c19577643697d2d29055350be670590003bc26f2fa WatchSource:0}: Error finding container 5459c3bef4a99f337d1599c19577643697d2d29055350be670590003bc26f2fa: Status 404 returned error can't find the container with id 5459c3bef4a99f337d1599c19577643697d2d29055350be670590003bc26f2fa
Nov 01 03:24:11.676792 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:11.676763    4400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 03:24:11.676866 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:11.676827    4400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 03:24:11.676866 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:11.676858    4400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 03:24:12.131774 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:12.131737    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-7pxrv" event={"ID":"925a7174-5bbd-4400-869a-c3b62c79199e","Type":"ContainerStarted","Data":"718caceaa9e3e2ecb3957206d063b432e74ff5f13c6dcc5aaa7bfc51d2bd4e2b"}
Nov 01 03:24:12.132187 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:12.131801    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-7pxrv" event={"ID":"925a7174-5bbd-4400-869a-c3b62c79199e","Type":"ContainerStarted","Data":"5459c3bef4a99f337d1599c19577643697d2d29055350be670590003bc26f2fa"}
Nov 01 03:24:12.240066 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:12.240027    4400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-7pxrv"
Nov 01 03:24:12.241969 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:12.241943    4400 patch_prober.go:29] interesting pod/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-7pxrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Nov 01 03:24:12.242090 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:12.241990    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-7pxrv" podUID="925a7174-5bbd-4400-869a-c3b62c79199e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Nov 01 03:24:12.255926 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:12.255486    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-7pxrv" podStartSLOduration=2.255469451 podStartE2EDuration="2.255469451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:24:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:24:12.254622238 +0000 UTC m=+1058.769268101" watchObservedRunningTime="2026-11-01 03:24:12.255469451 +0000 UTC m=+1058.770115315"
Nov 01 03:24:13.240509 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:13.240473    4400 patch_prober.go:29] interesting pod/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-7pxrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Nov 01 03:24:13.240911 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:13.240528    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-7pxrv" podUID="925a7174-5bbd-4400-869a-c3b62c79199e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Nov 01 03:24:14.241190 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:14.241155    4400 patch_prober.go:29] interesting pod/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-7pxrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Nov 01 03:24:14.241585 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:14.241225    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-7pxrv" podUID="925a7174-5bbd-4400-869a-c3b62c79199e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Nov 01 03:24:15.240668 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:15.240629    4400 patch_prober.go:29] interesting pod/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-7pxrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Nov 01 03:24:15.240842 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:15.240706    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-7pxrv" podUID="925a7174-5bbd-4400-869a-c3b62c79199e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Nov 01 03:24:15.570930 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:15.570895    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 03:24:16.241180 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:16.241147    4400 patch_prober.go:29] interesting pod/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-7pxrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Nov 01 03:24:16.241344 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:16.241204    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-7pxrv" podUID="925a7174-5bbd-4400-869a-c3b62c79199e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Nov 01 03:24:17.241343 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:17.241309    4400 patch_prober.go:29] interesting pod/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-7pxrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Nov 01 03:24:17.241800 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:17.241377    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-7pxrv" podUID="925a7174-5bbd-4400-869a-c3b62c79199e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Nov 01 03:24:17.526762 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:17.526148    4400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-w5-c22dd8-kserve-5c65c9bcd7-nnf7d"
Nov 01 03:24:17.543095 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:17.543061    4400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-w5-c22dd8-kserve-5c65c9bcd7-nnf7d"
Nov 01 03:24:18.197192 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:18.197166    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-w5-c22dd8-kserve-5c65c9bcd7-nnf7d"
Nov 01 03:24:18.240704 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:18.240666    4400 patch_prober.go:29] interesting pod/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-7pxrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Nov 01 03:24:18.240873 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:18.240743    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-7pxrv" podUID="925a7174-5bbd-4400-869a-c3b62c79199e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Nov 01 03:24:19.244592 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:19.244563    4400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-7pxrv"
Nov 01 03:24:19.245077 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:19.244896    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-7pxrv"
Nov 01 03:24:19.245519 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:19.245499    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-7pxrv"
Nov 01 03:24:31.729881 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:24:31.729838    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:24:45.191873 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:45.191835    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-d7f535/distinct-w3-d7f535-kserve-7d69fb9558-88lq8"]
Nov 01 03:24:45.197106 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:45.197090    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d7f535/distinct-w3-d7f535-kserve-7d69fb9558-88lq8"
Nov 01 03:24:45.256272 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:45.256237    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-d7f535/distinct-w3-d7f535-kserve-7d69fb9558-88lq8"]
Nov 01 03:24:45.263402 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:45.263378    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65358d08-83e6-45c5-a17e-f1dbde164345-home\") pod \"distinct-w3-d7f535-kserve-7d69fb9558-88lq8\" (UID: \"65358d08-83e6-45c5-a17e-f1dbde164345\") " pod="e2e-models-e2e-worker-w3-d7f535/distinct-w3-d7f535-kserve-7d69fb9558-88lq8"
Nov 01 03:24:45.263571 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:45.263410    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65358d08-83e6-45c5-a17e-f1dbde164345-model-cache\") pod \"distinct-w3-d7f535-kserve-7d69fb9558-88lq8\" (UID: \"65358d08-83e6-45c5-a17e-f1dbde164345\") " pod="e2e-models-e2e-worker-w3-d7f535/distinct-w3-d7f535-kserve-7d69fb9558-88lq8"
Nov 01 03:24:45.263571 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:45.263435    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65358d08-83e6-45c5-a17e-f1dbde164345-dshm\") pod \"distinct-w3-d7f535-kserve-7d69fb9558-88lq8\" (UID: \"65358d08-83e6-45c5-a17e-f1dbde164345\") " pod="e2e-models-e2e-worker-w3-d7f535/distinct-w3-d7f535-kserve-7d69fb9558-88lq8"
Nov 01 03:24:45.263713 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:45.263612    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65358d08-83e6-45c5-a17e-f1dbde164345-tmp-dir\") pod \"distinct-w3-d7f535-kserve-7d69fb9558-88lq8\" (UID: \"65358d08-83e6-45c5-a17e-f1dbde164345\") " pod="e2e-models-e2e-worker-w3-d7f535/distinct-w3-d7f535-kserve-7d69fb9558-88lq8"
Nov 01 03:24:45.263776 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:45.263755    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65358d08-83e6-45c5-a17e-f1dbde164345-tls-certs\") pod \"distinct-w3-d7f535-kserve-7d69fb9558-88lq8\" (UID: \"65358d08-83e6-45c5-a17e-f1dbde164345\") " pod="e2e-models-e2e-worker-w3-d7f535/distinct-w3-d7f535-kserve-7d69fb9558-88lq8"
Nov 01 03:24:45.263829 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:45.263787    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlfxt\" (UniqueName: \"kubernetes.io/projected/65358d08-83e6-45c5-a17e-f1dbde164345-kube-api-access-qlfxt\") pod \"distinct-w3-d7f535-kserve-7d69fb9558-88lq8\" (UID: \"65358d08-83e6-45c5-a17e-f1dbde164345\") " pod="e2e-models-e2e-worker-w3-d7f535/distinct-w3-d7f535-kserve-7d69fb9558-88lq8"
Nov 01 03:24:45.304004 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:45.303979    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-d7f535\"/\"openshift-service-ca.crt\""
Nov 01 03:24:45.304147 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:45.304013    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-d7f535\"/\"kube-root-ca.crt\""
Nov 01 03:24:45.304147 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:45.304013    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-d7f535\"/\"default-dockercfg-gw8jb\""
Nov 01 03:24:45.304147 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:45.304113    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-d7f535\"/\"distinct-w3-d7f535-kserve-self-signed-certs\""
Nov 01 03:24:45.375042 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:45.375003    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65358d08-83e6-45c5-a17e-f1dbde164345-tls-certs\") pod \"distinct-w3-d7f535-kserve-7d69fb9558-88lq8\" (UID: \"65358d08-83e6-45c5-a17e-f1dbde164345\") " pod="e2e-models-e2e-worker-w3-d7f535/distinct-w3-d7f535-kserve-7d69fb9558-88lq8"
Nov 01 03:24:45.377404 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:45.377366    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlfxt\" (UniqueName: \"kubernetes.io/projected/65358d08-83e6-45c5-a17e-f1dbde164345-kube-api-access-qlfxt\") pod \"distinct-w3-d7f535-kserve-7d69fb9558-88lq8\" (UID: \"65358d08-83e6-45c5-a17e-f1dbde164345\") " pod="e2e-models-e2e-worker-w3-d7f535/distinct-w3-d7f535-kserve-7d69fb9558-88lq8"
Nov 01 03:24:45.378244 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:45.378224    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65358d08-83e6-45c5-a17e-f1dbde164345-home\") pod \"distinct-w3-d7f535-kserve-7d69fb9558-88lq8\" (UID: \"65358d08-83e6-45c5-a17e-f1dbde164345\") " pod="e2e-models-e2e-worker-w3-d7f535/distinct-w3-d7f535-kserve-7d69fb9558-88lq8"
Nov 01 03:24:45.378423 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:45.378405    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65358d08-83e6-45c5-a17e-f1dbde164345-model-cache\") pod \"distinct-w3-d7f535-kserve-7d69fb9558-88lq8\" (UID: \"65358d08-83e6-45c5-a17e-f1dbde164345\") " pod="e2e-models-e2e-worker-w3-d7f535/distinct-w3-d7f535-kserve-7d69fb9558-88lq8"
Nov 01 03:24:45.378556 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:45.378541    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65358d08-83e6-45c5-a17e-f1dbde164345-dshm\") pod \"distinct-w3-d7f535-kserve-7d69fb9558-88lq8\" (UID: \"65358d08-83e6-45c5-a17e-f1dbde164345\") " pod="e2e-models-e2e-worker-w3-d7f535/distinct-w3-d7f535-kserve-7d69fb9558-88lq8"
Nov 01 03:24:45.378741 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:45.378725    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65358d08-83e6-45c5-a17e-f1dbde164345-tmp-dir\") pod \"distinct-w3-d7f535-kserve-7d69fb9558-88lq8\" (UID: \"65358d08-83e6-45c5-a17e-f1dbde164345\") " pod="e2e-models-e2e-worker-w3-d7f535/distinct-w3-d7f535-kserve-7d69fb9558-88lq8"
Nov 01 03:24:45.379271 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:45.379247    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65358d08-83e6-45c5-a17e-f1dbde164345-tmp-dir\") pod \"distinct-w3-d7f535-kserve-7d69fb9558-88lq8\" (UID: \"65358d08-83e6-45c5-a17e-f1dbde164345\") " pod="e2e-models-e2e-worker-w3-d7f535/distinct-w3-d7f535-kserve-7d69fb9558-88lq8"
Nov 01 03:24:45.379549 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:45.379517    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65358d08-83e6-45c5-a17e-f1dbde164345-home\") pod \"distinct-w3-d7f535-kserve-7d69fb9558-88lq8\" (UID: \"65358d08-83e6-45c5-a17e-f1dbde164345\") " pod="e2e-models-e2e-worker-w3-d7f535/distinct-w3-d7f535-kserve-7d69fb9558-88lq8"
Nov 01 03:24:45.381956 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:45.379783    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65358d08-83e6-45c5-a17e-f1dbde164345-model-cache\") pod \"distinct-w3-d7f535-kserve-7d69fb9558-88lq8\" (UID: \"65358d08-83e6-45c5-a17e-f1dbde164345\") " pod="e2e-models-e2e-worker-w3-d7f535/distinct-w3-d7f535-kserve-7d69fb9558-88lq8"
Nov 01 03:24:45.391017 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:45.390991    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65358d08-83e6-45c5-a17e-f1dbde164345-dshm\") pod \"distinct-w3-d7f535-kserve-7d69fb9558-88lq8\" (UID: \"65358d08-83e6-45c5-a17e-f1dbde164345\") " pod="e2e-models-e2e-worker-w3-d7f535/distinct-w3-d7f535-kserve-7d69fb9558-88lq8"
Nov 01 03:24:45.391293 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:45.391272    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65358d08-83e6-45c5-a17e-f1dbde164345-tls-certs\") pod \"distinct-w3-d7f535-kserve-7d69fb9558-88lq8\" (UID: \"65358d08-83e6-45c5-a17e-f1dbde164345\") " pod="e2e-models-e2e-worker-w3-d7f535/distinct-w3-d7f535-kserve-7d69fb9558-88lq8"
Nov 01 03:24:45.439903 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:45.439878    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlfxt\" (UniqueName: \"kubernetes.io/projected/65358d08-83e6-45c5-a17e-f1dbde164345-kube-api-access-qlfxt\") pod \"distinct-w3-d7f535-kserve-7d69fb9558-88lq8\" (UID: \"65358d08-83e6-45c5-a17e-f1dbde164345\") " pod="e2e-models-e2e-worker-w3-d7f535/distinct-w3-d7f535-kserve-7d69fb9558-88lq8"
Nov 01 03:24:45.509955 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:45.509890    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d7f535/distinct-w3-d7f535-kserve-7d69fb9558-88lq8"
Nov 01 03:24:45.922653 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:45.919450    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-d7f535/distinct-w3-d7f535-kserve-7d69fb9558-88lq8"]
Nov 01 03:24:45.922653 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:24:45.922530    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod65358d08_83e6_45c5_a17e_f1dbde164345.slice/crio-5340cf56d9db1941d947404f2bfc1632048158212a8c75f82ee4340f185a1cb4 WatchSource:0}: Error finding container 5340cf56d9db1941d947404f2bfc1632048158212a8c75f82ee4340f185a1cb4: Status 404 returned error can't find the container with id 5340cf56d9db1941d947404f2bfc1632048158212a8c75f82ee4340f185a1cb4
Nov 01 03:24:46.397979 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:46.397945    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d7f535/distinct-w3-d7f535-kserve-7d69fb9558-88lq8" event={"ID":"65358d08-83e6-45c5-a17e-f1dbde164345","Type":"ContainerStarted","Data":"5358aa2bcf8023a7cc57d249f7e1b5d1d2964a94922972b7c0dab7e001a04b16"}
Nov 01 03:24:46.398412 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:46.397992    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d7f535/distinct-w3-d7f535-kserve-7d69fb9558-88lq8" event={"ID":"65358d08-83e6-45c5-a17e-f1dbde164345","Type":"ContainerStarted","Data":"5340cf56d9db1941d947404f2bfc1632048158212a8c75f82ee4340f185a1cb4"}
Nov 01 03:24:46.510162 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:46.510121    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d7f535/distinct-w3-d7f535-kserve-7d69fb9558-88lq8"
Nov 01 03:24:46.519328 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:46.519280    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-d7f535/distinct-w3-d7f535-kserve-7d69fb9558-88lq8" podStartSLOduration=1.519267082 podStartE2EDuration="1.519267082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:24:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:24:46.516612405 +0000 UTC m=+1093.031258270" watchObservedRunningTime="2026-11-01 03:24:46.519267082 +0000 UTC m=+1093.033912944"
Nov 01 03:24:54.633184 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:54.633147    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-hnd5d"]
Nov 01 03:24:54.651535 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:54.651507    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-hnd5d"
Nov 01 03:24:54.676079 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:54.676055    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a03a2ca3-9463-4838-8c2f-a37293da9e6a-workload-socket\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-hnd5d\" (UID: \"a03a2ca3-9463-4838-8c2f-a37293da9e6a\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-hnd5d"
Nov 01 03:24:54.676208 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:54.676090    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77cr5\" (UniqueName: \"kubernetes.io/projected/a03a2ca3-9463-4838-8c2f-a37293da9e6a-kube-api-access-77cr5\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-hnd5d\" (UID: \"a03a2ca3-9463-4838-8c2f-a37293da9e6a\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-hnd5d"
Nov 01 03:24:54.676208 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:54.676110    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a03a2ca3-9463-4838-8c2f-a37293da9e6a-istio-envoy\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-hnd5d\" (UID: \"a03a2ca3-9463-4838-8c2f-a37293da9e6a\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-hnd5d"
Nov 01 03:24:54.676278 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:54.676204    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a03a2ca3-9463-4838-8c2f-a37293da9e6a-istio-podinfo\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-hnd5d\" (UID: \"a03a2ca3-9463-4838-8c2f-a37293da9e6a\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-hnd5d"
Nov 01 03:24:54.676278 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:54.676253    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a03a2ca3-9463-4838-8c2f-a37293da9e6a-credential-socket\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-hnd5d\" (UID: \"a03a2ca3-9463-4838-8c2f-a37293da9e6a\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-hnd5d"
Nov 01 03:24:54.676362 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:54.676346    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a03a2ca3-9463-4838-8c2f-a37293da9e6a-istio-token\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-hnd5d\" (UID: \"a03a2ca3-9463-4838-8c2f-a37293da9e6a\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-hnd5d"
Nov 01 03:24:54.676416 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:54.676401    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a03a2ca3-9463-4838-8c2f-a37293da9e6a-istiod-ca-cert\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-hnd5d\" (UID: \"a03a2ca3-9463-4838-8c2f-a37293da9e6a\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-hnd5d"
Nov 01 03:24:54.676549 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:54.676526    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a03a2ca3-9463-4838-8c2f-a37293da9e6a-workload-certs\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-hnd5d\" (UID: \"a03a2ca3-9463-4838-8c2f-a37293da9e6a\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-hnd5d"
Nov 01 03:24:54.676614 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:54.676598    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a03a2ca3-9463-4838-8c2f-a37293da9e6a-istio-data\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-hnd5d\" (UID: \"a03a2ca3-9463-4838-8c2f-a37293da9e6a\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-hnd5d"
Nov 01 03:24:54.723739 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:54.723713    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-hnd5d"]
Nov 01 03:24:54.777452 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:54.777423    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a03a2ca3-9463-4838-8c2f-a37293da9e6a-workload-certs\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-hnd5d\" (UID: \"a03a2ca3-9463-4838-8c2f-a37293da9e6a\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-hnd5d"
Nov 01 03:24:54.777593 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:54.777465    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a03a2ca3-9463-4838-8c2f-a37293da9e6a-istio-data\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-hnd5d\" (UID: \"a03a2ca3-9463-4838-8c2f-a37293da9e6a\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-hnd5d"
Nov 01 03:24:54.777593 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:54.777520    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a03a2ca3-9463-4838-8c2f-a37293da9e6a-workload-socket\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-hnd5d\" (UID: \"a03a2ca3-9463-4838-8c2f-a37293da9e6a\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-hnd5d"
Nov 01 03:24:54.777593 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:54.777542    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77cr5\" (UniqueName: \"kubernetes.io/projected/a03a2ca3-9463-4838-8c2f-a37293da9e6a-kube-api-access-77cr5\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-hnd5d\" (UID: \"a03a2ca3-9463-4838-8c2f-a37293da9e6a\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-hnd5d"
Nov 01 03:24:54.777593 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:54.777562    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a03a2ca3-9463-4838-8c2f-a37293da9e6a-istio-envoy\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-hnd5d\" (UID: \"a03a2ca3-9463-4838-8c2f-a37293da9e6a\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-hnd5d"
Nov 01 03:24:54.777593 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:54.777590    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a03a2ca3-9463-4838-8c2f-a37293da9e6a-istio-podinfo\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-hnd5d\" (UID: \"a03a2ca3-9463-4838-8c2f-a37293da9e6a\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-hnd5d"
Nov 01 03:24:54.777917 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:54.777639    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a03a2ca3-9463-4838-8c2f-a37293da9e6a-credential-socket\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-hnd5d\" (UID: \"a03a2ca3-9463-4838-8c2f-a37293da9e6a\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-hnd5d"
Nov 01 03:24:54.777917 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:54.777725    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a03a2ca3-9463-4838-8c2f-a37293da9e6a-istio-token\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-hnd5d\" (UID: \"a03a2ca3-9463-4838-8c2f-a37293da9e6a\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-hnd5d"
Nov 01 03:24:54.777917 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:54.777779    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a03a2ca3-9463-4838-8c2f-a37293da9e6a-istiod-ca-cert\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-hnd5d\" (UID: \"a03a2ca3-9463-4838-8c2f-a37293da9e6a\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-hnd5d"
Nov 01 03:24:54.777917 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:54.777904    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a03a2ca3-9463-4838-8c2f-a37293da9e6a-workload-certs\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-hnd5d\" (UID: \"a03a2ca3-9463-4838-8c2f-a37293da9e6a\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-hnd5d"
Nov 01 03:24:54.778130 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:54.777933    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a03a2ca3-9463-4838-8c2f-a37293da9e6a-istio-data\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-hnd5d\" (UID: \"a03a2ca3-9463-4838-8c2f-a37293da9e6a\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-hnd5d"
Nov 01 03:24:54.778130 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:54.777979    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a03a2ca3-9463-4838-8c2f-a37293da9e6a-workload-socket\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-hnd5d\" (UID: \"a03a2ca3-9463-4838-8c2f-a37293da9e6a\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-hnd5d"
Nov 01 03:24:54.778130 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:54.778126    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a03a2ca3-9463-4838-8c2f-a37293da9e6a-credential-socket\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-hnd5d\" (UID: \"a03a2ca3-9463-4838-8c2f-a37293da9e6a\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-hnd5d"
Nov 01 03:24:54.778476 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:54.778455    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a03a2ca3-9463-4838-8c2f-a37293da9e6a-istiod-ca-cert\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-hnd5d\" (UID: \"a03a2ca3-9463-4838-8c2f-a37293da9e6a\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-hnd5d"
Nov 01 03:24:54.779995 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:54.779975    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a03a2ca3-9463-4838-8c2f-a37293da9e6a-istio-envoy\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-hnd5d\" (UID: \"a03a2ca3-9463-4838-8c2f-a37293da9e6a\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-hnd5d"
Nov 01 03:24:54.780115 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:54.780097    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a03a2ca3-9463-4838-8c2f-a37293da9e6a-istio-podinfo\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-hnd5d\" (UID: \"a03a2ca3-9463-4838-8c2f-a37293da9e6a\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-hnd5d"
Nov 01 03:24:54.840186 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:54.840161    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a03a2ca3-9463-4838-8c2f-a37293da9e6a-istio-token\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-hnd5d\" (UID: \"a03a2ca3-9463-4838-8c2f-a37293da9e6a\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-hnd5d"
Nov 01 03:24:54.840421 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:54.840397    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77cr5\" (UniqueName: \"kubernetes.io/projected/a03a2ca3-9463-4838-8c2f-a37293da9e6a-kube-api-access-77cr5\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-hnd5d\" (UID: \"a03a2ca3-9463-4838-8c2f-a37293da9e6a\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-hnd5d"
Nov 01 03:24:54.967112 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:54.967054    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-hnd5d"
Nov 01 03:24:55.401010 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:55.400981    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-hnd5d"]
Nov 01 03:24:55.401821 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:24:55.401800    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda03a2ca3_9463_4838_8c2f_a37293da9e6a.slice/crio-7740dd97850cdfcc1d3efc7ad6bcdda355ad1b5a716758f628f399b8bd2414b8 WatchSource:0}: Error finding container 7740dd97850cdfcc1d3efc7ad6bcdda355ad1b5a716758f628f399b8bd2414b8: Status 404 returned error can't find the container with id 7740dd97850cdfcc1d3efc7ad6bcdda355ad1b5a716758f628f399b8bd2414b8
Nov 01 03:24:55.404461 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:55.404383    4400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 03:24:55.404567 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:55.404506    4400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 03:24:55.404613 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:55.404569    4400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 03:24:55.458114 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:55.458087    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-hnd5d" event={"ID":"a03a2ca3-9463-4838-8c2f-a37293da9e6a","Type":"ContainerStarted","Data":"7740dd97850cdfcc1d3efc7ad6bcdda355ad1b5a716758f628f399b8bd2414b8"}
Nov 01 03:24:55.510393 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:55.510370    4400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-d7f535/distinct-w3-d7f535-kserve-7d69fb9558-88lq8"
Nov 01 03:24:55.523810 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:55.523790    4400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-d7f535/distinct-w3-d7f535-kserve-7d69fb9558-88lq8"
Nov 01 03:24:56.289644 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:56.289609    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 03:24:56.465623 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:56.465589    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-hnd5d" event={"ID":"a03a2ca3-9463-4838-8c2f-a37293da9e6a","Type":"ContainerStarted","Data":"5239558be482e883d1a22ca33d40634a37aed71201d84fcf5751ba3a42045bfb"}
Nov 01 03:24:56.476907 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:56.476876    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d7f535/distinct-w3-d7f535-kserve-7d69fb9558-88lq8"
Nov 01 03:24:56.591350 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:56.591299    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-hnd5d" podStartSLOduration=2.591284316 podStartE2EDuration="2.591284316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:24:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:24:56.590214518 +0000 UTC m=+1103.104860380" watchObservedRunningTime="2026-11-01 03:24:56.591284316 +0000 UTC m=+1103.105930180"
Nov 01 03:24:56.968579 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:56.968499    4400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-hnd5d"
Nov 01 03:24:56.973061 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:56.973031    4400 patch_prober.go:29] interesting pod/e2e-worker-w1-b12199-openshift-default-774cb8c567-hnd5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Nov 01 03:24:56.973525 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:56.973496    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-hnd5d" podUID="a03a2ca3-9463-4838-8c2f-a37293da9e6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Nov 01 03:24:57.967842 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:57.967804    4400 patch_prober.go:29] interesting pod/e2e-worker-w1-b12199-openshift-default-774cb8c567-hnd5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Nov 01 03:24:57.968433 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:57.968401    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-hnd5d" podUID="a03a2ca3-9463-4838-8c2f-a37293da9e6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Nov 01 03:24:58.696177 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:58.696142    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-h7pkm-6lxwl"]
Nov 01 03:24:58.700400 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:58.700384    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-h7pkm-6lxwl"
Nov 01 03:24:58.760508 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:58.760474    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01fe53c4-ce4d-4696-b738-2e9e10c29238-trusted-ca-bundle\") pod \"periodic-gathering-h7pkm-6lxwl\" (UID: \"01fe53c4-ce4d-4696-b738-2e9e10c29238\") " pod="openshift-insights/periodic-gathering-h7pkm-6lxwl"
Nov 01 03:24:58.760701 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:58.760540    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01fe53c4-ce4d-4696-b738-2e9e10c29238-service-ca-bundle\") pod \"periodic-gathering-h7pkm-6lxwl\" (UID: \"01fe53c4-ce4d-4696-b738-2e9e10c29238\") " pod="openshift-insights/periodic-gathering-h7pkm-6lxwl"
Nov 01 03:24:58.760781 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:58.760734    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/01fe53c4-ce4d-4696-b738-2e9e10c29238-archives-path\") pod \"periodic-gathering-h7pkm-6lxwl\" (UID: \"01fe53c4-ce4d-4696-b738-2e9e10c29238\") " pod="openshift-insights/periodic-gathering-h7pkm-6lxwl"
Nov 01 03:24:58.760837 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:58.760776    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjdf9\" (UniqueName: \"kubernetes.io/projected/01fe53c4-ce4d-4696-b738-2e9e10c29238-kube-api-access-qjdf9\") pod \"periodic-gathering-h7pkm-6lxwl\" (UID: \"01fe53c4-ce4d-4696-b738-2e9e10c29238\") " pod="openshift-insights/periodic-gathering-h7pkm-6lxwl"
Nov 01 03:24:58.761245 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:58.761215    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-h7pkm-6lxwl"]
Nov 01 03:24:58.829709 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:58.828994    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 01 03:24:58.829709 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:58.829283    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-bt25w\""
Nov 01 03:24:58.861528 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:58.861503    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/01fe53c4-ce4d-4696-b738-2e9e10c29238-archives-path\") pod \"periodic-gathering-h7pkm-6lxwl\" (UID: \"01fe53c4-ce4d-4696-b738-2e9e10c29238\") " pod="openshift-insights/periodic-gathering-h7pkm-6lxwl"
Nov 01 03:24:58.862007 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:58.861993    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjdf9\" (UniqueName: \"kubernetes.io/projected/01fe53c4-ce4d-4696-b738-2e9e10c29238-kube-api-access-qjdf9\") pod \"periodic-gathering-h7pkm-6lxwl\" (UID: \"01fe53c4-ce4d-4696-b738-2e9e10c29238\") " pod="openshift-insights/periodic-gathering-h7pkm-6lxwl"
Nov 01 03:24:58.862413 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:58.862397    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01fe53c4-ce4d-4696-b738-2e9e10c29238-trusted-ca-bundle\") pod \"periodic-gathering-h7pkm-6lxwl\" (UID: \"01fe53c4-ce4d-4696-b738-2e9e10c29238\") " pod="openshift-insights/periodic-gathering-h7pkm-6lxwl"
Nov 01 03:24:58.862582 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:58.862570    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01fe53c4-ce4d-4696-b738-2e9e10c29238-service-ca-bundle\") pod \"periodic-gathering-h7pkm-6lxwl\" (UID: \"01fe53c4-ce4d-4696-b738-2e9e10c29238\") " pod="openshift-insights/periodic-gathering-h7pkm-6lxwl"
Nov 01 03:24:58.863128 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:58.863110    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01fe53c4-ce4d-4696-b738-2e9e10c29238-service-ca-bundle\") pod \"periodic-gathering-h7pkm-6lxwl\" (UID: \"01fe53c4-ce4d-4696-b738-2e9e10c29238\") " pod="openshift-insights/periodic-gathering-h7pkm-6lxwl"
Nov 01 03:24:58.863258 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:58.861946    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/01fe53c4-ce4d-4696-b738-2e9e10c29238-archives-path\") pod \"periodic-gathering-h7pkm-6lxwl\" (UID: \"01fe53c4-ce4d-4696-b738-2e9e10c29238\") " pod="openshift-insights/periodic-gathering-h7pkm-6lxwl"
Nov 01 03:24:58.933704 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:58.933661    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 01 03:24:58.943910 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:58.943880    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01fe53c4-ce4d-4696-b738-2e9e10c29238-trusted-ca-bundle\") pod \"periodic-gathering-h7pkm-6lxwl\" (UID: \"01fe53c4-ce4d-4696-b738-2e9e10c29238\") " pod="openshift-insights/periodic-gathering-h7pkm-6lxwl"
Nov 01 03:24:58.967640 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:58.967586    4400 patch_prober.go:29] interesting pod/e2e-worker-w1-b12199-openshift-default-774cb8c567-hnd5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Nov 01 03:24:58.967763 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:58.967652    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-hnd5d" podUID="a03a2ca3-9463-4838-8c2f-a37293da9e6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Nov 01 03:24:58.985245 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:58.985223    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjdf9\" (UniqueName: \"kubernetes.io/projected/01fe53c4-ce4d-4696-b738-2e9e10c29238-kube-api-access-qjdf9\") pod \"periodic-gathering-h7pkm-6lxwl\" (UID: \"01fe53c4-ce4d-4696-b738-2e9e10c29238\") " pod="openshift-insights/periodic-gathering-h7pkm-6lxwl"
Nov 01 03:24:59.029249 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:59.029226    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-h7pkm-6lxwl"
Nov 01 03:24:59.430858 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:59.430831    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-h7pkm-6lxwl"]
Nov 01 03:24:59.431363 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:24:59.431330    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01fe53c4_ce4d_4696_b738_2e9e10c29238.slice/crio-6b699d3caca2466107fb10f695896bfbcb7ecbf53619fde4aa57da2c006482f9 WatchSource:0}: Error finding container 6b699d3caca2466107fb10f695896bfbcb7ecbf53619fde4aa57da2c006482f9: Status 404 returned error can't find the container with id 6b699d3caca2466107fb10f695896bfbcb7ecbf53619fde4aa57da2c006482f9
Nov 01 03:24:59.488135 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:59.488112    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-h7pkm-6lxwl" event={"ID":"01fe53c4-ce4d-4696-b738-2e9e10c29238","Type":"ContainerStarted","Data":"6b699d3caca2466107fb10f695896bfbcb7ecbf53619fde4aa57da2c006482f9"}
Nov 01 03:24:59.967941 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:59.967903    4400 patch_prober.go:29] interesting pod/e2e-worker-w1-b12199-openshift-default-774cb8c567-hnd5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Nov 01 03:24:59.968112 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:24:59.967964    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-hnd5d" podUID="a03a2ca3-9463-4838-8c2f-a37293da9e6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Nov 01 03:25:00.496211 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:00.496178    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-h7pkm-6lxwl" event={"ID":"01fe53c4-ce4d-4696-b738-2e9e10c29238","Type":"ContainerStarted","Data":"8430914c265817a11143e856b491c91375ad50cc2ce3f51ac09fbe6d5a1b6437"}
Nov 01 03:25:00.623710 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:00.619653    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-h7pkm-6lxwl" podStartSLOduration=2.61963414 podStartE2EDuration="2.61963414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:24:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:25:00.617328921 +0000 UTC m=+1107.131974786" watchObservedRunningTime="2026-11-01 03:25:00.61963414 +0000 UTC m=+1107.134280008"
Nov 01 03:25:00.967364 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:00.967333    4400 patch_prober.go:29] interesting pod/e2e-worker-w1-b12199-openshift-default-774cb8c567-hnd5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Nov 01 03:25:00.967521 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:00.967386    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-hnd5d" podUID="a03a2ca3-9463-4838-8c2f-a37293da9e6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Nov 01 03:25:01.731751 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:25:01.731712    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:25:01.967789 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:01.967758    4400 patch_prober.go:29] interesting pod/e2e-worker-w1-b12199-openshift-default-774cb8c567-hnd5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Nov 01 03:25:01.967944 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:01.967807    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-hnd5d" podUID="a03a2ca3-9463-4838-8c2f-a37293da9e6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Nov 01 03:25:02.972025 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:02.971998    4400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-hnd5d"
Nov 01 03:25:02.972425 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:02.972287    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-hnd5d"
Nov 01 03:25:02.972904 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:02.972883    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-hnd5d"
Nov 01 03:25:10.465102 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:10.465069    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-2sbzb"]
Nov 01 03:25:10.469629 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:10.469608    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-2sbzb"
Nov 01 03:25:10.583093 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:10.583061    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-2sbzb"]
Nov 01 03:25:10.587880 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:10.587853    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/863e73b9-dddf-41c3-9a3d-eca8d7e2b727-workload-certs\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-2sbzb\" (UID: \"863e73b9-dddf-41c3-9a3d-eca8d7e2b727\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-2sbzb"
Nov 01 03:25:10.588018 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:10.587907    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/863e73b9-dddf-41c3-9a3d-eca8d7e2b727-istiod-ca-cert\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-2sbzb\" (UID: \"863e73b9-dddf-41c3-9a3d-eca8d7e2b727\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-2sbzb"
Nov 01 03:25:10.588018 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:10.587987    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27g8r\" (UniqueName: \"kubernetes.io/projected/863e73b9-dddf-41c3-9a3d-eca8d7e2b727-kube-api-access-27g8r\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-2sbzb\" (UID: \"863e73b9-dddf-41c3-9a3d-eca8d7e2b727\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-2sbzb"
Nov 01 03:25:10.588113 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:10.588023    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/863e73b9-dddf-41c3-9a3d-eca8d7e2b727-istio-podinfo\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-2sbzb\" (UID: \"863e73b9-dddf-41c3-9a3d-eca8d7e2b727\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-2sbzb"
Nov 01 03:25:10.588113 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:10.588057    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/863e73b9-dddf-41c3-9a3d-eca8d7e2b727-istio-token\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-2sbzb\" (UID: \"863e73b9-dddf-41c3-9a3d-eca8d7e2b727\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-2sbzb"
Nov 01 03:25:10.588222 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:10.588194    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/863e73b9-dddf-41c3-9a3d-eca8d7e2b727-workload-socket\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-2sbzb\" (UID: \"863e73b9-dddf-41c3-9a3d-eca8d7e2b727\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-2sbzb"
Nov 01 03:25:10.588358 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:10.588338    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/863e73b9-dddf-41c3-9a3d-eca8d7e2b727-credential-socket\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-2sbzb\" (UID: \"863e73b9-dddf-41c3-9a3d-eca8d7e2b727\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-2sbzb"
Nov 01 03:25:10.588433 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:10.588415    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/863e73b9-dddf-41c3-9a3d-eca8d7e2b727-istio-data\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-2sbzb\" (UID: \"863e73b9-dddf-41c3-9a3d-eca8d7e2b727\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-2sbzb"
Nov 01 03:25:10.588500 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:10.588463    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/863e73b9-dddf-41c3-9a3d-eca8d7e2b727-istio-envoy\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-2sbzb\" (UID: \"863e73b9-dddf-41c3-9a3d-eca8d7e2b727\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-2sbzb"
Nov 01 03:25:10.689597 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:10.689561    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/863e73b9-dddf-41c3-9a3d-eca8d7e2b727-istio-data\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-2sbzb\" (UID: \"863e73b9-dddf-41c3-9a3d-eca8d7e2b727\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-2sbzb"
Nov 01 03:25:10.689797 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:10.689599    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/863e73b9-dddf-41c3-9a3d-eca8d7e2b727-istio-envoy\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-2sbzb\" (UID: \"863e73b9-dddf-41c3-9a3d-eca8d7e2b727\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-2sbzb"
Nov 01 03:25:10.690008 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:10.689978    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/863e73b9-dddf-41c3-9a3d-eca8d7e2b727-workload-certs\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-2sbzb\" (UID: \"863e73b9-dddf-41c3-9a3d-eca8d7e2b727\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-2sbzb"
Nov 01 03:25:10.690008 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:10.690002    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/863e73b9-dddf-41c3-9a3d-eca8d7e2b727-istio-data\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-2sbzb\" (UID: \"863e73b9-dddf-41c3-9a3d-eca8d7e2b727\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-2sbzb"
Nov 01 03:25:10.690166 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:10.690031    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/863e73b9-dddf-41c3-9a3d-eca8d7e2b727-istiod-ca-cert\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-2sbzb\" (UID: \"863e73b9-dddf-41c3-9a3d-eca8d7e2b727\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-2sbzb"
Nov 01 03:25:10.690166 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:10.690051    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27g8r\" (UniqueName: \"kubernetes.io/projected/863e73b9-dddf-41c3-9a3d-eca8d7e2b727-kube-api-access-27g8r\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-2sbzb\" (UID: \"863e73b9-dddf-41c3-9a3d-eca8d7e2b727\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-2sbzb"
Nov 01 03:25:10.690166 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:10.690099    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/863e73b9-dddf-41c3-9a3d-eca8d7e2b727-istio-podinfo\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-2sbzb\" (UID: \"863e73b9-dddf-41c3-9a3d-eca8d7e2b727\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-2sbzb"
Nov 01 03:25:10.690166 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:10.690156    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/863e73b9-dddf-41c3-9a3d-eca8d7e2b727-istio-token\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-2sbzb\" (UID: \"863e73b9-dddf-41c3-9a3d-eca8d7e2b727\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-2sbzb"
Nov 01 03:25:10.690378 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:10.690194    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/863e73b9-dddf-41c3-9a3d-eca8d7e2b727-workload-certs\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-2sbzb\" (UID: \"863e73b9-dddf-41c3-9a3d-eca8d7e2b727\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-2sbzb"
Nov 01 03:25:10.690378 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:10.690198    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/863e73b9-dddf-41c3-9a3d-eca8d7e2b727-workload-socket\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-2sbzb\" (UID: \"863e73b9-dddf-41c3-9a3d-eca8d7e2b727\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-2sbzb"
Nov 01 03:25:10.690496 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:10.690481    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/863e73b9-dddf-41c3-9a3d-eca8d7e2b727-credential-socket\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-2sbzb\" (UID: \"863e73b9-dddf-41c3-9a3d-eca8d7e2b727\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-2sbzb"
Nov 01 03:25:10.690644 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:10.690621    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/863e73b9-dddf-41c3-9a3d-eca8d7e2b727-workload-socket\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-2sbzb\" (UID: \"863e73b9-dddf-41c3-9a3d-eca8d7e2b727\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-2sbzb"
Nov 01 03:25:10.690834 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:10.690807    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/863e73b9-dddf-41c3-9a3d-eca8d7e2b727-istiod-ca-cert\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-2sbzb\" (UID: \"863e73b9-dddf-41c3-9a3d-eca8d7e2b727\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-2sbzb"
Nov 01 03:25:10.690924 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:10.690904    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/863e73b9-dddf-41c3-9a3d-eca8d7e2b727-credential-socket\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-2sbzb\" (UID: \"863e73b9-dddf-41c3-9a3d-eca8d7e2b727\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-2sbzb"
Nov 01 03:25:10.692199 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:10.692173    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/863e73b9-dddf-41c3-9a3d-eca8d7e2b727-istio-envoy\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-2sbzb\" (UID: \"863e73b9-dddf-41c3-9a3d-eca8d7e2b727\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-2sbzb"
Nov 01 03:25:10.693183 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:10.693162    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/863e73b9-dddf-41c3-9a3d-eca8d7e2b727-istio-podinfo\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-2sbzb\" (UID: \"863e73b9-dddf-41c3-9a3d-eca8d7e2b727\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-2sbzb"
Nov 01 03:25:10.760035 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:10.759958    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27g8r\" (UniqueName: \"kubernetes.io/projected/863e73b9-dddf-41c3-9a3d-eca8d7e2b727-kube-api-access-27g8r\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-2sbzb\" (UID: \"863e73b9-dddf-41c3-9a3d-eca8d7e2b727\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-2sbzb"
Nov 01 03:25:10.761407 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:10.761384    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/863e73b9-dddf-41c3-9a3d-eca8d7e2b727-istio-token\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-2sbzb\" (UID: \"863e73b9-dddf-41c3-9a3d-eca8d7e2b727\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-2sbzb"
Nov 01 03:25:10.784346 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:10.784322    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-2sbzb"
Nov 01 03:25:11.227973 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:11.227944    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-2sbzb"]
Nov 01 03:25:11.228886 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:25:11.228861    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod863e73b9_dddf_41c3_9a3d_eca8d7e2b727.slice/crio-ddf44a4558e4529230be0a3f2b0b2b17d7dcfb9d93c5c92ef04ac52fb6a9df28 WatchSource:0}: Error finding container ddf44a4558e4529230be0a3f2b0b2b17d7dcfb9d93c5c92ef04ac52fb6a9df28: Status 404 returned error can't find the container with id ddf44a4558e4529230be0a3f2b0b2b17d7dcfb9d93c5c92ef04ac52fb6a9df28
Nov 01 03:25:11.231456 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:11.231424    4400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 03:25:11.231549 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:11.231499    4400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 03:25:11.231549 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:11.231527    4400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 03:25:11.564944 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:11.564858    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-2sbzb" event={"ID":"863e73b9-dddf-41c3-9a3d-eca8d7e2b727","Type":"ContainerStarted","Data":"c2165e9ef486f13dc4f334eb78392a42cf94525247353124496a23de97ac76e1"}
Nov 01 03:25:11.564944 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:11.564911    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-2sbzb" event={"ID":"863e73b9-dddf-41c3-9a3d-eca8d7e2b727","Type":"ContainerStarted","Data":"ddf44a4558e4529230be0a3f2b0b2b17d7dcfb9d93c5c92ef04ac52fb6a9df28"}
Nov 01 03:25:11.691273 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:11.691212    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-2sbzb" podStartSLOduration=1.691193319 podStartE2EDuration="1.691193319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:25:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:25:11.690392417 +0000 UTC m=+1118.205038281" watchObservedRunningTime="2026-11-01 03:25:11.691193319 +0000 UTC m=+1118.205839185"
Nov 01 03:25:11.784809 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:11.784772    4400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-2sbzb"
Nov 01 03:25:12.787202 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:12.787169    4400 patch_prober.go:29] interesting pod/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-2sbzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Nov 01 03:25:12.787601 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:12.787222    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-2sbzb" podUID="863e73b9-dddf-41c3-9a3d-eca8d7e2b727" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Nov 01 03:25:13.785282 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:13.785245    4400 patch_prober.go:29] interesting pod/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-2sbzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Nov 01 03:25:13.785458 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:13.785301    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-2sbzb" podUID="863e73b9-dddf-41c3-9a3d-eca8d7e2b727" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Nov 01 03:25:14.785234 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:14.785194    4400 patch_prober.go:29] interesting pod/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-2sbzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Nov 01 03:25:14.785783 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:14.785250    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-2sbzb" podUID="863e73b9-dddf-41c3-9a3d-eca8d7e2b727" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Nov 01 03:25:15.785651 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:15.785617    4400 patch_prober.go:29] interesting pod/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-2sbzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Nov 01 03:25:15.786200 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:15.785677    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-2sbzb" podUID="863e73b9-dddf-41c3-9a3d-eca8d7e2b727" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Nov 01 03:25:16.784614 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:16.784578    4400 patch_prober.go:29] interesting pod/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-2sbzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Nov 01 03:25:16.784836 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:16.784636    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-2sbzb" podUID="863e73b9-dddf-41c3-9a3d-eca8d7e2b727" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Nov 01 03:25:17.784763 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:17.784729    4400 patch_prober.go:29] interesting pod/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-2sbzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Nov 01 03:25:17.785166 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:17.784795    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-2sbzb" podUID="863e73b9-dddf-41c3-9a3d-eca8d7e2b727" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Nov 01 03:25:18.789532 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:18.789505    4400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-2sbzb"
Nov 01 03:25:18.789995 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:18.789980    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-2sbzb"
Nov 01 03:25:18.790603 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:18.790587    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-2sbzb"
Nov 01 03:25:31.444915 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:31.444881    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-c8cb9a/distinct-2-w2-c8cb9a-kserve-5c5cfb454d-fh2cj"]
Nov 01 03:25:31.450576 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:31.450559    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-2-w2-c8cb9a-kserve-5c5cfb454d-fh2cj"
Nov 01 03:25:31.513310 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:31.513281    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-c8cb9a/distinct-2-w2-c8cb9a-kserve-5c5cfb454d-fh2cj"]
Nov 01 03:25:31.536882 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:31.536849    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bcf3b85-3d4a-433f-94ad-0e954bf03e73-tls-certs\") pod \"distinct-2-w2-c8cb9a-kserve-5c5cfb454d-fh2cj\" (UID: \"2bcf3b85-3d4a-433f-94ad-0e954bf03e73\") " pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-2-w2-c8cb9a-kserve-5c5cfb454d-fh2cj"
Nov 01 03:25:31.537024 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:31.536941    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bcf3b85-3d4a-433f-94ad-0e954bf03e73-model-cache\") pod \"distinct-2-w2-c8cb9a-kserve-5c5cfb454d-fh2cj\" (UID: \"2bcf3b85-3d4a-433f-94ad-0e954bf03e73\") " pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-2-w2-c8cb9a-kserve-5c5cfb454d-fh2cj"
Nov 01 03:25:31.537090 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:31.537076    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bcf3b85-3d4a-433f-94ad-0e954bf03e73-tmp-dir\") pod \"distinct-2-w2-c8cb9a-kserve-5c5cfb454d-fh2cj\" (UID: \"2bcf3b85-3d4a-433f-94ad-0e954bf03e73\") " pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-2-w2-c8cb9a-kserve-5c5cfb454d-fh2cj"
Nov 01 03:25:31.537127 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:31.537106    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2bcf3b85-3d4a-433f-94ad-0e954bf03e73-home\") pod \"distinct-2-w2-c8cb9a-kserve-5c5cfb454d-fh2cj\" (UID: \"2bcf3b85-3d4a-433f-94ad-0e954bf03e73\") " pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-2-w2-c8cb9a-kserve-5c5cfb454d-fh2cj"
Nov 01 03:25:31.537160 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:31.537127    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bcf3b85-3d4a-433f-94ad-0e954bf03e73-dshm\") pod \"distinct-2-w2-c8cb9a-kserve-5c5cfb454d-fh2cj\" (UID: \"2bcf3b85-3d4a-433f-94ad-0e954bf03e73\") " pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-2-w2-c8cb9a-kserve-5c5cfb454d-fh2cj"
Nov 01 03:25:31.537266 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:31.537252    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrvhv\" (UniqueName: \"kubernetes.io/projected/2bcf3b85-3d4a-433f-94ad-0e954bf03e73-kube-api-access-zrvhv\") pod \"distinct-2-w2-c8cb9a-kserve-5c5cfb454d-fh2cj\" (UID: \"2bcf3b85-3d4a-433f-94ad-0e954bf03e73\") " pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-2-w2-c8cb9a-kserve-5c5cfb454d-fh2cj"
Nov 01 03:25:31.560997 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:31.560977    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-c8cb9a\"/\"distinct-2-w2-c8cb9a-kserve-self-signed-certs\""
Nov 01 03:25:31.561123 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:31.561101    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-c8cb9a\"/\"default-dockercfg-dxgc9\""
Nov 01 03:25:31.561164 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:31.561120    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-c8cb9a\"/\"openshift-service-ca.crt\""
Nov 01 03:25:31.561164 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:31.561130    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-c8cb9a\"/\"kube-root-ca.crt\""
Nov 01 03:25:31.642115 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:31.639887    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrvhv\" (UniqueName: \"kubernetes.io/projected/2bcf3b85-3d4a-433f-94ad-0e954bf03e73-kube-api-access-zrvhv\") pod \"distinct-2-w2-c8cb9a-kserve-5c5cfb454d-fh2cj\" (UID: \"2bcf3b85-3d4a-433f-94ad-0e954bf03e73\") " pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-2-w2-c8cb9a-kserve-5c5cfb454d-fh2cj"
Nov 01 03:25:31.642115 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:31.639969    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bcf3b85-3d4a-433f-94ad-0e954bf03e73-tls-certs\") pod \"distinct-2-w2-c8cb9a-kserve-5c5cfb454d-fh2cj\" (UID: \"2bcf3b85-3d4a-433f-94ad-0e954bf03e73\") " pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-2-w2-c8cb9a-kserve-5c5cfb454d-fh2cj"
Nov 01 03:25:31.642115 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:31.640014    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bcf3b85-3d4a-433f-94ad-0e954bf03e73-model-cache\") pod \"distinct-2-w2-c8cb9a-kserve-5c5cfb454d-fh2cj\" (UID: \"2bcf3b85-3d4a-433f-94ad-0e954bf03e73\") " pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-2-w2-c8cb9a-kserve-5c5cfb454d-fh2cj"
Nov 01 03:25:31.642115 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:31.640164    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bcf3b85-3d4a-433f-94ad-0e954bf03e73-tmp-dir\") pod \"distinct-2-w2-c8cb9a-kserve-5c5cfb454d-fh2cj\" (UID: \"2bcf3b85-3d4a-433f-94ad-0e954bf03e73\") " pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-2-w2-c8cb9a-kserve-5c5cfb454d-fh2cj"
Nov 01 03:25:31.642115 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:31.640202    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2bcf3b85-3d4a-433f-94ad-0e954bf03e73-home\") pod \"distinct-2-w2-c8cb9a-kserve-5c5cfb454d-fh2cj\" (UID: \"2bcf3b85-3d4a-433f-94ad-0e954bf03e73\") " pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-2-w2-c8cb9a-kserve-5c5cfb454d-fh2cj"
Nov 01 03:25:31.642115 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:31.640230    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bcf3b85-3d4a-433f-94ad-0e954bf03e73-dshm\") pod \"distinct-2-w2-c8cb9a-kserve-5c5cfb454d-fh2cj\" (UID: \"2bcf3b85-3d4a-433f-94ad-0e954bf03e73\") " pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-2-w2-c8cb9a-kserve-5c5cfb454d-fh2cj"
Nov 01 03:25:31.643755 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:31.643663    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bcf3b85-3d4a-433f-94ad-0e954bf03e73-model-cache\") pod \"distinct-2-w2-c8cb9a-kserve-5c5cfb454d-fh2cj\" (UID: \"2bcf3b85-3d4a-433f-94ad-0e954bf03e73\") " pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-2-w2-c8cb9a-kserve-5c5cfb454d-fh2cj"
Nov 01 03:25:31.647790 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:31.647758    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bcf3b85-3d4a-433f-94ad-0e954bf03e73-tmp-dir\") pod \"distinct-2-w2-c8cb9a-kserve-5c5cfb454d-fh2cj\" (UID: \"2bcf3b85-3d4a-433f-94ad-0e954bf03e73\") " pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-2-w2-c8cb9a-kserve-5c5cfb454d-fh2cj"
Nov 01 03:25:31.649241 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:31.649172    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2bcf3b85-3d4a-433f-94ad-0e954bf03e73-home\") pod \"distinct-2-w2-c8cb9a-kserve-5c5cfb454d-fh2cj\" (UID: \"2bcf3b85-3d4a-433f-94ad-0e954bf03e73\") " pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-2-w2-c8cb9a-kserve-5c5cfb454d-fh2cj"
Nov 01 03:25:31.650257 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:31.650234    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bcf3b85-3d4a-433f-94ad-0e954bf03e73-tls-certs\") pod \"distinct-2-w2-c8cb9a-kserve-5c5cfb454d-fh2cj\" (UID: \"2bcf3b85-3d4a-433f-94ad-0e954bf03e73\") " pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-2-w2-c8cb9a-kserve-5c5cfb454d-fh2cj"
Nov 01 03:25:31.652892 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:31.652869    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bcf3b85-3d4a-433f-94ad-0e954bf03e73-dshm\") pod \"distinct-2-w2-c8cb9a-kserve-5c5cfb454d-fh2cj\" (UID: \"2bcf3b85-3d4a-433f-94ad-0e954bf03e73\") " pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-2-w2-c8cb9a-kserve-5c5cfb454d-fh2cj"
Nov 01 03:25:31.706800 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:31.706709    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrvhv\" (UniqueName: \"kubernetes.io/projected/2bcf3b85-3d4a-433f-94ad-0e954bf03e73-kube-api-access-zrvhv\") pod \"distinct-2-w2-c8cb9a-kserve-5c5cfb454d-fh2cj\" (UID: \"2bcf3b85-3d4a-433f-94ad-0e954bf03e73\") " pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-2-w2-c8cb9a-kserve-5c5cfb454d-fh2cj"
Nov 01 03:25:31.732544 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:25:31.732496    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:25:31.764005 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:31.763971    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-2-w2-c8cb9a-kserve-5c5cfb454d-fh2cj"
Nov 01 03:25:32.171825 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:32.171800    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-c8cb9a/distinct-2-w2-c8cb9a-kserve-5c5cfb454d-fh2cj"]
Nov 01 03:25:32.172345 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:25:32.172322    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2bcf3b85_3d4a_433f_94ad_0e954bf03e73.slice/crio-2357d6f2a538784a4ef6fff4180d217eb9804ddd0a76b0117592aa2904de2ea8 WatchSource:0}: Error finding container 2357d6f2a538784a4ef6fff4180d217eb9804ddd0a76b0117592aa2904de2ea8: Status 404 returned error can't find the container with id 2357d6f2a538784a4ef6fff4180d217eb9804ddd0a76b0117592aa2904de2ea8
Nov 01 03:25:32.738250 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:32.738205    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-2-w2-c8cb9a-kserve-5c5cfb454d-fh2cj" event={"ID":"2bcf3b85-3d4a-433f-94ad-0e954bf03e73","Type":"ContainerStarted","Data":"50850fc9101f441bdefdec50b53aff9d82d329a0899d5d467e5f2e4eb4fe2e9f"}
Nov 01 03:25:32.738250 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:32.738261    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-2-w2-c8cb9a-kserve-5c5cfb454d-fh2cj" event={"ID":"2bcf3b85-3d4a-433f-94ad-0e954bf03e73","Type":"ContainerStarted","Data":"2357d6f2a538784a4ef6fff4180d217eb9804ddd0a76b0117592aa2904de2ea8"}
Nov 01 03:25:32.764567 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:32.764524    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-2-w2-c8cb9a-kserve-5c5cfb454d-fh2cj"
Nov 01 03:25:32.910780 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:32.910729    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-2-w2-c8cb9a-kserve-5c5cfb454d-fh2cj" podStartSLOduration=1.910714444 podStartE2EDuration="1.910714444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:25:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:25:32.907828447 +0000 UTC m=+1139.422474310" watchObservedRunningTime="2026-11-01 03:25:32.910714444 +0000 UTC m=+1139.425360304"
Nov 01 03:25:41.764781 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:41.764741    4400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-2-w2-c8cb9a-kserve-5c5cfb454d-fh2cj"
Nov 01 03:25:41.777548 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:41.777507    4400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-2-w2-c8cb9a-kserve-5c5cfb454d-fh2cj"
Nov 01 03:25:41.800572 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:41.800552    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-2-w2-c8cb9a-kserve-5c5cfb454d-fh2cj"
Nov 01 03:25:41.940062 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:41.940035    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 03:25:55.776106 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:55.776074    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lv9hr"]
Nov 01 03:25:55.780364 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:55.780342    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lv9hr"
Nov 01 03:25:55.890127 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:55.890092    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed1e2049-23c6-43e6-811c-fb0c5a9541b5-istio-podinfo\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lv9hr\" (UID: \"ed1e2049-23c6-43e6-811c-fb0c5a9541b5\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lv9hr"
Nov 01 03:25:55.890311 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:55.890173    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2s7p\" (UniqueName: \"kubernetes.io/projected/ed1e2049-23c6-43e6-811c-fb0c5a9541b5-kube-api-access-x2s7p\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lv9hr\" (UID: \"ed1e2049-23c6-43e6-811c-fb0c5a9541b5\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lv9hr"
Nov 01 03:25:55.890311 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:55.890215    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed1e2049-23c6-43e6-811c-fb0c5a9541b5-istio-token\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lv9hr\" (UID: \"ed1e2049-23c6-43e6-811c-fb0c5a9541b5\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lv9hr"
Nov 01 03:25:55.890311 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:55.890296    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed1e2049-23c6-43e6-811c-fb0c5a9541b5-credential-socket\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lv9hr\" (UID: \"ed1e2049-23c6-43e6-811c-fb0c5a9541b5\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lv9hr"
Nov 01 03:25:55.890495 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:55.890421    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed1e2049-23c6-43e6-811c-fb0c5a9541b5-istiod-ca-cert\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lv9hr\" (UID: \"ed1e2049-23c6-43e6-811c-fb0c5a9541b5\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lv9hr"
Nov 01 03:25:55.890495 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:55.890453    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed1e2049-23c6-43e6-811c-fb0c5a9541b5-istio-envoy\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lv9hr\" (UID: \"ed1e2049-23c6-43e6-811c-fb0c5a9541b5\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lv9hr"
Nov 01 03:25:55.890617 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:55.890580    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed1e2049-23c6-43e6-811c-fb0c5a9541b5-istio-data\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lv9hr\" (UID: \"ed1e2049-23c6-43e6-811c-fb0c5a9541b5\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lv9hr"
Nov 01 03:25:55.890617 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:55.890629    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed1e2049-23c6-43e6-811c-fb0c5a9541b5-workload-certs\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lv9hr\" (UID: \"ed1e2049-23c6-43e6-811c-fb0c5a9541b5\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lv9hr"
Nov 01 03:25:55.890847 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:55.890765    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed1e2049-23c6-43e6-811c-fb0c5a9541b5-workload-socket\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lv9hr\" (UID: \"ed1e2049-23c6-43e6-811c-fb0c5a9541b5\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lv9hr"
Nov 01 03:25:55.992113 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:55.992083    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed1e2049-23c6-43e6-811c-fb0c5a9541b5-istio-podinfo\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lv9hr\" (UID: \"ed1e2049-23c6-43e6-811c-fb0c5a9541b5\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lv9hr"
Nov 01 03:25:55.992113 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:55.992118    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2s7p\" (UniqueName: \"kubernetes.io/projected/ed1e2049-23c6-43e6-811c-fb0c5a9541b5-kube-api-access-x2s7p\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lv9hr\" (UID: \"ed1e2049-23c6-43e6-811c-fb0c5a9541b5\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lv9hr"
Nov 01 03:25:55.992363 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:55.992144    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed1e2049-23c6-43e6-811c-fb0c5a9541b5-istio-token\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lv9hr\" (UID: \"ed1e2049-23c6-43e6-811c-fb0c5a9541b5\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lv9hr"
Nov 01 03:25:55.992363 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:55.992192    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed1e2049-23c6-43e6-811c-fb0c5a9541b5-credential-socket\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lv9hr\" (UID: \"ed1e2049-23c6-43e6-811c-fb0c5a9541b5\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lv9hr"
Nov 01 03:25:55.992363 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:55.992347    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed1e2049-23c6-43e6-811c-fb0c5a9541b5-istiod-ca-cert\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lv9hr\" (UID: \"ed1e2049-23c6-43e6-811c-fb0c5a9541b5\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lv9hr"
Nov 01 03:25:55.992542 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:55.992383    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed1e2049-23c6-43e6-811c-fb0c5a9541b5-istio-envoy\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lv9hr\" (UID: \"ed1e2049-23c6-43e6-811c-fb0c5a9541b5\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lv9hr"
Nov 01 03:25:55.992542 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:55.992482    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed1e2049-23c6-43e6-811c-fb0c5a9541b5-istio-data\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lv9hr\" (UID: \"ed1e2049-23c6-43e6-811c-fb0c5a9541b5\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lv9hr"
Nov 01 03:25:55.992542 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:55.992514    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed1e2049-23c6-43e6-811c-fb0c5a9541b5-workload-certs\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lv9hr\" (UID: \"ed1e2049-23c6-43e6-811c-fb0c5a9541b5\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lv9hr"
Nov 01 03:25:55.992542 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:55.992540    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed1e2049-23c6-43e6-811c-fb0c5a9541b5-workload-socket\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lv9hr\" (UID: \"ed1e2049-23c6-43e6-811c-fb0c5a9541b5\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lv9hr"
Nov 01 03:25:55.993119 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:55.992839    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed1e2049-23c6-43e6-811c-fb0c5a9541b5-credential-socket\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lv9hr\" (UID: \"ed1e2049-23c6-43e6-811c-fb0c5a9541b5\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lv9hr"
Nov 01 03:25:55.993119 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:55.993001    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed1e2049-23c6-43e6-811c-fb0c5a9541b5-workload-socket\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lv9hr\" (UID: \"ed1e2049-23c6-43e6-811c-fb0c5a9541b5\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lv9hr"
Nov 01 03:25:55.993119 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:55.993038    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed1e2049-23c6-43e6-811c-fb0c5a9541b5-istio-data\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lv9hr\" (UID: \"ed1e2049-23c6-43e6-811c-fb0c5a9541b5\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lv9hr"
Nov 01 03:25:55.993119 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:55.993069    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed1e2049-23c6-43e6-811c-fb0c5a9541b5-workload-certs\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lv9hr\" (UID: \"ed1e2049-23c6-43e6-811c-fb0c5a9541b5\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lv9hr"
Nov 01 03:25:55.993575 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:55.993159    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed1e2049-23c6-43e6-811c-fb0c5a9541b5-istiod-ca-cert\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lv9hr\" (UID: \"ed1e2049-23c6-43e6-811c-fb0c5a9541b5\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lv9hr"
Nov 01 03:25:55.994891 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:55.994868    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed1e2049-23c6-43e6-811c-fb0c5a9541b5-istio-envoy\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lv9hr\" (UID: \"ed1e2049-23c6-43e6-811c-fb0c5a9541b5\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lv9hr"
Nov 01 03:25:55.995015 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:55.994996    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed1e2049-23c6-43e6-811c-fb0c5a9541b5-istio-podinfo\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lv9hr\" (UID: \"ed1e2049-23c6-43e6-811c-fb0c5a9541b5\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lv9hr"
Nov 01 03:25:56.003563 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:56.003543    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lv9hr"]
Nov 01 03:25:56.058066 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:56.058002    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed1e2049-23c6-43e6-811c-fb0c5a9541b5-istio-token\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lv9hr\" (UID: \"ed1e2049-23c6-43e6-811c-fb0c5a9541b5\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lv9hr"
Nov 01 03:25:56.058195 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:56.058175    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2s7p\" (UniqueName: \"kubernetes.io/projected/ed1e2049-23c6-43e6-811c-fb0c5a9541b5-kube-api-access-x2s7p\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lv9hr\" (UID: \"ed1e2049-23c6-43e6-811c-fb0c5a9541b5\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lv9hr"
Nov 01 03:25:56.095346 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:56.095327    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lv9hr"
Nov 01 03:25:56.586444 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:56.586411    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lv9hr"]
Nov 01 03:25:56.592939 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:25:56.592909    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded1e2049_23c6_43e6_811c_fb0c5a9541b5.slice/crio-3cff5ab83bf57f5ee546aad90aa96b33422e1ebddc7f1d147db05696788c3752 WatchSource:0}: Error finding container 3cff5ab83bf57f5ee546aad90aa96b33422e1ebddc7f1d147db05696788c3752: Status 404 returned error can't find the container with id 3cff5ab83bf57f5ee546aad90aa96b33422e1ebddc7f1d147db05696788c3752
Nov 01 03:25:56.595191 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:56.595161    4400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 03:25:56.595272 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:56.595230    4400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 03:25:56.595272 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:56.595259    4400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 03:25:56.958958 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:56.958861    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lv9hr" event={"ID":"ed1e2049-23c6-43e6-811c-fb0c5a9541b5","Type":"ContainerStarted","Data":"899f076ad230e0c666313f4c60979453677c213c000b89605566eaca2d7c7449"}
Nov 01 03:25:56.958958 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:56.958915    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lv9hr" event={"ID":"ed1e2049-23c6-43e6-811c-fb0c5a9541b5","Type":"ContainerStarted","Data":"3cff5ab83bf57f5ee546aad90aa96b33422e1ebddc7f1d147db05696788c3752"}
Nov 01 03:25:57.087895 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:57.087833    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lv9hr" podStartSLOduration=2.087811555 podStartE2EDuration="2.087811555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:25:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:25:57.083585437 +0000 UTC m=+1163.598231299" watchObservedRunningTime="2026-11-01 03:25:57.087811555 +0000 UTC m=+1163.602457422"
Nov 01 03:25:57.095737 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:57.095712    4400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lv9hr"
Nov 01 03:25:57.097224 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:57.097198    4400 patch_prober.go:29] interesting pod/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lv9hr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Nov 01 03:25:57.097326 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:57.097252    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lv9hr" podUID="ed1e2049-23c6-43e6-811c-fb0c5a9541b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Nov 01 03:25:58.096193 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:58.096159    4400 patch_prober.go:29] interesting pod/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lv9hr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Nov 01 03:25:58.096580 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:58.096220    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lv9hr" podUID="ed1e2049-23c6-43e6-811c-fb0c5a9541b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Nov 01 03:25:59.095662 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:59.095630    4400 patch_prober.go:29] interesting pod/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lv9hr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Nov 01 03:25:59.095852 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:25:59.095714    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lv9hr" podUID="ed1e2049-23c6-43e6-811c-fb0c5a9541b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Nov 01 03:26:00.096330 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:00.096299    4400 patch_prober.go:29] interesting pod/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lv9hr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Nov 01 03:26:00.096752 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:00.096358    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lv9hr" podUID="ed1e2049-23c6-43e6-811c-fb0c5a9541b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Nov 01 03:26:01.096705 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:01.096646    4400 patch_prober.go:29] interesting pod/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lv9hr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Nov 01 03:26:01.097077 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:01.096721    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lv9hr" podUID="ed1e2049-23c6-43e6-811c-fb0c5a9541b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Nov 01 03:26:01.749849 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:26:01.749739    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:26:02.096620 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:02.096583    4400 patch_prober.go:29] interesting pod/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lv9hr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Nov 01 03:26:02.096797 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:02.096652    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lv9hr" podUID="ed1e2049-23c6-43e6-811c-fb0c5a9541b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Nov 01 03:26:03.096519 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:03.096488    4400 patch_prober.go:29] interesting pod/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lv9hr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Nov 01 03:26:03.096728 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:03.096548    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lv9hr" podUID="ed1e2049-23c6-43e6-811c-fb0c5a9541b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Nov 01 03:26:04.100393 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:04.100365    4400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lv9hr"
Nov 01 03:26:04.100819 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:04.100802    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lv9hr"
Nov 01 03:26:04.101424 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:04.101404    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lv9hr"
Nov 01 03:26:04.378464 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:04.378386    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-19faa5/distinct-2-w6-19faa5-kserve-58db847577-hrzlw"]
Nov 01 03:26:04.382650 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:04.382626    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-19faa5/distinct-2-w6-19faa5-kserve-58db847577-hrzlw"
Nov 01 03:26:04.439023 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:04.438966    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-19faa5\"/\"default-dockercfg-8s7vl\""
Nov 01 03:26:04.441128 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:04.441104    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-19faa5\"/\"distinct-2-w6-19faa5-kserve-self-signed-certs\""
Nov 01 03:26:04.441508 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:04.441306    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-19faa5\"/\"openshift-service-ca.crt\""
Nov 01 03:26:04.441655 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:04.441455    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-19faa5\"/\"kube-root-ca.crt\""
Nov 01 03:26:04.444480 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:04.444457    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-19faa5/distinct-2-w6-19faa5-kserve-58db847577-hrzlw"]
Nov 01 03:26:04.522871 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:04.522836    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/062959ee-ec7c-4385-98d5-cf4a240852e5-dshm\") pod \"distinct-2-w6-19faa5-kserve-58db847577-hrzlw\" (UID: \"062959ee-ec7c-4385-98d5-cf4a240852e5\") " pod="e2e-models-e2e-worker-w6-19faa5/distinct-2-w6-19faa5-kserve-58db847577-hrzlw"
Nov 01 03:26:04.523042 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:04.522993    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/062959ee-ec7c-4385-98d5-cf4a240852e5-tmp-dir\") pod \"distinct-2-w6-19faa5-kserve-58db847577-hrzlw\" (UID: \"062959ee-ec7c-4385-98d5-cf4a240852e5\") " pod="e2e-models-e2e-worker-w6-19faa5/distinct-2-w6-19faa5-kserve-58db847577-hrzlw"
Nov 01 03:26:04.523116 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:04.523054    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/062959ee-ec7c-4385-98d5-cf4a240852e5-tls-certs\") pod \"distinct-2-w6-19faa5-kserve-58db847577-hrzlw\" (UID: \"062959ee-ec7c-4385-98d5-cf4a240852e5\") " pod="e2e-models-e2e-worker-w6-19faa5/distinct-2-w6-19faa5-kserve-58db847577-hrzlw"
Nov 01 03:26:04.523162 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:04.523118    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/062959ee-ec7c-4385-98d5-cf4a240852e5-model-cache\") pod \"distinct-2-w6-19faa5-kserve-58db847577-hrzlw\" (UID: \"062959ee-ec7c-4385-98d5-cf4a240852e5\") " pod="e2e-models-e2e-worker-w6-19faa5/distinct-2-w6-19faa5-kserve-58db847577-hrzlw"
Nov 01 03:26:04.523254 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:04.523228    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmptv\" (UniqueName: \"kubernetes.io/projected/062959ee-ec7c-4385-98d5-cf4a240852e5-kube-api-access-mmptv\") pod \"distinct-2-w6-19faa5-kserve-58db847577-hrzlw\" (UID: \"062959ee-ec7c-4385-98d5-cf4a240852e5\") " pod="e2e-models-e2e-worker-w6-19faa5/distinct-2-w6-19faa5-kserve-58db847577-hrzlw"
Nov 01 03:26:04.523300 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:04.523284    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/062959ee-ec7c-4385-98d5-cf4a240852e5-home\") pod \"distinct-2-w6-19faa5-kserve-58db847577-hrzlw\" (UID: \"062959ee-ec7c-4385-98d5-cf4a240852e5\") " pod="e2e-models-e2e-worker-w6-19faa5/distinct-2-w6-19faa5-kserve-58db847577-hrzlw"
Nov 01 03:26:04.624653 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:04.624621    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/062959ee-ec7c-4385-98d5-cf4a240852e5-tmp-dir\") pod \"distinct-2-w6-19faa5-kserve-58db847577-hrzlw\" (UID: \"062959ee-ec7c-4385-98d5-cf4a240852e5\") " pod="e2e-models-e2e-worker-w6-19faa5/distinct-2-w6-19faa5-kserve-58db847577-hrzlw"
Nov 01 03:26:04.624865 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:04.624732    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/062959ee-ec7c-4385-98d5-cf4a240852e5-tls-certs\") pod \"distinct-2-w6-19faa5-kserve-58db847577-hrzlw\" (UID: \"062959ee-ec7c-4385-98d5-cf4a240852e5\") " pod="e2e-models-e2e-worker-w6-19faa5/distinct-2-w6-19faa5-kserve-58db847577-hrzlw"
Nov 01 03:26:04.624865 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:04.624769    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/062959ee-ec7c-4385-98d5-cf4a240852e5-model-cache\") pod \"distinct-2-w6-19faa5-kserve-58db847577-hrzlw\" (UID: \"062959ee-ec7c-4385-98d5-cf4a240852e5\") " pod="e2e-models-e2e-worker-w6-19faa5/distinct-2-w6-19faa5-kserve-58db847577-hrzlw"
Nov 01 03:26:04.624865 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:04.624827    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmptv\" (UniqueName: \"kubernetes.io/projected/062959ee-ec7c-4385-98d5-cf4a240852e5-kube-api-access-mmptv\") pod \"distinct-2-w6-19faa5-kserve-58db847577-hrzlw\" (UID: \"062959ee-ec7c-4385-98d5-cf4a240852e5\") " pod="e2e-models-e2e-worker-w6-19faa5/distinct-2-w6-19faa5-kserve-58db847577-hrzlw"
Nov 01 03:26:04.624865 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:04.624861    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/062959ee-ec7c-4385-98d5-cf4a240852e5-home\") pod \"distinct-2-w6-19faa5-kserve-58db847577-hrzlw\" (UID: \"062959ee-ec7c-4385-98d5-cf4a240852e5\") " pod="e2e-models-e2e-worker-w6-19faa5/distinct-2-w6-19faa5-kserve-58db847577-hrzlw"
Nov 01 03:26:04.625076 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:04.624947    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/062959ee-ec7c-4385-98d5-cf4a240852e5-dshm\") pod \"distinct-2-w6-19faa5-kserve-58db847577-hrzlw\" (UID: \"062959ee-ec7c-4385-98d5-cf4a240852e5\") " pod="e2e-models-e2e-worker-w6-19faa5/distinct-2-w6-19faa5-kserve-58db847577-hrzlw"
Nov 01 03:26:04.625076 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:04.625042    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/062959ee-ec7c-4385-98d5-cf4a240852e5-tmp-dir\") pod \"distinct-2-w6-19faa5-kserve-58db847577-hrzlw\" (UID: \"062959ee-ec7c-4385-98d5-cf4a240852e5\") " pod="e2e-models-e2e-worker-w6-19faa5/distinct-2-w6-19faa5-kserve-58db847577-hrzlw"
Nov 01 03:26:04.625199 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:04.625181    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/062959ee-ec7c-4385-98d5-cf4a240852e5-model-cache\") pod \"distinct-2-w6-19faa5-kserve-58db847577-hrzlw\" (UID: \"062959ee-ec7c-4385-98d5-cf4a240852e5\") " pod="e2e-models-e2e-worker-w6-19faa5/distinct-2-w6-19faa5-kserve-58db847577-hrzlw"
Nov 01 03:26:04.625258 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:04.625211    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/062959ee-ec7c-4385-98d5-cf4a240852e5-home\") pod \"distinct-2-w6-19faa5-kserve-58db847577-hrzlw\" (UID: \"062959ee-ec7c-4385-98d5-cf4a240852e5\") " pod="e2e-models-e2e-worker-w6-19faa5/distinct-2-w6-19faa5-kserve-58db847577-hrzlw"
Nov 01 03:26:04.627218 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:04.627196    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/062959ee-ec7c-4385-98d5-cf4a240852e5-dshm\") pod \"distinct-2-w6-19faa5-kserve-58db847577-hrzlw\" (UID: \"062959ee-ec7c-4385-98d5-cf4a240852e5\") " pod="e2e-models-e2e-worker-w6-19faa5/distinct-2-w6-19faa5-kserve-58db847577-hrzlw"
Nov 01 03:26:04.627342 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:04.627310    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/062959ee-ec7c-4385-98d5-cf4a240852e5-tls-certs\") pod \"distinct-2-w6-19faa5-kserve-58db847577-hrzlw\" (UID: \"062959ee-ec7c-4385-98d5-cf4a240852e5\") " pod="e2e-models-e2e-worker-w6-19faa5/distinct-2-w6-19faa5-kserve-58db847577-hrzlw"
Nov 01 03:26:04.687416 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:04.687353    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmptv\" (UniqueName: \"kubernetes.io/projected/062959ee-ec7c-4385-98d5-cf4a240852e5-kube-api-access-mmptv\") pod \"distinct-2-w6-19faa5-kserve-58db847577-hrzlw\" (UID: \"062959ee-ec7c-4385-98d5-cf4a240852e5\") " pod="e2e-models-e2e-worker-w6-19faa5/distinct-2-w6-19faa5-kserve-58db847577-hrzlw"
Nov 01 03:26:04.695829 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:04.695798    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-19faa5/distinct-2-w6-19faa5-kserve-58db847577-hrzlw"
Nov 01 03:26:05.078524 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:05.078496    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-19faa5/distinct-2-w6-19faa5-kserve-58db847577-hrzlw"]
Nov 01 03:26:05.079302 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:26:05.079271    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod062959ee_ec7c_4385_98d5_cf4a240852e5.slice/crio-a9d95d204ab7acadfeecbaed868026e6e58093cf117d7e27f8054416a1c470de WatchSource:0}: Error finding container a9d95d204ab7acadfeecbaed868026e6e58093cf117d7e27f8054416a1c470de: Status 404 returned error can't find the container with id a9d95d204ab7acadfeecbaed868026e6e58093cf117d7e27f8054416a1c470de
Nov 01 03:26:06.012528 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:06.012483    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-19faa5/distinct-2-w6-19faa5-kserve-58db847577-hrzlw" event={"ID":"062959ee-ec7c-4385-98d5-cf4a240852e5","Type":"ContainerStarted","Data":"3391d65238eaaf6631628730676c33368ca0469d55c523ed38a27a089a79b54a"}
Nov 01 03:26:06.012528 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:06.012525    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-19faa5/distinct-2-w6-19faa5-kserve-58db847577-hrzlw" event={"ID":"062959ee-ec7c-4385-98d5-cf4a240852e5","Type":"ContainerStarted","Data":"a9d95d204ab7acadfeecbaed868026e6e58093cf117d7e27f8054416a1c470de"}
Nov 01 03:26:06.146389 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:06.146331    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-19faa5/distinct-2-w6-19faa5-kserve-58db847577-hrzlw" podStartSLOduration=2.146317003 podStartE2EDuration="2.146317003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:26:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:26:06.133803513 +0000 UTC m=+1172.648449389" watchObservedRunningTime="2026-11-01 03:26:06.146317003 +0000 UTC m=+1172.660962865"
Nov 01 03:26:06.696416 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:06.696384    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-19faa5/distinct-2-w6-19faa5-kserve-58db847577-hrzlw"
Nov 01 03:26:14.695987 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:14.695958    4400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-19faa5/distinct-2-w6-19faa5-kserve-58db847577-hrzlw"
Nov 01 03:26:14.708444 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:14.708421    4400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-19faa5/distinct-2-w6-19faa5-kserve-58db847577-hrzlw"
Nov 01 03:26:15.080501 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:15.080481    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-19faa5/distinct-2-w6-19faa5-kserve-58db847577-hrzlw"
Nov 01 03:26:23.873525 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:23.873494    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-e8c789/distinct-2-w4-e8c789-kserve-7977c64f4-wkxb2"]
Nov 01 03:26:23.877654 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:23.877633    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e8c789/distinct-2-w4-e8c789-kserve-7977c64f4-wkxb2"
Nov 01 03:26:23.932358 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:23.932121    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-e8c789\"/\"distinct-2-w4-e8c789-kserve-self-signed-certs\""
Nov 01 03:26:23.936858 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:23.936829    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-e8c789/distinct-2-w4-e8c789-kserve-7977c64f4-wkxb2"]
Nov 01 03:26:23.966647 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:23.966623    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdb1bb9e-14f5-4be6-a028-c980119f4aa8-model-cache\") pod \"distinct-2-w4-e8c789-kserve-7977c64f4-wkxb2\" (UID: \"bdb1bb9e-14f5-4be6-a028-c980119f4aa8\") " pod="e2e-models-e2e-worker-w4-e8c789/distinct-2-w4-e8c789-kserve-7977c64f4-wkxb2"
Nov 01 03:26:23.966776 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:23.966704    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdb1bb9e-14f5-4be6-a028-c980119f4aa8-tmp-dir\") pod \"distinct-2-w4-e8c789-kserve-7977c64f4-wkxb2\" (UID: \"bdb1bb9e-14f5-4be6-a028-c980119f4aa8\") " pod="e2e-models-e2e-worker-w4-e8c789/distinct-2-w4-e8c789-kserve-7977c64f4-wkxb2"
Nov 01 03:26:23.966776 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:23.966749    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdb1bb9e-14f5-4be6-a028-c980119f4aa8-tls-certs\") pod \"distinct-2-w4-e8c789-kserve-7977c64f4-wkxb2\" (UID: \"bdb1bb9e-14f5-4be6-a028-c980119f4aa8\") " pod="e2e-models-e2e-worker-w4-e8c789/distinct-2-w4-e8c789-kserve-7977c64f4-wkxb2"
Nov 01 03:26:23.966889 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:23.966868    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhwzn\" (UniqueName: \"kubernetes.io/projected/bdb1bb9e-14f5-4be6-a028-c980119f4aa8-kube-api-access-rhwzn\") pod \"distinct-2-w4-e8c789-kserve-7977c64f4-wkxb2\" (UID: \"bdb1bb9e-14f5-4be6-a028-c980119f4aa8\") " pod="e2e-models-e2e-worker-w4-e8c789/distinct-2-w4-e8c789-kserve-7977c64f4-wkxb2"
Nov 01 03:26:23.966936 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:23.966924    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdb1bb9e-14f5-4be6-a028-c980119f4aa8-dshm\") pod \"distinct-2-w4-e8c789-kserve-7977c64f4-wkxb2\" (UID: \"bdb1bb9e-14f5-4be6-a028-c980119f4aa8\") " pod="e2e-models-e2e-worker-w4-e8c789/distinct-2-w4-e8c789-kserve-7977c64f4-wkxb2"
Nov 01 03:26:23.967034 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:23.967023    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdb1bb9e-14f5-4be6-a028-c980119f4aa8-home\") pod \"distinct-2-w4-e8c789-kserve-7977c64f4-wkxb2\" (UID: \"bdb1bb9e-14f5-4be6-a028-c980119f4aa8\") " pod="e2e-models-e2e-worker-w4-e8c789/distinct-2-w4-e8c789-kserve-7977c64f4-wkxb2"
Nov 01 03:26:24.001410 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:24.001382    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 03:26:24.089714 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:24.073474    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdb1bb9e-14f5-4be6-a028-c980119f4aa8-home\") pod \"distinct-2-w4-e8c789-kserve-7977c64f4-wkxb2\" (UID: \"bdb1bb9e-14f5-4be6-a028-c980119f4aa8\") " pod="e2e-models-e2e-worker-w4-e8c789/distinct-2-w4-e8c789-kserve-7977c64f4-wkxb2"
Nov 01 03:26:24.089714 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:24.073734    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdb1bb9e-14f5-4be6-a028-c980119f4aa8-model-cache\") pod \"distinct-2-w4-e8c789-kserve-7977c64f4-wkxb2\" (UID: \"bdb1bb9e-14f5-4be6-a028-c980119f4aa8\") " pod="e2e-models-e2e-worker-w4-e8c789/distinct-2-w4-e8c789-kserve-7977c64f4-wkxb2"
Nov 01 03:26:24.089714 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:24.073835    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdb1bb9e-14f5-4be6-a028-c980119f4aa8-tmp-dir\") pod \"distinct-2-w4-e8c789-kserve-7977c64f4-wkxb2\" (UID: \"bdb1bb9e-14f5-4be6-a028-c980119f4aa8\") " pod="e2e-models-e2e-worker-w4-e8c789/distinct-2-w4-e8c789-kserve-7977c64f4-wkxb2"
Nov 01 03:26:24.089714 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:24.073990    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdb1bb9e-14f5-4be6-a028-c980119f4aa8-tls-certs\") pod \"distinct-2-w4-e8c789-kserve-7977c64f4-wkxb2\" (UID: \"bdb1bb9e-14f5-4be6-a028-c980119f4aa8\") " pod="e2e-models-e2e-worker-w4-e8c789/distinct-2-w4-e8c789-kserve-7977c64f4-wkxb2"
Nov 01 03:26:24.089714 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:24.074053    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhwzn\" (UniqueName: \"kubernetes.io/projected/bdb1bb9e-14f5-4be6-a028-c980119f4aa8-kube-api-access-rhwzn\") pod \"distinct-2-w4-e8c789-kserve-7977c64f4-wkxb2\" (UID: \"bdb1bb9e-14f5-4be6-a028-c980119f4aa8\") " pod="e2e-models-e2e-worker-w4-e8c789/distinct-2-w4-e8c789-kserve-7977c64f4-wkxb2"
Nov 01 03:26:24.089714 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:24.074200    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdb1bb9e-14f5-4be6-a028-c980119f4aa8-dshm\") pod \"distinct-2-w4-e8c789-kserve-7977c64f4-wkxb2\" (UID: \"bdb1bb9e-14f5-4be6-a028-c980119f4aa8\") " pod="e2e-models-e2e-worker-w4-e8c789/distinct-2-w4-e8c789-kserve-7977c64f4-wkxb2"
Nov 01 03:26:24.092446 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:24.092374    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdb1bb9e-14f5-4be6-a028-c980119f4aa8-tmp-dir\") pod \"distinct-2-w4-e8c789-kserve-7977c64f4-wkxb2\" (UID: \"bdb1bb9e-14f5-4be6-a028-c980119f4aa8\") " pod="e2e-models-e2e-worker-w4-e8c789/distinct-2-w4-e8c789-kserve-7977c64f4-wkxb2"
Nov 01 03:26:24.092886 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:24.092834    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdb1bb9e-14f5-4be6-a028-c980119f4aa8-model-cache\") pod \"distinct-2-w4-e8c789-kserve-7977c64f4-wkxb2\" (UID: \"bdb1bb9e-14f5-4be6-a028-c980119f4aa8\") " pod="e2e-models-e2e-worker-w4-e8c789/distinct-2-w4-e8c789-kserve-7977c64f4-wkxb2"
Nov 01 03:26:24.095113 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:24.095079    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdb1bb9e-14f5-4be6-a028-c980119f4aa8-home\") pod \"distinct-2-w4-e8c789-kserve-7977c64f4-wkxb2\" (UID: \"bdb1bb9e-14f5-4be6-a028-c980119f4aa8\") " pod="e2e-models-e2e-worker-w4-e8c789/distinct-2-w4-e8c789-kserve-7977c64f4-wkxb2"
Nov 01 03:26:24.129754 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:24.128202    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdb1bb9e-14f5-4be6-a028-c980119f4aa8-tls-certs\") pod \"distinct-2-w4-e8c789-kserve-7977c64f4-wkxb2\" (UID: \"bdb1bb9e-14f5-4be6-a028-c980119f4aa8\") " pod="e2e-models-e2e-worker-w4-e8c789/distinct-2-w4-e8c789-kserve-7977c64f4-wkxb2"
Nov 01 03:26:24.135015 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:24.134983    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdb1bb9e-14f5-4be6-a028-c980119f4aa8-dshm\") pod \"distinct-2-w4-e8c789-kserve-7977c64f4-wkxb2\" (UID: \"bdb1bb9e-14f5-4be6-a028-c980119f4aa8\") " pod="e2e-models-e2e-worker-w4-e8c789/distinct-2-w4-e8c789-kserve-7977c64f4-wkxb2"
Nov 01 03:26:24.159729 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:24.158520    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhwzn\" (UniqueName: \"kubernetes.io/projected/bdb1bb9e-14f5-4be6-a028-c980119f4aa8-kube-api-access-rhwzn\") pod \"distinct-2-w4-e8c789-kserve-7977c64f4-wkxb2\" (UID: \"bdb1bb9e-14f5-4be6-a028-c980119f4aa8\") " pod="e2e-models-e2e-worker-w4-e8c789/distinct-2-w4-e8c789-kserve-7977c64f4-wkxb2"
Nov 01 03:26:24.192201 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:24.192171    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e8c789/distinct-2-w4-e8c789-kserve-7977c64f4-wkxb2"
Nov 01 03:26:24.576840 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:26:24.576809    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbdb1bb9e_14f5_4be6_a028_c980119f4aa8.slice/crio-1da8d9451f703cd6903674237ef17bcd89db25bf6c63620a0771266436482224 WatchSource:0}: Error finding container 1da8d9451f703cd6903674237ef17bcd89db25bf6c63620a0771266436482224: Status 404 returned error can't find the container with id 1da8d9451f703cd6903674237ef17bcd89db25bf6c63620a0771266436482224
Nov 01 03:26:24.577262 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:24.577236    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-e8c789/distinct-2-w4-e8c789-kserve-7977c64f4-wkxb2"]
Nov 01 03:26:25.195644 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:25.195597    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e8c789/distinct-2-w4-e8c789-kserve-7977c64f4-wkxb2" event={"ID":"bdb1bb9e-14f5-4be6-a028-c980119f4aa8","Type":"ContainerStarted","Data":"14bb147afb82fe8ce409e1f8f4b10936e4771fed96d7a486fbb4200cab18dceb"}
Nov 01 03:26:25.196126 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:25.195658    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e8c789/distinct-2-w4-e8c789-kserve-7977c64f4-wkxb2" event={"ID":"bdb1bb9e-14f5-4be6-a028-c980119f4aa8","Type":"ContainerStarted","Data":"1da8d9451f703cd6903674237ef17bcd89db25bf6c63620a0771266436482224"}
Nov 01 03:26:26.192958 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:26.192929    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e8c789/distinct-2-w4-e8c789-kserve-7977c64f4-wkxb2"
Nov 01 03:26:31.730313 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:26:31.730278    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:26:34.195257 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:34.195228    4400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-e8c789/distinct-2-w4-e8c789-kserve-7977c64f4-wkxb2"
Nov 01 03:26:34.224127 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:34.224079    4400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-e8c789/distinct-2-w4-e8c789-kserve-7977c64f4-wkxb2"
Nov 01 03:26:34.285086 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:34.285060    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e8c789/distinct-2-w4-e8c789-kserve-7977c64f4-wkxb2"
Nov 01 03:26:34.346804 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:34.346756    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-e8c789/distinct-2-w4-e8c789-kserve-7977c64f4-wkxb2" podStartSLOduration=11.346742716 podStartE2EDuration="11.346742716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:26:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:26:25.358432455 +0000 UTC m=+1191.873078320" watchObservedRunningTime="2026-11-01 03:26:34.346742716 +0000 UTC m=+1200.861388630"
Nov 01 03:26:36.289718 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:36.289661    4400 generic.go:356] "Generic (PLEG): container finished" podID="01fe53c4-ce4d-4696-b738-2e9e10c29238" containerID="8430914c265817a11143e856b491c91375ad50cc2ce3f51ac09fbe6d5a1b6437" exitCode=0
Nov 01 03:26:36.290089 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:36.289742    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-h7pkm-6lxwl" event={"ID":"01fe53c4-ce4d-4696-b738-2e9e10c29238","Type":"ContainerDied","Data":"8430914c265817a11143e856b491c91375ad50cc2ce3f51ac09fbe6d5a1b6437"}
Nov 01 03:26:37.455293 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:37.455270    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-h7pkm-6lxwl"
Nov 01 03:26:37.588769 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:37.588745    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01fe53c4-ce4d-4696-b738-2e9e10c29238-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01fe53c4-ce4d-4696-b738-2e9e10c29238-service-ca-bundle\") pod \"01fe53c4-ce4d-4696-b738-2e9e10c29238\" (UID: \"01fe53c4-ce4d-4696-b738-2e9e10c29238\") "
Nov 01 03:26:37.588936 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:37.588777    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01fe53c4-ce4d-4696-b738-2e9e10c29238-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01fe53c4-ce4d-4696-b738-2e9e10c29238-trusted-ca-bundle\") pod \"01fe53c4-ce4d-4696-b738-2e9e10c29238\" (UID: \"01fe53c4-ce4d-4696-b738-2e9e10c29238\") "
Nov 01 03:26:37.588936 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:37.588816    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01fe53c4-ce4d-4696-b738-2e9e10c29238-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/01fe53c4-ce4d-4696-b738-2e9e10c29238-archives-path\") pod \"01fe53c4-ce4d-4696-b738-2e9e10c29238\" (UID: \"01fe53c4-ce4d-4696-b738-2e9e10c29238\") "
Nov 01 03:26:37.588936 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:37.588912    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01fe53c4-ce4d-4696-b738-2e9e10c29238-kube-api-access-qjdf9\" (UniqueName: \"kubernetes.io/projected/01fe53c4-ce4d-4696-b738-2e9e10c29238-kube-api-access-qjdf9\") pod \"01fe53c4-ce4d-4696-b738-2e9e10c29238\" (UID: \"01fe53c4-ce4d-4696-b738-2e9e10c29238\") "
Nov 01 03:26:37.589164 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:37.589140    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01fe53c4-ce4d-4696-b738-2e9e10c29238-service-ca-bundle" pod "01fe53c4-ce4d-4696-b738-2e9e10c29238" (UID: "01fe53c4-ce4d-4696-b738-2e9e10c29238"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:26:37.589228 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:37.589169    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01fe53c4-ce4d-4696-b738-2e9e10c29238-trusted-ca-bundle" pod "01fe53c4-ce4d-4696-b738-2e9e10c29238" (UID: "01fe53c4-ce4d-4696-b738-2e9e10c29238"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:26:37.589228 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:37.589186    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01fe53c4-ce4d-4696-b738-2e9e10c29238-archives-path" pod "01fe53c4-ce4d-4696-b738-2e9e10c29238" (UID: "01fe53c4-ce4d-4696-b738-2e9e10c29238"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:26:37.589451 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:37.589426    4400 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01fe53c4-ce4d-4696-b738-2e9e10c29238-service-ca-bundle\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:26:37.589745 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:37.589468    4400 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01fe53c4-ce4d-4696-b738-2e9e10c29238-trusted-ca-bundle\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:26:37.589745 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:37.589480    4400 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/01fe53c4-ce4d-4696-b738-2e9e10c29238-archives-path\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:26:37.591219 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:37.591193    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01fe53c4-ce4d-4696-b738-2e9e10c29238-kube-api-access-qjdf9" pod "01fe53c4-ce4d-4696-b738-2e9e10c29238" (UID: "01fe53c4-ce4d-4696-b738-2e9e10c29238"). InnerVolumeSpecName "kube-api-access-qjdf9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:26:37.690350 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:37.690328    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjdf9\" (UniqueName: \"kubernetes.io/projected/01fe53c4-ce4d-4696-b738-2e9e10c29238-kube-api-access-qjdf9\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:26:38.304921 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:38.304887    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-h7pkm-6lxwl" event={"ID":"01fe53c4-ce4d-4696-b738-2e9e10c29238","Type":"ContainerDied","Data":"6b699d3caca2466107fb10f695896bfbcb7ecbf53619fde4aa57da2c006482f9"}
Nov 01 03:26:38.304921 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:38.304918    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-h7pkm-6lxwl"
Nov 01 03:26:38.305122 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:38.304931    4400 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b699d3caca2466107fb10f695896bfbcb7ecbf53619fde4aa57da2c006482f9"
Nov 01 03:26:40.756562 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:40.756520    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wqmgx"]
Nov 01 03:26:40.757256 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:40.757238    4400 cpu_manager.go:414] "containerMap: removing container" podUID="01fe53c4-ce4d-4696-b738-2e9e10c29238" containerName="insights-gathering"
Nov 01 03:26:40.757317 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:40.757259    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01fe53c4-ce4d-4696-b738-2e9e10c29238" containerName="insights-gathering"
Nov 01 03:26:40.757401 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:40.757391    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="01fe53c4-ce4d-4696-b738-2e9e10c29238" containerName="insights-gathering"
Nov 01 03:26:40.761139 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:40.761120    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wqmgx"
Nov 01 03:26:40.817901 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:40.817877    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-19faa5-openshift-default-dockercfg-tk569\""
Nov 01 03:26:40.820610 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:40.820589    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66ca3614-7687-403d-a5c0-359421d8584b-istiod-ca-cert\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-wqmgx\" (UID: \"66ca3614-7687-403d-a5c0-359421d8584b\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wqmgx"
Nov 01 03:26:40.820738 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:40.820723    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66ca3614-7687-403d-a5c0-359421d8584b-credential-socket\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-wqmgx\" (UID: \"66ca3614-7687-403d-a5c0-359421d8584b\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wqmgx"
Nov 01 03:26:40.820787 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:40.820768    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66ca3614-7687-403d-a5c0-359421d8584b-workload-socket\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-wqmgx\" (UID: \"66ca3614-7687-403d-a5c0-359421d8584b\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wqmgx"
Nov 01 03:26:40.820876 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:40.820863    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkqv6\" (UniqueName: \"kubernetes.io/projected/66ca3614-7687-403d-a5c0-359421d8584b-kube-api-access-bkqv6\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-wqmgx\" (UID: \"66ca3614-7687-403d-a5c0-359421d8584b\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wqmgx"
Nov 01 03:26:40.820911 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:40.820899    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66ca3614-7687-403d-a5c0-359421d8584b-workload-certs\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-wqmgx\" (UID: \"66ca3614-7687-403d-a5c0-359421d8584b\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wqmgx"
Nov 01 03:26:40.820944 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:40.820916    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66ca3614-7687-403d-a5c0-359421d8584b-istio-token\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-wqmgx\" (UID: \"66ca3614-7687-403d-a5c0-359421d8584b\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wqmgx"
Nov 01 03:26:40.821057 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:40.821039    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66ca3614-7687-403d-a5c0-359421d8584b-istio-envoy\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-wqmgx\" (UID: \"66ca3614-7687-403d-a5c0-359421d8584b\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wqmgx"
Nov 01 03:26:40.821105 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:40.821093    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66ca3614-7687-403d-a5c0-359421d8584b-istio-podinfo\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-wqmgx\" (UID: \"66ca3614-7687-403d-a5c0-359421d8584b\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wqmgx"
Nov 01 03:26:40.821159 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:40.821147    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66ca3614-7687-403d-a5c0-359421d8584b-istio-data\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-wqmgx\" (UID: \"66ca3614-7687-403d-a5c0-359421d8584b\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wqmgx"
Nov 01 03:26:40.873507 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:40.873482    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wqmgx"]
Nov 01 03:26:40.921929 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:40.921906    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66ca3614-7687-403d-a5c0-359421d8584b-credential-socket\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-wqmgx\" (UID: \"66ca3614-7687-403d-a5c0-359421d8584b\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wqmgx"
Nov 01 03:26:40.922067 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:40.921951    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66ca3614-7687-403d-a5c0-359421d8584b-workload-socket\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-wqmgx\" (UID: \"66ca3614-7687-403d-a5c0-359421d8584b\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wqmgx"
Nov 01 03:26:40.922067 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:40.921996    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkqv6\" (UniqueName: \"kubernetes.io/projected/66ca3614-7687-403d-a5c0-359421d8584b-kube-api-access-bkqv6\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-wqmgx\" (UID: \"66ca3614-7687-403d-a5c0-359421d8584b\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wqmgx"
Nov 01 03:26:40.922067 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:40.922030    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66ca3614-7687-403d-a5c0-359421d8584b-workload-certs\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-wqmgx\" (UID: \"66ca3614-7687-403d-a5c0-359421d8584b\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wqmgx"
Nov 01 03:26:40.922067 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:40.922046    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66ca3614-7687-403d-a5c0-359421d8584b-istio-token\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-wqmgx\" (UID: \"66ca3614-7687-403d-a5c0-359421d8584b\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wqmgx"
Nov 01 03:26:40.922284 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:40.922092    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66ca3614-7687-403d-a5c0-359421d8584b-istio-envoy\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-wqmgx\" (UID: \"66ca3614-7687-403d-a5c0-359421d8584b\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wqmgx"
Nov 01 03:26:40.922284 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:40.922134    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66ca3614-7687-403d-a5c0-359421d8584b-istio-podinfo\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-wqmgx\" (UID: \"66ca3614-7687-403d-a5c0-359421d8584b\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wqmgx"
Nov 01 03:26:40.922284 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:40.922151    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66ca3614-7687-403d-a5c0-359421d8584b-istio-data\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-wqmgx\" (UID: \"66ca3614-7687-403d-a5c0-359421d8584b\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wqmgx"
Nov 01 03:26:40.922284 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:40.922186    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66ca3614-7687-403d-a5c0-359421d8584b-istiod-ca-cert\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-wqmgx\" (UID: \"66ca3614-7687-403d-a5c0-359421d8584b\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wqmgx"
Nov 01 03:26:40.922495 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:40.922343    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66ca3614-7687-403d-a5c0-359421d8584b-credential-socket\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-wqmgx\" (UID: \"66ca3614-7687-403d-a5c0-359421d8584b\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wqmgx"
Nov 01 03:26:40.922495 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:40.922341    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66ca3614-7687-403d-a5c0-359421d8584b-workload-socket\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-wqmgx\" (UID: \"66ca3614-7687-403d-a5c0-359421d8584b\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wqmgx"
Nov 01 03:26:40.922495 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:40.922447    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66ca3614-7687-403d-a5c0-359421d8584b-workload-certs\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-wqmgx\" (UID: \"66ca3614-7687-403d-a5c0-359421d8584b\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wqmgx"
Nov 01 03:26:40.922662 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:40.922566    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66ca3614-7687-403d-a5c0-359421d8584b-istio-data\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-wqmgx\" (UID: \"66ca3614-7687-403d-a5c0-359421d8584b\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wqmgx"
Nov 01 03:26:40.923073 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:40.923049    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66ca3614-7687-403d-a5c0-359421d8584b-istiod-ca-cert\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-wqmgx\" (UID: \"66ca3614-7687-403d-a5c0-359421d8584b\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wqmgx"
Nov 01 03:26:40.924895 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:40.924857    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66ca3614-7687-403d-a5c0-359421d8584b-istio-envoy\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-wqmgx\" (UID: \"66ca3614-7687-403d-a5c0-359421d8584b\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wqmgx"
Nov 01 03:26:40.925005 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:40.924986    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66ca3614-7687-403d-a5c0-359421d8584b-istio-podinfo\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-wqmgx\" (UID: \"66ca3614-7687-403d-a5c0-359421d8584b\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wqmgx"
Nov 01 03:26:40.988055 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:40.988031    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkqv6\" (UniqueName: \"kubernetes.io/projected/66ca3614-7687-403d-a5c0-359421d8584b-kube-api-access-bkqv6\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-wqmgx\" (UID: \"66ca3614-7687-403d-a5c0-359421d8584b\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wqmgx"
Nov 01 03:26:40.992769 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:40.992746    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66ca3614-7687-403d-a5c0-359421d8584b-istio-token\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-wqmgx\" (UID: \"66ca3614-7687-403d-a5c0-359421d8584b\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wqmgx"
Nov 01 03:26:41.075250 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:41.075230    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wqmgx"
Nov 01 03:26:41.505443 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:41.505420    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wqmgx"]
Nov 01 03:26:41.506233 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:26:41.506204    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66ca3614_7687_403d_a5c0_359421d8584b.slice/crio-5088e8235268c93be85c23d6ae9e80a1fb19ebd931518cb965c3d53999ed1add WatchSource:0}: Error finding container 5088e8235268c93be85c23d6ae9e80a1fb19ebd931518cb965c3d53999ed1add: Status 404 returned error can't find the container with id 5088e8235268c93be85c23d6ae9e80a1fb19ebd931518cb965c3d53999ed1add
Nov 01 03:26:41.508775 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:41.508745    4400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 03:26:41.508855 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:41.508806    4400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 03:26:41.508855 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:41.508832    4400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 03:26:42.364709 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:42.361161    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wqmgx" event={"ID":"66ca3614-7687-403d-a5c0-359421d8584b","Type":"ContainerStarted","Data":"c775e544a2c5155452178a9c1619ea5d76521fb7efdaaab69565bcd7645bd028"}
Nov 01 03:26:42.364709 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:42.361230    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wqmgx" event={"ID":"66ca3614-7687-403d-a5c0-359421d8584b","Type":"ContainerStarted","Data":"5088e8235268c93be85c23d6ae9e80a1fb19ebd931518cb965c3d53999ed1add"}
Nov 01 03:26:42.597292 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:42.597219    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wqmgx" podStartSLOduration=2.597198668 podStartE2EDuration="2.597198668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:26:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:26:42.545370334 +0000 UTC m=+1209.060016197" watchObservedRunningTime="2026-11-01 03:26:42.597198668 +0000 UTC m=+1209.111844531"
Nov 01 03:26:43.075850 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:43.075814    4400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wqmgx"
Nov 01 03:26:43.077827 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:43.077793    4400 patch_prober.go:29] interesting pod/e2e-worker-w6-19faa5-openshift-default-59c44967-wqmgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Nov 01 03:26:43.077962 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:43.077860    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wqmgx" podUID="66ca3614-7687-403d-a5c0-359421d8584b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Nov 01 03:26:44.085840 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:44.084957    4400 patch_prober.go:29] interesting pod/e2e-worker-w6-19faa5-openshift-default-59c44967-wqmgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Nov 01 03:26:44.085840 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:44.085017    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wqmgx" podUID="66ca3614-7687-403d-a5c0-359421d8584b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Nov 01 03:26:45.076261 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:45.076225    4400 patch_prober.go:29] interesting pod/e2e-worker-w6-19faa5-openshift-default-59c44967-wqmgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Nov 01 03:26:45.076453 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:45.076278    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wqmgx" podUID="66ca3614-7687-403d-a5c0-359421d8584b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Nov 01 03:26:46.076432 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:46.076399    4400 patch_prober.go:29] interesting pod/e2e-worker-w6-19faa5-openshift-default-59c44967-wqmgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Nov 01 03:26:46.077234 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:46.077193    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wqmgx" podUID="66ca3614-7687-403d-a5c0-359421d8584b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Nov 01 03:26:47.076367 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:47.076333    4400 patch_prober.go:29] interesting pod/e2e-worker-w6-19faa5-openshift-default-59c44967-wqmgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Nov 01 03:26:47.076548 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:47.076381    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wqmgx" podUID="66ca3614-7687-403d-a5c0-359421d8584b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Nov 01 03:26:48.076212 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:48.076177    4400 patch_prober.go:29] interesting pod/e2e-worker-w6-19faa5-openshift-default-59c44967-wqmgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Nov 01 03:26:48.076382 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:48.076244    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wqmgx" podUID="66ca3614-7687-403d-a5c0-359421d8584b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Nov 01 03:26:49.079614 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:49.079587    4400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wqmgx"
Nov 01 03:26:49.080111 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:49.080026    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wqmgx"
Nov 01 03:26:49.081463 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:49.081439    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wqmgx"
Nov 01 03:26:56.857269 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:56.857235    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-mk69d"]
Nov 01 03:26:56.861601 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:56.861572    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-mk69d"
Nov 01 03:26:56.915941 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:56.915920    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-c22dd8-openshift-default-dockercfg-64gt8\""
Nov 01 03:26:56.971887 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:56.971861    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-mk69d"]
Nov 01 03:26:56.973570 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:56.973547    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4988f434-dc04-41bd-8b3f-6e386bca1334-workload-socket\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-mk69d\" (UID: \"4988f434-dc04-41bd-8b3f-6e386bca1334\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-mk69d"
Nov 01 03:26:56.973747 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:56.973718    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4988f434-dc04-41bd-8b3f-6e386bca1334-istio-podinfo\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-mk69d\" (UID: \"4988f434-dc04-41bd-8b3f-6e386bca1334\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-mk69d"
Nov 01 03:26:56.973861 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:56.973837    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4988f434-dc04-41bd-8b3f-6e386bca1334-istio-data\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-mk69d\" (UID: \"4988f434-dc04-41bd-8b3f-6e386bca1334\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-mk69d"
Nov 01 03:26:56.973968 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:56.973949    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4988f434-dc04-41bd-8b3f-6e386bca1334-workload-certs\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-mk69d\" (UID: \"4988f434-dc04-41bd-8b3f-6e386bca1334\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-mk69d"
Nov 01 03:26:56.974021 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:56.974004    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvrpv\" (UniqueName: \"kubernetes.io/projected/4988f434-dc04-41bd-8b3f-6e386bca1334-kube-api-access-lvrpv\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-mk69d\" (UID: \"4988f434-dc04-41bd-8b3f-6e386bca1334\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-mk69d"
Nov 01 03:26:56.974056 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:56.974045    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4988f434-dc04-41bd-8b3f-6e386bca1334-istiod-ca-cert\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-mk69d\" (UID: \"4988f434-dc04-41bd-8b3f-6e386bca1334\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-mk69d"
Nov 01 03:26:56.974178 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:56.974161    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4988f434-dc04-41bd-8b3f-6e386bca1334-credential-socket\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-mk69d\" (UID: \"4988f434-dc04-41bd-8b3f-6e386bca1334\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-mk69d"
Nov 01 03:26:56.974256 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:56.974197    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4988f434-dc04-41bd-8b3f-6e386bca1334-istio-token\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-mk69d\" (UID: \"4988f434-dc04-41bd-8b3f-6e386bca1334\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-mk69d"
Nov 01 03:26:56.974343 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:56.974328    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4988f434-dc04-41bd-8b3f-6e386bca1334-istio-envoy\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-mk69d\" (UID: \"4988f434-dc04-41bd-8b3f-6e386bca1334\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-mk69d"
Nov 01 03:26:57.075972 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:57.075943    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4988f434-dc04-41bd-8b3f-6e386bca1334-credential-socket\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-mk69d\" (UID: \"4988f434-dc04-41bd-8b3f-6e386bca1334\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-mk69d"
Nov 01 03:26:57.076110 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:57.075985    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4988f434-dc04-41bd-8b3f-6e386bca1334-istio-token\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-mk69d\" (UID: \"4988f434-dc04-41bd-8b3f-6e386bca1334\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-mk69d"
Nov 01 03:26:57.076110 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:57.076052    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4988f434-dc04-41bd-8b3f-6e386bca1334-istio-envoy\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-mk69d\" (UID: \"4988f434-dc04-41bd-8b3f-6e386bca1334\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-mk69d"
Nov 01 03:26:57.076110 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:57.076097    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4988f434-dc04-41bd-8b3f-6e386bca1334-workload-socket\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-mk69d\" (UID: \"4988f434-dc04-41bd-8b3f-6e386bca1334\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-mk69d"
Nov 01 03:26:57.076282 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:57.076196    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4988f434-dc04-41bd-8b3f-6e386bca1334-istio-podinfo\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-mk69d\" (UID: \"4988f434-dc04-41bd-8b3f-6e386bca1334\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-mk69d"
Nov 01 03:26:57.076384 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:57.076309    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4988f434-dc04-41bd-8b3f-6e386bca1334-istio-data\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-mk69d\" (UID: \"4988f434-dc04-41bd-8b3f-6e386bca1334\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-mk69d"
Nov 01 03:26:57.076384 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:57.076340    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4988f434-dc04-41bd-8b3f-6e386bca1334-credential-socket\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-mk69d\" (UID: \"4988f434-dc04-41bd-8b3f-6e386bca1334\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-mk69d"
Nov 01 03:26:57.076496 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:57.076439    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4988f434-dc04-41bd-8b3f-6e386bca1334-workload-certs\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-mk69d\" (UID: \"4988f434-dc04-41bd-8b3f-6e386bca1334\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-mk69d"
Nov 01 03:26:57.076496 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:57.076485    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvrpv\" (UniqueName: \"kubernetes.io/projected/4988f434-dc04-41bd-8b3f-6e386bca1334-kube-api-access-lvrpv\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-mk69d\" (UID: \"4988f434-dc04-41bd-8b3f-6e386bca1334\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-mk69d"
Nov 01 03:26:57.076607 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:57.076519    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4988f434-dc04-41bd-8b3f-6e386bca1334-istiod-ca-cert\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-mk69d\" (UID: \"4988f434-dc04-41bd-8b3f-6e386bca1334\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-mk69d"
Nov 01 03:26:57.076607 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:57.076567    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4988f434-dc04-41bd-8b3f-6e386bca1334-workload-socket\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-mk69d\" (UID: \"4988f434-dc04-41bd-8b3f-6e386bca1334\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-mk69d"
Nov 01 03:26:57.076768 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:57.076737    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4988f434-dc04-41bd-8b3f-6e386bca1334-istio-data\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-mk69d\" (UID: \"4988f434-dc04-41bd-8b3f-6e386bca1334\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-mk69d"
Nov 01 03:26:57.076908 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:57.076862    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4988f434-dc04-41bd-8b3f-6e386bca1334-workload-certs\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-mk69d\" (UID: \"4988f434-dc04-41bd-8b3f-6e386bca1334\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-mk69d"
Nov 01 03:26:57.077191 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:57.077155    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4988f434-dc04-41bd-8b3f-6e386bca1334-istiod-ca-cert\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-mk69d\" (UID: \"4988f434-dc04-41bd-8b3f-6e386bca1334\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-mk69d"
Nov 01 03:26:57.078732 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:57.078713    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4988f434-dc04-41bd-8b3f-6e386bca1334-istio-envoy\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-mk69d\" (UID: \"4988f434-dc04-41bd-8b3f-6e386bca1334\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-mk69d"
Nov 01 03:26:57.078938 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:57.078921    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4988f434-dc04-41bd-8b3f-6e386bca1334-istio-podinfo\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-mk69d\" (UID: \"4988f434-dc04-41bd-8b3f-6e386bca1334\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-mk69d"
Nov 01 03:26:57.139784 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:57.139738    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4988f434-dc04-41bd-8b3f-6e386bca1334-istio-token\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-mk69d\" (UID: \"4988f434-dc04-41bd-8b3f-6e386bca1334\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-mk69d"
Nov 01 03:26:57.140621 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:57.140605    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvrpv\" (UniqueName: \"kubernetes.io/projected/4988f434-dc04-41bd-8b3f-6e386bca1334-kube-api-access-lvrpv\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-mk69d\" (UID: \"4988f434-dc04-41bd-8b3f-6e386bca1334\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-mk69d"
Nov 01 03:26:57.176174 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:57.176153    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-mk69d"
Nov 01 03:26:57.493047 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:57.492972    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-mp6pq"]
Nov 01 03:26:57.498580 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:57.498560    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-mp6pq"
Nov 01 03:26:57.553926 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:57.553902    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-e8c789-openshift-default-dockercfg-vpp4p\""
Nov 01 03:26:57.558290 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:57.558250    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-mp6pq"]
Nov 01 03:26:57.591002 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:57.590971    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd8db90d-adb2-4601-bd83-5f8dafbc02ce-istio-envoy\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-mp6pq\" (UID: \"bd8db90d-adb2-4601-bd83-5f8dafbc02ce\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-mp6pq"
Nov 01 03:26:57.591112 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:57.591022    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd8db90d-adb2-4601-bd83-5f8dafbc02ce-workload-certs\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-mp6pq\" (UID: \"bd8db90d-adb2-4601-bd83-5f8dafbc02ce\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-mp6pq"
Nov 01 03:26:57.591159 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:57.591127    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd8db90d-adb2-4601-bd83-5f8dafbc02ce-istio-data\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-mp6pq\" (UID: \"bd8db90d-adb2-4601-bd83-5f8dafbc02ce\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-mp6pq"
Nov 01 03:26:57.591268 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:57.591248    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd8db90d-adb2-4601-bd83-5f8dafbc02ce-workload-socket\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-mp6pq\" (UID: \"bd8db90d-adb2-4601-bd83-5f8dafbc02ce\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-mp6pq"
Nov 01 03:26:57.591305 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:57.591276    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd8db90d-adb2-4601-bd83-5f8dafbc02ce-istiod-ca-cert\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-mp6pq\" (UID: \"bd8db90d-adb2-4601-bd83-5f8dafbc02ce\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-mp6pq"
Nov 01 03:26:57.591374 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:57.591361    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd8db90d-adb2-4601-bd83-5f8dafbc02ce-istio-podinfo\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-mp6pq\" (UID: \"bd8db90d-adb2-4601-bd83-5f8dafbc02ce\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-mp6pq"
Nov 01 03:26:57.591405 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:57.591384    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd8db90d-adb2-4601-bd83-5f8dafbc02ce-istio-token\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-mp6pq\" (UID: \"bd8db90d-adb2-4601-bd83-5f8dafbc02ce\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-mp6pq"
Nov 01 03:26:57.591483 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:57.591472    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddxgr\" (UniqueName: \"kubernetes.io/projected/bd8db90d-adb2-4601-bd83-5f8dafbc02ce-kube-api-access-ddxgr\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-mp6pq\" (UID: \"bd8db90d-adb2-4601-bd83-5f8dafbc02ce\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-mp6pq"
Nov 01 03:26:57.591524 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:57.591496    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd8db90d-adb2-4601-bd83-5f8dafbc02ce-credential-socket\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-mp6pq\" (UID: \"bd8db90d-adb2-4601-bd83-5f8dafbc02ce\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-mp6pq"
Nov 01 03:26:57.657741 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:57.657719    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-mk69d"]
Nov 01 03:26:57.658352 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:26:57.658322    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4988f434_dc04_41bd_8b3f_6e386bca1334.slice/crio-32a1e9425bc4ad14558c06fcced5c07acec8b57c841ffa4801bf4053bfcd262e WatchSource:0}: Error finding container 32a1e9425bc4ad14558c06fcced5c07acec8b57c841ffa4801bf4053bfcd262e: Status 404 returned error can't find the container with id 32a1e9425bc4ad14558c06fcced5c07acec8b57c841ffa4801bf4053bfcd262e
Nov 01 03:26:57.660707 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:57.660665    4400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 03:26:57.660781 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:57.660739    4400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 03:26:57.660781 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:57.660768    4400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 03:26:57.692923 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:57.692892    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd8db90d-adb2-4601-bd83-5f8dafbc02ce-istio-envoy\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-mp6pq\" (UID: \"bd8db90d-adb2-4601-bd83-5f8dafbc02ce\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-mp6pq"
Nov 01 03:26:57.693056 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:57.693034    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd8db90d-adb2-4601-bd83-5f8dafbc02ce-workload-certs\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-mp6pq\" (UID: \"bd8db90d-adb2-4601-bd83-5f8dafbc02ce\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-mp6pq"
Nov 01 03:26:57.693133 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:57.693118    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd8db90d-adb2-4601-bd83-5f8dafbc02ce-istio-data\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-mp6pq\" (UID: \"bd8db90d-adb2-4601-bd83-5f8dafbc02ce\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-mp6pq"
Nov 01 03:26:57.693248 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:57.693224    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd8db90d-adb2-4601-bd83-5f8dafbc02ce-workload-socket\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-mp6pq\" (UID: \"bd8db90d-adb2-4601-bd83-5f8dafbc02ce\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-mp6pq"
Nov 01 03:26:57.693319 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:57.693267    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd8db90d-adb2-4601-bd83-5f8dafbc02ce-istiod-ca-cert\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-mp6pq\" (UID: \"bd8db90d-adb2-4601-bd83-5f8dafbc02ce\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-mp6pq"
Nov 01 03:26:57.693319 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:57.693306    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd8db90d-adb2-4601-bd83-5f8dafbc02ce-istio-podinfo\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-mp6pq\" (UID: \"bd8db90d-adb2-4601-bd83-5f8dafbc02ce\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-mp6pq"
Nov 01 03:26:57.693428 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:57.693329    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd8db90d-adb2-4601-bd83-5f8dafbc02ce-istio-token\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-mp6pq\" (UID: \"bd8db90d-adb2-4601-bd83-5f8dafbc02ce\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-mp6pq"
Nov 01 03:26:57.693428 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:57.693394    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddxgr\" (UniqueName: \"kubernetes.io/projected/bd8db90d-adb2-4601-bd83-5f8dafbc02ce-kube-api-access-ddxgr\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-mp6pq\" (UID: \"bd8db90d-adb2-4601-bd83-5f8dafbc02ce\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-mp6pq"
Nov 01 03:26:57.693532 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:57.693418    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd8db90d-adb2-4601-bd83-5f8dafbc02ce-workload-certs\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-mp6pq\" (UID: \"bd8db90d-adb2-4601-bd83-5f8dafbc02ce\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-mp6pq"
Nov 01 03:26:57.693532 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:57.693429    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd8db90d-adb2-4601-bd83-5f8dafbc02ce-credential-socket\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-mp6pq\" (UID: \"bd8db90d-adb2-4601-bd83-5f8dafbc02ce\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-mp6pq"
Nov 01 03:26:57.693532 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:57.693449    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd8db90d-adb2-4601-bd83-5f8dafbc02ce-istio-data\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-mp6pq\" (UID: \"bd8db90d-adb2-4601-bd83-5f8dafbc02ce\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-mp6pq"
Nov 01 03:26:57.693715 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:57.693523    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd8db90d-adb2-4601-bd83-5f8dafbc02ce-workload-socket\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-mp6pq\" (UID: \"bd8db90d-adb2-4601-bd83-5f8dafbc02ce\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-mp6pq"
Nov 01 03:26:57.693904 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:57.693880    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd8db90d-adb2-4601-bd83-5f8dafbc02ce-credential-socket\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-mp6pq\" (UID: \"bd8db90d-adb2-4601-bd83-5f8dafbc02ce\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-mp6pq"
Nov 01 03:26:57.693975 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:57.693944    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd8db90d-adb2-4601-bd83-5f8dafbc02ce-istiod-ca-cert\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-mp6pq\" (UID: \"bd8db90d-adb2-4601-bd83-5f8dafbc02ce\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-mp6pq"
Nov 01 03:26:57.695204 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:57.695185    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd8db90d-adb2-4601-bd83-5f8dafbc02ce-istio-envoy\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-mp6pq\" (UID: \"bd8db90d-adb2-4601-bd83-5f8dafbc02ce\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-mp6pq"
Nov 01 03:26:57.696618 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:57.696592    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd8db90d-adb2-4601-bd83-5f8dafbc02ce-istio-podinfo\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-mp6pq\" (UID: \"bd8db90d-adb2-4601-bd83-5f8dafbc02ce\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-mp6pq"
Nov 01 03:26:57.756220 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:57.756153    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddxgr\" (UniqueName: \"kubernetes.io/projected/bd8db90d-adb2-4601-bd83-5f8dafbc02ce-kube-api-access-ddxgr\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-mp6pq\" (UID: \"bd8db90d-adb2-4601-bd83-5f8dafbc02ce\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-mp6pq"
Nov 01 03:26:57.756220 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:57.756157    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd8db90d-adb2-4601-bd83-5f8dafbc02ce-istio-token\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-mp6pq\" (UID: \"bd8db90d-adb2-4601-bd83-5f8dafbc02ce\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-mp6pq"
Nov 01 03:26:57.814704 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:57.814657    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-mp6pq"
Nov 01 03:26:58.252406 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:58.252373    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-mp6pq"]
Nov 01 03:26:58.253438 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:26:58.253414    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd8db90d_adb2_4601_bd83_5f8dafbc02ce.slice/crio-7cf07daba7798e43432ef535f2de1cdda937afaff472c9862240318bcf4bcce3 WatchSource:0}: Error finding container 7cf07daba7798e43432ef535f2de1cdda937afaff472c9862240318bcf4bcce3: Status 404 returned error can't find the container with id 7cf07daba7798e43432ef535f2de1cdda937afaff472c9862240318bcf4bcce3
Nov 01 03:26:58.255956 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:58.255925    4400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 03:26:58.256050 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:58.255990    4400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 03:26:58.256050 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:58.256018    4400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 03:26:58.541858 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:58.541761    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-mp6pq" event={"ID":"bd8db90d-adb2-4601-bd83-5f8dafbc02ce","Type":"ContainerStarted","Data":"d46c3b33c85ec5ae8946575290fa3b1536711ce6d58cf5e3094b59917a754555"}
Nov 01 03:26:58.541858 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:58.541823    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-mp6pq" event={"ID":"bd8db90d-adb2-4601-bd83-5f8dafbc02ce","Type":"ContainerStarted","Data":"7cf07daba7798e43432ef535f2de1cdda937afaff472c9862240318bcf4bcce3"}
Nov 01 03:26:58.544167 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:58.544138    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-mk69d" event={"ID":"4988f434-dc04-41bd-8b3f-6e386bca1334","Type":"ContainerStarted","Data":"c8de3e02ecb3cc679bfa7ff7557f3c7baedf4eb48cdf77aaa4f24aca38b9302f"}
Nov 01 03:26:58.544297 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:58.544183    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-mk69d" event={"ID":"4988f434-dc04-41bd-8b3f-6e386bca1334","Type":"ContainerStarted","Data":"32a1e9425bc4ad14558c06fcced5c07acec8b57c841ffa4801bf4053bfcd262e"}
Nov 01 03:26:58.669904 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:58.669851    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-mp6pq" podStartSLOduration=1.6698372240000001 podStartE2EDuration="1.669837224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:26:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:26:58.665997698 +0000 UTC m=+1225.180643583" watchObservedRunningTime="2026-11-01 03:26:58.669837224 +0000 UTC m=+1225.184483086"
Nov 01 03:26:58.791975 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:58.791880    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-mk69d" podStartSLOduration=2.791860694 podStartE2EDuration="2.791860694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:26:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:26:58.790137821 +0000 UTC m=+1225.304783685" watchObservedRunningTime="2026-11-01 03:26:58.791860694 +0000 UTC m=+1225.306506607"
Nov 01 03:26:58.822710 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:58.814796    4400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-mp6pq"
Nov 01 03:26:59.176996 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:59.176965    4400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-mk69d"
Nov 01 03:26:59.178342 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:59.178320    4400 patch_prober.go:29] interesting pod/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-mk69d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Nov 01 03:26:59.178462 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:59.178366    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-mk69d" podUID="4988f434-dc04-41bd-8b3f-6e386bca1334" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Nov 01 03:26:59.823708 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:59.821968    4400 patch_prober.go:29] interesting pod/e2e-worker-w4-e8c789-openshift-default-774d997df-mp6pq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Nov 01 03:26:59.823708 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:26:59.822025    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-mp6pq" podUID="bd8db90d-adb2-4601-bd83-5f8dafbc02ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Nov 01 03:27:00.201805 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:00.182628    4400 patch_prober.go:29] interesting pod/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-mk69d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Nov 01 03:27:00.201805 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:00.182708    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-mk69d" podUID="4988f434-dc04-41bd-8b3f-6e386bca1334" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Nov 01 03:27:00.818710 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:00.815979    4400 patch_prober.go:29] interesting pod/e2e-worker-w4-e8c789-openshift-default-774d997df-mp6pq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Nov 01 03:27:00.818710 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:00.816036    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-mp6pq" podUID="bd8db90d-adb2-4601-bd83-5f8dafbc02ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Nov 01 03:27:00.854707 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:00.854196    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 03:27:01.177261 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:01.177223    4400 patch_prober.go:29] interesting pod/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-mk69d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Nov 01 03:27:01.177781 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:01.177748    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-mk69d" podUID="4988f434-dc04-41bd-8b3f-6e386bca1334" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Nov 01 03:27:01.816709 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:01.816109    4400 patch_prober.go:29] interesting pod/e2e-worker-w4-e8c789-openshift-default-774d997df-mp6pq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Nov 01 03:27:01.816709 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:01.816168    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-mp6pq" podUID="bd8db90d-adb2-4601-bd83-5f8dafbc02ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Nov 01 03:27:01.945801 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:27:01.944193    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:27:02.180799 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:02.180257    4400 patch_prober.go:29] interesting pod/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-mk69d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Nov 01 03:27:02.180799 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:02.180350    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-mk69d" podUID="4988f434-dc04-41bd-8b3f-6e386bca1334" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Nov 01 03:27:02.816136 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:02.816100    4400 patch_prober.go:29] interesting pod/e2e-worker-w4-e8c789-openshift-default-774d997df-mp6pq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Nov 01 03:27:02.816407 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:02.816372    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-mp6pq" podUID="bd8db90d-adb2-4601-bd83-5f8dafbc02ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Nov 01 03:27:02.992147 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:02.992101    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e8c789/premium-w4-e8c789-kserve-9458f8874-dtpnm" podUID="67bba91a-5ace-4d3f-89da-06bb9db17c78" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:03.071874 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:03.071836    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/distinct-2-w6-19faa5-kserve-58db847577-hrzlw" podUID="062959ee-ec7c-4385-98d5-cf4a240852e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Nov 01 03:27:03.177919 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:03.177884    4400 patch_prober.go:29] interesting pod/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-mk69d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Nov 01 03:27:03.178209 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:03.178182    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-mk69d" podUID="4988f434-dc04-41bd-8b3f-6e386bca1334" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Nov 01 03:27:03.193858 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:03.193819    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-w5-c22dd8-kserve-5c65c9bcd7-nnf7d" podUID="9cd440c6-7191-4fae-a806-f8728975b3cf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:03.328774 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:03.327820    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e8c789/distinct-2-w4-e8c789-kserve-7977c64f4-wkxb2" podUID="bdb1bb9e-14f5-4be6-a028-c980119f4aa8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:03.794763 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:03.790972    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-2-w2-c8cb9a-kserve-5c5cfb454d-fh2cj" podUID="2bcf3b85-3d4a-433f-94ad-0e954bf03e73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:03.815309 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:03.815267    4400 patch_prober.go:29] interesting pod/e2e-worker-w4-e8c789-openshift-default-774d997df-mp6pq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Nov 01 03:27:03.815604 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:03.815574    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-mp6pq" podUID="bd8db90d-adb2-4601-bd83-5f8dafbc02ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Nov 01 03:27:03.997846 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:03.997802    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e8c789/premium-w4-e8c789-kserve-9458f8874-dtpnm" podUID="67bba91a-5ace-4d3f-89da-06bb9db17c78" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:03.998903 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:03.998880    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e8c789/premium-w4-e8c789-kserve-9458f8874-dtpnm"
Nov 01 03:27:04.024911 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:04.024880    4400 patch_prober.go:29] interesting pod/e2e-worker-w1-b12199-openshift-default-774cb8c567-hnd5d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 03:27:04.025205 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:04.025158    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-hnd5d" podUID="a03a2ca3-9463-4838-8c2f-a37293da9e6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:04.074804 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:04.074763    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/distinct-2-w6-19faa5-kserve-58db847577-hrzlw" podUID="062959ee-ec7c-4385-98d5-cf4a240852e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:04.075409 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:04.075388    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-19faa5/distinct-2-w6-19faa5-kserve-58db847577-hrzlw"
Nov 01 03:27:04.187732 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:04.178536    4400 patch_prober.go:29] interesting pod/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-mk69d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Nov 01 03:27:04.187732 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:04.178590    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-mk69d" podUID="4988f434-dc04-41bd-8b3f-6e386bca1334" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Nov 01 03:27:04.194729 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:04.194674    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-w5-c22dd8-kserve-5c65c9bcd7-nnf7d" podUID="9cd440c6-7191-4fae-a806-f8728975b3cf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:04.195273 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:04.195252    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-w5-c22dd8-kserve-5c65c9bcd7-nnf7d"
Nov 01 03:27:04.340761 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:04.339830    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e8c789/distinct-2-w4-e8c789-kserve-7977c64f4-wkxb2" podUID="bdb1bb9e-14f5-4be6-a028-c980119f4aa8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:04.340761 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:04.340249    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e8c789/distinct-2-w4-e8c789-kserve-7977c64f4-wkxb2"
Nov 01 03:27:04.608769 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:04.604905    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7f535/distinct-w3-d7f535-kserve-7d69fb9558-88lq8" podUID="65358d08-83e6-45c5-a17e-f1dbde164345" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Nov 01 03:27:04.850710 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:04.847834    4400 patch_prober.go:29] interesting pod/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-2sbzb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 03:27:04.850710 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:04.847912    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-2sbzb" podUID="863e73b9-dddf-41c3-9a3d-eca8d7e2b727" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:04.882522 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:04.882437    4400 patch_prober.go:29] interesting pod/e2e-worker-w4-e8c789-openshift-default-774d997df-mp6pq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Nov 01 03:27:04.886811 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:04.886769    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-mp6pq" podUID="bd8db90d-adb2-4601-bd83-5f8dafbc02ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Nov 01 03:27:04.918714 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:04.916305    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-2-w2-c8cb9a-kserve-5c5cfb454d-fh2cj" podUID="2bcf3b85-3d4a-433f-94ad-0e954bf03e73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:04.918714 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:04.916735    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-2-w2-c8cb9a-kserve-5c5cfb454d-fh2cj"
Nov 01 03:27:05.007749 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:05.005329    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e8c789/premium-w4-e8c789-kserve-9458f8874-dtpnm" podUID="67bba91a-5ace-4d3f-89da-06bb9db17c78" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:05.084808 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:05.084767    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/distinct-2-w6-19faa5-kserve-58db847577-hrzlw" podUID="062959ee-ec7c-4385-98d5-cf4a240852e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Nov 01 03:27:05.164845 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:05.164761    4400 patch_prober.go:29] interesting pod/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lv9hr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 03:27:05.165082 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:05.165054    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lv9hr" podUID="ed1e2049-23c6-43e6-811c-fb0c5a9541b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:05.192705 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:05.192648    4400 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-e8c789/distinct-2-w4-e8c789-kserve-7977c64f4-wkxb2" podUID="bdb1bb9e-14f5-4be6-a028-c980119f4aa8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:05.315723 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:05.213784    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-w5-c22dd8-kserve-5c65c9bcd7-nnf7d" podUID="9cd440c6-7191-4fae-a806-f8728975b3cf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:05.315723 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:05.242316    4400 patch_prober.go:29] interesting pod/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-mk69d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Nov 01 03:27:05.315723 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:05.242359    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-mk69d" podUID="4988f434-dc04-41bd-8b3f-6e386bca1334" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Nov 01 03:27:05.315723 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:05.261137    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-2-w2-c8cb9a-kserve-5c5cfb454d-fh2cj"
Nov 01 03:27:05.341335 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:05.341280    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e8c789/distinct-2-w4-e8c789-kserve-7977c64f4-wkxb2" podUID="bdb1bb9e-14f5-4be6-a028-c980119f4aa8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:05.359269 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:05.359242    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e8c789/distinct-2-w4-e8c789-kserve-7977c64f4-wkxb2"
Nov 01 03:27:05.360489 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:05.360467    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-19faa5/distinct-2-w6-19faa5-kserve-58db847577-hrzlw"
Nov 01 03:27:05.362615 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:05.362512    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-w5-c22dd8-kserve-5c65c9bcd7-nnf7d"
Nov 01 03:27:05.369143 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:05.369122    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e8c789/premium-w4-e8c789-kserve-9458f8874-dtpnm"
Nov 01 03:27:05.821815 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:05.821773    4400 patch_prober.go:29] interesting pod/e2e-worker-w4-e8c789-openshift-default-774d997df-mp6pq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Nov 01 03:27:05.822695 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:05.822647    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-mp6pq" podUID="bd8db90d-adb2-4601-bd83-5f8dafbc02ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Nov 01 03:27:06.177067 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:06.176978    4400 patch_prober.go:29] interesting pod/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-mk69d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Nov 01 03:27:06.177067 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:06.177035    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-mk69d" podUID="4988f434-dc04-41bd-8b3f-6e386bca1334" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Nov 01 03:27:06.815003 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:06.814968    4400 patch_prober.go:29] interesting pod/e2e-worker-w4-e8c789-openshift-default-774d997df-mp6pq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Nov 01 03:27:06.815175 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:06.815035    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-mp6pq" podUID="bd8db90d-adb2-4601-bd83-5f8dafbc02ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Nov 01 03:27:07.182113 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:07.182082    4400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-mk69d"
Nov 01 03:27:07.182550 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:07.182496    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-mk69d"
Nov 01 03:27:07.183367 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:07.183346    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-mk69d"
Nov 01 03:27:07.819127 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:07.819094    4400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-mp6pq"
Nov 01 03:27:07.819550 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:07.819527    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-mp6pq"
Nov 01 03:27:07.820153 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:07.820133    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-mp6pq"
Nov 01 03:27:25.844179 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:25.844147    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-sfxhr"]
Nov 01 03:27:25.848599 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:25.848577    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-sfxhr"
Nov 01 03:27:25.961425 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:25.961391    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-sfxhr"]
Nov 01 03:27:25.971422 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:25.971388    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86018717-3f70-43de-843c-f9429c605a74-istio-envoy\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-sfxhr\" (UID: \"86018717-3f70-43de-843c-f9429c605a74\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-sfxhr"
Nov 01 03:27:25.971605 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:25.971501    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86018717-3f70-43de-843c-f9429c605a74-istiod-ca-cert\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-sfxhr\" (UID: \"86018717-3f70-43de-843c-f9429c605a74\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-sfxhr"
Nov 01 03:27:25.971605 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:25.971582    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86018717-3f70-43de-843c-f9429c605a74-workload-certs\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-sfxhr\" (UID: \"86018717-3f70-43de-843c-f9429c605a74\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-sfxhr"
Nov 01 03:27:25.971810 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:25.971663    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86018717-3f70-43de-843c-f9429c605a74-workload-socket\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-sfxhr\" (UID: \"86018717-3f70-43de-843c-f9429c605a74\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-sfxhr"
Nov 01 03:27:25.971810 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:25.971735    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86018717-3f70-43de-843c-f9429c605a74-istio-data\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-sfxhr\" (UID: \"86018717-3f70-43de-843c-f9429c605a74\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-sfxhr"
Nov 01 03:27:25.971810 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:25.971760    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86018717-3f70-43de-843c-f9429c605a74-istio-token\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-sfxhr\" (UID: \"86018717-3f70-43de-843c-f9429c605a74\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-sfxhr"
Nov 01 03:27:25.971810 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:25.971783    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dx52\" (UniqueName: \"kubernetes.io/projected/86018717-3f70-43de-843c-f9429c605a74-kube-api-access-6dx52\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-sfxhr\" (UID: \"86018717-3f70-43de-843c-f9429c605a74\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-sfxhr"
Nov 01 03:27:25.972028 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:25.971867    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86018717-3f70-43de-843c-f9429c605a74-credential-socket\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-sfxhr\" (UID: \"86018717-3f70-43de-843c-f9429c605a74\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-sfxhr"
Nov 01 03:27:25.972028 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:25.971906    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86018717-3f70-43de-843c-f9429c605a74-istio-podinfo\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-sfxhr\" (UID: \"86018717-3f70-43de-843c-f9429c605a74\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-sfxhr"
Nov 01 03:27:26.073019 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:26.072988    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86018717-3f70-43de-843c-f9429c605a74-istiod-ca-cert\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-sfxhr\" (UID: \"86018717-3f70-43de-843c-f9429c605a74\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-sfxhr"
Nov 01 03:27:26.073157 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:26.073051    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86018717-3f70-43de-843c-f9429c605a74-workload-certs\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-sfxhr\" (UID: \"86018717-3f70-43de-843c-f9429c605a74\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-sfxhr"
Nov 01 03:27:26.073157 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:26.073098    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86018717-3f70-43de-843c-f9429c605a74-workload-socket\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-sfxhr\" (UID: \"86018717-3f70-43de-843c-f9429c605a74\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-sfxhr"
Nov 01 03:27:26.073157 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:26.073128    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86018717-3f70-43de-843c-f9429c605a74-istio-data\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-sfxhr\" (UID: \"86018717-3f70-43de-843c-f9429c605a74\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-sfxhr"
Nov 01 03:27:26.073157 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:26.073151    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86018717-3f70-43de-843c-f9429c605a74-istio-token\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-sfxhr\" (UID: \"86018717-3f70-43de-843c-f9429c605a74\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-sfxhr"
Nov 01 03:27:26.073356 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:26.073174    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dx52\" (UniqueName: \"kubernetes.io/projected/86018717-3f70-43de-843c-f9429c605a74-kube-api-access-6dx52\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-sfxhr\" (UID: \"86018717-3f70-43de-843c-f9429c605a74\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-sfxhr"
Nov 01 03:27:26.073356 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:26.073237    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86018717-3f70-43de-843c-f9429c605a74-credential-socket\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-sfxhr\" (UID: \"86018717-3f70-43de-843c-f9429c605a74\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-sfxhr"
Nov 01 03:27:26.073356 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:26.073274    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86018717-3f70-43de-843c-f9429c605a74-istio-podinfo\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-sfxhr\" (UID: \"86018717-3f70-43de-843c-f9429c605a74\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-sfxhr"
Nov 01 03:27:26.073356 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:26.073335    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86018717-3f70-43de-843c-f9429c605a74-istio-envoy\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-sfxhr\" (UID: \"86018717-3f70-43de-843c-f9429c605a74\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-sfxhr"
Nov 01 03:27:26.073555 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:26.073535    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86018717-3f70-43de-843c-f9429c605a74-workload-certs\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-sfxhr\" (UID: \"86018717-3f70-43de-843c-f9429c605a74\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-sfxhr"
Nov 01 03:27:26.073615 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:26.073596    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86018717-3f70-43de-843c-f9429c605a74-istio-data\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-sfxhr\" (UID: \"86018717-3f70-43de-843c-f9429c605a74\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-sfxhr"
Nov 01 03:27:26.073815 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:26.073789    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86018717-3f70-43de-843c-f9429c605a74-workload-socket\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-sfxhr\" (UID: \"86018717-3f70-43de-843c-f9429c605a74\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-sfxhr"
Nov 01 03:27:26.073883 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:26.073826    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86018717-3f70-43de-843c-f9429c605a74-credential-socket\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-sfxhr\" (UID: \"86018717-3f70-43de-843c-f9429c605a74\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-sfxhr"
Nov 01 03:27:26.073883 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:26.073839    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86018717-3f70-43de-843c-f9429c605a74-istiod-ca-cert\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-sfxhr\" (UID: \"86018717-3f70-43de-843c-f9429c605a74\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-sfxhr"
Nov 01 03:27:26.076073 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:26.076046    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86018717-3f70-43de-843c-f9429c605a74-istio-envoy\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-sfxhr\" (UID: \"86018717-3f70-43de-843c-f9429c605a74\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-sfxhr"
Nov 01 03:27:26.076296 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:26.076277    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86018717-3f70-43de-843c-f9429c605a74-istio-podinfo\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-sfxhr\" (UID: \"86018717-3f70-43de-843c-f9429c605a74\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-sfxhr"
Nov 01 03:27:26.136085 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:26.136031    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dx52\" (UniqueName: \"kubernetes.io/projected/86018717-3f70-43de-843c-f9429c605a74-kube-api-access-6dx52\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-sfxhr\" (UID: \"86018717-3f70-43de-843c-f9429c605a74\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-sfxhr"
Nov 01 03:27:26.136191 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:26.136172    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86018717-3f70-43de-843c-f9429c605a74-istio-token\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-sfxhr\" (UID: \"86018717-3f70-43de-843c-f9429c605a74\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-sfxhr"
Nov 01 03:27:26.164551 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:26.164531    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-sfxhr"
Nov 01 03:27:26.617983 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:26.617943    4400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 03:27:26.618107 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:26.618021    4400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 03:27:26.618107 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:26.618061    4400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 03:27:26.618203 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:26.618100    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-sfxhr"]
Nov 01 03:27:27.157528 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:27.157494    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-sfxhr" event={"ID":"86018717-3f70-43de-843c-f9429c605a74","Type":"ContainerStarted","Data":"00d3725e28cb6802aa49e20c719b9cfdbf8f6f40d73845192eec92cb97b5cc91"}
Nov 01 03:27:27.157959 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:27.157560    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-sfxhr" event={"ID":"86018717-3f70-43de-843c-f9429c605a74","Type":"ContainerStarted","Data":"52fb57724b24a417530f0dee01df005e5e8852622721bdd8e9e7f5290fff42b1"}
Nov 01 03:27:27.165327 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:27.165305    4400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-sfxhr"
Nov 01 03:27:27.166907 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:27.166888    4400 patch_prober.go:29] interesting pod/e2e-worker-w1-b12199-openshift-default-774cb8c567-sfxhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Nov 01 03:27:27.167003 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:27.166935    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-sfxhr" podUID="86018717-3f70-43de-843c-f9429c605a74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Nov 01 03:27:27.297787 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:27.297742    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-sfxhr" podStartSLOduration=2.297727916 podStartE2EDuration="2.297727916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:27:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:27:27.293044518 +0000 UTC m=+1253.807690381" watchObservedRunningTime="2026-11-01 03:27:27.297727916 +0000 UTC m=+1253.812373778"
Nov 01 03:27:28.164912 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:28.164882    4400 patch_prober.go:29] interesting pod/e2e-worker-w1-b12199-openshift-default-774cb8c567-sfxhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Nov 01 03:27:28.165277 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:28.164935    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-sfxhr" podUID="86018717-3f70-43de-843c-f9429c605a74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Nov 01 03:27:29.165603 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:29.165564    4400 patch_prober.go:29] interesting pod/e2e-worker-w1-b12199-openshift-default-774cb8c567-sfxhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Nov 01 03:27:29.166099 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:29.165627    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-sfxhr" podUID="86018717-3f70-43de-843c-f9429c605a74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Nov 01 03:27:30.165764 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:30.165729    4400 patch_prober.go:29] interesting pod/e2e-worker-w1-b12199-openshift-default-774cb8c567-sfxhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Nov 01 03:27:30.166134 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:30.165784    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-sfxhr" podUID="86018717-3f70-43de-843c-f9429c605a74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Nov 01 03:27:31.165021 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:31.164984    4400 patch_prober.go:29] interesting pod/e2e-worker-w1-b12199-openshift-default-774cb8c567-sfxhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Nov 01 03:27:31.165183 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:31.165038    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-sfxhr" podUID="86018717-3f70-43de-843c-f9429c605a74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Nov 01 03:27:31.753141 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:27:31.753105    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:27:32.165341 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:32.165307    4400 patch_prober.go:29] interesting pod/e2e-worker-w1-b12199-openshift-default-774cb8c567-sfxhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Nov 01 03:27:32.165505 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:32.165369    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-sfxhr" podUID="86018717-3f70-43de-843c-f9429c605a74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Nov 01 03:27:33.171513 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:33.171476    4400 patch_prober.go:29] interesting pod/e2e-worker-w1-b12199-openshift-default-774cb8c567-sfxhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Nov 01 03:27:33.172122 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:33.172087    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-sfxhr" podUID="86018717-3f70-43de-843c-f9429c605a74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Nov 01 03:27:34.237475 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:34.237445    4400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-sfxhr"
Nov 01 03:27:34.237968 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:34.237772    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-sfxhr"
Nov 01 03:27:34.238819 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:34.238797    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-sfxhr"
Nov 01 03:27:41.939416 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:41.939387    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jq6dn"]
Nov 01 03:27:41.945165 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:41.945141    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jq6dn"
Nov 01 03:27:41.987596 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:41.987559    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-9qpm8"]
Nov 01 03:27:41.991871 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:41.991852    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-9qpm8"
Nov 01 03:27:42.046524 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.046495    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jq6dn"]
Nov 01 03:27:42.109578 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.109545    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-9qpm8"]
Nov 01 03:27:42.115643 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.115610    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/842a8e86-2464-4f2e-88ac-6be1b68b0d7e-istiod-ca-cert\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-jq6dn\" (UID: \"842a8e86-2464-4f2e-88ac-6be1b68b0d7e\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jq6dn"
Nov 01 03:27:42.115845 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.115672    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqlcx\" (UniqueName: \"kubernetes.io/projected/842a8e86-2464-4f2e-88ac-6be1b68b0d7e-kube-api-access-vqlcx\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-jq6dn\" (UID: \"842a8e86-2464-4f2e-88ac-6be1b68b0d7e\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jq6dn"
Nov 01 03:27:42.115845 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.115814    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/204735ae-bff5-4cdc-9df5-f85158677fdc-workload-socket\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-9qpm8\" (UID: \"204735ae-bff5-4cdc-9df5-f85158677fdc\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-9qpm8"
Nov 01 03:27:42.115975 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.115849    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/204735ae-bff5-4cdc-9df5-f85158677fdc-istio-token\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-9qpm8\" (UID: \"204735ae-bff5-4cdc-9df5-f85158677fdc\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-9qpm8"
Nov 01 03:27:42.115975 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.115876    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/204735ae-bff5-4cdc-9df5-f85158677fdc-istiod-ca-cert\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-9qpm8\" (UID: \"204735ae-bff5-4cdc-9df5-f85158677fdc\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-9qpm8"
Nov 01 03:27:42.115975 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.115917    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/842a8e86-2464-4f2e-88ac-6be1b68b0d7e-istio-podinfo\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-jq6dn\" (UID: \"842a8e86-2464-4f2e-88ac-6be1b68b0d7e\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jq6dn"
Nov 01 03:27:42.115975 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.115947    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/204735ae-bff5-4cdc-9df5-f85158677fdc-istio-podinfo\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-9qpm8\" (UID: \"204735ae-bff5-4cdc-9df5-f85158677fdc\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-9qpm8"
Nov 01 03:27:42.116228 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.115988    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb5xw\" (UniqueName: \"kubernetes.io/projected/204735ae-bff5-4cdc-9df5-f85158677fdc-kube-api-access-wb5xw\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-9qpm8\" (UID: \"204735ae-bff5-4cdc-9df5-f85158677fdc\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-9qpm8"
Nov 01 03:27:42.116228 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.116029    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/204735ae-bff5-4cdc-9df5-f85158677fdc-istio-data\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-9qpm8\" (UID: \"204735ae-bff5-4cdc-9df5-f85158677fdc\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-9qpm8"
Nov 01 03:27:42.116228 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.116080    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/842a8e86-2464-4f2e-88ac-6be1b68b0d7e-workload-certs\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-jq6dn\" (UID: \"842a8e86-2464-4f2e-88ac-6be1b68b0d7e\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jq6dn"
Nov 01 03:27:42.116228 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.116106    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/204735ae-bff5-4cdc-9df5-f85158677fdc-credential-socket\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-9qpm8\" (UID: \"204735ae-bff5-4cdc-9df5-f85158677fdc\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-9qpm8"
Nov 01 03:27:42.116228 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.116175    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/842a8e86-2464-4f2e-88ac-6be1b68b0d7e-istio-data\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-jq6dn\" (UID: \"842a8e86-2464-4f2e-88ac-6be1b68b0d7e\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jq6dn"
Nov 01 03:27:42.116228 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.116212    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/204735ae-bff5-4cdc-9df5-f85158677fdc-workload-certs\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-9qpm8\" (UID: \"204735ae-bff5-4cdc-9df5-f85158677fdc\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-9qpm8"
Nov 01 03:27:42.116539 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.116320    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/842a8e86-2464-4f2e-88ac-6be1b68b0d7e-workload-socket\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-jq6dn\" (UID: \"842a8e86-2464-4f2e-88ac-6be1b68b0d7e\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jq6dn"
Nov 01 03:27:42.116539 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.116344    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/842a8e86-2464-4f2e-88ac-6be1b68b0d7e-istio-envoy\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-jq6dn\" (UID: \"842a8e86-2464-4f2e-88ac-6be1b68b0d7e\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jq6dn"
Nov 01 03:27:42.116539 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.116374    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/204735ae-bff5-4cdc-9df5-f85158677fdc-istio-envoy\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-9qpm8\" (UID: \"204735ae-bff5-4cdc-9df5-f85158677fdc\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-9qpm8"
Nov 01 03:27:42.116539 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.116411    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/842a8e86-2464-4f2e-88ac-6be1b68b0d7e-istio-token\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-jq6dn\" (UID: \"842a8e86-2464-4f2e-88ac-6be1b68b0d7e\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jq6dn"
Nov 01 03:27:42.116539 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.116503    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/842a8e86-2464-4f2e-88ac-6be1b68b0d7e-credential-socket\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-jq6dn\" (UID: \"842a8e86-2464-4f2e-88ac-6be1b68b0d7e\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jq6dn"
Nov 01 03:27:42.218201 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.218109    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/842a8e86-2464-4f2e-88ac-6be1b68b0d7e-credential-socket\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-jq6dn\" (UID: \"842a8e86-2464-4f2e-88ac-6be1b68b0d7e\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jq6dn"
Nov 01 03:27:42.218201 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.218187    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/842a8e86-2464-4f2e-88ac-6be1b68b0d7e-istiod-ca-cert\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-jq6dn\" (UID: \"842a8e86-2464-4f2e-88ac-6be1b68b0d7e\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jq6dn"
Nov 01 03:27:42.218430 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.218213    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqlcx\" (UniqueName: \"kubernetes.io/projected/842a8e86-2464-4f2e-88ac-6be1b68b0d7e-kube-api-access-vqlcx\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-jq6dn\" (UID: \"842a8e86-2464-4f2e-88ac-6be1b68b0d7e\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jq6dn"
Nov 01 03:27:42.218430 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.218263    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/204735ae-bff5-4cdc-9df5-f85158677fdc-workload-socket\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-9qpm8\" (UID: \"204735ae-bff5-4cdc-9df5-f85158677fdc\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-9qpm8"
Nov 01 03:27:42.218430 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.218291    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/204735ae-bff5-4cdc-9df5-f85158677fdc-istio-token\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-9qpm8\" (UID: \"204735ae-bff5-4cdc-9df5-f85158677fdc\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-9qpm8"
Nov 01 03:27:42.218430 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.218315    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/204735ae-bff5-4cdc-9df5-f85158677fdc-istiod-ca-cert\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-9qpm8\" (UID: \"204735ae-bff5-4cdc-9df5-f85158677fdc\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-9qpm8"
Nov 01 03:27:42.218430 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.218348    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/842a8e86-2464-4f2e-88ac-6be1b68b0d7e-istio-podinfo\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-jq6dn\" (UID: \"842a8e86-2464-4f2e-88ac-6be1b68b0d7e\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jq6dn"
Nov 01 03:27:42.218430 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.218379    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/204735ae-bff5-4cdc-9df5-f85158677fdc-istio-podinfo\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-9qpm8\" (UID: \"204735ae-bff5-4cdc-9df5-f85158677fdc\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-9qpm8"
Nov 01 03:27:42.218430 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.218417    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb5xw\" (UniqueName: \"kubernetes.io/projected/204735ae-bff5-4cdc-9df5-f85158677fdc-kube-api-access-wb5xw\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-9qpm8\" (UID: \"204735ae-bff5-4cdc-9df5-f85158677fdc\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-9qpm8"
Nov 01 03:27:42.218838 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.218450    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/204735ae-bff5-4cdc-9df5-f85158677fdc-istio-data\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-9qpm8\" (UID: \"204735ae-bff5-4cdc-9df5-f85158677fdc\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-9qpm8"
Nov 01 03:27:42.218838 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.218494    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/842a8e86-2464-4f2e-88ac-6be1b68b0d7e-workload-certs\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-jq6dn\" (UID: \"842a8e86-2464-4f2e-88ac-6be1b68b0d7e\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jq6dn"
Nov 01 03:27:42.218838 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.218516    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/204735ae-bff5-4cdc-9df5-f85158677fdc-credential-socket\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-9qpm8\" (UID: \"204735ae-bff5-4cdc-9df5-f85158677fdc\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-9qpm8"
Nov 01 03:27:42.218838 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.218570    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/842a8e86-2464-4f2e-88ac-6be1b68b0d7e-credential-socket\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-jq6dn\" (UID: \"842a8e86-2464-4f2e-88ac-6be1b68b0d7e\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jq6dn"
Nov 01 03:27:42.218838 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.218603    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/842a8e86-2464-4f2e-88ac-6be1b68b0d7e-istio-data\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-jq6dn\" (UID: \"842a8e86-2464-4f2e-88ac-6be1b68b0d7e\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jq6dn"
Nov 01 03:27:42.218838 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.218636    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/204735ae-bff5-4cdc-9df5-f85158677fdc-workload-certs\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-9qpm8\" (UID: \"204735ae-bff5-4cdc-9df5-f85158677fdc\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-9qpm8"
Nov 01 03:27:42.218838 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.218752    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/842a8e86-2464-4f2e-88ac-6be1b68b0d7e-workload-socket\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-jq6dn\" (UID: \"842a8e86-2464-4f2e-88ac-6be1b68b0d7e\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jq6dn"
Nov 01 03:27:42.218838 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.218775    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/842a8e86-2464-4f2e-88ac-6be1b68b0d7e-istio-envoy\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-jq6dn\" (UID: \"842a8e86-2464-4f2e-88ac-6be1b68b0d7e\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jq6dn"
Nov 01 03:27:42.218838 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.218811    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/204735ae-bff5-4cdc-9df5-f85158677fdc-workload-socket\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-9qpm8\" (UID: \"204735ae-bff5-4cdc-9df5-f85158677fdc\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-9qpm8"
Nov 01 03:27:42.218838 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.218806    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/204735ae-bff5-4cdc-9df5-f85158677fdc-istio-envoy\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-9qpm8\" (UID: \"204735ae-bff5-4cdc-9df5-f85158677fdc\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-9qpm8"
Nov 01 03:27:42.219397 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.218885    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/842a8e86-2464-4f2e-88ac-6be1b68b0d7e-istio-token\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-jq6dn\" (UID: \"842a8e86-2464-4f2e-88ac-6be1b68b0d7e\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jq6dn"
Nov 01 03:27:42.219397 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.219068    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/842a8e86-2464-4f2e-88ac-6be1b68b0d7e-istiod-ca-cert\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-jq6dn\" (UID: \"842a8e86-2464-4f2e-88ac-6be1b68b0d7e\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jq6dn"
Nov 01 03:27:42.219397 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.219152    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/842a8e86-2464-4f2e-88ac-6be1b68b0d7e-workload-socket\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-jq6dn\" (UID: \"842a8e86-2464-4f2e-88ac-6be1b68b0d7e\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jq6dn"
Nov 01 03:27:42.219397 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.219232    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/842a8e86-2464-4f2e-88ac-6be1b68b0d7e-workload-certs\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-jq6dn\" (UID: \"842a8e86-2464-4f2e-88ac-6be1b68b0d7e\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jq6dn"
Nov 01 03:27:42.219397 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.219302    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/204735ae-bff5-4cdc-9df5-f85158677fdc-istiod-ca-cert\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-9qpm8\" (UID: \"204735ae-bff5-4cdc-9df5-f85158677fdc\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-9qpm8"
Nov 01 03:27:42.219629 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.219463    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/204735ae-bff5-4cdc-9df5-f85158677fdc-istio-data\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-9qpm8\" (UID: \"204735ae-bff5-4cdc-9df5-f85158677fdc\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-9qpm8"
Nov 01 03:27:42.219629 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.219524    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/204735ae-bff5-4cdc-9df5-f85158677fdc-workload-certs\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-9qpm8\" (UID: \"204735ae-bff5-4cdc-9df5-f85158677fdc\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-9qpm8"
Nov 01 03:27:42.219744 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.219668    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/204735ae-bff5-4cdc-9df5-f85158677fdc-credential-socket\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-9qpm8\" (UID: \"204735ae-bff5-4cdc-9df5-f85158677fdc\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-9qpm8"
Nov 01 03:27:42.219887 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.219867    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/842a8e86-2464-4f2e-88ac-6be1b68b0d7e-istio-data\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-jq6dn\" (UID: \"842a8e86-2464-4f2e-88ac-6be1b68b0d7e\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jq6dn"
Nov 01 03:27:42.222075 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.222049    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/842a8e86-2464-4f2e-88ac-6be1b68b0d7e-istio-envoy\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-jq6dn\" (UID: \"842a8e86-2464-4f2e-88ac-6be1b68b0d7e\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jq6dn"
Nov 01 03:27:42.222173 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.222058    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/204735ae-bff5-4cdc-9df5-f85158677fdc-istio-envoy\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-9qpm8\" (UID: \"204735ae-bff5-4cdc-9df5-f85158677fdc\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-9qpm8"
Nov 01 03:27:42.222296 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.222268    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/842a8e86-2464-4f2e-88ac-6be1b68b0d7e-istio-podinfo\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-jq6dn\" (UID: \"842a8e86-2464-4f2e-88ac-6be1b68b0d7e\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jq6dn"
Nov 01 03:27:42.222571 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.222554    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/204735ae-bff5-4cdc-9df5-f85158677fdc-istio-podinfo\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-9qpm8\" (UID: \"204735ae-bff5-4cdc-9df5-f85158677fdc\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-9qpm8"
Nov 01 03:27:42.281746 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.281673    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/204735ae-bff5-4cdc-9df5-f85158677fdc-istio-token\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-9qpm8\" (UID: \"204735ae-bff5-4cdc-9df5-f85158677fdc\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-9qpm8"
Nov 01 03:27:42.281899 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.281766    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqlcx\" (UniqueName: \"kubernetes.io/projected/842a8e86-2464-4f2e-88ac-6be1b68b0d7e-kube-api-access-vqlcx\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-jq6dn\" (UID: \"842a8e86-2464-4f2e-88ac-6be1b68b0d7e\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jq6dn"
Nov 01 03:27:42.281899 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.281874    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb5xw\" (UniqueName: \"kubernetes.io/projected/204735ae-bff5-4cdc-9df5-f85158677fdc-kube-api-access-wb5xw\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-9qpm8\" (UID: \"204735ae-bff5-4cdc-9df5-f85158677fdc\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-9qpm8"
Nov 01 03:27:42.282048 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.282032    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/842a8e86-2464-4f2e-88ac-6be1b68b0d7e-istio-token\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-jq6dn\" (UID: \"842a8e86-2464-4f2e-88ac-6be1b68b0d7e\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jq6dn"
Nov 01 03:27:42.305157 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.305130    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-9qpm8"
Nov 01 03:27:42.559993 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.559901    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jq6dn"
Nov 01 03:27:42.676138 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.676103    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-9xqhq"]
Nov 01 03:27:42.682339 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.682313    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-9xqhq"
Nov 01 03:27:42.722341 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.722311    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-l4xqq"]
Nov 01 03:27:42.726494 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.726471    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-l4xqq"
Nov 01 03:27:42.727117 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.727096    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c782468-958d-496b-afec-73b546553a57-istiod-ca-cert\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-9xqhq\" (UID: \"6c782468-958d-496b-afec-73b546553a57\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-9xqhq"
Nov 01 03:27:42.727195 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.727130    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c782468-958d-496b-afec-73b546553a57-workload-certs\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-9xqhq\" (UID: \"6c782468-958d-496b-afec-73b546553a57\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-9xqhq"
Nov 01 03:27:42.727195 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.727168    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c782468-958d-496b-afec-73b546553a57-workload-socket\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-9xqhq\" (UID: \"6c782468-958d-496b-afec-73b546553a57\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-9xqhq"
Nov 01 03:27:42.727285 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.727270    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c782468-958d-496b-afec-73b546553a57-istio-token\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-9xqhq\" (UID: \"6c782468-958d-496b-afec-73b546553a57\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-9xqhq"
Nov 01 03:27:42.727344 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.727300    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c782468-958d-496b-afec-73b546553a57-istio-envoy\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-9xqhq\" (UID: \"6c782468-958d-496b-afec-73b546553a57\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-9xqhq"
Nov 01 03:27:42.727344 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.727326    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cz2ff\" (UniqueName: \"kubernetes.io/projected/6c782468-958d-496b-afec-73b546553a57-kube-api-access-cz2ff\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-9xqhq\" (UID: \"6c782468-958d-496b-afec-73b546553a57\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-9xqhq"
Nov 01 03:27:42.727458 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.727437    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c782468-958d-496b-afec-73b546553a57-istio-data\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-9xqhq\" (UID: \"6c782468-958d-496b-afec-73b546553a57\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-9xqhq"
Nov 01 03:27:42.727519 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.727507    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c782468-958d-496b-afec-73b546553a57-istio-podinfo\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-9xqhq\" (UID: \"6c782468-958d-496b-afec-73b546553a57\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-9xqhq"
Nov 01 03:27:42.727551 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.727534    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c782468-958d-496b-afec-73b546553a57-credential-socket\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-9xqhq\" (UID: \"6c782468-958d-496b-afec-73b546553a57\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-9xqhq"
Nov 01 03:27:42.740336 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.740310    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-9xqhq"]
Nov 01 03:27:42.747820 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:27:42.747796    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod204735ae_bff5_4cdc_9df5_f85158677fdc.slice/crio-dcbcb0cce3820a1b009665ce9e8b9681673c0848579a96f4a17f5531bd27ee65 WatchSource:0}: Error finding container dcbcb0cce3820a1b009665ce9e8b9681673c0848579a96f4a17f5531bd27ee65: Status 404 returned error can't find the container with id dcbcb0cce3820a1b009665ce9e8b9681673c0848579a96f4a17f5531bd27ee65
Nov 01 03:27:42.751049 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.751017    4400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 03:27:42.751152 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.751094    4400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 03:27:42.751152 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.751139    4400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 03:27:42.775242 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.775203    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-9qpm8"]
Nov 01 03:27:42.790180 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.790155    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-l4xqq"]
Nov 01 03:27:42.829643 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.829615    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73ae6733-594c-47d9-89e9-8b92fde0b8f2-credential-socket\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-l4xqq\" (UID: \"73ae6733-594c-47d9-89e9-8b92fde0b8f2\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-l4xqq"
Nov 01 03:27:42.829740 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.829663    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73ae6733-594c-47d9-89e9-8b92fde0b8f2-workload-socket\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-l4xqq\" (UID: \"73ae6733-594c-47d9-89e9-8b92fde0b8f2\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-l4xqq"
Nov 01 03:27:42.829797 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.829768    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73ae6733-594c-47d9-89e9-8b92fde0b8f2-istio-data\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-l4xqq\" (UID: \"73ae6733-594c-47d9-89e9-8b92fde0b8f2\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-l4xqq"
Nov 01 03:27:42.829860 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.829852    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73ae6733-594c-47d9-89e9-8b92fde0b8f2-istio-token\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-l4xqq\" (UID: \"73ae6733-594c-47d9-89e9-8b92fde0b8f2\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-l4xqq"
Nov 01 03:27:42.829968 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.829947    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c782468-958d-496b-afec-73b546553a57-istiod-ca-cert\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-9xqhq\" (UID: \"6c782468-958d-496b-afec-73b546553a57\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-9xqhq"
Nov 01 03:27:42.830030 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.829999    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c782468-958d-496b-afec-73b546553a57-workload-certs\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-9xqhq\" (UID: \"6c782468-958d-496b-afec-73b546553a57\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-9xqhq"
Nov 01 03:27:42.830086 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.830047    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73ae6733-594c-47d9-89e9-8b92fde0b8f2-istiod-ca-cert\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-l4xqq\" (UID: \"73ae6733-594c-47d9-89e9-8b92fde0b8f2\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-l4xqq"
Nov 01 03:27:42.830134 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.830083    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c782468-958d-496b-afec-73b546553a57-workload-socket\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-9xqhq\" (UID: \"6c782468-958d-496b-afec-73b546553a57\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-9xqhq"
Nov 01 03:27:42.830174 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.830159    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c782468-958d-496b-afec-73b546553a57-istio-token\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-9xqhq\" (UID: \"6c782468-958d-496b-afec-73b546553a57\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-9xqhq"
Nov 01 03:27:42.830234 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.830218    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c782468-958d-496b-afec-73b546553a57-istio-envoy\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-9xqhq\" (UID: \"6c782468-958d-496b-afec-73b546553a57\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-9xqhq"
Nov 01 03:27:42.830282 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.830262    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cz2ff\" (UniqueName: \"kubernetes.io/projected/6c782468-958d-496b-afec-73b546553a57-kube-api-access-cz2ff\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-9xqhq\" (UID: \"6c782468-958d-496b-afec-73b546553a57\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-9xqhq"
Nov 01 03:27:42.830335 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.830293    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w42cj\" (UniqueName: \"kubernetes.io/projected/73ae6733-594c-47d9-89e9-8b92fde0b8f2-kube-api-access-w42cj\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-l4xqq\" (UID: \"73ae6733-594c-47d9-89e9-8b92fde0b8f2\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-l4xqq"
Nov 01 03:27:42.830335 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.830322    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c782468-958d-496b-afec-73b546553a57-istio-data\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-9xqhq\" (UID: \"6c782468-958d-496b-afec-73b546553a57\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-9xqhq"
Nov 01 03:27:42.830437 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.830363    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73ae6733-594c-47d9-89e9-8b92fde0b8f2-workload-certs\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-l4xqq\" (UID: \"73ae6733-594c-47d9-89e9-8b92fde0b8f2\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-l4xqq"
Nov 01 03:27:42.830437 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.830382    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c782468-958d-496b-afec-73b546553a57-workload-certs\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-9xqhq\" (UID: \"6c782468-958d-496b-afec-73b546553a57\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-9xqhq"
Nov 01 03:27:42.830528 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.830435    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73ae6733-594c-47d9-89e9-8b92fde0b8f2-istio-envoy\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-l4xqq\" (UID: \"73ae6733-594c-47d9-89e9-8b92fde0b8f2\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-l4xqq"
Nov 01 03:27:42.830528 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.830455    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c782468-958d-496b-afec-73b546553a57-workload-socket\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-9xqhq\" (UID: \"6c782468-958d-496b-afec-73b546553a57\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-9xqhq"
Nov 01 03:27:42.830625 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.830543    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c782468-958d-496b-afec-73b546553a57-istio-podinfo\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-9xqhq\" (UID: \"6c782468-958d-496b-afec-73b546553a57\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-9xqhq"
Nov 01 03:27:42.830625 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.830588    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c782468-958d-496b-afec-73b546553a57-credential-socket\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-9xqhq\" (UID: \"6c782468-958d-496b-afec-73b546553a57\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-9xqhq"
Nov 01 03:27:42.830769 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.830632    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73ae6733-594c-47d9-89e9-8b92fde0b8f2-istio-podinfo\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-l4xqq\" (UID: \"73ae6733-594c-47d9-89e9-8b92fde0b8f2\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-l4xqq"
Nov 01 03:27:42.830769 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.830668    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c782468-958d-496b-afec-73b546553a57-istiod-ca-cert\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-9xqhq\" (UID: \"6c782468-958d-496b-afec-73b546553a57\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-9xqhq"
Nov 01 03:27:42.830769 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.830729    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c782468-958d-496b-afec-73b546553a57-istio-data\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-9xqhq\" (UID: \"6c782468-958d-496b-afec-73b546553a57\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-9xqhq"
Nov 01 03:27:42.830983 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.830956    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c782468-958d-496b-afec-73b546553a57-credential-socket\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-9xqhq\" (UID: \"6c782468-958d-496b-afec-73b546553a57\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-9xqhq"
Nov 01 03:27:42.832625 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.832601    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c782468-958d-496b-afec-73b546553a57-istio-envoy\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-9xqhq\" (UID: \"6c782468-958d-496b-afec-73b546553a57\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-9xqhq"
Nov 01 03:27:42.832918 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.832899    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c782468-958d-496b-afec-73b546553a57-istio-podinfo\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-9xqhq\" (UID: \"6c782468-958d-496b-afec-73b546553a57\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-9xqhq"
Nov 01 03:27:42.894920 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.894890    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c782468-958d-496b-afec-73b546553a57-istio-token\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-9xqhq\" (UID: \"6c782468-958d-496b-afec-73b546553a57\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-9xqhq"
Nov 01 03:27:42.895092 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.895020    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cz2ff\" (UniqueName: \"kubernetes.io/projected/6c782468-958d-496b-afec-73b546553a57-kube-api-access-cz2ff\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-9xqhq\" (UID: \"6c782468-958d-496b-afec-73b546553a57\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-9xqhq"
Nov 01 03:27:42.931882 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.931853    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73ae6733-594c-47d9-89e9-8b92fde0b8f2-istio-podinfo\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-l4xqq\" (UID: \"73ae6733-594c-47d9-89e9-8b92fde0b8f2\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-l4xqq"
Nov 01 03:27:42.932055 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.931922    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73ae6733-594c-47d9-89e9-8b92fde0b8f2-credential-socket\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-l4xqq\" (UID: \"73ae6733-594c-47d9-89e9-8b92fde0b8f2\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-l4xqq"
Nov 01 03:27:42.932055 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.931959    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73ae6733-594c-47d9-89e9-8b92fde0b8f2-workload-socket\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-l4xqq\" (UID: \"73ae6733-594c-47d9-89e9-8b92fde0b8f2\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-l4xqq"
Nov 01 03:27:42.932055 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.931999    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73ae6733-594c-47d9-89e9-8b92fde0b8f2-istio-data\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-l4xqq\" (UID: \"73ae6733-594c-47d9-89e9-8b92fde0b8f2\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-l4xqq"
Nov 01 03:27:42.932232 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.932078    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73ae6733-594c-47d9-89e9-8b92fde0b8f2-istio-token\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-l4xqq\" (UID: \"73ae6733-594c-47d9-89e9-8b92fde0b8f2\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-l4xqq"
Nov 01 03:27:42.932232 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.932157    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73ae6733-594c-47d9-89e9-8b92fde0b8f2-istiod-ca-cert\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-l4xqq\" (UID: \"73ae6733-594c-47d9-89e9-8b92fde0b8f2\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-l4xqq"
Nov 01 03:27:42.932344 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.932316    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73ae6733-594c-47d9-89e9-8b92fde0b8f2-credential-socket\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-l4xqq\" (UID: \"73ae6733-594c-47d9-89e9-8b92fde0b8f2\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-l4xqq"
Nov 01 03:27:42.932449 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.932428    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73ae6733-594c-47d9-89e9-8b92fde0b8f2-istio-data\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-l4xqq\" (UID: \"73ae6733-594c-47d9-89e9-8b92fde0b8f2\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-l4xqq"
Nov 01 03:27:42.932491 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.932451    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73ae6733-594c-47d9-89e9-8b92fde0b8f2-workload-socket\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-l4xqq\" (UID: \"73ae6733-594c-47d9-89e9-8b92fde0b8f2\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-l4xqq"
Nov 01 03:27:42.932562 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.932546    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w42cj\" (UniqueName: \"kubernetes.io/projected/73ae6733-594c-47d9-89e9-8b92fde0b8f2-kube-api-access-w42cj\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-l4xqq\" (UID: \"73ae6733-594c-47d9-89e9-8b92fde0b8f2\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-l4xqq"
Nov 01 03:27:42.932610 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.932584    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73ae6733-594c-47d9-89e9-8b92fde0b8f2-workload-certs\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-l4xqq\" (UID: \"73ae6733-594c-47d9-89e9-8b92fde0b8f2\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-l4xqq"
Nov 01 03:27:42.932610 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.932605    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73ae6733-594c-47d9-89e9-8b92fde0b8f2-istio-envoy\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-l4xqq\" (UID: \"73ae6733-594c-47d9-89e9-8b92fde0b8f2\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-l4xqq"
Nov 01 03:27:42.933007 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.932983    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73ae6733-594c-47d9-89e9-8b92fde0b8f2-workload-certs\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-l4xqq\" (UID: \"73ae6733-594c-47d9-89e9-8b92fde0b8f2\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-l4xqq"
Nov 01 03:27:42.933379 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.933307    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73ae6733-594c-47d9-89e9-8b92fde0b8f2-istiod-ca-cert\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-l4xqq\" (UID: \"73ae6733-594c-47d9-89e9-8b92fde0b8f2\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-l4xqq"
Nov 01 03:27:42.934590 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.934571    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73ae6733-594c-47d9-89e9-8b92fde0b8f2-istio-podinfo\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-l4xqq\" (UID: \"73ae6733-594c-47d9-89e9-8b92fde0b8f2\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-l4xqq"
Nov 01 03:27:42.934896 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.934878    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73ae6733-594c-47d9-89e9-8b92fde0b8f2-istio-envoy\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-l4xqq\" (UID: \"73ae6733-594c-47d9-89e9-8b92fde0b8f2\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-l4xqq"
Nov 01 03:27:42.955916 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:27:42.955893    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod842a8e86_2464_4f2e_88ac_6be1b68b0d7e.slice/crio-bd2ee24343b9a9a9307912b55691710d831e1e84cec6bc648c9ddcf7f7a175a0 WatchSource:0}: Error finding container bd2ee24343b9a9a9307912b55691710d831e1e84cec6bc648c9ddcf7f7a175a0: Status 404 returned error can't find the container with id bd2ee24343b9a9a9307912b55691710d831e1e84cec6bc648c9ddcf7f7a175a0
Nov 01 03:27:42.956214 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.956041    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jq6dn"]
Nov 01 03:27:42.958812 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.958776    4400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 03:27:42.958907 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.958861    4400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 03:27:42.958949 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.958909    4400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 03:27:43.001659 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.999321    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73ae6733-594c-47d9-89e9-8b92fde0b8f2-istio-token\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-l4xqq\" (UID: \"73ae6733-594c-47d9-89e9-8b92fde0b8f2\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-l4xqq"
Nov 01 03:27:43.001659 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:42.999521    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w42cj\" (UniqueName: \"kubernetes.io/projected/73ae6733-594c-47d9-89e9-8b92fde0b8f2-kube-api-access-w42cj\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-l4xqq\" (UID: \"73ae6733-594c-47d9-89e9-8b92fde0b8f2\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-l4xqq"
Nov 01 03:27:43.002389 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:43.002364    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-9xqhq"
Nov 01 03:27:43.052300 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:43.051994    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-l4xqq"
Nov 01 03:27:43.291671 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:43.291630    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jq6dn" event={"ID":"842a8e86-2464-4f2e-88ac-6be1b68b0d7e","Type":"ContainerStarted","Data":"fac64634c4365b03c1ab82cb24c5cbd49bf564073dac524d897649a3e513dc1b"}
Nov 01 03:27:43.291868 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:43.291741    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jq6dn" event={"ID":"842a8e86-2464-4f2e-88ac-6be1b68b0d7e","Type":"ContainerStarted","Data":"bd2ee24343b9a9a9307912b55691710d831e1e84cec6bc648c9ddcf7f7a175a0"}
Nov 01 03:27:43.293516 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:43.293487    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-9qpm8" event={"ID":"204735ae-bff5-4cdc-9df5-f85158677fdc","Type":"ContainerStarted","Data":"d858e3bb3f335736e568ec1f96c85dc678c5a395bc9294a668d681221704d47d"}
Nov 01 03:27:43.293644 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:43.293538    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-9qpm8" event={"ID":"204735ae-bff5-4cdc-9df5-f85158677fdc","Type":"ContainerStarted","Data":"dcbcb0cce3820a1b009665ce9e8b9681673c0848579a96f4a17f5531bd27ee65"}
Nov 01 03:27:43.306453 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:43.305593    4400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-9qpm8"
Nov 01 03:27:43.306574 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:43.306497    4400 patch_prober.go:29] interesting pod/e2e-worker-w3-d7f535-openshift-default-856bb56cf-9qpm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Nov 01 03:27:43.306574 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:43.306554    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-9qpm8" podUID="204735ae-bff5-4cdc-9df5-f85158677fdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Nov 01 03:27:43.451333 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:43.451235    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jq6dn" podStartSLOduration=2.451222116 podStartE2EDuration="2.451222116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:27:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:27:43.444619492 +0000 UTC m=+1269.959265356" watchObservedRunningTime="2026-11-01 03:27:43.451222116 +0000 UTC m=+1269.965867978"
Nov 01 03:27:43.486559 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:27:43.486527    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c782468_958d_496b_afec_73b546553a57.slice/crio-f88d191b9d8e216a4b881510d23731cd3ccc5cac212696d55e0720eb16a8eb6a WatchSource:0}: Error finding container f88d191b9d8e216a4b881510d23731cd3ccc5cac212696d55e0720eb16a8eb6a: Status 404 returned error can't find the container with id f88d191b9d8e216a4b881510d23731cd3ccc5cac212696d55e0720eb16a8eb6a
Nov 01 03:27:43.487317 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:43.487293    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-9xqhq"]
Nov 01 03:27:43.489838 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:43.489801    4400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 03:27:43.489947 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:43.489880    4400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 03:27:43.489947 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:43.489928    4400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 03:27:43.560576 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:43.560551    4400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jq6dn"
Nov 01 03:27:43.571959 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:43.571917    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-9qpm8" podStartSLOduration=2.571902163 podStartE2EDuration="2.571902163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:27:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:27:43.570524819 +0000 UTC m=+1270.085170682" watchObservedRunningTime="2026-11-01 03:27:43.571902163 +0000 UTC m=+1270.086548026"
Nov 01 03:27:43.673219 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:43.673187    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-l4xqq"]
Nov 01 03:27:43.679985 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:43.679947    4400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 03:27:43.680067 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:43.680034    4400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 03:27:43.680107 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:43.680078    4400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 03:27:44.319708 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:44.318464    4400 patch_prober.go:29] interesting pod/e2e-worker-w3-d7f535-openshift-default-856bb56cf-9qpm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Nov 01 03:27:44.319708 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:44.318525    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-9qpm8" podUID="204735ae-bff5-4cdc-9df5-f85158677fdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Nov 01 03:27:44.319708 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:44.319042    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-9xqhq" event={"ID":"6c782468-958d-496b-afec-73b546553a57","Type":"ContainerStarted","Data":"12cee91983923b762f742ec2bd784e664907973dbfe2b4f377a7298d7a67a855"}
Nov 01 03:27:44.319708 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:44.319096    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-9xqhq" event={"ID":"6c782468-958d-496b-afec-73b546553a57","Type":"ContainerStarted","Data":"f88d191b9d8e216a4b881510d23731cd3ccc5cac212696d55e0720eb16a8eb6a"}
Nov 01 03:27:44.330706 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:44.328245    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-l4xqq" event={"ID":"73ae6733-594c-47d9-89e9-8b92fde0b8f2","Type":"ContainerStarted","Data":"9bc3adca49fba1b1c0c649e84f537f42d45a25e8ad129ecb62c4d63e2c63292e"}
Nov 01 03:27:44.330706 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:44.328398    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-l4xqq" event={"ID":"73ae6733-594c-47d9-89e9-8b92fde0b8f2","Type":"ContainerStarted","Data":"55ed45e1c6ffe60ef8329cf602f92e4d61932e535b3b379490b54bff7c9c3be8"}
Nov 01 03:27:44.455292 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:44.455034    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-9xqhq" podStartSLOduration=2.4550140049999998 podStartE2EDuration="2.455014005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:27:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:27:44.453051116 +0000 UTC m=+1270.967696979" watchObservedRunningTime="2026-11-01 03:27:44.455014005 +0000 UTC m=+1270.969659872"
Nov 01 03:27:44.571772 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:44.566809    4400 patch_prober.go:29] interesting pod/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jq6dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 01 03:27:44.571772 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:44.566878    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jq6dn" podUID="842a8e86-2464-4f2e-88ac-6be1b68b0d7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 01 03:27:45.003179 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:45.003086    4400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-9xqhq"
Nov 01 03:27:45.010847 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:45.010812    4400 patch_prober.go:29] interesting pod/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-9xqhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Nov 01 03:27:45.011075 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:45.011050    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-9xqhq" podUID="6c782468-958d-496b-afec-73b546553a57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Nov 01 03:27:45.054405 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:45.054374    4400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-l4xqq"
Nov 01 03:27:45.067711 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:45.061880    4400 patch_prober.go:29] interesting pod/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-l4xqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Nov 01 03:27:45.067711 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:45.061948    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-l4xqq" podUID="73ae6733-594c-47d9-89e9-8b92fde0b8f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Nov 01 03:27:45.307337 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:45.307244    4400 patch_prober.go:29] interesting pod/e2e-worker-w3-d7f535-openshift-default-856bb56cf-9qpm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Nov 01 03:27:45.307598 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:45.307565    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-9qpm8" podUID="204735ae-bff5-4cdc-9df5-f85158677fdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Nov 01 03:27:45.576153 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:45.576114    4400 patch_prober.go:29] interesting pod/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jq6dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 01 03:27:45.576773 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:45.576739    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jq6dn" podUID="842a8e86-2464-4f2e-88ac-6be1b68b0d7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 01 03:27:45.697817 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:45.697770    4400 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-19faa5/distinct-2-w6-19faa5-kserve-58db847577-hrzlw" podUID="062959ee-ec7c-4385-98d5-cf4a240852e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:46.021792 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:46.016056    4400 patch_prober.go:29] interesting pod/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-9xqhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Nov 01 03:27:46.021792 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:46.016115    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-9xqhq" podUID="6c782468-958d-496b-afec-73b546553a57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Nov 01 03:27:46.054710 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:46.054010    4400 patch_prober.go:29] interesting pod/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-l4xqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Nov 01 03:27:46.054710 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:46.054064    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-l4xqq" podUID="73ae6733-594c-47d9-89e9-8b92fde0b8f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Nov 01 03:27:46.104710 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:46.091764    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/distinct-2-w6-19faa5-kserve-58db847577-hrzlw" podUID="062959ee-ec7c-4385-98d5-cf4a240852e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:46.302858 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:46.302757    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-2-w2-c8cb9a-kserve-5c5cfb454d-fh2cj" podUID="2bcf3b85-3d4a-433f-94ad-0e954bf03e73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:46.342262 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:46.342224    4400 patch_prober.go:29] interesting pod/e2e-worker-w3-d7f535-openshift-default-856bb56cf-9qpm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Nov 01 03:27:46.342545 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:46.342514    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-9qpm8" podUID="204735ae-bff5-4cdc-9df5-f85158677fdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Nov 01 03:27:46.390711 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:46.365609    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e8c789/distinct-2-w4-e8c789-kserve-7977c64f4-wkxb2" podUID="bdb1bb9e-14f5-4be6-a028-c980119f4aa8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:46.391242 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:46.389746    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-w5-c22dd8-kserve-5c65c9bcd7-nnf7d" podUID="9cd440c6-7191-4fae-a806-f8728975b3cf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:46.414711 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:46.389784    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e8c789/premium-w4-e8c789-kserve-9458f8874-dtpnm" podUID="67bba91a-5ace-4d3f-89da-06bb9db17c78" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Nov 01 03:27:46.574708 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:46.574249    4400 patch_prober.go:29] interesting pod/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jq6dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 01 03:27:46.574708 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:46.574335    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jq6dn" podUID="842a8e86-2464-4f2e-88ac-6be1b68b0d7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 01 03:27:47.026766 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:47.013972    4400 patch_prober.go:29] interesting pod/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-9xqhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Nov 01 03:27:47.026766 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:47.014048    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-9xqhq" podUID="6c782468-958d-496b-afec-73b546553a57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Nov 01 03:27:47.060709 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:47.059777    4400 patch_prober.go:29] interesting pod/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-l4xqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Nov 01 03:27:47.060709 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:47.059849    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-l4xqq" podUID="73ae6733-594c-47d9-89e9-8b92fde0b8f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Nov 01 03:27:47.150711 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:47.149924    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/distinct-2-w6-19faa5-kserve-58db847577-hrzlw" podUID="062959ee-ec7c-4385-98d5-cf4a240852e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:47.150914 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:47.150887    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-19faa5/distinct-2-w6-19faa5-kserve-58db847577-hrzlw"
Nov 01 03:27:47.315300 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:47.315213    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-2-w2-c8cb9a-kserve-5c5cfb454d-fh2cj" podUID="2bcf3b85-3d4a-433f-94ad-0e954bf03e73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:47.315840 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:47.315818    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-2-w2-c8cb9a-kserve-5c5cfb454d-fh2cj"
Nov 01 03:27:47.336710 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:47.323201    4400 patch_prober.go:29] interesting pod/e2e-worker-w3-d7f535-openshift-default-856bb56cf-9qpm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Nov 01 03:27:47.336710 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:47.323247    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-9qpm8" podUID="204735ae-bff5-4cdc-9df5-f85158677fdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Nov 01 03:27:47.362710 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:47.339083    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-l4xqq" podStartSLOduration=5.339063513 podStartE2EDuration="5.339063513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:27:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:27:44.58007333 +0000 UTC m=+1271.094719195" watchObservedRunningTime="2026-11-01 03:27:47.339063513 +0000 UTC m=+1273.853709377"
Nov 01 03:27:47.419865 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:47.419824    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e8c789/distinct-2-w4-e8c789-kserve-7977c64f4-wkxb2" podUID="bdb1bb9e-14f5-4be6-a028-c980119f4aa8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded"
Nov 01 03:27:47.430709 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:47.420445    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e8c789/distinct-2-w4-e8c789-kserve-7977c64f4-wkxb2"
Nov 01 03:27:47.468714 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:47.451184    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-w5-c22dd8-kserve-5c65c9bcd7-nnf7d" podUID="9cd440c6-7191-4fae-a806-f8728975b3cf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:47.482710 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:47.469306    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-w5-c22dd8-kserve-5c65c9bcd7-nnf7d"
Nov 01 03:27:47.502712 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:47.483904    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e8c789/premium-w4-e8c789-kserve-9458f8874-dtpnm" podUID="67bba91a-5ace-4d3f-89da-06bb9db17c78" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Nov 01 03:27:47.503086 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:47.503047    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7f535/distinct-w3-d7f535-kserve-7d69fb9558-88lq8" podUID="65358d08-83e6-45c5-a17e-f1dbde164345" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:47.526720 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:47.504221    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e8c789/premium-w4-e8c789-kserve-9458f8874-dtpnm"
Nov 01 03:27:47.851813 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:47.827999    4400 patch_prober.go:29] interesting pod/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jq6dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 01 03:27:47.852097 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:47.852064    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jq6dn" podUID="842a8e86-2464-4f2e-88ac-6be1b68b0d7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 01 03:27:48.117771 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:48.110864    4400 patch_prober.go:29] interesting pod/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-l4xqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Nov 01 03:27:48.117771 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:48.110942    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-l4xqq" podUID="73ae6733-594c-47d9-89e9-8b92fde0b8f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Nov 01 03:27:48.120429 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:48.120399    4400 patch_prober.go:29] interesting pod/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-9xqhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Nov 01 03:27:48.122774 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:48.122741    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-9xqhq" podUID="6c782468-958d-496b-afec-73b546553a57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Nov 01 03:27:48.155833 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:48.155795    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/distinct-2-w6-19faa5-kserve-58db847577-hrzlw" podUID="062959ee-ec7c-4385-98d5-cf4a240852e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:48.366656 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:48.359519    4400 patch_prober.go:29] interesting pod/e2e-worker-w3-d7f535-openshift-default-856bb56cf-9qpm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Nov 01 03:27:48.367110 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:48.366869    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-9qpm8" podUID="204735ae-bff5-4cdc-9df5-f85158677fdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Nov 01 03:27:48.428299 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:48.427793    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-2-w2-c8cb9a-kserve-5c5cfb454d-fh2cj" podUID="2bcf3b85-3d4a-433f-94ad-0e954bf03e73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:48.429346 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:48.429313    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e8c789/distinct-2-w4-e8c789-kserve-7977c64f4-wkxb2" podUID="bdb1bb9e-14f5-4be6-a028-c980119f4aa8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:48.508860 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:48.508814    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e8c789/premium-w4-e8c789-kserve-9458f8874-dtpnm" podUID="67bba91a-5ace-4d3f-89da-06bb9db17c78" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:48.511511 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:48.511478    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-w5-c22dd8-kserve-5c65c9bcd7-nnf7d" podUID="9cd440c6-7191-4fae-a806-f8728975b3cf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Nov 01 03:27:48.576394 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:48.576281    4400 patch_prober.go:29] interesting pod/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jq6dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 01 03:27:48.576394 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:48.576337    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jq6dn" podUID="842a8e86-2464-4f2e-88ac-6be1b68b0d7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 01 03:27:48.809678 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:48.809360    4400 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-w5-c22dd8-kserve-5c65c9bcd7-nnf7d" podUID="9cd440c6-7191-4fae-a806-f8728975b3cf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:48.848201 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:48.848159    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7f535/distinct-w3-d7f535-kserve-7d69fb9558-88lq8" podUID="65358d08-83e6-45c5-a17e-f1dbde164345" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:48.855735 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:48.852804    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d7f535/distinct-w3-d7f535-kserve-7d69fb9558-88lq8"
Nov 01 03:27:48.985725 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:48.984297    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-w5-c22dd8-kserve-5c65c9bcd7-nnf7d"
Nov 01 03:27:49.015370 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:49.015329    4400 patch_prober.go:29] interesting pod/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-9xqhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Nov 01 03:27:49.015645 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:49.015617    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-9xqhq" podUID="6c782468-958d-496b-afec-73b546553a57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Nov 01 03:27:49.063258 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:49.063077    4400 patch_prober.go:29] interesting pod/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-l4xqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Nov 01 03:27:49.063258 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:49.063143    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-l4xqq" podUID="73ae6733-594c-47d9-89e9-8b92fde0b8f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Nov 01 03:27:49.221708 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:49.204858    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/distinct-2-w6-19faa5-kserve-58db847577-hrzlw" podUID="062959ee-ec7c-4385-98d5-cf4a240852e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Nov 01 03:27:49.344747 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:49.335629    4400 patch_prober.go:29] interesting pod/e2e-worker-w3-d7f535-openshift-default-856bb56cf-9qpm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Nov 01 03:27:49.344747 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:49.335705    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-9qpm8" podUID="204735ae-bff5-4cdc-9df5-f85158677fdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Nov 01 03:27:49.469972 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:49.468051    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e8c789/distinct-2-w4-e8c789-kserve-7977c64f4-wkxb2" podUID="bdb1bb9e-14f5-4be6-a028-c980119f4aa8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:49.469972 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:49.468398    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-2-w2-c8cb9a-kserve-5c5cfb454d-fh2cj" podUID="2bcf3b85-3d4a-433f-94ad-0e954bf03e73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Nov 01 03:27:49.530416 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:49.530308    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e8c789/premium-w4-e8c789-kserve-9458f8874-dtpnm" podUID="67bba91a-5ace-4d3f-89da-06bb9db17c78" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:49.562123 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:49.562008    4400 patch_prober.go:29] interesting pod/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jq6dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 01 03:27:49.562123 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:49.562064    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jq6dn" podUID="842a8e86-2464-4f2e-88ac-6be1b68b0d7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 01 03:27:49.863746 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:49.848810    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7f535/distinct-w3-d7f535-kserve-7d69fb9558-88lq8" podUID="65358d08-83e6-45c5-a17e-f1dbde164345" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:50.052726 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:50.014789    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-w5-c22dd8-kserve-5c65c9bcd7-nnf7d" podUID="9cd440c6-7191-4fae-a806-f8728975b3cf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Nov 01 03:27:50.100431 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:50.099825    4400 patch_prober.go:29] interesting pod/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-9xqhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Nov 01 03:27:50.100431 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:50.099887    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-9xqhq" podUID="6c782468-958d-496b-afec-73b546553a57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Nov 01 03:27:50.232876 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:50.219893    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/distinct-2-w6-19faa5-kserve-58db847577-hrzlw" podUID="062959ee-ec7c-4385-98d5-cf4a240852e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Nov 01 03:27:50.334061 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:50.334025    4400 patch_prober.go:29] interesting pod/e2e-worker-w3-d7f535-openshift-default-856bb56cf-9qpm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Nov 01 03:27:50.334328 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:50.334303    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-9qpm8" podUID="204735ae-bff5-4cdc-9df5-f85158677fdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Nov 01 03:27:50.334788 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:50.334766    4400 patch_prober.go:29] interesting pod/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-l4xqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Nov 01 03:27:50.334953 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:50.334935    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-l4xqq" podUID="73ae6733-594c-47d9-89e9-8b92fde0b8f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Nov 01 03:27:50.474894 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:50.474853    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-2-w2-c8cb9a-kserve-5c5cfb454d-fh2cj" podUID="2bcf3b85-3d4a-433f-94ad-0e954bf03e73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:50.475452 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:50.475425    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e8c789/distinct-2-w4-e8c789-kserve-7977c64f4-wkxb2" podUID="bdb1bb9e-14f5-4be6-a028-c980119f4aa8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded"
Nov 01 03:27:50.539853 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:50.539764    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e8c789/premium-w4-e8c789-kserve-9458f8874-dtpnm" podUID="67bba91a-5ace-4d3f-89da-06bb9db17c78" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Nov 01 03:27:50.565088 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:50.565056    4400 patch_prober.go:29] interesting pod/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jq6dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 01 03:27:50.565342 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:50.565318    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jq6dn" podUID="842a8e86-2464-4f2e-88ac-6be1b68b0d7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 01 03:27:50.978842 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:50.978768    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7f535/distinct-w3-d7f535-kserve-7d69fb9558-88lq8" podUID="65358d08-83e6-45c5-a17e-f1dbde164345" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:51.068760 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:51.011354    4400 patch_prober.go:29] interesting pod/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-9xqhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Nov 01 03:27:51.068760 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:51.011414    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-9xqhq" podUID="6c782468-958d-496b-afec-73b546553a57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Nov 01 03:27:51.068760 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:51.034362    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-w5-c22dd8-kserve-5c65c9bcd7-nnf7d" podUID="9cd440c6-7191-4fae-a806-f8728975b3cf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Nov 01 03:27:51.068760 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:51.035243    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-w5-c22dd8-kserve-5c65c9bcd7-nnf7d"
Nov 01 03:27:51.083137 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:51.080819    4400 patch_prober.go:29] interesting pod/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-l4xqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Nov 01 03:27:51.083137 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:51.080887    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-l4xqq" podUID="73ae6733-594c-47d9-89e9-8b92fde0b8f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Nov 01 03:27:51.244220 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:51.241465    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d7f535/distinct-w3-d7f535-kserve-7d69fb9558-88lq8"
Nov 01 03:27:51.244220 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:51.243495    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/distinct-2-w6-19faa5-kserve-58db847577-hrzlw" podUID="062959ee-ec7c-4385-98d5-cf4a240852e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:51.334297 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:51.334268    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-19faa5/distinct-2-w6-19faa5-kserve-58db847577-hrzlw"
Nov 01 03:27:51.344665 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:51.344626    4400 patch_prober.go:29] interesting pod/e2e-worker-w3-d7f535-openshift-default-856bb56cf-9qpm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Nov 01 03:27:51.344992 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:51.344912    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-9qpm8" podUID="204735ae-bff5-4cdc-9df5-f85158677fdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Nov 01 03:27:51.354026 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:51.354003    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e8c789/premium-w4-e8c789-kserve-9458f8874-dtpnm"
Nov 01 03:27:51.361412 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:51.358625    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-w5-c22dd8-kserve-5c65c9bcd7-nnf7d"
Nov 01 03:27:51.366007 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:51.365986    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-2-w2-c8cb9a-kserve-5c5cfb454d-fh2cj"
Nov 01 03:27:51.371990 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:51.371967    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e8c789/distinct-2-w4-e8c789-kserve-7977c64f4-wkxb2"
Nov 01 03:27:51.566081 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:51.566002    4400 patch_prober.go:29] interesting pod/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jq6dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 01 03:27:51.566342 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:51.566312    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jq6dn" podUID="842a8e86-2464-4f2e-88ac-6be1b68b0d7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 01 03:27:52.004223 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:52.003946    4400 patch_prober.go:29] interesting pod/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-9xqhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Nov 01 03:27:52.004223 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:52.004008    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-9xqhq" podUID="6c782468-958d-496b-afec-73b546553a57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Nov 01 03:27:52.060184 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:52.058940    4400 patch_prober.go:29] interesting pod/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-l4xqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Nov 01 03:27:52.060184 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:52.058998    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-l4xqq" podUID="73ae6733-594c-47d9-89e9-8b92fde0b8f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Nov 01 03:27:52.330889 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:52.330017    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 03:27:52.338055 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:52.338029    4400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-9qpm8"
Nov 01 03:27:52.339752 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:52.339731    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-9qpm8"
Nov 01 03:27:52.340551 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:52.340529    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-9qpm8"
Nov 01 03:27:53.083541 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:53.083511    4400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-l4xqq"
Nov 01 03:27:53.086088 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:53.086067    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-l4xqq"
Nov 01 03:27:53.102034 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:53.102004    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-l4xqq"
Nov 01 03:27:53.283712 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:53.282950    4400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jq6dn"
Nov 01 03:27:53.283712 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:53.283622    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jq6dn"
Nov 01 03:27:53.293990 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:53.293927    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jq6dn"
Nov 01 03:27:54.017711 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:54.014800    4400 patch_prober.go:29] interesting pod/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-9xqhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 03:27:54.017711 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:54.014877    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-9xqhq" podUID="6c782468-958d-496b-afec-73b546553a57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:55.059711 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:55.058798    4400 patch_prober.go:29] interesting pod/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-9xqhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 03:27:55.059711 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:55.058861    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-9xqhq" podUID="6c782468-958d-496b-afec-73b546553a57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:55.203720 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:55.199778    4400 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-e8c789/distinct-2-w4-e8c789-kserve-7977c64f4-wkxb2" podUID="bdb1bb9e-14f5-4be6-a028-c980119f4aa8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:55.442721 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:55.440757    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-2-w2-c8cb9a-kserve-5c5cfb454d-fh2cj" podUID="2bcf3b85-3d4a-433f-94ad-0e954bf03e73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Nov 01 03:27:55.973711 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:55.973463    4400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-9xqhq"
Nov 01 03:27:55.975276 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:55.975107    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-9xqhq"
Nov 01 03:27:55.985034 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:27:55.984980    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-9xqhq"
Nov 01 03:28:01.738869 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:28:01.738832    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:28:11.510350 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:11.510320    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-8sx2k"]
Nov 01 03:28:11.515815 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:11.515791    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-8sx2k"
Nov 01 03:28:11.639218 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:11.639180    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-8sx2k"]
Nov 01 03:28:11.655138 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:11.655108    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3532ce0-59fa-4822-bff3-2d0d356a3f13-workload-socket\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-8sx2k\" (UID: \"b3532ce0-59fa-4822-bff3-2d0d356a3f13\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-8sx2k"
Nov 01 03:28:11.655288 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:11.655141    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3532ce0-59fa-4822-bff3-2d0d356a3f13-istio-data\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-8sx2k\" (UID: \"b3532ce0-59fa-4822-bff3-2d0d356a3f13\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-8sx2k"
Nov 01 03:28:11.655288 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:11.655173    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3532ce0-59fa-4822-bff3-2d0d356a3f13-istio-podinfo\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-8sx2k\" (UID: \"b3532ce0-59fa-4822-bff3-2d0d356a3f13\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-8sx2k"
Nov 01 03:28:11.655358 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:11.655297    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3532ce0-59fa-4822-bff3-2d0d356a3f13-istiod-ca-cert\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-8sx2k\" (UID: \"b3532ce0-59fa-4822-bff3-2d0d356a3f13\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-8sx2k"
Nov 01 03:28:11.655358 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:11.655330    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3532ce0-59fa-4822-bff3-2d0d356a3f13-istio-token\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-8sx2k\" (UID: \"b3532ce0-59fa-4822-bff3-2d0d356a3f13\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-8sx2k"
Nov 01 03:28:11.655358 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:11.655349    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3532ce0-59fa-4822-bff3-2d0d356a3f13-istio-envoy\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-8sx2k\" (UID: \"b3532ce0-59fa-4822-bff3-2d0d356a3f13\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-8sx2k"
Nov 01 03:28:11.655524 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:11.655506    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3532ce0-59fa-4822-bff3-2d0d356a3f13-credential-socket\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-8sx2k\" (UID: \"b3532ce0-59fa-4822-bff3-2d0d356a3f13\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-8sx2k"
Nov 01 03:28:11.655559 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:11.655536    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dt952\" (UniqueName: \"kubernetes.io/projected/b3532ce0-59fa-4822-bff3-2d0d356a3f13-kube-api-access-dt952\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-8sx2k\" (UID: \"b3532ce0-59fa-4822-bff3-2d0d356a3f13\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-8sx2k"
Nov 01 03:28:11.655592 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:11.655561    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3532ce0-59fa-4822-bff3-2d0d356a3f13-workload-certs\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-8sx2k\" (UID: \"b3532ce0-59fa-4822-bff3-2d0d356a3f13\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-8sx2k"
Nov 01 03:28:11.757031 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:11.756999    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3532ce0-59fa-4822-bff3-2d0d356a3f13-istio-token\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-8sx2k\" (UID: \"b3532ce0-59fa-4822-bff3-2d0d356a3f13\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-8sx2k"
Nov 01 03:28:11.757031 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:11.757028    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3532ce0-59fa-4822-bff3-2d0d356a3f13-istio-envoy\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-8sx2k\" (UID: \"b3532ce0-59fa-4822-bff3-2d0d356a3f13\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-8sx2k"
Nov 01 03:28:11.757236 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:11.757085    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3532ce0-59fa-4822-bff3-2d0d356a3f13-credential-socket\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-8sx2k\" (UID: \"b3532ce0-59fa-4822-bff3-2d0d356a3f13\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-8sx2k"
Nov 01 03:28:11.757236 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:11.757113    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt952\" (UniqueName: \"kubernetes.io/projected/b3532ce0-59fa-4822-bff3-2d0d356a3f13-kube-api-access-dt952\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-8sx2k\" (UID: \"b3532ce0-59fa-4822-bff3-2d0d356a3f13\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-8sx2k"
Nov 01 03:28:11.757236 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:11.757145    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3532ce0-59fa-4822-bff3-2d0d356a3f13-workload-certs\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-8sx2k\" (UID: \"b3532ce0-59fa-4822-bff3-2d0d356a3f13\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-8sx2k"
Nov 01 03:28:11.757372 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:11.757334    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3532ce0-59fa-4822-bff3-2d0d356a3f13-workload-socket\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-8sx2k\" (UID: \"b3532ce0-59fa-4822-bff3-2d0d356a3f13\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-8sx2k"
Nov 01 03:28:11.757372 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:11.757364    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3532ce0-59fa-4822-bff3-2d0d356a3f13-istio-data\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-8sx2k\" (UID: \"b3532ce0-59fa-4822-bff3-2d0d356a3f13\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-8sx2k"
Nov 01 03:28:11.757494 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:11.757395    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3532ce0-59fa-4822-bff3-2d0d356a3f13-istio-podinfo\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-8sx2k\" (UID: \"b3532ce0-59fa-4822-bff3-2d0d356a3f13\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-8sx2k"
Nov 01 03:28:11.757494 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:11.757481    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3532ce0-59fa-4822-bff3-2d0d356a3f13-istiod-ca-cert\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-8sx2k\" (UID: \"b3532ce0-59fa-4822-bff3-2d0d356a3f13\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-8sx2k"
Nov 01 03:28:11.757642 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:11.757592    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3532ce0-59fa-4822-bff3-2d0d356a3f13-workload-certs\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-8sx2k\" (UID: \"b3532ce0-59fa-4822-bff3-2d0d356a3f13\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-8sx2k"
Nov 01 03:28:11.757740 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:11.757622    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3532ce0-59fa-4822-bff3-2d0d356a3f13-credential-socket\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-8sx2k\" (UID: \"b3532ce0-59fa-4822-bff3-2d0d356a3f13\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-8sx2k"
Nov 01 03:28:11.757740 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:11.757650    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3532ce0-59fa-4822-bff3-2d0d356a3f13-istio-data\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-8sx2k\" (UID: \"b3532ce0-59fa-4822-bff3-2d0d356a3f13\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-8sx2k"
Nov 01 03:28:11.757943 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:11.757924    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3532ce0-59fa-4822-bff3-2d0d356a3f13-workload-socket\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-8sx2k\" (UID: \"b3532ce0-59fa-4822-bff3-2d0d356a3f13\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-8sx2k"
Nov 01 03:28:11.758216 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:11.758195    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3532ce0-59fa-4822-bff3-2d0d356a3f13-istiod-ca-cert\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-8sx2k\" (UID: \"b3532ce0-59fa-4822-bff3-2d0d356a3f13\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-8sx2k"
Nov 01 03:28:11.759654 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:11.759629    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3532ce0-59fa-4822-bff3-2d0d356a3f13-istio-envoy\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-8sx2k\" (UID: \"b3532ce0-59fa-4822-bff3-2d0d356a3f13\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-8sx2k"
Nov 01 03:28:11.760243 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:11.760223    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3532ce0-59fa-4822-bff3-2d0d356a3f13-istio-podinfo\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-8sx2k\" (UID: \"b3532ce0-59fa-4822-bff3-2d0d356a3f13\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-8sx2k"
Nov 01 03:28:11.819772 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:11.819750    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt952\" (UniqueName: \"kubernetes.io/projected/b3532ce0-59fa-4822-bff3-2d0d356a3f13-kube-api-access-dt952\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-8sx2k\" (UID: \"b3532ce0-59fa-4822-bff3-2d0d356a3f13\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-8sx2k"
Nov 01 03:28:11.819877 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:11.819863    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3532ce0-59fa-4822-bff3-2d0d356a3f13-istio-token\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-8sx2k\" (UID: \"b3532ce0-59fa-4822-bff3-2d0d356a3f13\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-8sx2k"
Nov 01 03:28:11.829856 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:11.829835    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-8sx2k"
Nov 01 03:28:12.263153 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:28:12.263122    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3532ce0_59fa_4822_bff3_2d0d356a3f13.slice/crio-4ad24337d9a122978a7cb41fdafdb42c4066d619c8a90ee0a33e1dac690f8788 WatchSource:0}: Error finding container 4ad24337d9a122978a7cb41fdafdb42c4066d619c8a90ee0a33e1dac690f8788: Status 404 returned error can't find the container with id 4ad24337d9a122978a7cb41fdafdb42c4066d619c8a90ee0a33e1dac690f8788
Nov 01 03:28:12.264136 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:12.264111    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-8sx2k"]
Nov 01 03:28:12.265868 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:12.265841    4400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 03:28:12.265924 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:12.265906    4400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 03:28:12.265959 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:12.265937    4400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 03:28:12.857728 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:12.857672    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-8sx2k" event={"ID":"b3532ce0-59fa-4822-bff3-2d0d356a3f13","Type":"ContainerStarted","Data":"660447dc25d90263f45c91c2528a9e508a36a7eccf1457163368f6d54492bd36"}
Nov 01 03:28:12.858393 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:12.857744    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-8sx2k" event={"ID":"b3532ce0-59fa-4822-bff3-2d0d356a3f13","Type":"ContainerStarted","Data":"4ad24337d9a122978a7cb41fdafdb42c4066d619c8a90ee0a33e1dac690f8788"}
Nov 01 03:28:12.997501 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:12.997446    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-8sx2k" podStartSLOduration=1.997430284 podStartE2EDuration="1.997430284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:28:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:28:12.99590856 +0000 UTC m=+1299.510554425" watchObservedRunningTime="2026-11-01 03:28:12.997430284 +0000 UTC m=+1299.512076147"
Nov 01 03:28:13.839713 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:13.833290    4400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-8sx2k"
Nov 01 03:28:13.839713 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:13.833424    4400 patch_prober.go:29] interesting pod/e2e-worker-w6-19faa5-openshift-default-59c44967-8sx2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Nov 01 03:28:13.839713 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:13.833471    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-8sx2k" podUID="b3532ce0-59fa-4822-bff3-2d0d356a3f13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Nov 01 03:28:14.830435 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:14.830394    4400 patch_prober.go:29] interesting pod/e2e-worker-w6-19faa5-openshift-default-59c44967-8sx2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Nov 01 03:28:14.830963 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:14.830443    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-8sx2k" podUID="b3532ce0-59fa-4822-bff3-2d0d356a3f13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Nov 01 03:28:15.830954 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:15.830907    4400 patch_prober.go:29] interesting pod/e2e-worker-w6-19faa5-openshift-default-59c44967-8sx2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Nov 01 03:28:15.831427 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:15.830970    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-8sx2k" podUID="b3532ce0-59fa-4822-bff3-2d0d356a3f13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Nov 01 03:28:16.831169 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:16.831135    4400 patch_prober.go:29] interesting pod/e2e-worker-w6-19faa5-openshift-default-59c44967-8sx2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Nov 01 03:28:16.831588 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:16.831206    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-8sx2k" podUID="b3532ce0-59fa-4822-bff3-2d0d356a3f13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Nov 01 03:28:17.830177 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:17.830142    4400 patch_prober.go:29] interesting pod/e2e-worker-w6-19faa5-openshift-default-59c44967-8sx2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Nov 01 03:28:17.830359 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:17.830208    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-8sx2k" podUID="b3532ce0-59fa-4822-bff3-2d0d356a3f13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Nov 01 03:28:18.830456 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:18.830422    4400 patch_prober.go:29] interesting pod/e2e-worker-w6-19faa5-openshift-default-59c44967-8sx2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Nov 01 03:28:18.837800 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:18.837759    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-8sx2k" podUID="b3532ce0-59fa-4822-bff3-2d0d356a3f13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Nov 01 03:28:19.834206 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:19.834176    4400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-8sx2k"
Nov 01 03:28:19.834724 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:19.834513    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-8sx2k"
Nov 01 03:28:19.835333 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:19.835311    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-8sx2k"
Nov 01 03:28:23.900969 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:23.900937    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7066-670757"]
Nov 01 03:28:23.905107 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:23.905090    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7066-670757"
Nov 01 03:28:23.960622 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:23.960597    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7066-670757"]
Nov 01 03:28:23.969185 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:23.969160    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sp5j4\" (UniqueName: \"kubernetes.io/projected/2e12986c-b627-4c63-a2cc-1f09574ed3c5-kube-api-access-sp5j4\") pod \"test-curl-7066-670757\" (UID: \"2e12986c-b627-4c63-a2cc-1f09574ed3c5\") " pod="opendatahub/test-curl-7066-670757"
Nov 01 03:28:24.007596 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:24.007568    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-h2kzj\""
Nov 01 03:28:24.070964 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:24.070934    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sp5j4\" (UniqueName: \"kubernetes.io/projected/2e12986c-b627-4c63-a2cc-1f09574ed3c5-kube-api-access-sp5j4\") pod \"test-curl-7066-670757\" (UID: \"2e12986c-b627-4c63-a2cc-1f09574ed3c5\") " pod="opendatahub/test-curl-7066-670757"
Nov 01 03:28:24.133514 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:24.133485    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sp5j4\" (UniqueName: \"kubernetes.io/projected/2e12986c-b627-4c63-a2cc-1f09574ed3c5-kube-api-access-sp5j4\") pod \"test-curl-7066-670757\" (UID: \"2e12986c-b627-4c63-a2cc-1f09574ed3c5\") " pod="opendatahub/test-curl-7066-670757"
Nov 01 03:28:24.218661 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:24.218595    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7066-670757"
Nov 01 03:28:24.604585 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:28:24.604551    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2e12986c_b627_4c63_a2cc_1f09574ed3c5.slice/crio-6420d0136fbfd943d26c4d28b04666c320c6fdc5a43c0f952ecb40a49364884e WatchSource:0}: Error finding container 6420d0136fbfd943d26c4d28b04666c320c6fdc5a43c0f952ecb40a49364884e: Status 404 returned error can't find the container with id 6420d0136fbfd943d26c4d28b04666c320c6fdc5a43c0f952ecb40a49364884e
Nov 01 03:28:24.605164 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:24.605136    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7066-670757"]
Nov 01 03:28:24.607017 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:24.606995    4400 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 03:28:25.050804 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:25.050710    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7066-670757" event={"ID":"2e12986c-b627-4c63-a2cc-1f09574ed3c5","Type":"ContainerStarted","Data":"6420d0136fbfd943d26c4d28b04666c320c6fdc5a43c0f952ecb40a49364884e"}
Nov 01 03:28:26.530721 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:26.526981    4400 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-d7f535/distinct-w3-d7f535-kserve-7d69fb9558-88lq8" podUID="65358d08-83e6-45c5-a17e-f1dbde164345" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": context deadline exceeded"
Nov 01 03:28:26.865721 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:26.861771    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7f535/distinct-w3-d7f535-kserve-7d69fb9558-88lq8" podUID="65358d08-83e6-45c5-a17e-f1dbde164345" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Nov 01 03:28:27.213133 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:27.213038    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/distinct-2-w6-19faa5-kserve-58db847577-hrzlw" podUID="062959ee-ec7c-4385-98d5-cf4a240852e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:28:27.457709 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:27.449189    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-2-w2-c8cb9a-kserve-5c5cfb454d-fh2cj" podUID="2bcf3b85-3d4a-433f-94ad-0e954bf03e73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Nov 01 03:28:27.551711 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:27.545737    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e8c789/premium-w4-e8c789-kserve-9458f8874-dtpnm" podUID="67bba91a-5ace-4d3f-89da-06bb9db17c78" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Nov 01 03:28:27.862829 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:27.862782    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7f535/distinct-w3-d7f535-kserve-7d69fb9558-88lq8" podUID="65358d08-83e6-45c5-a17e-f1dbde164345" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:28:27.872715 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:27.863840    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d7f535/distinct-w3-d7f535-kserve-7d69fb9558-88lq8"
Nov 01 03:28:28.308058 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:28.227744    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/distinct-2-w6-19faa5-kserve-58db847577-hrzlw" podUID="062959ee-ec7c-4385-98d5-cf4a240852e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Nov 01 03:28:28.308058 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:28.228437    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-19faa5/distinct-2-w6-19faa5-kserve-58db847577-hrzlw"
Nov 01 03:28:28.382024 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:28.381964    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-w5-c22dd8-kserve-5c65c9bcd7-nnf7d" podUID="9cd440c6-7191-4fae-a806-f8728975b3cf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:28:28.482075 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:28.482023    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-2-w2-c8cb9a-kserve-5c5cfb454d-fh2cj" podUID="2bcf3b85-3d4a-433f-94ad-0e954bf03e73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Nov 01 03:28:28.507748 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:28.507720    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-2-w2-c8cb9a-kserve-5c5cfb454d-fh2cj"
Nov 01 03:28:28.509785 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:28.509749    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e8c789/distinct-2-w4-e8c789-kserve-7977c64f4-wkxb2" podUID="bdb1bb9e-14f5-4be6-a028-c980119f4aa8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded"
Nov 01 03:28:28.541426 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:28.541392    4400 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-w5-c22dd8-kserve-5c65c9bcd7-nnf7d" podUID="9cd440c6-7191-4fae-a806-f8728975b3cf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": context deadline exceeded"
Nov 01 03:28:28.566585 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:28.566360    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e8c789/premium-w4-e8c789-kserve-9458f8874-dtpnm" podUID="67bba91a-5ace-4d3f-89da-06bb9db17c78" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:28:28.621092 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:28.594933    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e8c789/premium-w4-e8c789-kserve-9458f8874-dtpnm"
Nov 01 03:28:28.863772 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:28.863717    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7f535/distinct-w3-d7f535-kserve-7d69fb9558-88lq8" podUID="65358d08-83e6-45c5-a17e-f1dbde164345" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Nov 01 03:28:29.246871 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:29.246773    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/distinct-2-w6-19faa5-kserve-58db847577-hrzlw" podUID="062959ee-ec7c-4385-98d5-cf4a240852e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Nov 01 03:28:29.424464 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:29.415529    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-w5-c22dd8-kserve-5c65c9bcd7-nnf7d" podUID="9cd440c6-7191-4fae-a806-f8728975b3cf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Nov 01 03:28:29.424464 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:29.418962    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-w5-c22dd8-kserve-5c65c9bcd7-nnf7d"
Nov 01 03:28:29.519884 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:29.519777    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-2-w2-c8cb9a-kserve-5c5cfb454d-fh2cj" podUID="2bcf3b85-3d4a-433f-94ad-0e954bf03e73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:28:29.576710 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:29.558312    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e8c789/distinct-2-w4-e8c789-kserve-7977c64f4-wkxb2" podUID="bdb1bb9e-14f5-4be6-a028-c980119f4aa8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded"
Nov 01 03:28:29.576710 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:29.559753    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e8c789/distinct-2-w4-e8c789-kserve-7977c64f4-wkxb2"
Nov 01 03:28:29.655727 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:29.619533    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e8c789/premium-w4-e8c789-kserve-9458f8874-dtpnm" podUID="67bba91a-5ace-4d3f-89da-06bb9db17c78" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:28:29.883008 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:29.882960    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7f535/distinct-w3-d7f535-kserve-7d69fb9558-88lq8" podUID="65358d08-83e6-45c5-a17e-f1dbde164345" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Nov 01 03:28:30.522344 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:30.522277    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/distinct-2-w6-19faa5-kserve-58db847577-hrzlw" podUID="062959ee-ec7c-4385-98d5-cf4a240852e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:28:30.530726 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:30.528030    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-2-w2-c8cb9a-kserve-5c5cfb454d-fh2cj" podUID="2bcf3b85-3d4a-433f-94ad-0e954bf03e73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:28:30.955717 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:30.941065    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e8c789/distinct-2-w4-e8c789-kserve-7977c64f4-wkxb2" podUID="bdb1bb9e-14f5-4be6-a028-c980119f4aa8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:28:30.955717 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:30.943098    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7f535/distinct-w3-d7f535-kserve-7d69fb9558-88lq8" podUID="65358d08-83e6-45c5-a17e-f1dbde164345" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Nov 01 03:28:30.981712 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:30.980877    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-d7f535-67cfcb74d9-lzhln" podUID="ed1ec2ab-443d-42d9-9221-20cdf668121a" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.56:50052\" within 1s: context deadline exceeded"
Nov 01 03:28:31.003711 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:30.987492    4400 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-e8c789/premium-w4-e8c789-kserve-9458f8874-dtpnm" podUID="67bba91a-5ace-4d3f-89da-06bb9db17c78" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": context deadline exceeded"
Nov 01 03:28:31.047921 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:28:31.047885    4400 logging.go:55] [core] [Channel #2141 SubChannel #2142]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.56:50052", ServerName: "10.132.0.56:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: failed to write client preface: write tcp 10.132.0.2:35724->10.132.0.56:50052: use of closed network connection"
Nov 01 03:28:31.052560 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:31.052521    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-w5-c22dd8-kserve-5c65c9bcd7-nnf7d" podUID="9cd440c6-7191-4fae-a806-f8728975b3cf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:28:31.067618 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:31.067581    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e8c789/premium-w4-e8c789-kserve-9458f8874-dtpnm" podUID="67bba91a-5ace-4d3f-89da-06bb9db17c78" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:28:31.579795 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:31.530756    4400 patch_prober.go:29] interesting pod/e2e-worker-w1-b12199-openshift-default-774cb8c567-57cp9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 03:28:31.579795 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:31.530806    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-57cp9" podUID="543cb5b7-afc2-4042-af65-69bf92772c80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:28:31.579795 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:28:31.555856    4400 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.886s"
Nov 01 03:28:31.588952 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:31.587517    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/distinct-2-w6-19faa5-kserve-58db847577-hrzlw" podUID="062959ee-ec7c-4385-98d5-cf4a240852e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Nov 01 03:28:31.588952 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:31.587888    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-2-w2-c8cb9a-kserve-5c5cfb454d-fh2cj" podUID="2bcf3b85-3d4a-433f-94ad-0e954bf03e73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Nov 01 03:28:31.944016 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:31.943897    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7f535/distinct-w3-d7f535-kserve-7d69fb9558-88lq8" podUID="65358d08-83e6-45c5-a17e-f1dbde164345" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:28:32.015712 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:31.991045    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e8c789/distinct-2-w4-e8c789-kserve-7977c64f4-wkxb2" podUID="bdb1bb9e-14f5-4be6-a028-c980119f4aa8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:28:32.057944 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:32.057905    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-w5-c22dd8-kserve-5c65c9bcd7-nnf7d" podUID="9cd440c6-7191-4fae-a806-f8728975b3cf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:28:32.079924 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:32.079886    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e8c789/premium-w4-e8c789-kserve-9458f8874-dtpnm" podUID="67bba91a-5ace-4d3f-89da-06bb9db17c78" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:28:32.244868 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:32.242669    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-2-w2-c8cb9a-kserve-5c5cfb454d-fh2cj"
Nov 01 03:28:32.345430 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:28:32.345346    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:28:32.383984 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:32.383955    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e8c789/premium-w4-e8c789-kserve-9458f8874-dtpnm"
Nov 01 03:28:32.617763 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:32.570808    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e8c789/distinct-2-w4-e8c789-kserve-7977c64f4-wkxb2"
Nov 01 03:28:32.617763 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:32.588154    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/distinct-2-w6-19faa5-kserve-58db847577-hrzlw" podUID="062959ee-ec7c-4385-98d5-cf4a240852e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:28:32.617763 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:32.596999    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7066-670757" event={"ID":"2e12986c-b627-4c63-a2cc-1f09574ed3c5","Type":"ContainerStarted","Data":"62c3b95bff1e23558203a09bcffb21fb4ae1bb790a8c933668d9b4ce80bae8d9"}
Nov 01 03:28:32.681199 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:32.681172    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-19faa5/distinct-2-w6-19faa5-kserve-58db847577-hrzlw"
Nov 01 03:28:32.864734 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:32.824009    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-w5-c22dd8-kserve-5c65c9bcd7-nnf7d"
Nov 01 03:28:32.883972 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:32.883888    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d7f535/distinct-w3-d7f535-kserve-7d69fb9558-88lq8"
Nov 01 03:28:32.917433 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:32.917076    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7066-670757" podStartSLOduration=8.502692955 podStartE2EDuration="9.91705234s" totalImagesPullingTime="1.414359385s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:28:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:28:32.761005535 +0000 UTC m=+1319.275651432" watchObservedRunningTime="2026-11-01 03:28:32.91705234 +0000 UTC m=+1319.431698259"
Nov 01 03:28:34.384070 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:34.384037    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7066-670757"]
Nov 01 03:28:34.389431 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:34.389401    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7066-670757"]
Nov 01 03:28:34.537911 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:34.537878    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7066-9e5918"]
Nov 01 03:28:34.538622 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:34.538607    4400 cpu_manager.go:414] "containerMap: removing container" podUID="2e12986c-b627-4c63-a2cc-1f09574ed3c5" containerName="test-curl-7066-670757"
Nov 01 03:28:34.538662 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:34.538624    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e12986c-b627-4c63-a2cc-1f09574ed3c5" containerName="test-curl-7066-670757"
Nov 01 03:28:34.538829 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:34.538819    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e12986c-b627-4c63-a2cc-1f09574ed3c5" containerName="test-curl-7066-670757"
Nov 01 03:28:34.542702 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:34.542669    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7066-9e5918"
Nov 01 03:28:34.604139 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:34.604107    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7066-9e5918"]
Nov 01 03:28:34.628122 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:34.628073    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7066-670757" podUID="2e12986c-b627-4c63-a2cc-1f09574ed3c5" containerName="test-curl-7066-670757" containerID="cri-o://62c3b95bff1e23558203a09bcffb21fb4ae1bb790a8c933668d9b4ce80bae8d9" gracePeriod=2
Nov 01 03:28:34.653341 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:34.653279    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vs8gs"]
Nov 01 03:28:34.653467 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:34.653443    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vs8gs" podUID="c94ffb2b-3e93-4898-97e8-1de6587c38a7" containerName="istio-proxy" containerID="cri-o://6f9527523882d778e6155135752f0e21ddccb5ac129164e1a3b9ed20a392c0c3" gracePeriod=30
Nov 01 03:28:34.674075 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:34.674045    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vttd9"]
Nov 01 03:28:34.674282 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:34.674257    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vttd9" podUID="ca8197ec-0240-48b6-ae32-c5ad205f9d2d" containerName="istio-proxy" containerID="cri-o://b637a2b17b7136b5451f57c211b80db2fad3d1c0f7fd119463e9eaca636d7de8" gracePeriod=30
Nov 01 03:28:34.684840 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:34.684810    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g54zf\" (UniqueName: \"kubernetes.io/projected/9ace3b5b-e63a-495f-960e-0d47df550f59-kube-api-access-g54zf\") pod \"test-curl-7066-9e5918\" (UID: \"9ace3b5b-e63a-495f-960e-0d47df550f59\") " pod="opendatahub/test-curl-7066-9e5918"
Nov 01 03:28:34.786601 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:34.786552    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g54zf\" (UniqueName: \"kubernetes.io/projected/9ace3b5b-e63a-495f-960e-0d47df550f59-kube-api-access-g54zf\") pod \"test-curl-7066-9e5918\" (UID: \"9ace3b5b-e63a-495f-960e-0d47df550f59\") " pod="opendatahub/test-curl-7066-9e5918"
Nov 01 03:28:34.846241 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:28:34.846200    4400 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7066-670757\" is forbidden: User \"system:node:ip-10-0-133-54.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-54.us-west-1.compute.internal' and this object" podUID="2e12986c-b627-4c63-a2cc-1f09574ed3c5" pod="opendatahub/test-curl-7066-670757"
Nov 01 03:28:34.849223 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:34.849186    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g54zf\" (UniqueName: \"kubernetes.io/projected/9ace3b5b-e63a-495f-960e-0d47df550f59-kube-api-access-g54zf\") pod \"test-curl-7066-9e5918\" (UID: \"9ace3b5b-e63a-495f-960e-0d47df550f59\") " pod="opendatahub/test-curl-7066-9e5918"
Nov 01 03:28:34.856093 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:34.856070    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7066-9e5918"
Nov 01 03:28:35.021832 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:28:35.021799    4400 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7066-670757\" is forbidden: User \"system:node:ip-10-0-133-54.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-54.us-west-1.compute.internal' and this object" podUID="2e12986c-b627-4c63-a2cc-1f09574ed3c5" pod="opendatahub/test-curl-7066-670757"
Nov 01 03:28:35.249054 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:35.249026    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7066-9e5918"]
Nov 01 03:28:35.249724 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:28:35.249667    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9ace3b5b_e63a_495f_960e_0d47df550f59.slice/crio-611f9f4135f2af0afe44e3e126904c7b67e4e53f08d1421b513ac3b6b81aea3e WatchSource:0}: Error finding container 611f9f4135f2af0afe44e3e126904c7b67e4e53f08d1421b513ac3b6b81aea3e: Status 404 returned error can't find the container with id 611f9f4135f2af0afe44e3e126904c7b67e4e53f08d1421b513ac3b6b81aea3e
Nov 01 03:28:35.635288 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:35.635260    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7066-9e5918" event={"ID":"9ace3b5b-e63a-495f-960e-0d47df550f59","Type":"ContainerStarted","Data":"74afcd71374d9527390139281b7b3a6bcbba875f8be6d2ca8532531cfcd7b554"}
Nov 01 03:28:35.635665 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:35.635296    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7066-9e5918" event={"ID":"9ace3b5b-e63a-495f-960e-0d47df550f59","Type":"ContainerStarted","Data":"611f9f4135f2af0afe44e3e126904c7b67e4e53f08d1421b513ac3b6b81aea3e"}
Nov 01 03:28:35.779544 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:35.779488    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7066-9e5918" podStartSLOduration=1.779472718 podStartE2EDuration="1.779472718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:28:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:28:35.77606452 +0000 UTC m=+1322.290710384" watchObservedRunningTime="2026-11-01 03:28:35.779472718 +0000 UTC m=+1322.294118575"
Nov 01 03:28:36.896048 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:36.895987    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7066-670757"
Nov 01 03:28:37.023900 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:37.023868    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e12986c-b627-4c63-a2cc-1f09574ed3c5-kube-api-access-sp5j4\" (UniqueName: \"kubernetes.io/projected/2e12986c-b627-4c63-a2cc-1f09574ed3c5-kube-api-access-sp5j4\") pod \"2e12986c-b627-4c63-a2cc-1f09574ed3c5\" (UID: \"2e12986c-b627-4c63-a2cc-1f09574ed3c5\") "
Nov 01 03:28:37.026054 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:37.026027    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e12986c-b627-4c63-a2cc-1f09574ed3c5-kube-api-access-sp5j4" pod "2e12986c-b627-4c63-a2cc-1f09574ed3c5" (UID: "2e12986c-b627-4c63-a2cc-1f09574ed3c5"). InnerVolumeSpecName "kube-api-access-sp5j4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:28:37.134748 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:37.134094    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sp5j4\" (UniqueName: \"kubernetes.io/projected/2e12986c-b627-4c63-a2cc-1f09574ed3c5-kube-api-access-sp5j4\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:28:37.397812 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:37.397756    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7066-9e5918"]
Nov 01 03:28:37.424949 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:37.424123    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7066-9e5918"]
Nov 01 03:28:37.616167 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:37.616124    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e12986c-b627-4c63-a2cc-1f09574ed3c5" path="/var/lib/kubelet/pods/2e12986c-b627-4c63-a2cc-1f09574ed3c5/volumes"
Nov 01 03:28:37.651728 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:37.651656    4400 generic.go:356] "Generic (PLEG): container finished" podID="2e12986c-b627-4c63-a2cc-1f09574ed3c5" containerID="62c3b95bff1e23558203a09bcffb21fb4ae1bb790a8c933668d9b4ce80bae8d9" exitCode=137
Nov 01 03:28:37.651858 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:37.651746    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7066-670757"
Nov 01 03:28:37.651858 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:37.651750    4400 scope.go:122] "RemoveContainer" containerID="62c3b95bff1e23558203a09bcffb21fb4ae1bb790a8c933668d9b4ce80bae8d9"
Nov 01 03:28:37.652060 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:37.652015    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7066-9e5918" podUID="9ace3b5b-e63a-495f-960e-0d47df550f59" containerName="test-curl-7066-9e5918" containerID="cri-o://74afcd71374d9527390139281b7b3a6bcbba875f8be6d2ca8532531cfcd7b554" gracePeriod=2
Nov 01 03:28:37.661547 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:37.661530    4400 scope.go:122] "RemoveContainer" containerID="62c3b95bff1e23558203a09bcffb21fb4ae1bb790a8c933668d9b4ce80bae8d9"
Nov 01 03:28:37.661814 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:28:37.661798    4400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62c3b95bff1e23558203a09bcffb21fb4ae1bb790a8c933668d9b4ce80bae8d9\": container with ID starting with 62c3b95bff1e23558203a09bcffb21fb4ae1bb790a8c933668d9b4ce80bae8d9 not found: ID does not exist" containerID="62c3b95bff1e23558203a09bcffb21fb4ae1bb790a8c933668d9b4ce80bae8d9"
Nov 01 03:28:37.661870 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:37.661822    4400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62c3b95bff1e23558203a09bcffb21fb4ae1bb790a8c933668d9b4ce80bae8d9"} err="failed to get container status \"62c3b95bff1e23558203a09bcffb21fb4ae1bb790a8c933668d9b4ce80bae8d9\": rpc error: code = NotFound desc = could not find container \"62c3b95bff1e23558203a09bcffb21fb4ae1bb790a8c933668d9b4ce80bae8d9\": container with ID starting with 62c3b95bff1e23558203a09bcffb21fb4ae1bb790a8c933668d9b4ce80bae8d9 not found: ID does not exist"
Nov 01 03:28:39.962430 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:39.962403    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vs8gs"
Nov 01 03:28:39.992851 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:39.992111    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c94ffb2b-3e93-4898-97e8-1de6587c38a7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c94ffb2b-3e93-4898-97e8-1de6587c38a7-workload-certs\") pod \"c94ffb2b-3e93-4898-97e8-1de6587c38a7\" (UID: \"c94ffb2b-3e93-4898-97e8-1de6587c38a7\") "
Nov 01 03:28:39.992851 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:39.992241    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c94ffb2b-3e93-4898-97e8-1de6587c38a7-kube-api-access-bxxsr\" (UniqueName: \"kubernetes.io/projected/c94ffb2b-3e93-4898-97e8-1de6587c38a7-kube-api-access-bxxsr\") pod \"c94ffb2b-3e93-4898-97e8-1de6587c38a7\" (UID: \"c94ffb2b-3e93-4898-97e8-1de6587c38a7\") "
Nov 01 03:28:39.992851 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:39.992290    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c94ffb2b-3e93-4898-97e8-1de6587c38a7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c94ffb2b-3e93-4898-97e8-1de6587c38a7-credential-socket\") pod \"c94ffb2b-3e93-4898-97e8-1de6587c38a7\" (UID: \"c94ffb2b-3e93-4898-97e8-1de6587c38a7\") "
Nov 01 03:28:39.992851 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:39.992340    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c94ffb2b-3e93-4898-97e8-1de6587c38a7-istio-token\" (UniqueName: \"kubernetes.io/projected/c94ffb2b-3e93-4898-97e8-1de6587c38a7-istio-token\") pod \"c94ffb2b-3e93-4898-97e8-1de6587c38a7\" (UID: \"c94ffb2b-3e93-4898-97e8-1de6587c38a7\") "
Nov 01 03:28:39.992851 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:39.992382    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c94ffb2b-3e93-4898-97e8-1de6587c38a7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c94ffb2b-3e93-4898-97e8-1de6587c38a7-workload-socket\") pod \"c94ffb2b-3e93-4898-97e8-1de6587c38a7\" (UID: \"c94ffb2b-3e93-4898-97e8-1de6587c38a7\") "
Nov 01 03:28:39.992851 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:39.992418    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c94ffb2b-3e93-4898-97e8-1de6587c38a7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c94ffb2b-3e93-4898-97e8-1de6587c38a7-istio-envoy\") pod \"c94ffb2b-3e93-4898-97e8-1de6587c38a7\" (UID: \"c94ffb2b-3e93-4898-97e8-1de6587c38a7\") "
Nov 01 03:28:39.992851 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:39.992458    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c94ffb2b-3e93-4898-97e8-1de6587c38a7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c94ffb2b-3e93-4898-97e8-1de6587c38a7-istio-podinfo\") pod \"c94ffb2b-3e93-4898-97e8-1de6587c38a7\" (UID: \"c94ffb2b-3e93-4898-97e8-1de6587c38a7\") "
Nov 01 03:28:39.992851 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:39.992571    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c94ffb2b-3e93-4898-97e8-1de6587c38a7-credential-socket" pod "c94ffb2b-3e93-4898-97e8-1de6587c38a7" (UID: "c94ffb2b-3e93-4898-97e8-1de6587c38a7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:28:39.992851 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:39.992642    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c94ffb2b-3e93-4898-97e8-1de6587c38a7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c94ffb2b-3e93-4898-97e8-1de6587c38a7-istiod-ca-cert\") pod \"c94ffb2b-3e93-4898-97e8-1de6587c38a7\" (UID: \"c94ffb2b-3e93-4898-97e8-1de6587c38a7\") "
Nov 01 03:28:39.992851 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:39.992763    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c94ffb2b-3e93-4898-97e8-1de6587c38a7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c94ffb2b-3e93-4898-97e8-1de6587c38a7-istio-data\") pod \"c94ffb2b-3e93-4898-97e8-1de6587c38a7\" (UID: \"c94ffb2b-3e93-4898-97e8-1de6587c38a7\") "
Nov 01 03:28:39.993667 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:39.993475    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c94ffb2b-3e93-4898-97e8-1de6587c38a7-workload-socket" pod "c94ffb2b-3e93-4898-97e8-1de6587c38a7" (UID: "c94ffb2b-3e93-4898-97e8-1de6587c38a7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:28:39.993667 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:39.993527    4400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c94ffb2b-3e93-4898-97e8-1de6587c38a7-credential-socket\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:28:39.993667 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:39.993533    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c94ffb2b-3e93-4898-97e8-1de6587c38a7-workload-certs" pod "c94ffb2b-3e93-4898-97e8-1de6587c38a7" (UID: "c94ffb2b-3e93-4898-97e8-1de6587c38a7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:28:39.993911 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:39.993888    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c94ffb2b-3e93-4898-97e8-1de6587c38a7-istiod-ca-cert" pod "c94ffb2b-3e93-4898-97e8-1de6587c38a7" (UID: "c94ffb2b-3e93-4898-97e8-1de6587c38a7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:28:39.995385 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:39.995330    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c94ffb2b-3e93-4898-97e8-1de6587c38a7-kube-api-access-bxxsr" pod "c94ffb2b-3e93-4898-97e8-1de6587c38a7" (UID: "c94ffb2b-3e93-4898-97e8-1de6587c38a7"). InnerVolumeSpecName "kube-api-access-bxxsr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:28:39.996468 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:39.996439    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c94ffb2b-3e93-4898-97e8-1de6587c38a7-istio-envoy" pod "c94ffb2b-3e93-4898-97e8-1de6587c38a7" (UID: "c94ffb2b-3e93-4898-97e8-1de6587c38a7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:28:39.996576 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:39.996533    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c94ffb2b-3e93-4898-97e8-1de6587c38a7-istio-token" pod "c94ffb2b-3e93-4898-97e8-1de6587c38a7" (UID: "c94ffb2b-3e93-4898-97e8-1de6587c38a7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:28:39.996670 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:39.996645    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c94ffb2b-3e93-4898-97e8-1de6587c38a7-istio-podinfo" pod "c94ffb2b-3e93-4898-97e8-1de6587c38a7" (UID: "c94ffb2b-3e93-4898-97e8-1de6587c38a7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 03:28:39.996825 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:39.996800    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c94ffb2b-3e93-4898-97e8-1de6587c38a7-istio-data" pod "c94ffb2b-3e93-4898-97e8-1de6587c38a7" (UID: "c94ffb2b-3e93-4898-97e8-1de6587c38a7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:28:39.999204 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:39.999181    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vttd9"
Nov 01 03:28:40.008236 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:40.008217    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7066-9e5918"
Nov 01 03:28:40.094604 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:40.094573    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca8197ec-0240-48b6-ae32-c5ad205f9d2d-istio-token\" (UniqueName: \"kubernetes.io/projected/ca8197ec-0240-48b6-ae32-c5ad205f9d2d-istio-token\") pod \"ca8197ec-0240-48b6-ae32-c5ad205f9d2d\" (UID: \"ca8197ec-0240-48b6-ae32-c5ad205f9d2d\") "
Nov 01 03:28:40.094761 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:40.094621    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca8197ec-0240-48b6-ae32-c5ad205f9d2d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca8197ec-0240-48b6-ae32-c5ad205f9d2d-istio-data\") pod \"ca8197ec-0240-48b6-ae32-c5ad205f9d2d\" (UID: \"ca8197ec-0240-48b6-ae32-c5ad205f9d2d\") "
Nov 01 03:28:40.094761 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:40.094746    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca8197ec-0240-48b6-ae32-c5ad205f9d2d-kube-api-access-bn6x2\" (UniqueName: \"kubernetes.io/projected/ca8197ec-0240-48b6-ae32-c5ad205f9d2d-kube-api-access-bn6x2\") pod \"ca8197ec-0240-48b6-ae32-c5ad205f9d2d\" (UID: \"ca8197ec-0240-48b6-ae32-c5ad205f9d2d\") "
Nov 01 03:28:40.094894 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:40.094779    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca8197ec-0240-48b6-ae32-c5ad205f9d2d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca8197ec-0240-48b6-ae32-c5ad205f9d2d-workload-socket\") pod \"ca8197ec-0240-48b6-ae32-c5ad205f9d2d\" (UID: \"ca8197ec-0240-48b6-ae32-c5ad205f9d2d\") "
Nov 01 03:28:40.094894 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:40.094808    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca8197ec-0240-48b6-ae32-c5ad205f9d2d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca8197ec-0240-48b6-ae32-c5ad205f9d2d-istiod-ca-cert\") pod \"ca8197ec-0240-48b6-ae32-c5ad205f9d2d\" (UID: \"ca8197ec-0240-48b6-ae32-c5ad205f9d2d\") "
Nov 01 03:28:40.094894 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:40.094836    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ca8197ec-0240-48b6-ae32-c5ad205f9d2d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca8197ec-0240-48b6-ae32-c5ad205f9d2d-istio-podinfo\") pod \"ca8197ec-0240-48b6-ae32-c5ad205f9d2d\" (UID: \"ca8197ec-0240-48b6-ae32-c5ad205f9d2d\") "
Nov 01 03:28:40.095052 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:40.094922    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca8197ec-0240-48b6-ae32-c5ad205f9d2d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca8197ec-0240-48b6-ae32-c5ad205f9d2d-credential-socket\") pod \"ca8197ec-0240-48b6-ae32-c5ad205f9d2d\" (UID: \"ca8197ec-0240-48b6-ae32-c5ad205f9d2d\") "
Nov 01 03:28:40.095052 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:40.094955    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca8197ec-0240-48b6-ae32-c5ad205f9d2d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca8197ec-0240-48b6-ae32-c5ad205f9d2d-workload-certs\") pod \"ca8197ec-0240-48b6-ae32-c5ad205f9d2d\" (UID: \"ca8197ec-0240-48b6-ae32-c5ad205f9d2d\") "
Nov 01 03:28:40.095052 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:40.094975    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca8197ec-0240-48b6-ae32-c5ad205f9d2d-istio-data" pod "ca8197ec-0240-48b6-ae32-c5ad205f9d2d" (UID: "ca8197ec-0240-48b6-ae32-c5ad205f9d2d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:28:40.095052 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:40.095011    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca8197ec-0240-48b6-ae32-c5ad205f9d2d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca8197ec-0240-48b6-ae32-c5ad205f9d2d-istio-envoy\") pod \"ca8197ec-0240-48b6-ae32-c5ad205f9d2d\" (UID: \"ca8197ec-0240-48b6-ae32-c5ad205f9d2d\") "
Nov 01 03:28:40.095255 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:40.095074    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca8197ec-0240-48b6-ae32-c5ad205f9d2d-workload-socket" pod "ca8197ec-0240-48b6-ae32-c5ad205f9d2d" (UID: "ca8197ec-0240-48b6-ae32-c5ad205f9d2d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:28:40.095255 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:40.095135    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ace3b5b-e63a-495f-960e-0d47df550f59-kube-api-access-g54zf\" (UniqueName: \"kubernetes.io/projected/9ace3b5b-e63a-495f-960e-0d47df550f59-kube-api-access-g54zf\") pod \"9ace3b5b-e63a-495f-960e-0d47df550f59\" (UID: \"9ace3b5b-e63a-495f-960e-0d47df550f59\") "
Nov 01 03:28:40.095255 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:40.095190    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca8197ec-0240-48b6-ae32-c5ad205f9d2d-credential-socket" pod "ca8197ec-0240-48b6-ae32-c5ad205f9d2d" (UID: "ca8197ec-0240-48b6-ae32-c5ad205f9d2d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:28:40.095255 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:40.095203    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca8197ec-0240-48b6-ae32-c5ad205f9d2d-workload-certs" pod "ca8197ec-0240-48b6-ae32-c5ad205f9d2d" (UID: "ca8197ec-0240-48b6-ae32-c5ad205f9d2d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:28:40.095746 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:40.095677    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca8197ec-0240-48b6-ae32-c5ad205f9d2d-istiod-ca-cert" pod "ca8197ec-0240-48b6-ae32-c5ad205f9d2d" (UID: "ca8197ec-0240-48b6-ae32-c5ad205f9d2d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:28:40.096062 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:40.096039    4400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c94ffb2b-3e93-4898-97e8-1de6587c38a7-istio-data\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:28:40.096181 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:40.096068    4400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca8197ec-0240-48b6-ae32-c5ad205f9d2d-istio-data\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:28:40.096181 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:40.096083    4400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c94ffb2b-3e93-4898-97e8-1de6587c38a7-workload-certs\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:28:40.096181 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:40.096098    4400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca8197ec-0240-48b6-ae32-c5ad205f9d2d-workload-socket\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:28:40.096181 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:40.096114    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxxsr\" (UniqueName: \"kubernetes.io/projected/c94ffb2b-3e93-4898-97e8-1de6587c38a7-kube-api-access-bxxsr\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:28:40.096181 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:40.096130    4400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca8197ec-0240-48b6-ae32-c5ad205f9d2d-istiod-ca-cert\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:28:40.096181 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:40.096144    4400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c94ffb2b-3e93-4898-97e8-1de6587c38a7-istio-token\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:28:40.096181 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:40.096158    4400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c94ffb2b-3e93-4898-97e8-1de6587c38a7-workload-socket\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:28:40.096181 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:40.096171    4400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c94ffb2b-3e93-4898-97e8-1de6587c38a7-istio-envoy\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:28:40.096640 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:40.096185    4400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca8197ec-0240-48b6-ae32-c5ad205f9d2d-credential-socket\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:28:40.096640 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:40.096200    4400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c94ffb2b-3e93-4898-97e8-1de6587c38a7-istio-podinfo\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:28:40.096640 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:40.096215    4400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca8197ec-0240-48b6-ae32-c5ad205f9d2d-workload-certs\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:28:40.096640 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:40.096228    4400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c94ffb2b-3e93-4898-97e8-1de6587c38a7-istiod-ca-cert\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:28:40.097398 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:40.097377    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca8197ec-0240-48b6-ae32-c5ad205f9d2d-istio-token" pod "ca8197ec-0240-48b6-ae32-c5ad205f9d2d" (UID: "ca8197ec-0240-48b6-ae32-c5ad205f9d2d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:28:40.098119 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:40.098097    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ca8197ec-0240-48b6-ae32-c5ad205f9d2d-istio-podinfo" pod "ca8197ec-0240-48b6-ae32-c5ad205f9d2d" (UID: "ca8197ec-0240-48b6-ae32-c5ad205f9d2d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 03:28:40.098210 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:40.098179    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca8197ec-0240-48b6-ae32-c5ad205f9d2d-kube-api-access-bn6x2" pod "ca8197ec-0240-48b6-ae32-c5ad205f9d2d" (UID: "ca8197ec-0240-48b6-ae32-c5ad205f9d2d"). InnerVolumeSpecName "kube-api-access-bn6x2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:28:40.098468 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:40.098450    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca8197ec-0240-48b6-ae32-c5ad205f9d2d-istio-envoy" pod "ca8197ec-0240-48b6-ae32-c5ad205f9d2d" (UID: "ca8197ec-0240-48b6-ae32-c5ad205f9d2d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:28:40.098517 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:40.098498    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ace3b5b-e63a-495f-960e-0d47df550f59-kube-api-access-g54zf" pod "9ace3b5b-e63a-495f-960e-0d47df550f59" (UID: "9ace3b5b-e63a-495f-960e-0d47df550f59"). InnerVolumeSpecName "kube-api-access-g54zf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:28:40.197792 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:40.197715    4400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca8197ec-0240-48b6-ae32-c5ad205f9d2d-istio-token\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:28:40.197792 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:40.197750    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bn6x2\" (UniqueName: \"kubernetes.io/projected/ca8197ec-0240-48b6-ae32-c5ad205f9d2d-kube-api-access-bn6x2\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:28:40.197792 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:40.197766    4400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca8197ec-0240-48b6-ae32-c5ad205f9d2d-istio-podinfo\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:28:40.197792 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:40.197780    4400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca8197ec-0240-48b6-ae32-c5ad205f9d2d-istio-envoy\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:28:40.198019 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:40.197795    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g54zf\" (UniqueName: \"kubernetes.io/projected/9ace3b5b-e63a-495f-960e-0d47df550f59-kube-api-access-g54zf\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:28:40.676131 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:40.676101    4400 generic.go:356] "Generic (PLEG): container finished" podID="ca8197ec-0240-48b6-ae32-c5ad205f9d2d" containerID="b637a2b17b7136b5451f57c211b80db2fad3d1c0f7fd119463e9eaca636d7de8" exitCode=0
Nov 01 03:28:40.676273 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:40.676152    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vttd9"
Nov 01 03:28:40.676273 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:40.676186    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vttd9" event={"ID":"ca8197ec-0240-48b6-ae32-c5ad205f9d2d","Type":"ContainerDied","Data":"b637a2b17b7136b5451f57c211b80db2fad3d1c0f7fd119463e9eaca636d7de8"}
Nov 01 03:28:40.676273 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:40.676247    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vttd9" event={"ID":"ca8197ec-0240-48b6-ae32-c5ad205f9d2d","Type":"ContainerDied","Data":"1140de05843223be09dcb11dc23a1d75ee36c6f7d3e3741891e13658c46e5803"}
Nov 01 03:28:40.676490 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:40.676277    4400 scope.go:122] "RemoveContainer" containerID="b637a2b17b7136b5451f57c211b80db2fad3d1c0f7fd119463e9eaca636d7de8"
Nov 01 03:28:40.678134 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:40.678108    4400 generic.go:356] "Generic (PLEG): container finished" podID="9ace3b5b-e63a-495f-960e-0d47df550f59" containerID="74afcd71374d9527390139281b7b3a6bcbba875f8be6d2ca8532531cfcd7b554" exitCode=137
Nov 01 03:28:40.678254 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:40.678152    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7066-9e5918"
Nov 01 03:28:40.679861 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:40.679840    4400 generic.go:356] "Generic (PLEG): container finished" podID="c94ffb2b-3e93-4898-97e8-1de6587c38a7" containerID="6f9527523882d778e6155135752f0e21ddccb5ac129164e1a3b9ed20a392c0c3" exitCode=0
Nov 01 03:28:40.679968 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:40.679897    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vs8gs"
Nov 01 03:28:40.679968 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:40.679914    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vs8gs" event={"ID":"c94ffb2b-3e93-4898-97e8-1de6587c38a7","Type":"ContainerDied","Data":"6f9527523882d778e6155135752f0e21ddccb5ac129164e1a3b9ed20a392c0c3"}
Nov 01 03:28:40.679968 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:40.679957    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vs8gs" event={"ID":"c94ffb2b-3e93-4898-97e8-1de6587c38a7","Type":"ContainerDied","Data":"a493a4cc4bac3470c5ee404f4632607653e51f05c3e1f9b2ea0c6fda3a245441"}
Nov 01 03:28:40.696181 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:40.695856    4400 scope.go:122] "RemoveContainer" containerID="b637a2b17b7136b5451f57c211b80db2fad3d1c0f7fd119463e9eaca636d7de8"
Nov 01 03:28:40.696302 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:28:40.696201    4400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b637a2b17b7136b5451f57c211b80db2fad3d1c0f7fd119463e9eaca636d7de8\": container with ID starting with b637a2b17b7136b5451f57c211b80db2fad3d1c0f7fd119463e9eaca636d7de8 not found: ID does not exist" containerID="b637a2b17b7136b5451f57c211b80db2fad3d1c0f7fd119463e9eaca636d7de8"
Nov 01 03:28:40.696302 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:40.696234    4400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b637a2b17b7136b5451f57c211b80db2fad3d1c0f7fd119463e9eaca636d7de8"} err="failed to get container status \"b637a2b17b7136b5451f57c211b80db2fad3d1c0f7fd119463e9eaca636d7de8\": rpc error: code = NotFound desc = could not find container \"b637a2b17b7136b5451f57c211b80db2fad3d1c0f7fd119463e9eaca636d7de8\": container with ID starting with b637a2b17b7136b5451f57c211b80db2fad3d1c0f7fd119463e9eaca636d7de8 not found: ID does not exist"
Nov 01 03:28:40.696302 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:40.696267    4400 scope.go:122] "RemoveContainer" containerID="74afcd71374d9527390139281b7b3a6bcbba875f8be6d2ca8532531cfcd7b554"
Nov 01 03:28:40.707253 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:40.707222    4400 scope.go:122] "RemoveContainer" containerID="74afcd71374d9527390139281b7b3a6bcbba875f8be6d2ca8532531cfcd7b554"
Nov 01 03:28:40.707574 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:28:40.707551    4400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74afcd71374d9527390139281b7b3a6bcbba875f8be6d2ca8532531cfcd7b554\": container with ID starting with 74afcd71374d9527390139281b7b3a6bcbba875f8be6d2ca8532531cfcd7b554 not found: ID does not exist" containerID="74afcd71374d9527390139281b7b3a6bcbba875f8be6d2ca8532531cfcd7b554"
Nov 01 03:28:40.707622 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:40.707582    4400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74afcd71374d9527390139281b7b3a6bcbba875f8be6d2ca8532531cfcd7b554"} err="failed to get container status \"74afcd71374d9527390139281b7b3a6bcbba875f8be6d2ca8532531cfcd7b554\": rpc error: code = NotFound desc = could not find container \"74afcd71374d9527390139281b7b3a6bcbba875f8be6d2ca8532531cfcd7b554\": container with ID starting with 74afcd71374d9527390139281b7b3a6bcbba875f8be6d2ca8532531cfcd7b554 not found: ID does not exist"
Nov 01 03:28:40.707622 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:40.707599    4400 scope.go:122] "RemoveContainer" containerID="6f9527523882d778e6155135752f0e21ddccb5ac129164e1a3b9ed20a392c0c3"
Nov 01 03:28:40.718830 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:40.718801    4400 scope.go:122] "RemoveContainer" containerID="6f9527523882d778e6155135752f0e21ddccb5ac129164e1a3b9ed20a392c0c3"
Nov 01 03:28:40.719098 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:28:40.719078    4400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f9527523882d778e6155135752f0e21ddccb5ac129164e1a3b9ed20a392c0c3\": container with ID starting with 6f9527523882d778e6155135752f0e21ddccb5ac129164e1a3b9ed20a392c0c3 not found: ID does not exist" containerID="6f9527523882d778e6155135752f0e21ddccb5ac129164e1a3b9ed20a392c0c3"
Nov 01 03:28:40.719145 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:40.719106    4400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f9527523882d778e6155135752f0e21ddccb5ac129164e1a3b9ed20a392c0c3"} err="failed to get container status \"6f9527523882d778e6155135752f0e21ddccb5ac129164e1a3b9ed20a392c0c3\": rpc error: code = NotFound desc = could not find container \"6f9527523882d778e6155135752f0e21ddccb5ac129164e1a3b9ed20a392c0c3\": container with ID starting with 6f9527523882d778e6155135752f0e21ddccb5ac129164e1a3b9ed20a392c0c3 not found: ID does not exist"
Nov 01 03:28:40.887501 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:40.887472    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vs8gs"]
Nov 01 03:28:40.938165 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:40.938092    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vs8gs"]
Nov 01 03:28:40.943336 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:28:40.943305    4400 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7066-9e5918\" is forbidden: User \"system:node:ip-10-0-133-54.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-54.us-west-1.compute.internal' and this object" podUID="9ace3b5b-e63a-495f-960e-0d47df550f59" pod="opendatahub/test-curl-7066-9e5918"
Nov 01 03:28:41.062720 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:41.062647    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vttd9"]
Nov 01 03:28:41.068825 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:41.068797    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-vttd9"]
Nov 01 03:28:41.616022 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:41.615993    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ace3b5b-e63a-495f-960e-0d47df550f59" path="/var/lib/kubelet/pods/9ace3b5b-e63a-495f-960e-0d47df550f59/volumes"
Nov 01 03:28:41.616526 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:41.616511    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c94ffb2b-3e93-4898-97e8-1de6587c38a7" path="/var/lib/kubelet/pods/c94ffb2b-3e93-4898-97e8-1de6587c38a7/volumes"
Nov 01 03:28:41.617010 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:41.616997    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca8197ec-0240-48b6-ae32-c5ad205f9d2d" path="/var/lib/kubelet/pods/ca8197ec-0240-48b6-ae32-c5ad205f9d2d/volumes"
Nov 01 03:28:42.793707 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:42.777187    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hf2dc"]
Nov 01 03:28:42.793707 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:42.778125    4400 cpu_manager.go:414] "containerMap: removing container" podUID="c94ffb2b-3e93-4898-97e8-1de6587c38a7" containerName="istio-proxy"
Nov 01 03:28:42.793707 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:42.778142    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c94ffb2b-3e93-4898-97e8-1de6587c38a7" containerName="istio-proxy"
Nov 01 03:28:42.793707 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:42.778171    4400 cpu_manager.go:414] "containerMap: removing container" podUID="ca8197ec-0240-48b6-ae32-c5ad205f9d2d" containerName="istio-proxy"
Nov 01 03:28:42.793707 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:42.778178    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca8197ec-0240-48b6-ae32-c5ad205f9d2d" containerName="istio-proxy"
Nov 01 03:28:42.793707 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:42.778243    4400 cpu_manager.go:414] "containerMap: removing container" podUID="9ace3b5b-e63a-495f-960e-0d47df550f59" containerName="test-curl-7066-9e5918"
Nov 01 03:28:42.793707 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:42.778251    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ace3b5b-e63a-495f-960e-0d47df550f59" containerName="test-curl-7066-9e5918"
Nov 01 03:28:42.793707 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:42.778389    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ace3b5b-e63a-495f-960e-0d47df550f59" containerName="test-curl-7066-9e5918"
Nov 01 03:28:42.793707 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:42.778402    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="c94ffb2b-3e93-4898-97e8-1de6587c38a7" containerName="istio-proxy"
Nov 01 03:28:42.793707 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:42.778415    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca8197ec-0240-48b6-ae32-c5ad205f9d2d" containerName="istio-proxy"
Nov 01 03:28:42.793707 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:42.785043    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hf2dc"
Nov 01 03:28:42.843078 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:42.843049    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51938745-c46e-4917-b963-5d1f98b0298d-workload-certs\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hf2dc\" (UID: \"51938745-c46e-4917-b963-5d1f98b0298d\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hf2dc"
Nov 01 03:28:42.843260 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:42.843085    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51938745-c46e-4917-b963-5d1f98b0298d-istio-data\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hf2dc\" (UID: \"51938745-c46e-4917-b963-5d1f98b0298d\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hf2dc"
Nov 01 03:28:42.843260 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:42.843128    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51938745-c46e-4917-b963-5d1f98b0298d-credential-socket\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hf2dc\" (UID: \"51938745-c46e-4917-b963-5d1f98b0298d\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hf2dc"
Nov 01 03:28:42.843260 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:42.843218    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97r95\" (UniqueName: \"kubernetes.io/projected/51938745-c46e-4917-b963-5d1f98b0298d-kube-api-access-97r95\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hf2dc\" (UID: \"51938745-c46e-4917-b963-5d1f98b0298d\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hf2dc"
Nov 01 03:28:42.843446 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:42.843377    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51938745-c46e-4917-b963-5d1f98b0298d-istio-envoy\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hf2dc\" (UID: \"51938745-c46e-4917-b963-5d1f98b0298d\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hf2dc"
Nov 01 03:28:42.843446 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:42.843404    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51938745-c46e-4917-b963-5d1f98b0298d-istio-podinfo\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hf2dc\" (UID: \"51938745-c46e-4917-b963-5d1f98b0298d\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hf2dc"
Nov 01 03:28:42.843549 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:42.843526    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51938745-c46e-4917-b963-5d1f98b0298d-istio-token\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hf2dc\" (UID: \"51938745-c46e-4917-b963-5d1f98b0298d\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hf2dc"
Nov 01 03:28:42.843610 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:42.843552    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51938745-c46e-4917-b963-5d1f98b0298d-workload-socket\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hf2dc\" (UID: \"51938745-c46e-4917-b963-5d1f98b0298d\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hf2dc"
Nov 01 03:28:42.843610 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:42.843580    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51938745-c46e-4917-b963-5d1f98b0298d-istiod-ca-cert\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hf2dc\" (UID: \"51938745-c46e-4917-b963-5d1f98b0298d\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hf2dc"
Nov 01 03:28:42.897399 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:42.897368    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hf2dc"]
Nov 01 03:28:42.945094 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:42.945065    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51938745-c46e-4917-b963-5d1f98b0298d-istio-envoy\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hf2dc\" (UID: \"51938745-c46e-4917-b963-5d1f98b0298d\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hf2dc"
Nov 01 03:28:42.945241 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:42.945112    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51938745-c46e-4917-b963-5d1f98b0298d-istio-podinfo\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hf2dc\" (UID: \"51938745-c46e-4917-b963-5d1f98b0298d\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hf2dc"
Nov 01 03:28:42.945241 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:42.945235    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51938745-c46e-4917-b963-5d1f98b0298d-istio-token\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hf2dc\" (UID: \"51938745-c46e-4917-b963-5d1f98b0298d\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hf2dc"
Nov 01 03:28:42.945384 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:42.945267    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51938745-c46e-4917-b963-5d1f98b0298d-workload-socket\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hf2dc\" (UID: \"51938745-c46e-4917-b963-5d1f98b0298d\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hf2dc"
Nov 01 03:28:42.945384 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:42.945308    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51938745-c46e-4917-b963-5d1f98b0298d-istiod-ca-cert\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hf2dc\" (UID: \"51938745-c46e-4917-b963-5d1f98b0298d\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hf2dc"
Nov 01 03:28:42.945384 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:42.945373    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51938745-c46e-4917-b963-5d1f98b0298d-workload-certs\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hf2dc\" (UID: \"51938745-c46e-4917-b963-5d1f98b0298d\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hf2dc"
Nov 01 03:28:42.945539 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:42.945406    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51938745-c46e-4917-b963-5d1f98b0298d-istio-data\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hf2dc\" (UID: \"51938745-c46e-4917-b963-5d1f98b0298d\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hf2dc"
Nov 01 03:28:42.945539 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:42.945462    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51938745-c46e-4917-b963-5d1f98b0298d-credential-socket\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hf2dc\" (UID: \"51938745-c46e-4917-b963-5d1f98b0298d\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hf2dc"
Nov 01 03:28:42.945539 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:42.945503    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97r95\" (UniqueName: \"kubernetes.io/projected/51938745-c46e-4917-b963-5d1f98b0298d-kube-api-access-97r95\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hf2dc\" (UID: \"51938745-c46e-4917-b963-5d1f98b0298d\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hf2dc"
Nov 01 03:28:42.945755 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:42.945727    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51938745-c46e-4917-b963-5d1f98b0298d-workload-socket\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hf2dc\" (UID: \"51938745-c46e-4917-b963-5d1f98b0298d\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hf2dc"
Nov 01 03:28:42.945899 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:42.945873    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51938745-c46e-4917-b963-5d1f98b0298d-credential-socket\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hf2dc\" (UID: \"51938745-c46e-4917-b963-5d1f98b0298d\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hf2dc"
Nov 01 03:28:42.946007 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:42.945925    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51938745-c46e-4917-b963-5d1f98b0298d-istio-data\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hf2dc\" (UID: \"51938745-c46e-4917-b963-5d1f98b0298d\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hf2dc"
Nov 01 03:28:42.946069 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:42.946049    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51938745-c46e-4917-b963-5d1f98b0298d-workload-certs\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hf2dc\" (UID: \"51938745-c46e-4917-b963-5d1f98b0298d\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hf2dc"
Nov 01 03:28:42.946163 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:42.946145    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51938745-c46e-4917-b963-5d1f98b0298d-istiod-ca-cert\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hf2dc\" (UID: \"51938745-c46e-4917-b963-5d1f98b0298d\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hf2dc"
Nov 01 03:28:42.947733 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:42.947712    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51938745-c46e-4917-b963-5d1f98b0298d-istio-podinfo\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hf2dc\" (UID: \"51938745-c46e-4917-b963-5d1f98b0298d\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hf2dc"
Nov 01 03:28:42.947945 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:42.947928    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51938745-c46e-4917-b963-5d1f98b0298d-istio-envoy\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hf2dc\" (UID: \"51938745-c46e-4917-b963-5d1f98b0298d\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hf2dc"
Nov 01 03:28:43.010038 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:43.009992    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51938745-c46e-4917-b963-5d1f98b0298d-istio-token\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hf2dc\" (UID: \"51938745-c46e-4917-b963-5d1f98b0298d\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hf2dc"
Nov 01 03:28:43.010821 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:43.010660    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97r95\" (UniqueName: \"kubernetes.io/projected/51938745-c46e-4917-b963-5d1f98b0298d-kube-api-access-97r95\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hf2dc\" (UID: \"51938745-c46e-4917-b963-5d1f98b0298d\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hf2dc"
Nov 01 03:28:43.104088 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:43.104062    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hf2dc"
Nov 01 03:28:43.427293 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:43.427215    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-pbw4s"]
Nov 01 03:28:43.432745 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:43.432722    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-pbw4s"
Nov 01 03:28:43.453941 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:43.453920    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82b7e135-bf31-448d-ae07-f8b859f15420-istiod-ca-cert\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-pbw4s\" (UID: \"82b7e135-bf31-448d-ae07-f8b859f15420\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-pbw4s"
Nov 01 03:28:43.454060 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:43.453956    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2kgf\" (UniqueName: \"kubernetes.io/projected/82b7e135-bf31-448d-ae07-f8b859f15420-kube-api-access-k2kgf\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-pbw4s\" (UID: \"82b7e135-bf31-448d-ae07-f8b859f15420\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-pbw4s"
Nov 01 03:28:43.454099 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:43.454067    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82b7e135-bf31-448d-ae07-f8b859f15420-workload-certs\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-pbw4s\" (UID: \"82b7e135-bf31-448d-ae07-f8b859f15420\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-pbw4s"
Nov 01 03:28:43.454135 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:43.454116    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82b7e135-bf31-448d-ae07-f8b859f15420-istio-envoy\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-pbw4s\" (UID: \"82b7e135-bf31-448d-ae07-f8b859f15420\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-pbw4s"
Nov 01 03:28:43.454346 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:43.454321    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82b7e135-bf31-448d-ae07-f8b859f15420-istio-data\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-pbw4s\" (UID: \"82b7e135-bf31-448d-ae07-f8b859f15420\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-pbw4s"
Nov 01 03:28:43.454395 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:43.454379    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82b7e135-bf31-448d-ae07-f8b859f15420-istio-podinfo\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-pbw4s\" (UID: \"82b7e135-bf31-448d-ae07-f8b859f15420\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-pbw4s"
Nov 01 03:28:43.454514 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:43.454496    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82b7e135-bf31-448d-ae07-f8b859f15420-credential-socket\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-pbw4s\" (UID: \"82b7e135-bf31-448d-ae07-f8b859f15420\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-pbw4s"
Nov 01 03:28:43.454554 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:43.454537    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82b7e135-bf31-448d-ae07-f8b859f15420-istio-token\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-pbw4s\" (UID: \"82b7e135-bf31-448d-ae07-f8b859f15420\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-pbw4s"
Nov 01 03:28:43.454611 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:43.454599    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82b7e135-bf31-448d-ae07-f8b859f15420-workload-socket\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-pbw4s\" (UID: \"82b7e135-bf31-448d-ae07-f8b859f15420\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-pbw4s"
Nov 01 03:28:43.467466 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:43.467442    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-bpgj8"]
Nov 01 03:28:43.471579 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:43.471561    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-bpgj8"
Nov 01 03:28:43.498832 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:43.498811    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-pbw4s"]
Nov 01 03:28:43.538579 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:43.538552    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-bpgj8"]
Nov 01 03:28:43.556061 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:43.556040    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82b7e135-bf31-448d-ae07-f8b859f15420-istio-data\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-pbw4s\" (UID: \"82b7e135-bf31-448d-ae07-f8b859f15420\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-pbw4s"
Nov 01 03:28:43.556233 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:43.556098    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82b7e135-bf31-448d-ae07-f8b859f15420-istio-podinfo\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-pbw4s\" (UID: \"82b7e135-bf31-448d-ae07-f8b859f15420\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-pbw4s"
Nov 01 03:28:43.556233 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:43.556128    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0-workload-socket\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-bpgj8\" (UID: \"bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-bpgj8"
Nov 01 03:28:43.556233 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:43.556205    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82b7e135-bf31-448d-ae07-f8b859f15420-credential-socket\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-pbw4s\" (UID: \"82b7e135-bf31-448d-ae07-f8b859f15420\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-pbw4s"
Nov 01 03:28:43.556413 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:43.556241    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0-istio-token\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-bpgj8\" (UID: \"bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-bpgj8"
Nov 01 03:28:43.556413 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:43.556323    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0-istio-data\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-bpgj8\" (UID: \"bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-bpgj8"
Nov 01 03:28:43.556546 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:43.556432    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82b7e135-bf31-448d-ae07-f8b859f15420-istio-token\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-pbw4s\" (UID: \"82b7e135-bf31-448d-ae07-f8b859f15420\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-pbw4s"
Nov 01 03:28:43.556546 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:43.556482    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82b7e135-bf31-448d-ae07-f8b859f15420-istio-data\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-pbw4s\" (UID: \"82b7e135-bf31-448d-ae07-f8b859f15420\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-pbw4s"
Nov 01 03:28:43.556546 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:43.556491    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82b7e135-bf31-448d-ae07-f8b859f15420-workload-socket\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-pbw4s\" (UID: \"82b7e135-bf31-448d-ae07-f8b859f15420\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-pbw4s"
Nov 01 03:28:43.556713 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:43.556572    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8s86\" (UniqueName: \"kubernetes.io/projected/bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0-kube-api-access-v8s86\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-bpgj8\" (UID: \"bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-bpgj8"
Nov 01 03:28:43.556713 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:43.556604    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82b7e135-bf31-448d-ae07-f8b859f15420-credential-socket\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-pbw4s\" (UID: \"82b7e135-bf31-448d-ae07-f8b859f15420\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-pbw4s"
Nov 01 03:28:43.556713 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:43.556641    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0-credential-socket\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-bpgj8\" (UID: \"bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-bpgj8"
Nov 01 03:28:43.556713 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:43.556708    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0-istio-envoy\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-bpgj8\" (UID: \"bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-bpgj8"
Nov 01 03:28:43.557113 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:43.556775    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82b7e135-bf31-448d-ae07-f8b859f15420-istiod-ca-cert\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-pbw4s\" (UID: \"82b7e135-bf31-448d-ae07-f8b859f15420\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-pbw4s"
Nov 01 03:28:43.557113 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:43.556801    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0-workload-certs\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-bpgj8\" (UID: \"bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-bpgj8"
Nov 01 03:28:43.557113 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:43.556826    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82b7e135-bf31-448d-ae07-f8b859f15420-workload-socket\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-pbw4s\" (UID: \"82b7e135-bf31-448d-ae07-f8b859f15420\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-pbw4s"
Nov 01 03:28:43.557113 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:43.556961    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2kgf\" (UniqueName: \"kubernetes.io/projected/82b7e135-bf31-448d-ae07-f8b859f15420-kube-api-access-k2kgf\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-pbw4s\" (UID: \"82b7e135-bf31-448d-ae07-f8b859f15420\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-pbw4s"
Nov 01 03:28:43.557113 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:43.556989    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82b7e135-bf31-448d-ae07-f8b859f15420-workload-certs\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-pbw4s\" (UID: \"82b7e135-bf31-448d-ae07-f8b859f15420\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-pbw4s"
Nov 01 03:28:43.557113 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:43.557022    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82b7e135-bf31-448d-ae07-f8b859f15420-istio-envoy\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-pbw4s\" (UID: \"82b7e135-bf31-448d-ae07-f8b859f15420\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-pbw4s"
Nov 01 03:28:43.557113 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:43.557051    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0-istio-podinfo\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-bpgj8\" (UID: \"bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-bpgj8"
Nov 01 03:28:43.557113 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:43.557094    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0-istiod-ca-cert\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-bpgj8\" (UID: \"bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-bpgj8"
Nov 01 03:28:43.557546 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:43.557480    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82b7e135-bf31-448d-ae07-f8b859f15420-istiod-ca-cert\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-pbw4s\" (UID: \"82b7e135-bf31-448d-ae07-f8b859f15420\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-pbw4s"
Nov 01 03:28:43.557601 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:43.557557    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82b7e135-bf31-448d-ae07-f8b859f15420-workload-certs\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-pbw4s\" (UID: \"82b7e135-bf31-448d-ae07-f8b859f15420\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-pbw4s"
Nov 01 03:28:43.559112 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:43.559091    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82b7e135-bf31-448d-ae07-f8b859f15420-istio-podinfo\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-pbw4s\" (UID: \"82b7e135-bf31-448d-ae07-f8b859f15420\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-pbw4s"
Nov 01 03:28:43.559664 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:43.559640    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82b7e135-bf31-448d-ae07-f8b859f15420-istio-envoy\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-pbw4s\" (UID: \"82b7e135-bf31-448d-ae07-f8b859f15420\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-pbw4s"
Nov 01 03:28:43.594120 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:28:43.594095    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51938745_c46e_4917_b963_5d1f98b0298d.slice/crio-8e7fd7e6acd965a5f3af5d0be21d77d43aa4bca947958ca3270e9b2ffb241459 WatchSource:0}: Error finding container 8e7fd7e6acd965a5f3af5d0be21d77d43aa4bca947958ca3270e9b2ffb241459: Status 404 returned error can't find the container with id 8e7fd7e6acd965a5f3af5d0be21d77d43aa4bca947958ca3270e9b2ffb241459
Nov 01 03:28:43.594292 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:43.594275    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hf2dc"]
Nov 01 03:28:43.605702 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:43.597117    4400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 03:28:43.605702 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:43.597267    4400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 03:28:43.605702 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:43.597350    4400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 03:28:43.620555 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:43.620528    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2kgf\" (UniqueName: \"kubernetes.io/projected/82b7e135-bf31-448d-ae07-f8b859f15420-kube-api-access-k2kgf\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-pbw4s\" (UID: \"82b7e135-bf31-448d-ae07-f8b859f15420\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-pbw4s"
Nov 01 03:28:43.630351 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:43.630325    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82b7e135-bf31-448d-ae07-f8b859f15420-istio-token\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-pbw4s\" (UID: \"82b7e135-bf31-448d-ae07-f8b859f15420\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-pbw4s"
Nov 01 03:28:43.658338 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:43.658310    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0-workload-socket\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-bpgj8\" (UID: \"bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-bpgj8"
Nov 01 03:28:43.658447 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:43.658424    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0-istio-token\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-bpgj8\" (UID: \"bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-bpgj8"
Nov 01 03:28:43.658517 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:43.658460    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0-istio-data\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-bpgj8\" (UID: \"bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-bpgj8"
Nov 01 03:28:43.658573 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:43.658514    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8s86\" (UniqueName: \"kubernetes.io/projected/bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0-kube-api-access-v8s86\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-bpgj8\" (UID: \"bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-bpgj8"
Nov 01 03:28:43.658628 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:43.658578    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0-credential-socket\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-bpgj8\" (UID: \"bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-bpgj8"
Nov 01 03:28:43.658628 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:43.658604    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0-istio-envoy\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-bpgj8\" (UID: \"bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-bpgj8"
Nov 01 03:28:43.658761 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:43.658628    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0-workload-socket\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-bpgj8\" (UID: \"bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-bpgj8"
Nov 01 03:28:43.658761 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:43.658714    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0-workload-certs\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-bpgj8\" (UID: \"bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-bpgj8"
Nov 01 03:28:43.658923 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:43.658837    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0-istio-podinfo\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-bpgj8\" (UID: \"bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-bpgj8"
Nov 01 03:28:43.658923 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:43.658909    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0-istio-data\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-bpgj8\" (UID: \"bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-bpgj8"
Nov 01 03:28:43.659022 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:43.658948    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0-credential-socket\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-bpgj8\" (UID: \"bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-bpgj8"
Nov 01 03:28:43.659022 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:43.659013    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0-istiod-ca-cert\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-bpgj8\" (UID: \"bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-bpgj8"
Nov 01 03:28:43.659090 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:43.659050    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0-workload-certs\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-bpgj8\" (UID: \"bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-bpgj8"
Nov 01 03:28:43.659616 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:43.659595    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0-istiod-ca-cert\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-bpgj8\" (UID: \"bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-bpgj8"
Nov 01 03:28:43.660998 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:43.660973    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0-istio-envoy\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-bpgj8\" (UID: \"bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-bpgj8"
Nov 01 03:28:43.661520 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:43.661503    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0-istio-podinfo\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-bpgj8\" (UID: \"bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-bpgj8"
Nov 01 03:28:43.724636 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:43.724601    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0-istio-token\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-bpgj8\" (UID: \"bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-bpgj8"
Nov 01 03:28:43.725240 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:43.725212    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8s86\" (UniqueName: \"kubernetes.io/projected/bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0-kube-api-access-v8s86\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-bpgj8\" (UID: \"bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-bpgj8"
Nov 01 03:28:43.728589 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:43.728554    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hf2dc" event={"ID":"51938745-c46e-4917-b963-5d1f98b0298d","Type":"ContainerStarted","Data":"6cff9759bad6097d625173d1c685932d01aeea4f36bb735534b7df550af291a8"}
Nov 01 03:28:43.728708 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:43.728623    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hf2dc" event={"ID":"51938745-c46e-4917-b963-5d1f98b0298d","Type":"ContainerStarted","Data":"8e7fd7e6acd965a5f3af5d0be21d77d43aa4bca947958ca3270e9b2ffb241459"}
Nov 01 03:28:43.748527 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:43.748499    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-pbw4s"
Nov 01 03:28:43.784934 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:43.784903    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-bpgj8"
Nov 01 03:28:43.865905 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:43.864925    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hf2dc" podStartSLOduration=1.8649052739999998 podStartE2EDuration="1.864905274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:28:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:28:43.863028588 +0000 UTC m=+1330.377674453" watchObservedRunningTime="2026-11-01 03:28:43.864905274 +0000 UTC m=+1330.379551140"
Nov 01 03:28:44.105286 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:44.105256    4400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hf2dc"
Nov 01 03:28:44.106583 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:44.106558    4400 patch_prober.go:29] interesting pod/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hf2dc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Nov 01 03:28:44.106744 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:44.106610    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hf2dc" podUID="51938745-c46e-4917-b963-5d1f98b0298d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Nov 01 03:28:44.140136 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:44.140106    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-pbw4s"]
Nov 01 03:28:44.140784 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:28:44.140752    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82b7e135_bf31_448d_ae07_f8b859f15420.slice/crio-f57894cca6b7d92496bf12533edf0b528f87a7b4caa94c0c395887a3c4ea390b WatchSource:0}: Error finding container f57894cca6b7d92496bf12533edf0b528f87a7b4caa94c0c395887a3c4ea390b: Status 404 returned error can't find the container with id f57894cca6b7d92496bf12533edf0b528f87a7b4caa94c0c395887a3c4ea390b
Nov 01 03:28:44.143643 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:44.143612    4400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 03:28:44.143767 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:44.143676    4400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 03:28:44.143767 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:44.143729    4400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 03:28:44.207816 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:44.203463    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-bpgj8"]
Nov 01 03:28:44.239025 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:28:44.238996    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdfd43b2_b7ab_4d1a_a31e_5ebbd8ad50d0.slice/crio-bc60717e987690c1bb2aedd0576c7fce97ba2ff99bd686c7064c69c6f404cfa4 WatchSource:0}: Error finding container bc60717e987690c1bb2aedd0576c7fce97ba2ff99bd686c7064c69c6f404cfa4: Status 404 returned error can't find the container with id bc60717e987690c1bb2aedd0576c7fce97ba2ff99bd686c7064c69c6f404cfa4
Nov 01 03:28:44.242751 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:44.242672    4400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 03:28:44.242906 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:44.242782    4400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 03:28:44.242906 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:44.242824    4400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 03:28:44.745677 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:44.745590    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-pbw4s" event={"ID":"82b7e135-bf31-448d-ae07-f8b859f15420","Type":"ContainerStarted","Data":"79be6af514c393018879b4f1dbd9f898188f176d545aa6bcccadbd8b3683c44f"}
Nov 01 03:28:44.745677 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:44.745655    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-pbw4s" event={"ID":"82b7e135-bf31-448d-ae07-f8b859f15420","Type":"ContainerStarted","Data":"f57894cca6b7d92496bf12533edf0b528f87a7b4caa94c0c395887a3c4ea390b"}
Nov 01 03:28:44.747496 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:44.747469    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-bpgj8" event={"ID":"bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0","Type":"ContainerStarted","Data":"19377fb2c7b55ba8ad1a164d262d13e684c7ab57434d86ed7c4f7f2309ff9c23"}
Nov 01 03:28:44.747625 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:44.747519    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-bpgj8" event={"ID":"bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0","Type":"ContainerStarted","Data":"bc60717e987690c1bb2aedd0576c7fce97ba2ff99bd686c7064c69c6f404cfa4"}
Nov 01 03:28:44.748846 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:44.748831    4400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-pbw4s"
Nov 01 03:28:44.785308 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:44.785282    4400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-bpgj8"
Nov 01 03:28:44.872359 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:44.872304    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-pbw4s" podStartSLOduration=1.872284976 podStartE2EDuration="1.872284976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:28:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:28:44.869030816 +0000 UTC m=+1331.383676686" watchObservedRunningTime="2026-11-01 03:28:44.872284976 +0000 UTC m=+1331.386930842"
Nov 01 03:28:45.031771 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:45.023066    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-bpgj8" podStartSLOduration=2.023042433 podStartE2EDuration="2.023042433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:28:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:28:44.998489337 +0000 UTC m=+1331.513135201" watchObservedRunningTime="2026-11-01 03:28:45.023042433 +0000 UTC m=+1331.537688298"
Nov 01 03:28:45.124709 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:45.124620    4400 patch_prober.go:29] interesting pod/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hf2dc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Nov 01 03:28:45.124709 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:45.124695    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hf2dc" podUID="51938745-c46e-4917-b963-5d1f98b0298d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Nov 01 03:28:45.783796 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:45.783756    4400 patch_prober.go:29] interesting pod/e2e-worker-w4-e8c789-openshift-default-774d997df-pbw4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Nov 01 03:28:45.784000 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:45.783832    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-pbw4s" podUID="82b7e135-bf31-448d-ae07-f8b859f15420" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Nov 01 03:28:45.788436 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:45.788377    4400 patch_prober.go:29] interesting pod/e2e-worker-w4-e8c789-openshift-default-774d997df-bpgj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Nov 01 03:28:45.788599 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:45.788447    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-bpgj8" podUID="bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Nov 01 03:28:46.109711 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:46.107172    4400 patch_prober.go:29] interesting pod/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hf2dc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Nov 01 03:28:46.109711 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:46.107228    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hf2dc" podUID="51938745-c46e-4917-b963-5d1f98b0298d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Nov 01 03:28:46.757763 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:46.751146    4400 patch_prober.go:29] interesting pod/e2e-worker-w4-e8c789-openshift-default-774d997df-pbw4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Nov 01 03:28:46.757763 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:46.751205    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-pbw4s" podUID="82b7e135-bf31-448d-ae07-f8b859f15420" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Nov 01 03:28:46.906708 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:46.905780    4400 patch_prober.go:29] interesting pod/e2e-worker-w4-e8c789-openshift-default-774d997df-bpgj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Nov 01 03:28:47.192873 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:46.905860    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-bpgj8" podUID="bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Nov 01 03:28:47.192873 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:47.133128    4400 patch_prober.go:29] interesting pod/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hf2dc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Nov 01 03:28:47.192873 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:47.162234    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hf2dc" podUID="51938745-c46e-4917-b963-5d1f98b0298d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Nov 01 03:28:47.783894 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:47.760765    4400 patch_prober.go:29] interesting pod/e2e-worker-w4-e8c789-openshift-default-774d997df-pbw4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Nov 01 03:28:47.783894 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:47.760822    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-pbw4s" podUID="82b7e135-bf31-448d-ae07-f8b859f15420" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Nov 01 03:28:47.800895 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:47.800860    4400 patch_prober.go:29] interesting pod/e2e-worker-w4-e8c789-openshift-default-774d997df-bpgj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Nov 01 03:28:47.801175 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:47.801146    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-bpgj8" podUID="bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Nov 01 03:28:47.831996 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:47.831965    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d5c42938-5b46444559-n968k"]
Nov 01 03:28:47.881915 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:47.881884    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d5c42938-5b46444559-n968k"]
Nov 01 03:28:47.882132 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:47.882117    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d5c42938-6c6bf65c67-lqdcx"]
Nov 01 03:28:47.895717 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:47.890304    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d5c42938-5b46444559-n968k"
Nov 01 03:28:47.907907 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:47.907878    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d5c42938-6c6bf65c67-lqdcx"]
Nov 01 03:28:47.925741 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:47.908928    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d5c42938-6c6bf65c67-lqdcx"
Nov 01 03:28:47.955813 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:47.955787    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-d5c42938\""
Nov 01 03:28:48.002357 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:48.002322    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71e81e40-e46d-4981-a403-d9a74bdc5957-plugins-config-volume\") pod \"payload-processing-e2e-ait-d5c42938-5b46444559-n968k\" (UID: \"71e81e40-e46d-4981-a403-d9a74bdc5957\") " pod="openshift-ingress/payload-processing-e2e-ait-d5c42938-5b46444559-n968k"
Nov 01 03:28:48.025812 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:48.025782    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c264853-8d3b-4e19-8872-224f7aee939c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d5c42938-6c6bf65c67-lqdcx\" (UID: \"0c264853-8d3b-4e19-8872-224f7aee939c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d5c42938-6c6bf65c67-lqdcx"
Nov 01 03:28:48.026200 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:48.026177    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n224h\" (UniqueName: \"kubernetes.io/projected/0c264853-8d3b-4e19-8872-224f7aee939c-kube-api-access-n224h\") pod \"payload-pre-processing-e2e-ait-d5c42938-6c6bf65c67-lqdcx\" (UID: \"0c264853-8d3b-4e19-8872-224f7aee939c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d5c42938-6c6bf65c67-lqdcx"
Nov 01 03:28:48.026397 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:48.026379    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbt2w\" (UniqueName: \"kubernetes.io/projected/71e81e40-e46d-4981-a403-d9a74bdc5957-kube-api-access-zbt2w\") pod \"payload-processing-e2e-ait-d5c42938-5b46444559-n968k\" (UID: \"71e81e40-e46d-4981-a403-d9a74bdc5957\") " pod="openshift-ingress/payload-processing-e2e-ait-d5c42938-5b46444559-n968k"
Nov 01 03:28:48.026620 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:48.005875    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-d5c42938-dockercfg-ds65k\""
Nov 01 03:28:48.116710 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:48.114851    4400 patch_prober.go:29] interesting pod/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hf2dc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Nov 01 03:28:48.116710 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:48.114919    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hf2dc" podUID="51938745-c46e-4917-b963-5d1f98b0298d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Nov 01 03:28:48.127653 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:48.127615    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c264853-8d3b-4e19-8872-224f7aee939c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d5c42938-6c6bf65c67-lqdcx\" (UID: \"0c264853-8d3b-4e19-8872-224f7aee939c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d5c42938-6c6bf65c67-lqdcx"
Nov 01 03:28:48.129055 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:48.129020    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c264853-8d3b-4e19-8872-224f7aee939c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d5c42938-6c6bf65c67-lqdcx\" (UID: \"0c264853-8d3b-4e19-8872-224f7aee939c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d5c42938-6c6bf65c67-lqdcx"
Nov 01 03:28:48.129244 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:48.129039    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n224h\" (UniqueName: \"kubernetes.io/projected/0c264853-8d3b-4e19-8872-224f7aee939c-kube-api-access-n224h\") pod \"payload-pre-processing-e2e-ait-d5c42938-6c6bf65c67-lqdcx\" (UID: \"0c264853-8d3b-4e19-8872-224f7aee939c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d5c42938-6c6bf65c67-lqdcx"
Nov 01 03:28:48.129775 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:48.129750    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbt2w\" (UniqueName: \"kubernetes.io/projected/71e81e40-e46d-4981-a403-d9a74bdc5957-kube-api-access-zbt2w\") pod \"payload-processing-e2e-ait-d5c42938-5b46444559-n968k\" (UID: \"71e81e40-e46d-4981-a403-d9a74bdc5957\") " pod="openshift-ingress/payload-processing-e2e-ait-d5c42938-5b46444559-n968k"
Nov 01 03:28:48.130247 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:48.130226    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71e81e40-e46d-4981-a403-d9a74bdc5957-plugins-config-volume\") pod \"payload-processing-e2e-ait-d5c42938-5b46444559-n968k\" (UID: \"71e81e40-e46d-4981-a403-d9a74bdc5957\") " pod="openshift-ingress/payload-processing-e2e-ait-d5c42938-5b46444559-n968k"
Nov 01 03:28:48.137714 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:48.131781    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71e81e40-e46d-4981-a403-d9a74bdc5957-plugins-config-volume\") pod \"payload-processing-e2e-ait-d5c42938-5b46444559-n968k\" (UID: \"71e81e40-e46d-4981-a403-d9a74bdc5957\") " pod="openshift-ingress/payload-processing-e2e-ait-d5c42938-5b46444559-n968k"
Nov 01 03:28:48.197594 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:48.197439    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n224h\" (UniqueName: \"kubernetes.io/projected/0c264853-8d3b-4e19-8872-224f7aee939c-kube-api-access-n224h\") pod \"payload-pre-processing-e2e-ait-d5c42938-6c6bf65c67-lqdcx\" (UID: \"0c264853-8d3b-4e19-8872-224f7aee939c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d5c42938-6c6bf65c67-lqdcx"
Nov 01 03:28:48.198320 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:48.198072    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbt2w\" (UniqueName: \"kubernetes.io/projected/71e81e40-e46d-4981-a403-d9a74bdc5957-kube-api-access-zbt2w\") pod \"payload-processing-e2e-ait-d5c42938-5b46444559-n968k\" (UID: \"71e81e40-e46d-4981-a403-d9a74bdc5957\") " pod="openshift-ingress/payload-processing-e2e-ait-d5c42938-5b46444559-n968k"
Nov 01 03:28:48.316269 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:48.316236    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d5c42938-5b46444559-n968k"
Nov 01 03:28:48.362007 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:48.361975    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d5c42938-6c6bf65c67-lqdcx"
Nov 01 03:28:48.771770 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:48.757526    4400 patch_prober.go:29] interesting pod/e2e-worker-w4-e8c789-openshift-default-774d997df-pbw4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Nov 01 03:28:48.771770 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:48.757587    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-pbw4s" podUID="82b7e135-bf31-448d-ae07-f8b859f15420" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Nov 01 03:28:48.833407 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:48.831805    4400 patch_prober.go:29] interesting pod/e2e-worker-w4-e8c789-openshift-default-774d997df-bpgj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Nov 01 03:28:48.833407 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:48.831871    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-bpgj8" podUID="bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Nov 01 03:28:49.178711 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:49.144658    4400 patch_prober.go:29] interesting pod/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hf2dc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Nov 01 03:28:49.178711 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:49.174990    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hf2dc" podUID="51938745-c46e-4917-b963-5d1f98b0298d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Nov 01 03:28:49.498950 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:49.497524    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d5c42938-6c6bf65c67-lqdcx"]
Nov 01 03:28:49.801090 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:49.801051    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d5c42938-5b46444559-n968k"]
Nov 01 03:28:49.805712 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:49.801831    4400 patch_prober.go:29] interesting pod/e2e-worker-w4-e8c789-openshift-default-774d997df-bpgj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Nov 01 03:28:49.805712 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:49.801886    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-bpgj8" podUID="bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Nov 01 03:28:49.890190 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:49.890145    4400 patch_prober.go:29] interesting pod/e2e-worker-w4-e8c789-openshift-default-774d997df-pbw4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Nov 01 03:28:49.890292 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:49.890215    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-pbw4s" podUID="82b7e135-bf31-448d-ae07-f8b859f15420" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Nov 01 03:28:50.201753 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:50.201716    4400 patch_prober.go:29] interesting pod/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hf2dc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Nov 01 03:28:50.202076 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:50.202030    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hf2dc" podUID="51938745-c46e-4917-b963-5d1f98b0298d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Nov 01 03:28:50.519443 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:50.465949    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7911e5a0-68f8dd6b5c-jrj6z"]
Nov 01 03:28:50.622121 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:50.622087    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7911e5a0-68f8dd6b5c-jrj6z"
Nov 01 03:28:50.708710 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:50.708066    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71f320ef-9f70-41a4-887e-413230d65992-plugins-config-volume\") pod \"payload-processing-e2e-ait-7911e5a0-68f8dd6b5c-jrj6z\" (UID: \"71f320ef-9f70-41a4-887e-413230d65992\") " pod="openshift-ingress/payload-processing-e2e-ait-7911e5a0-68f8dd6b5c-jrj6z"
Nov 01 03:28:50.708710 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:50.708214    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghbbc\" (UniqueName: \"kubernetes.io/projected/71f320ef-9f70-41a4-887e-413230d65992-kube-api-access-ghbbc\") pod \"payload-processing-e2e-ait-7911e5a0-68f8dd6b5c-jrj6z\" (UID: \"71f320ef-9f70-41a4-887e-413230d65992\") " pod="openshift-ingress/payload-processing-e2e-ait-7911e5a0-68f8dd6b5c-jrj6z"
Nov 01 03:28:50.801568 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:50.801535    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-7911e5a0-dockercfg-rpbrl\""
Nov 01 03:28:50.822577 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:50.814624    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-7911e5a0\""
Nov 01 03:28:50.822805 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:50.815677    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71f320ef-9f70-41a4-887e-413230d65992-plugins-config-volume\") pod \"payload-processing-e2e-ait-7911e5a0-68f8dd6b5c-jrj6z\" (UID: \"71f320ef-9f70-41a4-887e-413230d65992\") " pod="openshift-ingress/payload-processing-e2e-ait-7911e5a0-68f8dd6b5c-jrj6z"
Nov 01 03:28:50.823044 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:50.823023    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghbbc\" (UniqueName: \"kubernetes.io/projected/71f320ef-9f70-41a4-887e-413230d65992-kube-api-access-ghbbc\") pod \"payload-processing-e2e-ait-7911e5a0-68f8dd6b5c-jrj6z\" (UID: \"71f320ef-9f70-41a4-887e-413230d65992\") " pod="openshift-ingress/payload-processing-e2e-ait-7911e5a0-68f8dd6b5c-jrj6z"
Nov 01 03:28:50.823775 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:50.816647    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71f320ef-9f70-41a4-887e-413230d65992-plugins-config-volume\") pod \"payload-processing-e2e-ait-7911e5a0-68f8dd6b5c-jrj6z\" (UID: \"71f320ef-9f70-41a4-887e-413230d65992\") " pod="openshift-ingress/payload-processing-e2e-ait-7911e5a0-68f8dd6b5c-jrj6z"
Nov 01 03:28:50.893706 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:50.824850    4400 patch_prober.go:29] interesting pod/e2e-worker-w4-e8c789-openshift-default-774d997df-bpgj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Nov 01 03:28:50.893706 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:50.824900    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-bpgj8" podUID="bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Nov 01 03:28:50.893706 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:50.880936    4400 patch_prober.go:29] interesting pod/e2e-worker-w4-e8c789-openshift-default-774d997df-pbw4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Nov 01 03:28:50.893706 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:50.880990    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-pbw4s" podUID="82b7e135-bf31-448d-ae07-f8b859f15420" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Nov 01 03:28:51.003314 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:51.003223    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e8c789/distinct-2-w4-e8c789-kserve-7977c64f4-wkxb2" podUID="bdb1bb9e-14f5-4be6-a028-c980119f4aa8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:28:51.078903 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:51.070731    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7911e5a0-68f8dd6b5c-jrj6z"]
Nov 01 03:28:51.204786 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:51.202898    4400 patch_prober.go:29] interesting pod/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hf2dc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Nov 01 03:28:51.204786 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:51.202962    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hf2dc" podUID="51938745-c46e-4917-b963-5d1f98b0298d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Nov 01 03:28:51.208790 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:51.208758    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghbbc\" (UniqueName: \"kubernetes.io/projected/71f320ef-9f70-41a4-887e-413230d65992-kube-api-access-ghbbc\") pod \"payload-processing-e2e-ait-7911e5a0-68f8dd6b5c-jrj6z\" (UID: \"71f320ef-9f70-41a4-887e-413230d65992\") " pod="openshift-ingress/payload-processing-e2e-ait-7911e5a0-68f8dd6b5c-jrj6z"
Nov 01 03:28:51.209245 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:51.209221    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d5c42938-6c6bf65c67-lqdcx" event={"ID":"0c264853-8d3b-4e19-8872-224f7aee939c","Type":"ContainerStarted","Data":"68d624ce390342eae619c6ac68c815e625408abf528480dbd626b7d1d5765ad2"}
Nov 01 03:28:51.209394 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:51.209382    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d5c42938-6c6bf65c67-lqdcx" event={"ID":"0c264853-8d3b-4e19-8872-224f7aee939c","Type":"ContainerStarted","Data":"7ea26ee582432ab4a9c1a5eb24f324834bf314f8904945faa52d4c0f9892deb7"}
Nov 01 03:28:51.209781 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:51.209723    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-d5c42938-6c6bf65c67-lqdcx"
Nov 01 03:28:51.213313 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:51.213262    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d5c42938-5b46444559-n968k" event={"ID":"71e81e40-e46d-4981-a403-d9a74bdc5957","Type":"ContainerStarted","Data":"7b21b37e95147db2c03bca6480f7afd3d1188040e5691a26cf0214a15421cf3b"}
Nov 01 03:28:51.221316 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:51.221287    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-w5-c22dd8-kserve-5c65c9bcd7-nnf7d" podUID="9cd440c6-7191-4fae-a806-f8728975b3cf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:28:51.314758 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:51.307045    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7911e5a0-68f8dd6b5c-jrj6z"
Nov 01 03:28:51.378767 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:51.359599    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-d5c42938-6c6bf65c67-lqdcx" podStartSLOduration=4.359560307 podStartE2EDuration="4.359560307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:28:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:28:51.337511722 +0000 UTC m=+1337.852157587" watchObservedRunningTime="2026-11-01 03:28:51.359560307 +0000 UTC m=+1337.874206171"
Nov 01 03:28:51.752525 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:51.752058    4400 patch_prober.go:29] interesting pod/e2e-worker-w4-e8c789-openshift-default-774d997df-pbw4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Nov 01 03:28:51.752525 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:51.752131    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-pbw4s" podUID="82b7e135-bf31-448d-ae07-f8b859f15420" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Nov 01 03:28:51.787914 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:51.785627    4400 patch_prober.go:29] interesting pod/e2e-worker-w4-e8c789-openshift-default-774d997df-bpgj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Nov 01 03:28:51.787914 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:51.785735    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-bpgj8" podUID="bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Nov 01 03:28:52.081445 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:52.081406    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7911e5a0-68f8dd6b5c-jrj6z"]
Nov 01 03:28:52.083064 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:28:52.082976    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71f320ef_9f70_41a4_887e_413230d65992.slice/crio-30efa34d9bb8c47e126dd49fa17aba5017574ef1f7bf379c5655cc0567477fea WatchSource:0}: Error finding container 30efa34d9bb8c47e126dd49fa17aba5017574ef1f7bf379c5655cc0567477fea: Status 404 returned error can't find the container with id 30efa34d9bb8c47e126dd49fa17aba5017574ef1f7bf379c5655cc0567477fea
Nov 01 03:28:52.105048 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:52.105022    4400 patch_prober.go:29] interesting pod/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hf2dc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Nov 01 03:28:52.105227 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:52.105203    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hf2dc" podUID="51938745-c46e-4917-b963-5d1f98b0298d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Nov 01 03:28:52.226729 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:52.226522    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7911e5a0-68f8dd6b5c-jrj6z" event={"ID":"71f320ef-9f70-41a4-887e-413230d65992","Type":"ContainerStarted","Data":"319cd592c45077cae76ed97bf3586b1261a73a2fb059bde8c4173ce3eca8ea7b"}
Nov 01 03:28:52.226729 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:52.226593    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7911e5a0-68f8dd6b5c-jrj6z" event={"ID":"71f320ef-9f70-41a4-887e-413230d65992","Type":"ContainerStarted","Data":"30efa34d9bb8c47e126dd49fa17aba5017574ef1f7bf379c5655cc0567477fea"}
Nov 01 03:28:52.227722 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:52.227657    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-7911e5a0-68f8dd6b5c-jrj6z"
Nov 01 03:28:52.229902 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:52.229876    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d5c42938-5b46444559-n968k" event={"ID":"71e81e40-e46d-4981-a403-d9a74bdc5957","Type":"ContainerStarted","Data":"cc3be2b7f6478296d2523e159b6c60903a6999c9669ee3ca8caf55403710a7e1"}
Nov 01 03:28:52.230020 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:52.229947    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-d5c42938-5b46444559-n968k"
Nov 01 03:28:52.353004 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:52.352896    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-7911e5a0-68f8dd6b5c-jrj6z" podStartSLOduration=2.352877612 podStartE2EDuration="2.352877612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:28:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:28:52.346069132 +0000 UTC m=+1338.860714994" watchObservedRunningTime="2026-11-01 03:28:52.352877612 +0000 UTC m=+1338.867523477"
Nov 01 03:28:52.756044 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:52.755501    4400 patch_prober.go:29] interesting pod/e2e-worker-w4-e8c789-openshift-default-774d997df-pbw4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Nov 01 03:28:52.756044 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:52.755560    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-pbw4s" podUID="82b7e135-bf31-448d-ae07-f8b859f15420" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Nov 01 03:28:52.805729 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:52.795092    4400 patch_prober.go:29] interesting pod/e2e-worker-w4-e8c789-openshift-default-774d997df-bpgj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Nov 01 03:28:52.805729 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:52.795153    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-bpgj8" podUID="bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Nov 01 03:28:53.573379 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:53.573346    4400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hf2dc"
Nov 01 03:28:53.574569 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:53.574537    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hf2dc"
Nov 01 03:28:53.576496 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:53.576473    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hf2dc"
Nov 01 03:28:53.773867 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:53.773801    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-d5c42938-5b46444559-n968k" podStartSLOduration=6.773782594 podStartE2EDuration="6.773782594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:28:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:28:52.473979361 +0000 UTC m=+1338.988625223" watchObservedRunningTime="2026-11-01 03:28:53.773782594 +0000 UTC m=+1340.288428456"
Nov 01 03:28:53.918777 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:53.914930    4400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-bpgj8"
Nov 01 03:28:53.918777 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:53.916661    4400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-pbw4s"
Nov 01 03:28:53.918777 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:53.916807    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-bpgj8"
Nov 01 03:28:53.918777 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:53.917187    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-pbw4s"
Nov 01 03:28:53.918777 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:53.918182    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-bpgj8"
Nov 01 03:28:53.918777 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:28:53.918235    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-pbw4s"
Nov 01 03:29:00.569709 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:00.566322    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e6e7ea12-55895cd555-6g2bl"]
Nov 01 03:29:00.572895 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:00.572868    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e6e7ea12-55895cd555-6g2bl"
Nov 01 03:29:00.671722 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:00.670632    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e6e7ea12-55895cd555-6g2bl"]
Nov 01 03:29:00.679189 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:00.679154    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-e6e7ea12\""
Nov 01 03:29:00.679496 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:00.679479    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-e6e7ea12-dockercfg-8j2gc\""
Nov 01 03:29:00.751767 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:00.739605    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pwzs\" (UniqueName: \"kubernetes.io/projected/d7e99137-7930-47f4-9f70-06f8cbd811cd-kube-api-access-8pwzs\") pod \"payload-processing-e2e-ait-e6e7ea12-55895cd555-6g2bl\" (UID: \"d7e99137-7930-47f4-9f70-06f8cbd811cd\") " pod="openshift-ingress/payload-processing-e2e-ait-e6e7ea12-55895cd555-6g2bl"
Nov 01 03:29:00.752125 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:00.752089    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7e99137-7930-47f4-9f70-06f8cbd811cd-plugins-config-volume\") pod \"payload-processing-e2e-ait-e6e7ea12-55895cd555-6g2bl\" (UID: \"d7e99137-7930-47f4-9f70-06f8cbd811cd\") " pod="openshift-ingress/payload-processing-e2e-ait-e6e7ea12-55895cd555-6g2bl"
Nov 01 03:29:00.854159 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:00.854122    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pwzs\" (UniqueName: \"kubernetes.io/projected/d7e99137-7930-47f4-9f70-06f8cbd811cd-kube-api-access-8pwzs\") pod \"payload-processing-e2e-ait-e6e7ea12-55895cd555-6g2bl\" (UID: \"d7e99137-7930-47f4-9f70-06f8cbd811cd\") " pod="openshift-ingress/payload-processing-e2e-ait-e6e7ea12-55895cd555-6g2bl"
Nov 01 03:29:00.854342 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:00.854185    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7e99137-7930-47f4-9f70-06f8cbd811cd-plugins-config-volume\") pod \"payload-processing-e2e-ait-e6e7ea12-55895cd555-6g2bl\" (UID: \"d7e99137-7930-47f4-9f70-06f8cbd811cd\") " pod="openshift-ingress/payload-processing-e2e-ait-e6e7ea12-55895cd555-6g2bl"
Nov 01 03:29:00.855239 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:00.855210    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7e99137-7930-47f4-9f70-06f8cbd811cd-plugins-config-volume\") pod \"payload-processing-e2e-ait-e6e7ea12-55895cd555-6g2bl\" (UID: \"d7e99137-7930-47f4-9f70-06f8cbd811cd\") " pod="openshift-ingress/payload-processing-e2e-ait-e6e7ea12-55895cd555-6g2bl"
Nov 01 03:29:00.916637 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:00.916614    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pwzs\" (UniqueName: \"kubernetes.io/projected/d7e99137-7930-47f4-9f70-06f8cbd811cd-kube-api-access-8pwzs\") pod \"payload-processing-e2e-ait-e6e7ea12-55895cd555-6g2bl\" (UID: \"d7e99137-7930-47f4-9f70-06f8cbd811cd\") " pod="openshift-ingress/payload-processing-e2e-ait-e6e7ea12-55895cd555-6g2bl"
Nov 01 03:29:01.209641 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:01.209572    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e6e7ea12-55895cd555-6g2bl"
Nov 01 03:29:01.592577 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:29:01.592539    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7e99137_7930_47f4_9f70_06f8cbd811cd.slice/crio-396b4a7385129a9767d674dedc9c4c40abcebe88025a43760870fbe72e7e8c33 WatchSource:0}: Error finding container 396b4a7385129a9767d674dedc9c4c40abcebe88025a43760870fbe72e7e8c33: Status 404 returned error can't find the container with id 396b4a7385129a9767d674dedc9c4c40abcebe88025a43760870fbe72e7e8c33
Nov 01 03:29:01.592979 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:01.592673    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e6e7ea12-55895cd555-6g2bl"]
Nov 01 03:29:01.730401 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:29:01.730368    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:29:02.232648 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:02.232624    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-d5c42938-6c6bf65c67-lqdcx"
Nov 01 03:29:02.476123 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:02.476083    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e6e7ea12-55895cd555-6g2bl" event={"ID":"d7e99137-7930-47f4-9f70-06f8cbd811cd","Type":"ContainerStarted","Data":"dd6618575471879e4fb749b557e155f70a393df4e91eeefab6fc749d4dfc7e9f"}
Nov 01 03:29:02.476333 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:02.476175    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e6e7ea12-55895cd555-6g2bl" event={"ID":"d7e99137-7930-47f4-9f70-06f8cbd811cd","Type":"ContainerStarted","Data":"396b4a7385129a9767d674dedc9c4c40abcebe88025a43760870fbe72e7e8c33"}
Nov 01 03:29:02.476414 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:02.476384    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-e6e7ea12-55895cd555-6g2bl"
Nov 01 03:29:02.596621 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:02.596575    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-e6e7ea12-55895cd555-6g2bl" podStartSLOduration=2.5965627209999997 podStartE2EDuration="2.596562721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:29:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:29:02.595552322 +0000 UTC m=+1349.110198196" watchObservedRunningTime="2026-11-01 03:29:02.596562721 +0000 UTC m=+1349.111208586"
Nov 01 03:29:03.358845 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:03.358817    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-d5c42938-5b46444559-n968k"
Nov 01 03:29:04.366071 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:04.366038    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-7911e5a0-68f8dd6b5c-jrj6z"
Nov 01 03:29:06.861299 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:06.861258    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 03:29:10.251810 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:10.251761    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/distinct-2-w6-19faa5-kserve-58db847577-hrzlw" podUID="062959ee-ec7c-4385-98d5-cf4a240852e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:29:10.433710 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:10.408761    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e8c789/premium-w4-e8c789-kserve-9458f8874-dtpnm" podUID="67bba91a-5ace-4d3f-89da-06bb9db17c78" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:29:10.550767 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:10.542439    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-2-w2-c8cb9a-kserve-5c5cfb454d-fh2cj" podUID="2bcf3b85-3d4a-433f-94ad-0e954bf03e73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:29:10.760742 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:10.759540    4400 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-e8c789/premium-w4-e8c789-kserve-9458f8874-dtpnm" podUID="67bba91a-5ace-4d3f-89da-06bb9db17c78" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:29:11.036757 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:11.019287    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e8c789/distinct-2-w4-e8c789-kserve-7977c64f4-wkxb2" podUID="bdb1bb9e-14f5-4be6-a028-c980119f4aa8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:29:11.079827 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:11.079778    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-w5-c22dd8-kserve-5c65c9bcd7-nnf7d" podUID="9cd440c6-7191-4fae-a806-f8728975b3cf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:29:11.271705 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:11.271641    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/distinct-2-w6-19faa5-kserve-58db847577-hrzlw" podUID="062959ee-ec7c-4385-98d5-cf4a240852e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:29:11.272594 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:11.272459    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-19faa5/distinct-2-w6-19faa5-kserve-58db847577-hrzlw"
Nov 01 03:29:11.458944 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:11.458904    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e8c789/premium-w4-e8c789-kserve-9458f8874-dtpnm" podUID="67bba91a-5ace-4d3f-89da-06bb9db17c78" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Nov 01 03:29:11.464720 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:11.459826    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e8c789/premium-w4-e8c789-kserve-9458f8874-dtpnm"
Nov 01 03:29:11.687672 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:11.680353    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-2-w2-c8cb9a-kserve-5c5cfb454d-fh2cj" podUID="2bcf3b85-3d4a-433f-94ad-0e954bf03e73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:29:11.960709 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:11.947793    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-2-w2-c8cb9a-kserve-5c5cfb454d-fh2cj"
Nov 01 03:29:12.007711 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:12.007151    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-19faa5/distinct-2-w6-19faa5-kserve-58db847577-hrzlw"
Nov 01 03:29:12.071719 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:12.069900    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e8c789/distinct-2-w4-e8c789-kserve-7977c64f4-wkxb2" podUID="bdb1bb9e-14f5-4be6-a028-c980119f4aa8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:29:12.071719 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:12.070566    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e8c789/distinct-2-w4-e8c789-kserve-7977c64f4-wkxb2"
Nov 01 03:29:12.082826 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:12.082787    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-w5-c22dd8-kserve-5c65c9bcd7-nnf7d" podUID="9cd440c6-7191-4fae-a806-f8728975b3cf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:29:12.083714 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:12.083665    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-w5-c22dd8-kserve-5c65c9bcd7-nnf7d"
Nov 01 03:29:12.215800 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:12.215717    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e8c789/premium-w4-e8c789-kserve-9458f8874-dtpnm"
Nov 01 03:29:12.336225 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:12.336182    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-2-w2-c8cb9a-kserve-5c5cfb454d-fh2cj"
Nov 01 03:29:12.338885 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:12.338317    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e8c789/distinct-2-w4-e8c789-kserve-7977c64f4-wkxb2"
Nov 01 03:29:12.355990 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:12.355968    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-w5-c22dd8-kserve-5c65c9bcd7-nnf7d"
Nov 01 03:29:13.486934 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:13.486908    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-e6e7ea12-55895cd555-6g2bl"
Nov 01 03:29:14.193767 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:14.193737    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-44379c05-68dbc8565f-dg4s6"]
Nov 01 03:29:14.199363 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:14.199344    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-44379c05-68dbc8565f-dg4s6"
Nov 01 03:29:14.254964 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:14.254938    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-44379c05-68dbc8565f-dg4s6"]
Nov 01 03:29:14.301783 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:14.301749    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-44379c05\""
Nov 01 03:29:14.301783 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:14.301759    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-44379c05-dockercfg-wvshn\""
Nov 01 03:29:14.316200 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:14.316176    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lv2bm\" (UniqueName: \"kubernetes.io/projected/19fdf16d-cb31-448c-a99a-2d81d14622ab-kube-api-access-lv2bm\") pod \"payload-pre-processing-e2e-mt-44379c05-68dbc8565f-dg4s6\" (UID: \"19fdf16d-cb31-448c-a99a-2d81d14622ab\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-44379c05-68dbc8565f-dg4s6"
Nov 01 03:29:14.316357 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:14.316239    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19fdf16d-cb31-448c-a99a-2d81d14622ab-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-44379c05-68dbc8565f-dg4s6\" (UID: \"19fdf16d-cb31-448c-a99a-2d81d14622ab\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-44379c05-68dbc8565f-dg4s6"
Nov 01 03:29:14.368910 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:14.368886    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-44379c05-f9696f457-b2k6w"]
Nov 01 03:29:14.372077 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:14.372061    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-44379c05-f9696f457-b2k6w"
Nov 01 03:29:14.417879 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:14.417832    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lv2bm\" (UniqueName: \"kubernetes.io/projected/19fdf16d-cb31-448c-a99a-2d81d14622ab-kube-api-access-lv2bm\") pod \"payload-pre-processing-e2e-mt-44379c05-68dbc8565f-dg4s6\" (UID: \"19fdf16d-cb31-448c-a99a-2d81d14622ab\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-44379c05-68dbc8565f-dg4s6"
Nov 01 03:29:14.418536 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:14.418508    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19fdf16d-cb31-448c-a99a-2d81d14622ab-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-44379c05-68dbc8565f-dg4s6\" (UID: \"19fdf16d-cb31-448c-a99a-2d81d14622ab\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-44379c05-68dbc8565f-dg4s6"
Nov 01 03:29:14.420773 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:14.420204    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19fdf16d-cb31-448c-a99a-2d81d14622ab-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-44379c05-68dbc8565f-dg4s6\" (UID: \"19fdf16d-cb31-448c-a99a-2d81d14622ab\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-44379c05-68dbc8565f-dg4s6"
Nov 01 03:29:14.421199 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:14.421182    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68faceb9-2a06-4170-b200-37c4133ea373-plugins-config-volume\") pod \"payload-processing-e2e-mt-44379c05-f9696f457-b2k6w\" (UID: \"68faceb9-2a06-4170-b200-37c4133ea373\") " pod="openshift-ingress/payload-processing-e2e-mt-44379c05-f9696f457-b2k6w"
Nov 01 03:29:14.421699 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:14.421662    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c27gd\" (UniqueName: \"kubernetes.io/projected/68faceb9-2a06-4170-b200-37c4133ea373-kube-api-access-c27gd\") pod \"payload-processing-e2e-mt-44379c05-f9696f457-b2k6w\" (UID: \"68faceb9-2a06-4170-b200-37c4133ea373\") " pod="openshift-ingress/payload-processing-e2e-mt-44379c05-f9696f457-b2k6w"
Nov 01 03:29:14.461092 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:14.460935    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-44379c05-f9696f457-b2k6w"]
Nov 01 03:29:14.554782 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:14.547564    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68faceb9-2a06-4170-b200-37c4133ea373-plugins-config-volume\") pod \"payload-processing-e2e-mt-44379c05-f9696f457-b2k6w\" (UID: \"68faceb9-2a06-4170-b200-37c4133ea373\") " pod="openshift-ingress/payload-processing-e2e-mt-44379c05-f9696f457-b2k6w"
Nov 01 03:29:14.554782 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:14.547633    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c27gd\" (UniqueName: \"kubernetes.io/projected/68faceb9-2a06-4170-b200-37c4133ea373-kube-api-access-c27gd\") pod \"payload-processing-e2e-mt-44379c05-f9696f457-b2k6w\" (UID: \"68faceb9-2a06-4170-b200-37c4133ea373\") " pod="openshift-ingress/payload-processing-e2e-mt-44379c05-f9696f457-b2k6w"
Nov 01 03:29:14.554782 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:14.550082    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68faceb9-2a06-4170-b200-37c4133ea373-plugins-config-volume\") pod \"payload-processing-e2e-mt-44379c05-f9696f457-b2k6w\" (UID: \"68faceb9-2a06-4170-b200-37c4133ea373\") " pod="openshift-ingress/payload-processing-e2e-mt-44379c05-f9696f457-b2k6w"
Nov 01 03:29:14.643741 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:14.643694    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lv2bm\" (UniqueName: \"kubernetes.io/projected/19fdf16d-cb31-448c-a99a-2d81d14622ab-kube-api-access-lv2bm\") pod \"payload-pre-processing-e2e-mt-44379c05-68dbc8565f-dg4s6\" (UID: \"19fdf16d-cb31-448c-a99a-2d81d14622ab\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-44379c05-68dbc8565f-dg4s6"
Nov 01 03:29:14.704513 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:14.704469    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c27gd\" (UniqueName: \"kubernetes.io/projected/68faceb9-2a06-4170-b200-37c4133ea373-kube-api-access-c27gd\") pod \"payload-processing-e2e-mt-44379c05-f9696f457-b2k6w\" (UID: \"68faceb9-2a06-4170-b200-37c4133ea373\") " pod="openshift-ingress/payload-processing-e2e-mt-44379c05-f9696f457-b2k6w"
Nov 01 03:29:14.815635 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:14.815560    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-44379c05-68dbc8565f-dg4s6"
Nov 01 03:29:14.999481 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:14.999451    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-44379c05-f9696f457-b2k6w"
Nov 01 03:29:15.195553 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:29:15.195520    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19fdf16d_cb31_448c_a99a_2d81d14622ab.slice/crio-84a6b56d70ab125a410e609ad87b15b44e3e698f778ffa62be666fc0127c04bb WatchSource:0}: Error finding container 84a6b56d70ab125a410e609ad87b15b44e3e698f778ffa62be666fc0127c04bb: Status 404 returned error can't find the container with id 84a6b56d70ab125a410e609ad87b15b44e3e698f778ffa62be666fc0127c04bb
Nov 01 03:29:15.196106 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:15.196040    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-44379c05-68dbc8565f-dg4s6"]
Nov 01 03:29:15.204281 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:15.204258    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-44379c05-68dbc8565f-dg4s6" event={"ID":"19fdf16d-cb31-448c-a99a-2d81d14622ab","Type":"ContainerStarted","Data":"84a6b56d70ab125a410e609ad87b15b44e3e698f778ffa62be666fc0127c04bb"}
Nov 01 03:29:15.378390 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:29:15.378364    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68faceb9_2a06_4170_b200_37c4133ea373.slice/crio-33213d0dc291ab238d015d0584e011796454f9c87287ff00244a33eae7c077fa WatchSource:0}: Error finding container 33213d0dc291ab238d015d0584e011796454f9c87287ff00244a33eae7c077fa: Status 404 returned error can't find the container with id 33213d0dc291ab238d015d0584e011796454f9c87287ff00244a33eae7c077fa
Nov 01 03:29:15.379878 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:15.379858    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-44379c05-f9696f457-b2k6w"]
Nov 01 03:29:16.213079 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:16.213031    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-44379c05-f9696f457-b2k6w" event={"ID":"68faceb9-2a06-4170-b200-37c4133ea373","Type":"ContainerStarted","Data":"85c9599de4bde60eee361a62a4d65acc7c7e65f9469b22f0595e3b8c9bf3402f"}
Nov 01 03:29:16.213484 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:16.213133    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-44379c05-f9696f457-b2k6w" event={"ID":"68faceb9-2a06-4170-b200-37c4133ea373","Type":"ContainerStarted","Data":"33213d0dc291ab238d015d0584e011796454f9c87287ff00244a33eae7c077fa"}
Nov 01 03:29:16.213484 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:16.213163    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-44379c05-f9696f457-b2k6w"
Nov 01 03:29:16.218950 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:16.215157    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-44379c05-68dbc8565f-dg4s6" event={"ID":"19fdf16d-cb31-448c-a99a-2d81d14622ab","Type":"ContainerStarted","Data":"bcfc68868f6f2a246925b16ee77e45b8e2cf519bf534618d57382b6b4c6abf26"}
Nov 01 03:29:16.218950 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:16.215903    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-44379c05-68dbc8565f-dg4s6"
Nov 01 03:29:16.333243 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:16.333201    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-44379c05-f9696f457-b2k6w" podStartSLOduration=2.333189176 podStartE2EDuration="2.333189176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:29:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:29:16.330404509 +0000 UTC m=+1362.845050371" watchObservedRunningTime="2026-11-01 03:29:16.333189176 +0000 UTC m=+1362.847835072"
Nov 01 03:29:25.904553 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:25.904487    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-44379c05-68dbc8565f-dg4s6" podStartSLOduration=11.904473356 podStartE2EDuration="11.904473356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:29:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:29:16.465370932 +0000 UTC m=+1362.980016806" watchObservedRunningTime="2026-11-01 03:29:25.904473356 +0000 UTC m=+1372.419119219"
Nov 01 03:29:25.907870 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:25.907843    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d5c42938-7d76f878dc-knnh7"]
Nov 01 03:29:25.912622 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:25.912602    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5c42938-7d76f878dc-knnh7"
Nov 01 03:29:25.971646 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:25.971613    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d5c42938-7d76f878dc-knnh7"]
Nov 01 03:29:26.001395 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:26.001362    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2ztl\" (UniqueName: \"kubernetes.io/projected/442765eb-e690-44fe-ba14-c627900cb8fc-kube-api-access-t2ztl\") pod \"maas-api-e2e-ait-d5c42938-7d76f878dc-knnh7\" (UID: \"442765eb-e690-44fe-ba14-c627900cb8fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5c42938-7d76f878dc-knnh7"
Nov 01 03:29:26.001563 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:26.001412    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/442765eb-e690-44fe-ba14-c627900cb8fc-openshift-service-ca\") pod \"maas-api-e2e-ait-d5c42938-7d76f878dc-knnh7\" (UID: \"442765eb-e690-44fe-ba14-c627900cb8fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5c42938-7d76f878dc-knnh7"
Nov 01 03:29:26.001673 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:26.001652    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/442765eb-e690-44fe-ba14-c627900cb8fc-maas-api-tls\") pod \"maas-api-e2e-ait-d5c42938-7d76f878dc-knnh7\" (UID: \"442765eb-e690-44fe-ba14-c627900cb8fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5c42938-7d76f878dc-knnh7"
Nov 01 03:29:26.001836 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:26.001800    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/442765eb-e690-44fe-ba14-c627900cb8fc-metrics-certs\") pod \"maas-api-e2e-ait-d5c42938-7d76f878dc-knnh7\" (UID: \"442765eb-e690-44fe-ba14-c627900cb8fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5c42938-7d76f878dc-knnh7"
Nov 01 03:29:26.017034 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:26.017010    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-d5c42938-serving-cert\""
Nov 01 03:29:26.103194 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:26.103157    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/442765eb-e690-44fe-ba14-c627900cb8fc-maas-api-tls\") pod \"maas-api-e2e-ait-d5c42938-7d76f878dc-knnh7\" (UID: \"442765eb-e690-44fe-ba14-c627900cb8fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5c42938-7d76f878dc-knnh7"
Nov 01 03:29:26.103361 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:26.103227    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/442765eb-e690-44fe-ba14-c627900cb8fc-metrics-certs\") pod \"maas-api-e2e-ait-d5c42938-7d76f878dc-knnh7\" (UID: \"442765eb-e690-44fe-ba14-c627900cb8fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5c42938-7d76f878dc-knnh7"
Nov 01 03:29:26.103361 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:29:26.103248    4400 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-d5c42938-serving-cert: secret "maas-api-e2e-ait-d5c42938-serving-cert" not found
Nov 01 03:29:26.103361 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:29:26.103300    4400 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/442765eb-e690-44fe-ba14-c627900cb8fc-maas-api-tls podName:442765eb-e690-44fe-ba14-c627900cb8fc nodeName:}" failed. No retries permitted until 2026-11-01 03:29:26.603286023 +0000 UTC m=+1373.117931865 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/442765eb-e690-44fe-ba14-c627900cb8fc-maas-api-tls") pod "maas-api-e2e-ait-d5c42938-7d76f878dc-knnh7" (UID: "442765eb-e690-44fe-ba14-c627900cb8fc") : secret "maas-api-e2e-ait-d5c42938-serving-cert" not found
Nov 01 03:29:26.103487 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:26.103400    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2ztl\" (UniqueName: \"kubernetes.io/projected/442765eb-e690-44fe-ba14-c627900cb8fc-kube-api-access-t2ztl\") pod \"maas-api-e2e-ait-d5c42938-7d76f878dc-knnh7\" (UID: \"442765eb-e690-44fe-ba14-c627900cb8fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5c42938-7d76f878dc-knnh7"
Nov 01 03:29:26.103487 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:26.103423    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/442765eb-e690-44fe-ba14-c627900cb8fc-openshift-service-ca\") pod \"maas-api-e2e-ait-d5c42938-7d76f878dc-knnh7\" (UID: \"442765eb-e690-44fe-ba14-c627900cb8fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5c42938-7d76f878dc-knnh7"
Nov 01 03:29:26.104108 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:26.104090    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/442765eb-e690-44fe-ba14-c627900cb8fc-openshift-service-ca\") pod \"maas-api-e2e-ait-d5c42938-7d76f878dc-knnh7\" (UID: \"442765eb-e690-44fe-ba14-c627900cb8fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5c42938-7d76f878dc-knnh7"
Nov 01 03:29:26.105929 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:26.105910    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/442765eb-e690-44fe-ba14-c627900cb8fc-metrics-certs\") pod \"maas-api-e2e-ait-d5c42938-7d76f878dc-knnh7\" (UID: \"442765eb-e690-44fe-ba14-c627900cb8fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5c42938-7d76f878dc-knnh7"
Nov 01 03:29:26.191762 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:26.191576    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2ztl\" (UniqueName: \"kubernetes.io/projected/442765eb-e690-44fe-ba14-c627900cb8fc-kube-api-access-t2ztl\") pod \"maas-api-e2e-ait-d5c42938-7d76f878dc-knnh7\" (UID: \"442765eb-e690-44fe-ba14-c627900cb8fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5c42938-7d76f878dc-knnh7"
Nov 01 03:29:26.628560 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:26.628520    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/442765eb-e690-44fe-ba14-c627900cb8fc-maas-api-tls\") pod \"maas-api-e2e-ait-d5c42938-7d76f878dc-knnh7\" (UID: \"442765eb-e690-44fe-ba14-c627900cb8fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5c42938-7d76f878dc-knnh7"
Nov 01 03:29:26.631019 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:26.630998    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/442765eb-e690-44fe-ba14-c627900cb8fc-maas-api-tls\") pod \"maas-api-e2e-ait-d5c42938-7d76f878dc-knnh7\" (UID: \"442765eb-e690-44fe-ba14-c627900cb8fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5c42938-7d76f878dc-knnh7"
Nov 01 03:29:26.828747 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:26.828675    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5c42938-7d76f878dc-knnh7"
Nov 01 03:29:27.212772 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:29:27.212745    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod442765eb_e690_44fe_ba14_c627900cb8fc.slice/crio-f3d4b11e532f677efa4ffdab7f78391c40f4724d744e642aefcdb324ce588b08 WatchSource:0}: Error finding container f3d4b11e532f677efa4ffdab7f78391c40f4724d744e642aefcdb324ce588b08: Status 404 returned error can't find the container with id f3d4b11e532f677efa4ffdab7f78391c40f4724d744e642aefcdb324ce588b08
Nov 01 03:29:27.213524 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:27.213497    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d5c42938-7d76f878dc-knnh7"]
Nov 01 03:29:27.227514 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:27.227495    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-44379c05-f9696f457-b2k6w"
Nov 01 03:29:27.322637 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:27.322599    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5c42938-7d76f878dc-knnh7" event={"ID":"442765eb-e690-44fe-ba14-c627900cb8fc","Type":"ContainerStarted","Data":"f3d4b11e532f677efa4ffdab7f78391c40f4724d744e642aefcdb324ce588b08"}
Nov 01 03:29:28.138526 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:28.138499    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hsfpq"]
Nov 01 03:29:28.145235 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:28.145214    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hsfpq"
Nov 01 03:29:28.235856 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:28.235811    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-44379c05-68dbc8565f-dg4s6"
Nov 01 03:29:28.257774 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:28.257738    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cb9c447-7466-4f2e-9449-d153b9b8ec2c-credential-socket\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hsfpq\" (UID: \"3cb9c447-7466-4f2e-9449-d153b9b8ec2c\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hsfpq"
Nov 01 03:29:28.257914 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:28.257805    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cb9c447-7466-4f2e-9449-d153b9b8ec2c-istio-token\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hsfpq\" (UID: \"3cb9c447-7466-4f2e-9449-d153b9b8ec2c\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hsfpq"
Nov 01 03:29:28.257914 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:28.257854    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cb9c447-7466-4f2e-9449-d153b9b8ec2c-istio-podinfo\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hsfpq\" (UID: \"3cb9c447-7466-4f2e-9449-d153b9b8ec2c\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hsfpq"
Nov 01 03:29:28.258043 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:28.257917    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cb9c447-7466-4f2e-9449-d153b9b8ec2c-istio-envoy\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hsfpq\" (UID: \"3cb9c447-7466-4f2e-9449-d153b9b8ec2c\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hsfpq"
Nov 01 03:29:28.258043 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:28.258018    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cb9c447-7466-4f2e-9449-d153b9b8ec2c-workload-socket\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hsfpq\" (UID: \"3cb9c447-7466-4f2e-9449-d153b9b8ec2c\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hsfpq"
Nov 01 03:29:28.258169 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:28.258102    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cb9c447-7466-4f2e-9449-d153b9b8ec2c-istiod-ca-cert\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hsfpq\" (UID: \"3cb9c447-7466-4f2e-9449-d153b9b8ec2c\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hsfpq"
Nov 01 03:29:28.258169 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:28.258133    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cb9c447-7466-4f2e-9449-d153b9b8ec2c-workload-certs\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hsfpq\" (UID: \"3cb9c447-7466-4f2e-9449-d153b9b8ec2c\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hsfpq"
Nov 01 03:29:28.258279 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:28.258206    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dz6ct\" (UniqueName: \"kubernetes.io/projected/3cb9c447-7466-4f2e-9449-d153b9b8ec2c-kube-api-access-dz6ct\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hsfpq\" (UID: \"3cb9c447-7466-4f2e-9449-d153b9b8ec2c\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hsfpq"
Nov 01 03:29:28.258279 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:28.258229    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cb9c447-7466-4f2e-9449-d153b9b8ec2c-istio-data\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hsfpq\" (UID: \"3cb9c447-7466-4f2e-9449-d153b9b8ec2c\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hsfpq"
Nov 01 03:29:28.332746 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:28.332701    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5c42938-7d76f878dc-knnh7" event={"ID":"442765eb-e690-44fe-ba14-c627900cb8fc","Type":"ContainerStarted","Data":"f4aeded30bad1d316ee72cbcc2ca08ac0cb35be28fa3ee2d24aba98730440f81"}
Nov 01 03:29:28.332888 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:28.332772    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5c42938-7d76f878dc-knnh7"
Nov 01 03:29:28.356347 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:28.356323    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hsfpq"]
Nov 01 03:29:28.359636 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:28.359612    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz6ct\" (UniqueName: \"kubernetes.io/projected/3cb9c447-7466-4f2e-9449-d153b9b8ec2c-kube-api-access-dz6ct\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hsfpq\" (UID: \"3cb9c447-7466-4f2e-9449-d153b9b8ec2c\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hsfpq"
Nov 01 03:29:28.359814 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:28.359660    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cb9c447-7466-4f2e-9449-d153b9b8ec2c-istio-data\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hsfpq\" (UID: \"3cb9c447-7466-4f2e-9449-d153b9b8ec2c\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hsfpq"
Nov 01 03:29:28.359814 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:28.359703    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cb9c447-7466-4f2e-9449-d153b9b8ec2c-credential-socket\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hsfpq\" (UID: \"3cb9c447-7466-4f2e-9449-d153b9b8ec2c\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hsfpq"
Nov 01 03:29:28.359814 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:28.359738    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cb9c447-7466-4f2e-9449-d153b9b8ec2c-istio-token\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hsfpq\" (UID: \"3cb9c447-7466-4f2e-9449-d153b9b8ec2c\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hsfpq"
Nov 01 03:29:28.359814 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:28.359775    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cb9c447-7466-4f2e-9449-d153b9b8ec2c-istio-podinfo\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hsfpq\" (UID: \"3cb9c447-7466-4f2e-9449-d153b9b8ec2c\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hsfpq"
Nov 01 03:29:28.359814 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:28.359814    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cb9c447-7466-4f2e-9449-d153b9b8ec2c-istio-envoy\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hsfpq\" (UID: \"3cb9c447-7466-4f2e-9449-d153b9b8ec2c\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hsfpq"
Nov 01 03:29:28.360093 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:28.359926    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cb9c447-7466-4f2e-9449-d153b9b8ec2c-workload-socket\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hsfpq\" (UID: \"3cb9c447-7466-4f2e-9449-d153b9b8ec2c\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hsfpq"
Nov 01 03:29:28.360093 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:28.360049    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cb9c447-7466-4f2e-9449-d153b9b8ec2c-istiod-ca-cert\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hsfpq\" (UID: \"3cb9c447-7466-4f2e-9449-d153b9b8ec2c\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hsfpq"
Nov 01 03:29:28.360093 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:28.360089    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cb9c447-7466-4f2e-9449-d153b9b8ec2c-workload-certs\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hsfpq\" (UID: \"3cb9c447-7466-4f2e-9449-d153b9b8ec2c\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hsfpq"
Nov 01 03:29:28.360244 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:28.360102    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cb9c447-7466-4f2e-9449-d153b9b8ec2c-istio-data\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hsfpq\" (UID: \"3cb9c447-7466-4f2e-9449-d153b9b8ec2c\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hsfpq"
Nov 01 03:29:28.360443 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:28.360357    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cb9c447-7466-4f2e-9449-d153b9b8ec2c-workload-certs\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hsfpq\" (UID: \"3cb9c447-7466-4f2e-9449-d153b9b8ec2c\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hsfpq"
Nov 01 03:29:28.360443 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:28.360460    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cb9c447-7466-4f2e-9449-d153b9b8ec2c-credential-socket\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hsfpq\" (UID: \"3cb9c447-7466-4f2e-9449-d153b9b8ec2c\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hsfpq"
Nov 01 03:29:28.361022 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:28.360967    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cb9c447-7466-4f2e-9449-d153b9b8ec2c-workload-socket\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hsfpq\" (UID: \"3cb9c447-7466-4f2e-9449-d153b9b8ec2c\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hsfpq"
Nov 01 03:29:28.361440 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:28.361421    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cb9c447-7466-4f2e-9449-d153b9b8ec2c-istiod-ca-cert\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hsfpq\" (UID: \"3cb9c447-7466-4f2e-9449-d153b9b8ec2c\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hsfpq"
Nov 01 03:29:28.362893 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:28.362869    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cb9c447-7466-4f2e-9449-d153b9b8ec2c-istio-envoy\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hsfpq\" (UID: \"3cb9c447-7466-4f2e-9449-d153b9b8ec2c\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hsfpq"
Nov 01 03:29:28.363014 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:28.362990    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cb9c447-7466-4f2e-9449-d153b9b8ec2c-istio-podinfo\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hsfpq\" (UID: \"3cb9c447-7466-4f2e-9449-d153b9b8ec2c\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hsfpq"
Nov 01 03:29:28.423184 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:28.423085    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dz6ct\" (UniqueName: \"kubernetes.io/projected/3cb9c447-7466-4f2e-9449-d153b9b8ec2c-kube-api-access-dz6ct\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hsfpq\" (UID: \"3cb9c447-7466-4f2e-9449-d153b9b8ec2c\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hsfpq"
Nov 01 03:29:28.424672 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:28.424604    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cb9c447-7466-4f2e-9449-d153b9b8ec2c-istio-token\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hsfpq\" (UID: \"3cb9c447-7466-4f2e-9449-d153b9b8ec2c\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hsfpq"
Nov 01 03:29:28.492171 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:28.492142    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hsfpq"
Nov 01 03:29:28.524307 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:28.524197    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5c42938-7d76f878dc-knnh7" podStartSLOduration=2.607697893 podStartE2EDuration="3.524177902s" totalImagesPullingTime="916.480009ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:29:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:29:28.516677179 +0000 UTC m=+1375.031323054" watchObservedRunningTime="2026-11-01 03:29:28.524177902 +0000 UTC m=+1375.038823767"
Nov 01 03:29:28.983050 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:28.983011    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hsfpq"]
Nov 01 03:29:29.033288 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:29:29.013855    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cb9c447_7466_4f2e_9449_d153b9b8ec2c.slice/crio-42e311494b0cbc676556b8ec9ba25d8bad9b8bf1ade2142eb04e147c1fb862a0 WatchSource:0}: Error finding container 42e311494b0cbc676556b8ec9ba25d8bad9b8bf1ade2142eb04e147c1fb862a0: Status 404 returned error can't find the container with id 42e311494b0cbc676556b8ec9ba25d8bad9b8bf1ade2142eb04e147c1fb862a0
Nov 01 03:29:29.050016 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:29.047012    4400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 03:29:29.050016 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:29.047129    4400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 03:29:29.050016 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:29.047170    4400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 03:29:29.345149 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:29.345110    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hsfpq" event={"ID":"3cb9c447-7466-4f2e-9449-d153b9b8ec2c","Type":"ContainerStarted","Data":"490c384a9a528bcb5a9456833fe3ada4aca2f2907dcaf15580536eb4ff4eb40c"}
Nov 01 03:29:29.345633 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:29.345176    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hsfpq" event={"ID":"3cb9c447-7466-4f2e-9449-d153b9b8ec2c","Type":"ContainerStarted","Data":"42e311494b0cbc676556b8ec9ba25d8bad9b8bf1ade2142eb04e147c1fb862a0"}
Nov 01 03:29:29.471047 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:29.470947    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hsfpq" podStartSLOduration=1.470931161 podStartE2EDuration="1.470931161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:29:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:29:29.470349663 +0000 UTC m=+1375.984995550" watchObservedRunningTime="2026-11-01 03:29:29.470931161 +0000 UTC m=+1375.985577025"
Nov 01 03:29:29.493297 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:29.493266    4400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hsfpq"
Nov 01 03:29:30.495063 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:30.495025    4400 patch_prober.go:29] interesting pod/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hsfpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Nov 01 03:29:30.495463 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:30.495080    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hsfpq" podUID="3cb9c447-7466-4f2e-9449-d153b9b8ec2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Nov 01 03:29:30.829855 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:30.829822    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7911e5a0-6dbc75fc4d-grqqm"]
Nov 01 03:29:30.834811 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:30.834792    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7911e5a0-6dbc75fc4d-grqqm"
Nov 01 03:29:30.890341 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:30.890310    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7911e5a0-6dbc75fc4d-grqqm"]
Nov 01 03:29:30.936372 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:30.936335    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-7911e5a0-serving-cert\""
Nov 01 03:29:30.961532 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:30.961503    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9569e45-00e1-4480-9335-6b41732856e9-maas-api-tls\") pod \"maas-api-e2e-ait-7911e5a0-6dbc75fc4d-grqqm\" (UID: \"c9569e45-00e1-4480-9335-6b41732856e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7911e5a0-6dbc75fc4d-grqqm"
Nov 01 03:29:30.961741 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:30.961590    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9569e45-00e1-4480-9335-6b41732856e9-metrics-certs\") pod \"maas-api-e2e-ait-7911e5a0-6dbc75fc4d-grqqm\" (UID: \"c9569e45-00e1-4480-9335-6b41732856e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7911e5a0-6dbc75fc4d-grqqm"
Nov 01 03:29:30.961741 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:30.961640    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9569e45-00e1-4480-9335-6b41732856e9-openshift-service-ca\") pod \"maas-api-e2e-ait-7911e5a0-6dbc75fc4d-grqqm\" (UID: \"c9569e45-00e1-4480-9335-6b41732856e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7911e5a0-6dbc75fc4d-grqqm"
Nov 01 03:29:30.961898 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:30.961816    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwl8v\" (UniqueName: \"kubernetes.io/projected/c9569e45-00e1-4480-9335-6b41732856e9-kube-api-access-dwl8v\") pod \"maas-api-e2e-ait-7911e5a0-6dbc75fc4d-grqqm\" (UID: \"c9569e45-00e1-4480-9335-6b41732856e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7911e5a0-6dbc75fc4d-grqqm"
Nov 01 03:29:31.063610 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:31.063579    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwl8v\" (UniqueName: \"kubernetes.io/projected/c9569e45-00e1-4480-9335-6b41732856e9-kube-api-access-dwl8v\") pod \"maas-api-e2e-ait-7911e5a0-6dbc75fc4d-grqqm\" (UID: \"c9569e45-00e1-4480-9335-6b41732856e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7911e5a0-6dbc75fc4d-grqqm"
Nov 01 03:29:31.063846 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:31.063662    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9569e45-00e1-4480-9335-6b41732856e9-maas-api-tls\") pod \"maas-api-e2e-ait-7911e5a0-6dbc75fc4d-grqqm\" (UID: \"c9569e45-00e1-4480-9335-6b41732856e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7911e5a0-6dbc75fc4d-grqqm"
Nov 01 03:29:31.063846 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:31.063733    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9569e45-00e1-4480-9335-6b41732856e9-metrics-certs\") pod \"maas-api-e2e-ait-7911e5a0-6dbc75fc4d-grqqm\" (UID: \"c9569e45-00e1-4480-9335-6b41732856e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7911e5a0-6dbc75fc4d-grqqm"
Nov 01 03:29:31.063846 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:31.063775    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9569e45-00e1-4480-9335-6b41732856e9-openshift-service-ca\") pod \"maas-api-e2e-ait-7911e5a0-6dbc75fc4d-grqqm\" (UID: \"c9569e45-00e1-4480-9335-6b41732856e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7911e5a0-6dbc75fc4d-grqqm"
Nov 01 03:29:31.064470 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:31.064447    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9569e45-00e1-4480-9335-6b41732856e9-openshift-service-ca\") pod \"maas-api-e2e-ait-7911e5a0-6dbc75fc4d-grqqm\" (UID: \"c9569e45-00e1-4480-9335-6b41732856e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7911e5a0-6dbc75fc4d-grqqm"
Nov 01 03:29:31.066332 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:31.066301    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9569e45-00e1-4480-9335-6b41732856e9-maas-api-tls\") pod \"maas-api-e2e-ait-7911e5a0-6dbc75fc4d-grqqm\" (UID: \"c9569e45-00e1-4480-9335-6b41732856e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7911e5a0-6dbc75fc4d-grqqm"
Nov 01 03:29:31.066437 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:31.066368    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9569e45-00e1-4480-9335-6b41732856e9-metrics-certs\") pod \"maas-api-e2e-ait-7911e5a0-6dbc75fc4d-grqqm\" (UID: \"c9569e45-00e1-4480-9335-6b41732856e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7911e5a0-6dbc75fc4d-grqqm"
Nov 01 03:29:31.133224 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:31.133155    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwl8v\" (UniqueName: \"kubernetes.io/projected/c9569e45-00e1-4480-9335-6b41732856e9-kube-api-access-dwl8v\") pod \"maas-api-e2e-ait-7911e5a0-6dbc75fc4d-grqqm\" (UID: \"c9569e45-00e1-4480-9335-6b41732856e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7911e5a0-6dbc75fc4d-grqqm"
Nov 01 03:29:31.150614 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:31.150575    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7911e5a0-6dbc75fc4d-grqqm"
Nov 01 03:29:31.503756 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:31.503243    4400 patch_prober.go:29] interesting pod/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hsfpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Nov 01 03:29:31.503756 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:31.503296    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hsfpq" podUID="3cb9c447-7466-4f2e-9449-d153b9b8ec2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Nov 01 03:29:31.755861 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:29:31.755777    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:29:31.789996 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:29:31.789966    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9569e45_00e1_4480_9335_6b41732856e9.slice/crio-487462f7aeea5026ce2c7ba7ed717c6f472e729b2cb1d58a536db92abe20dab9 WatchSource:0}: Error finding container 487462f7aeea5026ce2c7ba7ed717c6f472e729b2cb1d58a536db92abe20dab9: Status 404 returned error can't find the container with id 487462f7aeea5026ce2c7ba7ed717c6f472e729b2cb1d58a536db92abe20dab9
Nov 01 03:29:31.790412 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:31.790390    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7911e5a0-6dbc75fc4d-grqqm"]
Nov 01 03:29:32.493539 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:32.493499    4400 patch_prober.go:29] interesting pod/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hsfpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Nov 01 03:29:32.493776 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:32.493563    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hsfpq" podUID="3cb9c447-7466-4f2e-9449-d153b9b8ec2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Nov 01 03:29:32.532462 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:32.532423    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7911e5a0-6dbc75fc4d-grqqm" event={"ID":"c9569e45-00e1-4480-9335-6b41732856e9","Type":"ContainerStarted","Data":"487462f7aeea5026ce2c7ba7ed717c6f472e729b2cb1d58a536db92abe20dab9"}
Nov 01 03:29:33.493459 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:33.493414    4400 patch_prober.go:29] interesting pod/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hsfpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Nov 01 03:29:33.493642 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:33.493499    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hsfpq" podUID="3cb9c447-7466-4f2e-9449-d153b9b8ec2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Nov 01 03:29:33.550006 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:33.549973    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7911e5a0-6dbc75fc4d-grqqm" event={"ID":"c9569e45-00e1-4480-9335-6b41732856e9","Type":"ContainerStarted","Data":"800df96e28090adeb79cab13255b55959836dcf017de355d82ccf7df1e7a4aba"}
Nov 01 03:29:33.550482 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:33.550462    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7911e5a0-6dbc75fc4d-grqqm"
Nov 01 03:29:33.694234 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:33.694182    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7911e5a0-6dbc75fc4d-grqqm" podStartSLOduration=2.886085488 podStartE2EDuration="3.69416663s" totalImagesPullingTime="808.081142ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:29:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:29:33.69320711 +0000 UTC m=+1380.207852975" watchObservedRunningTime="2026-11-01 03:29:33.69416663 +0000 UTC m=+1380.208812511"
Nov 01 03:29:34.352421 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:34.352398    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5c42938-7d76f878dc-knnh7"
Nov 01 03:29:34.493149 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:34.493116    4400 patch_prober.go:29] interesting pod/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hsfpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Nov 01 03:29:34.493419 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:34.493395    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hsfpq" podUID="3cb9c447-7466-4f2e-9449-d153b9b8ec2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Nov 01 03:29:35.493404 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:35.493362    4400 patch_prober.go:29] interesting pod/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hsfpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Nov 01 03:29:35.493863 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:35.493427    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hsfpq" podUID="3cb9c447-7466-4f2e-9449-d153b9b8ec2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Nov 01 03:29:36.756567 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:36.756533    4400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hsfpq"
Nov 01 03:29:36.757333 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:36.757308    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hsfpq"
Nov 01 03:29:36.758070 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:36.758045    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hsfpq"
Nov 01 03:29:40.576093 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:40.576065    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7911e5a0-6dbc75fc4d-grqqm"
Nov 01 03:29:45.713806 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:45.713456    4400 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-19faa5/distinct-2-w6-19faa5-kserve-58db847577-hrzlw" podUID="062959ee-ec7c-4385-98d5-cf4a240852e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:29:46.370936 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:46.370875    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-w5-c22dd8-kserve-5c65c9bcd7-nnf7d" podUID="9cd440c6-7191-4fae-a806-f8728975b3cf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Nov 01 03:29:46.425866 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:46.425819    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-2-w2-c8cb9a-kserve-5c5cfb454d-fh2cj" podUID="2bcf3b85-3d4a-433f-94ad-0e954bf03e73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:29:46.426580 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:46.426544    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e8c789/distinct-2-w4-e8c789-kserve-7977c64f4-wkxb2" podUID="bdb1bb9e-14f5-4be6-a028-c980119f4aa8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded"
Nov 01 03:29:46.521720 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:46.518837    4400 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-d7f535/distinct-w3-d7f535-kserve-7d69fb9558-88lq8" podUID="65358d08-83e6-45c5-a17e-f1dbde164345" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:29:46.868051 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:46.865187    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7f535/distinct-w3-d7f535-kserve-7d69fb9558-88lq8" podUID="65358d08-83e6-45c5-a17e-f1dbde164345" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:29:46.931576 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:46.931537    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e6e7ea12-5fc4d966cf-fllxv"]
Nov 01 03:29:46.974750 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:46.940806    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e6e7ea12-5fc4d966cf-fllxv"
Nov 01 03:29:46.974750 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:46.962646    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5kh9\" (UniqueName: \"kubernetes.io/projected/1c79d2fd-b23e-486b-b49c-1c8e401a75a7-kube-api-access-h5kh9\") pod \"maas-api-e2e-ait-e6e7ea12-5fc4d966cf-fllxv\" (UID: \"1c79d2fd-b23e-486b-b49c-1c8e401a75a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e6e7ea12-5fc4d966cf-fllxv"
Nov 01 03:29:46.974750 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:46.962717    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c79d2fd-b23e-486b-b49c-1c8e401a75a7-maas-api-tls\") pod \"maas-api-e2e-ait-e6e7ea12-5fc4d966cf-fllxv\" (UID: \"1c79d2fd-b23e-486b-b49c-1c8e401a75a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e6e7ea12-5fc4d966cf-fllxv"
Nov 01 03:29:46.974750 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:46.962775    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c79d2fd-b23e-486b-b49c-1c8e401a75a7-metrics-certs\") pod \"maas-api-e2e-ait-e6e7ea12-5fc4d966cf-fllxv\" (UID: \"1c79d2fd-b23e-486b-b49c-1c8e401a75a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e6e7ea12-5fc4d966cf-fllxv"
Nov 01 03:29:46.974750 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:46.962923    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c79d2fd-b23e-486b-b49c-1c8e401a75a7-openshift-service-ca\") pod \"maas-api-e2e-ait-e6e7ea12-5fc4d966cf-fllxv\" (UID: \"1c79d2fd-b23e-486b-b49c-1c8e401a75a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e6e7ea12-5fc4d966cf-fllxv"
Nov 01 03:29:46.978564 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:46.978518    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e6e7ea12-5fc4d966cf-fllxv"]
Nov 01 03:29:47.031460 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:47.030754    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/distinct-2-w6-19faa5-kserve-58db847577-hrzlw" podUID="062959ee-ec7c-4385-98d5-cf4a240852e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:29:47.040236 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:47.040195    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-e6e7ea12-serving-cert\""
Nov 01 03:29:47.073153 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:47.064561    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5kh9\" (UniqueName: \"kubernetes.io/projected/1c79d2fd-b23e-486b-b49c-1c8e401a75a7-kube-api-access-h5kh9\") pod \"maas-api-e2e-ait-e6e7ea12-5fc4d966cf-fllxv\" (UID: \"1c79d2fd-b23e-486b-b49c-1c8e401a75a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e6e7ea12-5fc4d966cf-fllxv"
Nov 01 03:29:47.145933 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:47.145841    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c79d2fd-b23e-486b-b49c-1c8e401a75a7-maas-api-tls\") pod \"maas-api-e2e-ait-e6e7ea12-5fc4d966cf-fllxv\" (UID: \"1c79d2fd-b23e-486b-b49c-1c8e401a75a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e6e7ea12-5fc4d966cf-fllxv"
Nov 01 03:29:47.146219 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:47.146199    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c79d2fd-b23e-486b-b49c-1c8e401a75a7-metrics-certs\") pod \"maas-api-e2e-ait-e6e7ea12-5fc4d966cf-fllxv\" (UID: \"1c79d2fd-b23e-486b-b49c-1c8e401a75a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e6e7ea12-5fc4d966cf-fllxv"
Nov 01 03:29:47.146485 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:47.146463    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c79d2fd-b23e-486b-b49c-1c8e401a75a7-openshift-service-ca\") pod \"maas-api-e2e-ait-e6e7ea12-5fc4d966cf-fllxv\" (UID: \"1c79d2fd-b23e-486b-b49c-1c8e401a75a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e6e7ea12-5fc4d966cf-fllxv"
Nov 01 03:29:47.148007 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:29:47.147984    4400 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-e6e7ea12-serving-cert: secret "maas-api-e2e-ait-e6e7ea12-serving-cert" not found
Nov 01 03:29:47.158435 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:29:47.148200    4400 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c79d2fd-b23e-486b-b49c-1c8e401a75a7-maas-api-tls podName:1c79d2fd-b23e-486b-b49c-1c8e401a75a7 nodeName:}" failed. No retries permitted until 2026-11-01 03:29:47.648176829 +0000 UTC m=+1394.162822673 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1c79d2fd-b23e-486b-b49c-1c8e401a75a7-maas-api-tls") pod "maas-api-e2e-ait-e6e7ea12-5fc4d966cf-fllxv" (UID: "1c79d2fd-b23e-486b-b49c-1c8e401a75a7") : secret "maas-api-e2e-ait-e6e7ea12-serving-cert" not found
Nov 01 03:29:47.160628 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:47.160599    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c79d2fd-b23e-486b-b49c-1c8e401a75a7-openshift-service-ca\") pod \"maas-api-e2e-ait-e6e7ea12-5fc4d966cf-fllxv\" (UID: \"1c79d2fd-b23e-486b-b49c-1c8e401a75a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e6e7ea12-5fc4d966cf-fllxv"
Nov 01 03:29:47.176888 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:47.174236    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5kh9\" (UniqueName: \"kubernetes.io/projected/1c79d2fd-b23e-486b-b49c-1c8e401a75a7-kube-api-access-h5kh9\") pod \"maas-api-e2e-ait-e6e7ea12-5fc4d966cf-fllxv\" (UID: \"1c79d2fd-b23e-486b-b49c-1c8e401a75a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e6e7ea12-5fc4d966cf-fllxv"
Nov 01 03:29:47.187146 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:47.187115    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c79d2fd-b23e-486b-b49c-1c8e401a75a7-metrics-certs\") pod \"maas-api-e2e-ait-e6e7ea12-5fc4d966cf-fllxv\" (UID: \"1c79d2fd-b23e-486b-b49c-1c8e401a75a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e6e7ea12-5fc4d966cf-fllxv"
Nov 01 03:29:47.748002 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:47.747955    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c79d2fd-b23e-486b-b49c-1c8e401a75a7-maas-api-tls\") pod \"maas-api-e2e-ait-e6e7ea12-5fc4d966cf-fllxv\" (UID: \"1c79d2fd-b23e-486b-b49c-1c8e401a75a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e6e7ea12-5fc4d966cf-fllxv"
Nov 01 03:29:47.751783 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:47.751752    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c79d2fd-b23e-486b-b49c-1c8e401a75a7-maas-api-tls\") pod \"maas-api-e2e-ait-e6e7ea12-5fc4d966cf-fllxv\" (UID: \"1c79d2fd-b23e-486b-b49c-1c8e401a75a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e6e7ea12-5fc4d966cf-fllxv"
Nov 01 03:29:47.900861 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:47.900822    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e6e7ea12-5fc4d966cf-fllxv"
Nov 01 03:29:48.352515 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:48.352467    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e6e7ea12-5fc4d966cf-fllxv"]
Nov 01 03:29:48.354952 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:29:48.354920    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c79d2fd_b23e_486b_b49c_1c8e401a75a7.slice/crio-cd5adf3b3cea6d4f7e8d2fb63757a215529586c9efcf81b027be54060661981d WatchSource:0}: Error finding container cd5adf3b3cea6d4f7e8d2fb63757a215529586c9efcf81b027be54060661981d: Status 404 returned error can't find the container with id cd5adf3b3cea6d4f7e8d2fb63757a215529586c9efcf81b027be54060661981d
Nov 01 03:29:48.858204 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:48.858172    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e6e7ea12-5fc4d966cf-fllxv" event={"ID":"1c79d2fd-b23e-486b-b49c-1c8e401a75a7","Type":"ContainerStarted","Data":"cd5adf3b3cea6d4f7e8d2fb63757a215529586c9efcf81b027be54060661981d"}
Nov 01 03:29:49.867052 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:49.867016    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e6e7ea12-5fc4d966cf-fllxv" event={"ID":"1c79d2fd-b23e-486b-b49c-1c8e401a75a7","Type":"ContainerStarted","Data":"60941bada9101829cc831e399dca3c549b1c2be28cdfc1ea9d822326ae1b589f"}
Nov 01 03:29:49.867544 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:49.867082    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e6e7ea12-5fc4d966cf-fllxv"
Nov 01 03:29:49.986970 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:49.986924    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e6e7ea12-5fc4d966cf-fllxv" podStartSLOduration=3.197313256 podStartE2EDuration="3.986908654s" totalImagesPullingTime="789.595398ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:29:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:29:49.986836338 +0000 UTC m=+1396.501482215" watchObservedRunningTime="2026-11-01 03:29:49.986908654 +0000 UTC m=+1396.501554514"
Nov 01 03:29:55.880912 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:29:55.880876    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e6e7ea12-5fc4d966cf-fllxv"
Nov 01 03:30:01.731855 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:30:01.731818    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:30:06.894431 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:06.894399    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-44379c05-686994b9f-q8vrd"]
Nov 01 03:30:06.900135 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:06.900118    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-44379c05-686994b9f-q8vrd"
Nov 01 03:30:06.957585 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:06.957556    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-44379c05-686994b9f-q8vrd"]
Nov 01 03:30:06.979173 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:06.979126    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjj9h\" (UniqueName: \"kubernetes.io/projected/a54ae264-a89c-4a76-b513-9d85150cd488-kube-api-access-kjj9h\") pod \"maas-api-e2e-mt-44379c05-686994b9f-q8vrd\" (UID: \"a54ae264-a89c-4a76-b513-9d85150cd488\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-44379c05-686994b9f-q8vrd"
Nov 01 03:30:06.979311 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:06.979180    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a54ae264-a89c-4a76-b513-9d85150cd488-maas-api-tls\") pod \"maas-api-e2e-mt-44379c05-686994b9f-q8vrd\" (UID: \"a54ae264-a89c-4a76-b513-9d85150cd488\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-44379c05-686994b9f-q8vrd"
Nov 01 03:30:06.979402 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:06.979386    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a54ae264-a89c-4a76-b513-9d85150cd488-metrics-certs\") pod \"maas-api-e2e-mt-44379c05-686994b9f-q8vrd\" (UID: \"a54ae264-a89c-4a76-b513-9d85150cd488\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-44379c05-686994b9f-q8vrd"
Nov 01 03:30:06.979581 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:06.979558    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a54ae264-a89c-4a76-b513-9d85150cd488-openshift-service-ca\") pod \"maas-api-e2e-mt-44379c05-686994b9f-q8vrd\" (UID: \"a54ae264-a89c-4a76-b513-9d85150cd488\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-44379c05-686994b9f-q8vrd"
Nov 01 03:30:07.003728 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:07.003696    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-44379c05-serving-cert\""
Nov 01 03:30:07.080842 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:07.080814    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a54ae264-a89c-4a76-b513-9d85150cd488-metrics-certs\") pod \"maas-api-e2e-mt-44379c05-686994b9f-q8vrd\" (UID: \"a54ae264-a89c-4a76-b513-9d85150cd488\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-44379c05-686994b9f-q8vrd"
Nov 01 03:30:07.080992 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:07.080916    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a54ae264-a89c-4a76-b513-9d85150cd488-openshift-service-ca\") pod \"maas-api-e2e-mt-44379c05-686994b9f-q8vrd\" (UID: \"a54ae264-a89c-4a76-b513-9d85150cd488\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-44379c05-686994b9f-q8vrd"
Nov 01 03:30:07.081055 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:07.081039    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjj9h\" (UniqueName: \"kubernetes.io/projected/a54ae264-a89c-4a76-b513-9d85150cd488-kube-api-access-kjj9h\") pod \"maas-api-e2e-mt-44379c05-686994b9f-q8vrd\" (UID: \"a54ae264-a89c-4a76-b513-9d85150cd488\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-44379c05-686994b9f-q8vrd"
Nov 01 03:30:07.081114 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:07.081060    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a54ae264-a89c-4a76-b513-9d85150cd488-maas-api-tls\") pod \"maas-api-e2e-mt-44379c05-686994b9f-q8vrd\" (UID: \"a54ae264-a89c-4a76-b513-9d85150cd488\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-44379c05-686994b9f-q8vrd"
Nov 01 03:30:07.081287 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:30:07.081264    4400 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-44379c05-serving-cert: secret "maas-api-e2e-mt-44379c05-serving-cert" not found
Nov 01 03:30:07.081358 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:30:07.081337    4400 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a54ae264-a89c-4a76-b513-9d85150cd488-maas-api-tls podName:a54ae264-a89c-4a76-b513-9d85150cd488 nodeName:}" failed. No retries permitted until 2026-11-01 03:30:07.581322359 +0000 UTC m=+1414.095968200 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a54ae264-a89c-4a76-b513-9d85150cd488-maas-api-tls") pod "maas-api-e2e-mt-44379c05-686994b9f-q8vrd" (UID: "a54ae264-a89c-4a76-b513-9d85150cd488") : secret "maas-api-e2e-mt-44379c05-serving-cert" not found
Nov 01 03:30:07.081610 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:07.081588    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a54ae264-a89c-4a76-b513-9d85150cd488-openshift-service-ca\") pod \"maas-api-e2e-mt-44379c05-686994b9f-q8vrd\" (UID: \"a54ae264-a89c-4a76-b513-9d85150cd488\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-44379c05-686994b9f-q8vrd"
Nov 01 03:30:07.083328 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:07.083309    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a54ae264-a89c-4a76-b513-9d85150cd488-metrics-certs\") pod \"maas-api-e2e-mt-44379c05-686994b9f-q8vrd\" (UID: \"a54ae264-a89c-4a76-b513-9d85150cd488\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-44379c05-686994b9f-q8vrd"
Nov 01 03:30:07.145593 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:07.145538    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjj9h\" (UniqueName: \"kubernetes.io/projected/a54ae264-a89c-4a76-b513-9d85150cd488-kube-api-access-kjj9h\") pod \"maas-api-e2e-mt-44379c05-686994b9f-q8vrd\" (UID: \"a54ae264-a89c-4a76-b513-9d85150cd488\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-44379c05-686994b9f-q8vrd"
Nov 01 03:30:07.589962 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:07.589928    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a54ae264-a89c-4a76-b513-9d85150cd488-maas-api-tls\") pod \"maas-api-e2e-mt-44379c05-686994b9f-q8vrd\" (UID: \"a54ae264-a89c-4a76-b513-9d85150cd488\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-44379c05-686994b9f-q8vrd"
Nov 01 03:30:07.592467 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:07.592449    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a54ae264-a89c-4a76-b513-9d85150cd488-maas-api-tls\") pod \"maas-api-e2e-mt-44379c05-686994b9f-q8vrd\" (UID: \"a54ae264-a89c-4a76-b513-9d85150cd488\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-44379c05-686994b9f-q8vrd"
Nov 01 03:30:07.814514 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:07.814480    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-44379c05-686994b9f-q8vrd"
Nov 01 03:30:08.199277 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:08.199248    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-44379c05-686994b9f-q8vrd"]
Nov 01 03:30:08.200726 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:30:08.200662    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda54ae264_a89c_4a76_b513_9d85150cd488.slice/crio-fc8fb2187c39fd0d4b28005b1a93c3eae4a1067aa20fdd380d93da14d5083e6f WatchSource:0}: Error finding container fc8fb2187c39fd0d4b28005b1a93c3eae4a1067aa20fdd380d93da14d5083e6f: Status 404 returned error can't find the container with id fc8fb2187c39fd0d4b28005b1a93c3eae4a1067aa20fdd380d93da14d5083e6f
Nov 01 03:30:09.073115 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:09.073073    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-44379c05-686994b9f-q8vrd" event={"ID":"a54ae264-a89c-4a76-b513-9d85150cd488","Type":"ContainerStarted","Data":"fc8fb2187c39fd0d4b28005b1a93c3eae4a1067aa20fdd380d93da14d5083e6f"}
Nov 01 03:30:10.103797 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:10.103757    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-44379c05-686994b9f-q8vrd" event={"ID":"a54ae264-a89c-4a76-b513-9d85150cd488","Type":"ContainerStarted","Data":"7fbed071b17c876a467b67df3b47d089a885e20552beed5b72161a51c2c3a6f7"}
Nov 01 03:30:10.105757 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:10.105734    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-44379c05-686994b9f-q8vrd"
Nov 01 03:30:10.229515 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:10.229467    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-44379c05-686994b9f-q8vrd" podStartSLOduration=3.340259933 podStartE2EDuration="4.229453627s" totalImagesPullingTime="889.193694ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:30:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:30:10.226242345 +0000 UTC m=+1416.740888208" watchObservedRunningTime="2026-11-01 03:30:10.229453627 +0000 UTC m=+1416.744099489"
Nov 01 03:30:17.162506 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:17.162475    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-44379c05-686994b9f-q8vrd"
Nov 01 03:30:20.550545 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:20.550510    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-8xrcf"]
Nov 01 03:30:20.556524 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:20.556506    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-8xrcf"
Nov 01 03:30:20.623929 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:20.623883    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa4d3ec1-8b26-4c06-8290-87c055cdca0a-istio-envoy\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-8xrcf\" (UID: \"fa4d3ec1-8b26-4c06-8290-87c055cdca0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-8xrcf"
Nov 01 03:30:20.625116 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:20.623935    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86dhb\" (UniqueName: \"kubernetes.io/projected/fa4d3ec1-8b26-4c06-8290-87c055cdca0a-kube-api-access-86dhb\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-8xrcf\" (UID: \"fa4d3ec1-8b26-4c06-8290-87c055cdca0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-8xrcf"
Nov 01 03:30:20.625116 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:20.623990    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa4d3ec1-8b26-4c06-8290-87c055cdca0a-istio-data\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-8xrcf\" (UID: \"fa4d3ec1-8b26-4c06-8290-87c055cdca0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-8xrcf"
Nov 01 03:30:20.629276 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:20.629245    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa4d3ec1-8b26-4c06-8290-87c055cdca0a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-8xrcf\" (UID: \"fa4d3ec1-8b26-4c06-8290-87c055cdca0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-8xrcf"
Nov 01 03:30:20.629414 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:20.629294    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa4d3ec1-8b26-4c06-8290-87c055cdca0a-workload-certs\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-8xrcf\" (UID: \"fa4d3ec1-8b26-4c06-8290-87c055cdca0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-8xrcf"
Nov 01 03:30:20.629475 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:20.629457    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa4d3ec1-8b26-4c06-8290-87c055cdca0a-workload-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-8xrcf\" (UID: \"fa4d3ec1-8b26-4c06-8290-87c055cdca0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-8xrcf"
Nov 01 03:30:20.629579 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:20.629564    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa4d3ec1-8b26-4c06-8290-87c055cdca0a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-8xrcf\" (UID: \"fa4d3ec1-8b26-4c06-8290-87c055cdca0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-8xrcf"
Nov 01 03:30:20.629674 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:20.629661    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa4d3ec1-8b26-4c06-8290-87c055cdca0a-credential-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-8xrcf\" (UID: \"fa4d3ec1-8b26-4c06-8290-87c055cdca0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-8xrcf"
Nov 01 03:30:20.629777 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:20.629714    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa4d3ec1-8b26-4c06-8290-87c055cdca0a-istio-token\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-8xrcf\" (UID: \"fa4d3ec1-8b26-4c06-8290-87c055cdca0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-8xrcf"
Nov 01 03:30:20.667771 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:20.667743    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-8xrcf"]
Nov 01 03:30:20.730838 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:20.730807    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa4d3ec1-8b26-4c06-8290-87c055cdca0a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-8xrcf\" (UID: \"fa4d3ec1-8b26-4c06-8290-87c055cdca0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-8xrcf"
Nov 01 03:30:20.730985 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:20.730845    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa4d3ec1-8b26-4c06-8290-87c055cdca0a-workload-certs\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-8xrcf\" (UID: \"fa4d3ec1-8b26-4c06-8290-87c055cdca0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-8xrcf"
Nov 01 03:30:20.730985 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:20.730926    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa4d3ec1-8b26-4c06-8290-87c055cdca0a-workload-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-8xrcf\" (UID: \"fa4d3ec1-8b26-4c06-8290-87c055cdca0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-8xrcf"
Nov 01 03:30:20.730985 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:20.730981    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa4d3ec1-8b26-4c06-8290-87c055cdca0a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-8xrcf\" (UID: \"fa4d3ec1-8b26-4c06-8290-87c055cdca0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-8xrcf"
Nov 01 03:30:20.731145 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:20.731034    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa4d3ec1-8b26-4c06-8290-87c055cdca0a-credential-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-8xrcf\" (UID: \"fa4d3ec1-8b26-4c06-8290-87c055cdca0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-8xrcf"
Nov 01 03:30:20.731145 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:20.731057    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa4d3ec1-8b26-4c06-8290-87c055cdca0a-istio-token\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-8xrcf\" (UID: \"fa4d3ec1-8b26-4c06-8290-87c055cdca0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-8xrcf"
Nov 01 03:30:20.731250 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:20.731152    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa4d3ec1-8b26-4c06-8290-87c055cdca0a-istio-envoy\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-8xrcf\" (UID: \"fa4d3ec1-8b26-4c06-8290-87c055cdca0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-8xrcf"
Nov 01 03:30:20.731250 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:20.731180    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86dhb\" (UniqueName: \"kubernetes.io/projected/fa4d3ec1-8b26-4c06-8290-87c055cdca0a-kube-api-access-86dhb\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-8xrcf\" (UID: \"fa4d3ec1-8b26-4c06-8290-87c055cdca0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-8xrcf"
Nov 01 03:30:20.731250 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:20.731236    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa4d3ec1-8b26-4c06-8290-87c055cdca0a-istio-data\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-8xrcf\" (UID: \"fa4d3ec1-8b26-4c06-8290-87c055cdca0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-8xrcf"
Nov 01 03:30:20.731506 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:20.731481    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa4d3ec1-8b26-4c06-8290-87c055cdca0a-workload-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-8xrcf\" (UID: \"fa4d3ec1-8b26-4c06-8290-87c055cdca0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-8xrcf"
Nov 01 03:30:20.731587 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:20.731542    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa4d3ec1-8b26-4c06-8290-87c055cdca0a-credential-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-8xrcf\" (UID: \"fa4d3ec1-8b26-4c06-8290-87c055cdca0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-8xrcf"
Nov 01 03:30:20.731831 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:20.731805    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa4d3ec1-8b26-4c06-8290-87c055cdca0a-istio-data\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-8xrcf\" (UID: \"fa4d3ec1-8b26-4c06-8290-87c055cdca0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-8xrcf"
Nov 01 03:30:20.731893 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:20.731853    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa4d3ec1-8b26-4c06-8290-87c055cdca0a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-8xrcf\" (UID: \"fa4d3ec1-8b26-4c06-8290-87c055cdca0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-8xrcf"
Nov 01 03:30:20.732027 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:20.731966    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa4d3ec1-8b26-4c06-8290-87c055cdca0a-workload-certs\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-8xrcf\" (UID: \"fa4d3ec1-8b26-4c06-8290-87c055cdca0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-8xrcf"
Nov 01 03:30:20.733783 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:20.733760    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa4d3ec1-8b26-4c06-8290-87c055cdca0a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-8xrcf\" (UID: \"fa4d3ec1-8b26-4c06-8290-87c055cdca0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-8xrcf"
Nov 01 03:30:20.733954 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:20.733780    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa4d3ec1-8b26-4c06-8290-87c055cdca0a-istio-envoy\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-8xrcf\" (UID: \"fa4d3ec1-8b26-4c06-8290-87c055cdca0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-8xrcf"
Nov 01 03:30:20.795017 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:20.794988    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86dhb\" (UniqueName: \"kubernetes.io/projected/fa4d3ec1-8b26-4c06-8290-87c055cdca0a-kube-api-access-86dhb\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-8xrcf\" (UID: \"fa4d3ec1-8b26-4c06-8290-87c055cdca0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-8xrcf"
Nov 01 03:30:20.795017 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:20.795116    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa4d3ec1-8b26-4c06-8290-87c055cdca0a-istio-token\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-8xrcf\" (UID: \"fa4d3ec1-8b26-4c06-8290-87c055cdca0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-8xrcf"
Nov 01 03:30:20.870777 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:20.870747    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-8xrcf"
Nov 01 03:30:21.318558 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:30:21.318529    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa4d3ec1_8b26_4c06_8290_87c055cdca0a.slice/crio-80b362c272f71d2fc77ba38dc51e3d6bb1487e6355012750c523342fb1b9bf29 WatchSource:0}: Error finding container 80b362c272f71d2fc77ba38dc51e3d6bb1487e6355012750c523342fb1b9bf29: Status 404 returned error can't find the container with id 80b362c272f71d2fc77ba38dc51e3d6bb1487e6355012750c523342fb1b9bf29
Nov 01 03:30:21.319744 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:21.319702    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-8xrcf"]
Nov 01 03:30:21.321289 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:21.321260    4400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 03:30:21.321368 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:21.321323    4400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 03:30:21.321368 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:21.321350    4400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 03:30:22.238698 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:22.236900    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-8xrcf" event={"ID":"fa4d3ec1-8b26-4c06-8290-87c055cdca0a","Type":"ContainerStarted","Data":"2fb0f4b2425bae17a2955645ac67d00aab109c92dff5c1ebbecf5dca5312ca7e"}
Nov 01 03:30:22.238698 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:22.236978    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-8xrcf" event={"ID":"fa4d3ec1-8b26-4c06-8290-87c055cdca0a","Type":"ContainerStarted","Data":"80b362c272f71d2fc77ba38dc51e3d6bb1487e6355012750c523342fb1b9bf29"}
Nov 01 03:30:22.426376 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:22.426306    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-8xrcf" podStartSLOduration=2.426286502 podStartE2EDuration="2.426286502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:30:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:30:22.424213941 +0000 UTC m=+1428.938859846" watchObservedRunningTime="2026-11-01 03:30:22.426286502 +0000 UTC m=+1428.940932366"
Nov 01 03:30:22.871414 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:22.871382    4400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-8xrcf"
Nov 01 03:30:22.872490 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:22.872461    4400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-8xrcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Nov 01 03:30:22.872622 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:22.872528    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-8xrcf" podUID="fa4d3ec1-8b26-4c06-8290-87c055cdca0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Nov 01 03:30:23.871205 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:23.871169    4400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-8xrcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Nov 01 03:30:23.871735 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:23.871222    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-8xrcf" podUID="fa4d3ec1-8b26-4c06-8290-87c055cdca0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Nov 01 03:30:24.871833 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:24.871788    4400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-8xrcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Nov 01 03:30:24.872240 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:24.871857    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-8xrcf" podUID="fa4d3ec1-8b26-4c06-8290-87c055cdca0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Nov 01 03:30:25.871844 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:25.871808    4400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-8xrcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Nov 01 03:30:25.872275 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:25.871894    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-8xrcf" podUID="fa4d3ec1-8b26-4c06-8290-87c055cdca0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Nov 01 03:30:26.870994 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:26.870965    4400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-8xrcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Nov 01 03:30:26.871220 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:26.871200    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-8xrcf" podUID="fa4d3ec1-8b26-4c06-8290-87c055cdca0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Nov 01 03:30:27.871650 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:27.871612    4400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-8xrcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Nov 01 03:30:27.872186 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:27.871708    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-8xrcf" podUID="fa4d3ec1-8b26-4c06-8290-87c055cdca0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Nov 01 03:30:28.871168 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:28.871134    4400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-8xrcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Nov 01 03:30:28.871339 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:28.871185    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-8xrcf" podUID="fa4d3ec1-8b26-4c06-8290-87c055cdca0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Nov 01 03:30:29.940959 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:29.940922    4400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-8xrcf"
Nov 01 03:30:29.941462 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:29.941310    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-8xrcf"
Nov 01 03:30:29.942580 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:29.942558    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-8xrcf"
Nov 01 03:30:31.731258 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:30:31.731218    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:30:35.683916 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:35.683885    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-g8ft5"]
Nov 01 03:30:35.689071 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:35.689053    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-g8ft5"
Nov 01 03:30:35.783484 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:35.783444    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-g8ft5"]
Nov 01 03:30:36.001627 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:36.001541    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f593b1bc-8cc9-4147-88d9-9f441b0d7159-istio-data\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-g8ft5\" (UID: \"f593b1bc-8cc9-4147-88d9-9f441b0d7159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-g8ft5"
Nov 01 03:30:36.021496 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:36.021336    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f593b1bc-8cc9-4147-88d9-9f441b0d7159-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-g8ft5\" (UID: \"f593b1bc-8cc9-4147-88d9-9f441b0d7159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-g8ft5"
Nov 01 03:30:36.021980 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:36.021942    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f593b1bc-8cc9-4147-88d9-9f441b0d7159-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-g8ft5\" (UID: \"f593b1bc-8cc9-4147-88d9-9f441b0d7159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-g8ft5"
Nov 01 03:30:36.022299 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:36.022266    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f593b1bc-8cc9-4147-88d9-9f441b0d7159-credential-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-g8ft5\" (UID: \"f593b1bc-8cc9-4147-88d9-9f441b0d7159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-g8ft5"
Nov 01 03:30:36.022618 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:36.022600    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f593b1bc-8cc9-4147-88d9-9f441b0d7159-istio-token\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-g8ft5\" (UID: \"f593b1bc-8cc9-4147-88d9-9f441b0d7159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-g8ft5"
Nov 01 03:30:36.023034 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:36.023012    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xf8v\" (UniqueName: \"kubernetes.io/projected/f593b1bc-8cc9-4147-88d9-9f441b0d7159-kube-api-access-9xf8v\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-g8ft5\" (UID: \"f593b1bc-8cc9-4147-88d9-9f441b0d7159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-g8ft5"
Nov 01 03:30:36.023558 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:36.023537    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f593b1bc-8cc9-4147-88d9-9f441b0d7159-workload-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-g8ft5\" (UID: \"f593b1bc-8cc9-4147-88d9-9f441b0d7159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-g8ft5"
Nov 01 03:30:36.024188 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:36.024165    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f593b1bc-8cc9-4147-88d9-9f441b0d7159-istio-envoy\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-g8ft5\" (UID: \"f593b1bc-8cc9-4147-88d9-9f441b0d7159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-g8ft5"
Nov 01 03:30:36.024361 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:36.024345    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f593b1bc-8cc9-4147-88d9-9f441b0d7159-workload-certs\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-g8ft5\" (UID: \"f593b1bc-8cc9-4147-88d9-9f441b0d7159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-g8ft5"
Nov 01 03:30:36.127068 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:36.127025    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f593b1bc-8cc9-4147-88d9-9f441b0d7159-workload-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-g8ft5\" (UID: \"f593b1bc-8cc9-4147-88d9-9f441b0d7159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-g8ft5"
Nov 01 03:30:36.127822 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:36.127655    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f593b1bc-8cc9-4147-88d9-9f441b0d7159-workload-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-g8ft5\" (UID: \"f593b1bc-8cc9-4147-88d9-9f441b0d7159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-g8ft5"
Nov 01 03:30:36.128486 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:36.128128    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f593b1bc-8cc9-4147-88d9-9f441b0d7159-istio-envoy\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-g8ft5\" (UID: \"f593b1bc-8cc9-4147-88d9-9f441b0d7159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-g8ft5"
Nov 01 03:30:36.128819 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:36.128774    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f593b1bc-8cc9-4147-88d9-9f441b0d7159-workload-certs\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-g8ft5\" (UID: \"f593b1bc-8cc9-4147-88d9-9f441b0d7159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-g8ft5"
Nov 01 03:30:36.129293 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:36.129276    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f593b1bc-8cc9-4147-88d9-9f441b0d7159-istio-data\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-g8ft5\" (UID: \"f593b1bc-8cc9-4147-88d9-9f441b0d7159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-g8ft5"
Nov 01 03:30:36.129428 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:36.129415    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f593b1bc-8cc9-4147-88d9-9f441b0d7159-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-g8ft5\" (UID: \"f593b1bc-8cc9-4147-88d9-9f441b0d7159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-g8ft5"
Nov 01 03:30:36.129612 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:36.129599    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f593b1bc-8cc9-4147-88d9-9f441b0d7159-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-g8ft5\" (UID: \"f593b1bc-8cc9-4147-88d9-9f441b0d7159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-g8ft5"
Nov 01 03:30:36.129805 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:36.129165    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f593b1bc-8cc9-4147-88d9-9f441b0d7159-workload-certs\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-g8ft5\" (UID: \"f593b1bc-8cc9-4147-88d9-9f441b0d7159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-g8ft5"
Nov 01 03:30:36.129882 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:36.129859    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f593b1bc-8cc9-4147-88d9-9f441b0d7159-credential-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-g8ft5\" (UID: \"f593b1bc-8cc9-4147-88d9-9f441b0d7159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-g8ft5"
Nov 01 03:30:36.129954 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:36.129936    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f593b1bc-8cc9-4147-88d9-9f441b0d7159-istio-token\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-g8ft5\" (UID: \"f593b1bc-8cc9-4147-88d9-9f441b0d7159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-g8ft5"
Nov 01 03:30:36.130008 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:36.129971    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xf8v\" (UniqueName: \"kubernetes.io/projected/f593b1bc-8cc9-4147-88d9-9f441b0d7159-kube-api-access-9xf8v\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-g8ft5\" (UID: \"f593b1bc-8cc9-4147-88d9-9f441b0d7159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-g8ft5"
Nov 01 03:30:36.130062 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:36.130024    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f593b1bc-8cc9-4147-88d9-9f441b0d7159-istio-data\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-g8ft5\" (UID: \"f593b1bc-8cc9-4147-88d9-9f441b0d7159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-g8ft5"
Nov 01 03:30:36.132732 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:36.130410    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f593b1bc-8cc9-4147-88d9-9f441b0d7159-credential-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-g8ft5\" (UID: \"f593b1bc-8cc9-4147-88d9-9f441b0d7159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-g8ft5"
Nov 01 03:30:36.133610 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:36.133586    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f593b1bc-8cc9-4147-88d9-9f441b0d7159-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-g8ft5\" (UID: \"f593b1bc-8cc9-4147-88d9-9f441b0d7159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-g8ft5"
Nov 01 03:30:36.135197 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:36.135176    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f593b1bc-8cc9-4147-88d9-9f441b0d7159-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-g8ft5\" (UID: \"f593b1bc-8cc9-4147-88d9-9f441b0d7159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-g8ft5"
Nov 01 03:30:36.136976 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:36.136772    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f593b1bc-8cc9-4147-88d9-9f441b0d7159-istio-envoy\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-g8ft5\" (UID: \"f593b1bc-8cc9-4147-88d9-9f441b0d7159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-g8ft5"
Nov 01 03:30:36.192503 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:36.192469    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f593b1bc-8cc9-4147-88d9-9f441b0d7159-istio-token\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-g8ft5\" (UID: \"f593b1bc-8cc9-4147-88d9-9f441b0d7159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-g8ft5"
Nov 01 03:30:36.192636 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:36.192614    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xf8v\" (UniqueName: \"kubernetes.io/projected/f593b1bc-8cc9-4147-88d9-9f441b0d7159-kube-api-access-9xf8v\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-g8ft5\" (UID: \"f593b1bc-8cc9-4147-88d9-9f441b0d7159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-g8ft5"
Nov 01 03:30:36.303966 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:36.303913    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-g8ft5"
Nov 01 03:30:36.801644 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:30:36.801613    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf593b1bc_8cc9_4147_88d9_9f441b0d7159.slice/crio-dda22dad46011c94f25a868920fdd2eae4542f3eb542e6dff259daea2d10f7ff WatchSource:0}: Error finding container dda22dad46011c94f25a868920fdd2eae4542f3eb542e6dff259daea2d10f7ff: Status 404 returned error can't find the container with id dda22dad46011c94f25a868920fdd2eae4542f3eb542e6dff259daea2d10f7ff
Nov 01 03:30:36.802158 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:36.802133    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-g8ft5"]
Nov 01 03:30:36.804160 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:36.804129    4400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 03:30:36.804249 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:36.804193    4400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 03:30:36.804249 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:36.804228    4400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 03:30:37.112710 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:37.110257    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 03:30:37.360566 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:37.360533    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-g8ft5" event={"ID":"f593b1bc-8cc9-4147-88d9-9f441b0d7159","Type":"ContainerStarted","Data":"f8322c8af545e7159a93ede36f7338261eb7539c557a2eb9ab7d200d487ca7bb"}
Nov 01 03:30:37.360774 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:37.360597    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-g8ft5" event={"ID":"f593b1bc-8cc9-4147-88d9-9f441b0d7159","Type":"ContainerStarted","Data":"dda22dad46011c94f25a868920fdd2eae4542f3eb542e6dff259daea2d10f7ff"}
Nov 01 03:30:37.492424 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:37.492326    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-g8ft5" podStartSLOduration=2.492311021 podStartE2EDuration="2.492311021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:30:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:30:37.488881566 +0000 UTC m=+1444.003527436" watchObservedRunningTime="2026-11-01 03:30:37.492311021 +0000 UTC m=+1444.006956953"
Nov 01 03:30:38.328708 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:38.328215    4400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-g8ft5"
Nov 01 03:30:38.329520 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:38.329484    4400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-g8ft5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Nov 01 03:30:38.329637 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:38.329558    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-g8ft5" podUID="f593b1bc-8cc9-4147-88d9-9f441b0d7159" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Nov 01 03:30:39.326712 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:39.325775    4400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-g8ft5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Nov 01 03:30:39.326712 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:39.325840    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-g8ft5" podUID="f593b1bc-8cc9-4147-88d9-9f441b0d7159" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Nov 01 03:30:40.327095 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:40.327061    4400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-g8ft5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Nov 01 03:30:40.327507 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:40.327119    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-g8ft5" podUID="f593b1bc-8cc9-4147-88d9-9f441b0d7159" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Nov 01 03:30:41.327233 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:41.327188    4400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-g8ft5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Nov 01 03:30:41.327669 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:41.327271    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-g8ft5" podUID="f593b1bc-8cc9-4147-88d9-9f441b0d7159" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Nov 01 03:30:42.316712 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:42.311107    4400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-g8ft5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Nov 01 03:30:42.316712 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:42.311180    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-g8ft5" podUID="f593b1bc-8cc9-4147-88d9-9f441b0d7159" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Nov 01 03:30:43.307712 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:43.307133    4400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-g8ft5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Nov 01 03:30:43.307712 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:43.307210    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-g8ft5" podUID="f593b1bc-8cc9-4147-88d9-9f441b0d7159" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Nov 01 03:30:44.305134 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:44.305093    4400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-g8ft5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Nov 01 03:30:44.305321 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:44.305156    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-g8ft5" podUID="f593b1bc-8cc9-4147-88d9-9f441b0d7159" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Nov 01 03:30:45.905118 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:45.905088    4400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-g8ft5"
Nov 01 03:30:45.905596 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:45.905474    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-g8ft5"
Nov 01 03:30:45.906754 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:45.906733    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-g8ft5"
Nov 01 03:30:46.051344 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:46.051316    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d5c42938-5b46444559-n968k"]
Nov 01 03:30:46.168937 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:46.168847    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d5c42938-6c6bf65c67-lqdcx"]
Nov 01 03:30:51.052044 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:51.051997    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-d5c42938-5b46444559-n968k" podUID="71e81e40-e46d-4981-a403-d9a74bdc5957" containerName="payload-processing" containerID="cri-o://cc3be2b7f6478296d2523e159b6c60903a6999c9669ee3ca8caf55403710a7e1" gracePeriod=25
Nov 01 03:30:51.170617 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:51.170579    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-d5c42938-6c6bf65c67-lqdcx" podUID="0c264853-8d3b-4e19-8872-224f7aee939c" containerName="payload-pre-processing" containerID="cri-o://68d624ce390342eae619c6ac68c815e625408abf528480dbd626b7d1d5765ad2" gracePeriod=25
Nov 01 03:30:51.355510 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:51.355484    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d5c42938-5b46444559-n968k"
Nov 01 03:30:51.466362 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:51.466332    4400 generic.go:356] "Generic (PLEG): container finished" podID="0c264853-8d3b-4e19-8872-224f7aee939c" containerID="68d624ce390342eae619c6ac68c815e625408abf528480dbd626b7d1d5765ad2" exitCode=0
Nov 01 03:30:51.466537 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:51.466421    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d5c42938-6c6bf65c67-lqdcx" event={"ID":"0c264853-8d3b-4e19-8872-224f7aee939c","Type":"ContainerDied","Data":"68d624ce390342eae619c6ac68c815e625408abf528480dbd626b7d1d5765ad2"}
Nov 01 03:30:51.466537 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:51.466491    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d5c42938-6c6bf65c67-lqdcx" event={"ID":"0c264853-8d3b-4e19-8872-224f7aee939c","Type":"ContainerDied","Data":"7ea26ee582432ab4a9c1a5eb24f324834bf314f8904945faa52d4c0f9892deb7"}
Nov 01 03:30:51.466537 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:51.466514    4400 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ea26ee582432ab4a9c1a5eb24f324834bf314f8904945faa52d4c0f9892deb7"
Nov 01 03:30:51.468439 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:51.468414    4400 generic.go:356] "Generic (PLEG): container finished" podID="71e81e40-e46d-4981-a403-d9a74bdc5957" containerID="cc3be2b7f6478296d2523e159b6c60903a6999c9669ee3ca8caf55403710a7e1" exitCode=0
Nov 01 03:30:51.468569 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:51.468475    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d5c42938-5b46444559-n968k"
Nov 01 03:30:51.468569 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:51.468504    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d5c42938-5b46444559-n968k" event={"ID":"71e81e40-e46d-4981-a403-d9a74bdc5957","Type":"ContainerDied","Data":"cc3be2b7f6478296d2523e159b6c60903a6999c9669ee3ca8caf55403710a7e1"}
Nov 01 03:30:51.468569 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:51.468550    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d5c42938-5b46444559-n968k" event={"ID":"71e81e40-e46d-4981-a403-d9a74bdc5957","Type":"ContainerDied","Data":"7b21b37e95147db2c03bca6480f7afd3d1188040e5691a26cf0214a15421cf3b"}
Nov 01 03:30:51.468760 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:51.468571    4400 scope.go:122] "RemoveContainer" containerID="cc3be2b7f6478296d2523e159b6c60903a6999c9669ee3ca8caf55403710a7e1"
Nov 01 03:30:51.485059 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:51.480823    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71e81e40-e46d-4981-a403-d9a74bdc5957-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71e81e40-e46d-4981-a403-d9a74bdc5957-plugins-config-volume\") pod \"71e81e40-e46d-4981-a403-d9a74bdc5957\" (UID: \"71e81e40-e46d-4981-a403-d9a74bdc5957\") "
Nov 01 03:30:51.485059 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:51.481000    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71e81e40-e46d-4981-a403-d9a74bdc5957-kube-api-access-zbt2w\" (UniqueName: \"kubernetes.io/projected/71e81e40-e46d-4981-a403-d9a74bdc5957-kube-api-access-zbt2w\") pod \"71e81e40-e46d-4981-a403-d9a74bdc5957\" (UID: \"71e81e40-e46d-4981-a403-d9a74bdc5957\") "
Nov 01 03:30:51.485059 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:51.482325    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71e81e40-e46d-4981-a403-d9a74bdc5957-plugins-config-volume" pod "71e81e40-e46d-4981-a403-d9a74bdc5957" (UID: "71e81e40-e46d-4981-a403-d9a74bdc5957"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:30:51.485549 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:51.485523    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71e81e40-e46d-4981-a403-d9a74bdc5957-kube-api-access-zbt2w" pod "71e81e40-e46d-4981-a403-d9a74bdc5957" (UID: "71e81e40-e46d-4981-a403-d9a74bdc5957"). InnerVolumeSpecName "kube-api-access-zbt2w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:30:51.488627 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:51.488607    4400 scope.go:122] "RemoveContainer" containerID="cc3be2b7f6478296d2523e159b6c60903a6999c9669ee3ca8caf55403710a7e1"
Nov 01 03:30:51.494136 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:30:51.494102    4400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc3be2b7f6478296d2523e159b6c60903a6999c9669ee3ca8caf55403710a7e1\": container with ID starting with cc3be2b7f6478296d2523e159b6c60903a6999c9669ee3ca8caf55403710a7e1 not found: ID does not exist" containerID="cc3be2b7f6478296d2523e159b6c60903a6999c9669ee3ca8caf55403710a7e1"
Nov 01 03:30:51.494258 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:51.494161    4400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc3be2b7f6478296d2523e159b6c60903a6999c9669ee3ca8caf55403710a7e1"} err="failed to get container status \"cc3be2b7f6478296d2523e159b6c60903a6999c9669ee3ca8caf55403710a7e1\": rpc error: code = NotFound desc = could not find container \"cc3be2b7f6478296d2523e159b6c60903a6999c9669ee3ca8caf55403710a7e1\": container with ID starting with cc3be2b7f6478296d2523e159b6c60903a6999c9669ee3ca8caf55403710a7e1 not found: ID does not exist"
Nov 01 03:30:51.497792 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:51.497774    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d5c42938-6c6bf65c67-lqdcx"
Nov 01 03:30:51.583133 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:51.583106    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c264853-8d3b-4e19-8872-224f7aee939c-kube-api-access-n224h\" (UniqueName: \"kubernetes.io/projected/0c264853-8d3b-4e19-8872-224f7aee939c-kube-api-access-n224h\") pod \"0c264853-8d3b-4e19-8872-224f7aee939c\" (UID: \"0c264853-8d3b-4e19-8872-224f7aee939c\") "
Nov 01 03:30:51.583317 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:51.583168    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c264853-8d3b-4e19-8872-224f7aee939c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c264853-8d3b-4e19-8872-224f7aee939c-plugins-config-volume\") pod \"0c264853-8d3b-4e19-8872-224f7aee939c\" (UID: \"0c264853-8d3b-4e19-8872-224f7aee939c\") "
Nov 01 03:30:51.583586 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:51.583561    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c264853-8d3b-4e19-8872-224f7aee939c-plugins-config-volume" pod "0c264853-8d3b-4e19-8872-224f7aee939c" (UID: "0c264853-8d3b-4e19-8872-224f7aee939c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:30:51.583936 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:51.583921    4400 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71e81e40-e46d-4981-a403-d9a74bdc5957-plugins-config-volume\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:30:51.583999 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:51.583939    4400 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c264853-8d3b-4e19-8872-224f7aee939c-plugins-config-volume\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:30:51.583999 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:51.583949    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbt2w\" (UniqueName: \"kubernetes.io/projected/71e81e40-e46d-4981-a403-d9a74bdc5957-kube-api-access-zbt2w\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:30:51.585349 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:51.585326    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c264853-8d3b-4e19-8872-224f7aee939c-kube-api-access-n224h" pod "0c264853-8d3b-4e19-8872-224f7aee939c" (UID: "0c264853-8d3b-4e19-8872-224f7aee939c"). InnerVolumeSpecName "kube-api-access-n224h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:30:51.685972 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:51.685937    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n224h\" (UniqueName: \"kubernetes.io/projected/0c264853-8d3b-4e19-8872-224f7aee939c-kube-api-access-n224h\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:30:51.889663 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:51.889587    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d5c42938-5b46444559-n968k"]
Nov 01 03:30:51.893587 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:51.893559    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d5c42938-5b46444559-n968k"]
Nov 01 03:30:52.476966 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:52.476931    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d5c42938-6c6bf65c67-lqdcx"
Nov 01 03:30:52.649949 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:52.649917    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d5c42938-6c6bf65c67-lqdcx"]
Nov 01 03:30:52.653804 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:52.653784    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d5c42938-6c6bf65c67-lqdcx"]
Nov 01 03:30:53.214667 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:53.214630    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7911e5a0-68f8dd6b5c-jrj6z"]
Nov 01 03:30:53.825786 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:53.825729    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c264853-8d3b-4e19-8872-224f7aee939c" path="/var/lib/kubelet/pods/0c264853-8d3b-4e19-8872-224f7aee939c/volumes"
Nov 01 03:30:53.827458 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:53.827421    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71e81e40-e46d-4981-a403-d9a74bdc5957" path="/var/lib/kubelet/pods/71e81e40-e46d-4981-a403-d9a74bdc5957/volumes"
Nov 01 03:30:58.215007 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:58.214931    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-7911e5a0-68f8dd6b5c-jrj6z" podUID="71f320ef-9f70-41a4-887e-413230d65992" containerName="payload-processing" containerID="cri-o://319cd592c45077cae76ed97bf3586b1261a73a2fb059bde8c4173ce3eca8ea7b" gracePeriod=25
Nov 01 03:30:58.505162 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:58.505135    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7911e5a0-68f8dd6b5c-jrj6z"
Nov 01 03:30:58.525205 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:58.525174    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71f320ef-9f70-41a4-887e-413230d65992-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71f320ef-9f70-41a4-887e-413230d65992-plugins-config-volume\") pod \"71f320ef-9f70-41a4-887e-413230d65992\" (UID: \"71f320ef-9f70-41a4-887e-413230d65992\") "
Nov 01 03:30:58.525383 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:58.525311    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71f320ef-9f70-41a4-887e-413230d65992-kube-api-access-ghbbc\" (UniqueName: \"kubernetes.io/projected/71f320ef-9f70-41a4-887e-413230d65992-kube-api-access-ghbbc\") pod \"71f320ef-9f70-41a4-887e-413230d65992\" (UID: \"71f320ef-9f70-41a4-887e-413230d65992\") "
Nov 01 03:30:58.525590 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:58.525565    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71f320ef-9f70-41a4-887e-413230d65992-plugins-config-volume" pod "71f320ef-9f70-41a4-887e-413230d65992" (UID: "71f320ef-9f70-41a4-887e-413230d65992"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:30:58.526119 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:58.526099    4400 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71f320ef-9f70-41a4-887e-413230d65992-plugins-config-volume\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:30:58.527725 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:58.527673    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71f320ef-9f70-41a4-887e-413230d65992-kube-api-access-ghbbc" pod "71f320ef-9f70-41a4-887e-413230d65992" (UID: "71f320ef-9f70-41a4-887e-413230d65992"). InnerVolumeSpecName "kube-api-access-ghbbc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:30:58.539982 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:58.539955    4400 generic.go:356] "Generic (PLEG): container finished" podID="71f320ef-9f70-41a4-887e-413230d65992" containerID="319cd592c45077cae76ed97bf3586b1261a73a2fb059bde8c4173ce3eca8ea7b" exitCode=0
Nov 01 03:30:58.540122 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:58.540011    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7911e5a0-68f8dd6b5c-jrj6z"
Nov 01 03:30:58.540122 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:58.540032    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7911e5a0-68f8dd6b5c-jrj6z" event={"ID":"71f320ef-9f70-41a4-887e-413230d65992","Type":"ContainerDied","Data":"319cd592c45077cae76ed97bf3586b1261a73a2fb059bde8c4173ce3eca8ea7b"}
Nov 01 03:30:58.540122 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:58.540094    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7911e5a0-68f8dd6b5c-jrj6z" event={"ID":"71f320ef-9f70-41a4-887e-413230d65992","Type":"ContainerDied","Data":"30efa34d9bb8c47e126dd49fa17aba5017574ef1f7bf379c5655cc0567477fea"}
Nov 01 03:30:58.540122 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:58.540121    4400 scope.go:122] "RemoveContainer" containerID="319cd592c45077cae76ed97bf3586b1261a73a2fb059bde8c4173ce3eca8ea7b"
Nov 01 03:30:58.554207 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:58.554188    4400 scope.go:122] "RemoveContainer" containerID="319cd592c45077cae76ed97bf3586b1261a73a2fb059bde8c4173ce3eca8ea7b"
Nov 01 03:30:58.554538 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:30:58.554520    4400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"319cd592c45077cae76ed97bf3586b1261a73a2fb059bde8c4173ce3eca8ea7b\": container with ID starting with 319cd592c45077cae76ed97bf3586b1261a73a2fb059bde8c4173ce3eca8ea7b not found: ID does not exist" containerID="319cd592c45077cae76ed97bf3586b1261a73a2fb059bde8c4173ce3eca8ea7b"
Nov 01 03:30:58.554609 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:58.554547    4400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"319cd592c45077cae76ed97bf3586b1261a73a2fb059bde8c4173ce3eca8ea7b"} err="failed to get container status \"319cd592c45077cae76ed97bf3586b1261a73a2fb059bde8c4173ce3eca8ea7b\": rpc error: code = NotFound desc = could not find container \"319cd592c45077cae76ed97bf3586b1261a73a2fb059bde8c4173ce3eca8ea7b\": container with ID starting with 319cd592c45077cae76ed97bf3586b1261a73a2fb059bde8c4173ce3eca8ea7b not found: ID does not exist"
Nov 01 03:30:58.630882 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:58.630822    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghbbc\" (UniqueName: \"kubernetes.io/projected/71f320ef-9f70-41a4-887e-413230d65992-kube-api-access-ghbbc\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:30:58.743911 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:58.743831    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7911e5a0-68f8dd6b5c-jrj6z"]
Nov 01 03:30:58.747610 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:58.747587    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7911e5a0-68f8dd6b5c-jrj6z"]
Nov 01 03:30:59.617557 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:30:59.617527    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71f320ef-9f70-41a4-887e-413230d65992" path="/var/lib/kubelet/pods/71f320ef-9f70-41a4-887e-413230d65992/volumes"
Nov 01 03:31:01.732078 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:31:01.732040    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:31:09.193323 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:09.193287    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e6e7ea12-55895cd555-6g2bl"]
Nov 01 03:31:14.193813 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:14.193731    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-e6e7ea12-55895cd555-6g2bl" podUID="d7e99137-7930-47f4-9f70-06f8cbd811cd" containerName="payload-processing" containerID="cri-o://dd6618575471879e4fb749b557e155f70a393df4e91eeefab6fc749d4dfc7e9f" gracePeriod=25
Nov 01 03:31:15.090729 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:15.056059    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e6e7ea12-55895cd555-6g2bl"
Nov 01 03:31:15.149367 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:15.149280    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7e99137-7930-47f4-9f70-06f8cbd811cd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7e99137-7930-47f4-9f70-06f8cbd811cd-plugins-config-volume\") pod \"d7e99137-7930-47f4-9f70-06f8cbd811cd\" (UID: \"d7e99137-7930-47f4-9f70-06f8cbd811cd\") "
Nov 01 03:31:15.176713 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:15.171520    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7e99137-7930-47f4-9f70-06f8cbd811cd-kube-api-access-8pwzs\" (UniqueName: \"kubernetes.io/projected/d7e99137-7930-47f4-9f70-06f8cbd811cd-kube-api-access-8pwzs\") pod \"d7e99137-7930-47f4-9f70-06f8cbd811cd\" (UID: \"d7e99137-7930-47f4-9f70-06f8cbd811cd\") "
Nov 01 03:31:15.190663 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:15.171200    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7e99137-7930-47f4-9f70-06f8cbd811cd-plugins-config-volume" pod "d7e99137-7930-47f4-9f70-06f8cbd811cd" (UID: "d7e99137-7930-47f4-9f70-06f8cbd811cd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:31:15.198569 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:15.197891    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7e99137-7930-47f4-9f70-06f8cbd811cd-kube-api-access-8pwzs" pod "d7e99137-7930-47f4-9f70-06f8cbd811cd" (UID: "d7e99137-7930-47f4-9f70-06f8cbd811cd"). InnerVolumeSpecName "kube-api-access-8pwzs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:31:15.254612 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:15.254574    4400 generic.go:356] "Generic (PLEG): container finished" podID="d7e99137-7930-47f4-9f70-06f8cbd811cd" containerID="dd6618575471879e4fb749b557e155f70a393df4e91eeefab6fc749d4dfc7e9f" exitCode=0
Nov 01 03:31:15.255092 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:15.255065    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e6e7ea12-55895cd555-6g2bl" event={"ID":"d7e99137-7930-47f4-9f70-06f8cbd811cd","Type":"ContainerDied","Data":"dd6618575471879e4fb749b557e155f70a393df4e91eeefab6fc749d4dfc7e9f"}
Nov 01 03:31:15.259797 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:15.259767    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e6e7ea12-55895cd555-6g2bl" event={"ID":"d7e99137-7930-47f4-9f70-06f8cbd811cd","Type":"ContainerDied","Data":"396b4a7385129a9767d674dedc9c4c40abcebe88025a43760870fbe72e7e8c33"}
Nov 01 03:31:15.259985 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:15.259968    4400 scope.go:122] "RemoveContainer" containerID="dd6618575471879e4fb749b557e155f70a393df4e91eeefab6fc749d4dfc7e9f"
Nov 01 03:31:15.260259 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:15.260240    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e6e7ea12-55895cd555-6g2bl"
Nov 01 03:31:15.281040 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:15.280640    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pwzs\" (UniqueName: \"kubernetes.io/projected/d7e99137-7930-47f4-9f70-06f8cbd811cd-kube-api-access-8pwzs\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:31:15.281040 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:15.280671    4400 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7e99137-7930-47f4-9f70-06f8cbd811cd-plugins-config-volume\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:31:15.300901 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:15.300878    4400 scope.go:122] "RemoveContainer" containerID="dd6618575471879e4fb749b557e155f70a393df4e91eeefab6fc749d4dfc7e9f"
Nov 01 03:31:15.301325 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:31:15.301292    4400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd6618575471879e4fb749b557e155f70a393df4e91eeefab6fc749d4dfc7e9f\": container with ID starting with dd6618575471879e4fb749b557e155f70a393df4e91eeefab6fc749d4dfc7e9f not found: ID does not exist" containerID="dd6618575471879e4fb749b557e155f70a393df4e91eeefab6fc749d4dfc7e9f"
Nov 01 03:31:15.301434 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:15.301337    4400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd6618575471879e4fb749b557e155f70a393df4e91eeefab6fc749d4dfc7e9f"} err="failed to get container status \"dd6618575471879e4fb749b557e155f70a393df4e91eeefab6fc749d4dfc7e9f\": rpc error: code = NotFound desc = could not find container \"dd6618575471879e4fb749b557e155f70a393df4e91eeefab6fc749d4dfc7e9f\": container with ID starting with dd6618575471879e4fb749b557e155f70a393df4e91eeefab6fc749d4dfc7e9f not found: ID does not exist"
Nov 01 03:31:15.453148 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:15.453064    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e6e7ea12-55895cd555-6g2bl"]
Nov 01 03:31:15.458576 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:15.458476    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e6e7ea12-55895cd555-6g2bl"]
Nov 01 03:31:15.676100 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:15.676066    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7e99137-7930-47f4-9f70-06f8cbd811cd" path="/var/lib/kubelet/pods/d7e99137-7930-47f4-9f70-06f8cbd811cd/volumes"
Nov 01 03:31:30.002964 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:30.002929    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d5c42938-7d76f878dc-knnh7"]
Nov 01 03:31:30.003428 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:30.003176    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5c42938-7d76f878dc-knnh7" podUID="442765eb-e690-44fe-ba14-c627900cb8fc" containerName="maas-api" containerID="cri-o://f4aeded30bad1d316ee72cbcc2ca08ac0cb35be28fa3ee2d24aba98730440f81" gracePeriod=30
Nov 01 03:31:30.572378 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:30.571474    4400 generic.go:356] "Generic (PLEG): container finished" podID="442765eb-e690-44fe-ba14-c627900cb8fc" containerID="f4aeded30bad1d316ee72cbcc2ca08ac0cb35be28fa3ee2d24aba98730440f81" exitCode=0
Nov 01 03:31:30.572378 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:30.571519    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5c42938-7d76f878dc-knnh7" event={"ID":"442765eb-e690-44fe-ba14-c627900cb8fc","Type":"ContainerDied","Data":"f4aeded30bad1d316ee72cbcc2ca08ac0cb35be28fa3ee2d24aba98730440f81"}
Nov 01 03:31:30.679160 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:30.679133    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5c42938-7d76f878dc-knnh7"
Nov 01 03:31:30.697451 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:30.697425    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/442765eb-e690-44fe-ba14-c627900cb8fc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/442765eb-e690-44fe-ba14-c627900cb8fc-metrics-certs\") pod \"442765eb-e690-44fe-ba14-c627900cb8fc\" (UID: \"442765eb-e690-44fe-ba14-c627900cb8fc\") "
Nov 01 03:31:30.697607 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:30.697480    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/442765eb-e690-44fe-ba14-c627900cb8fc-kube-api-access-t2ztl\" (UniqueName: \"kubernetes.io/projected/442765eb-e690-44fe-ba14-c627900cb8fc-kube-api-access-t2ztl\") pod \"442765eb-e690-44fe-ba14-c627900cb8fc\" (UID: \"442765eb-e690-44fe-ba14-c627900cb8fc\") "
Nov 01 03:31:30.697607 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:30.697516    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/442765eb-e690-44fe-ba14-c627900cb8fc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/442765eb-e690-44fe-ba14-c627900cb8fc-maas-api-tls\") pod \"442765eb-e690-44fe-ba14-c627900cb8fc\" (UID: \"442765eb-e690-44fe-ba14-c627900cb8fc\") "
Nov 01 03:31:30.697779 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:30.697643    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/442765eb-e690-44fe-ba14-c627900cb8fc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/442765eb-e690-44fe-ba14-c627900cb8fc-openshift-service-ca\") pod \"442765eb-e690-44fe-ba14-c627900cb8fc\" (UID: \"442765eb-e690-44fe-ba14-c627900cb8fc\") "
Nov 01 03:31:30.698014 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:30.697985    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/442765eb-e690-44fe-ba14-c627900cb8fc-openshift-service-ca" pod "442765eb-e690-44fe-ba14-c627900cb8fc" (UID: "442765eb-e690-44fe-ba14-c627900cb8fc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:31:30.698971 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:30.698950    4400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/442765eb-e690-44fe-ba14-c627900cb8fc-openshift-service-ca\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:31:30.700077 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:30.700051    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/442765eb-e690-44fe-ba14-c627900cb8fc-metrics-certs" pod "442765eb-e690-44fe-ba14-c627900cb8fc" (UID: "442765eb-e690-44fe-ba14-c627900cb8fc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:31:30.700205 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:30.700180    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/442765eb-e690-44fe-ba14-c627900cb8fc-kube-api-access-t2ztl" pod "442765eb-e690-44fe-ba14-c627900cb8fc" (UID: "442765eb-e690-44fe-ba14-c627900cb8fc"). InnerVolumeSpecName "kube-api-access-t2ztl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:31:30.701117 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:30.701092    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/442765eb-e690-44fe-ba14-c627900cb8fc-maas-api-tls" pod "442765eb-e690-44fe-ba14-c627900cb8fc" (UID: "442765eb-e690-44fe-ba14-c627900cb8fc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:31:30.800830 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:30.800799    4400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/442765eb-e690-44fe-ba14-c627900cb8fc-metrics-certs\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:31:30.801054 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:30.801032    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2ztl\" (UniqueName: \"kubernetes.io/projected/442765eb-e690-44fe-ba14-c627900cb8fc-kube-api-access-t2ztl\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:31:30.801168 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:30.801146    4400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/442765eb-e690-44fe-ba14-c627900cb8fc-maas-api-tls\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:31:31.186711 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:31.183701    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7911e5a0-6dbc75fc4d-grqqm"]
Nov 01 03:31:31.186711 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:31.183971    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7911e5a0-6dbc75fc4d-grqqm" podUID="c9569e45-00e1-4480-9335-6b41732856e9" containerName="maas-api" containerID="cri-o://800df96e28090adeb79cab13255b55959836dcf017de355d82ccf7df1e7a4aba" gracePeriod=30
Nov 01 03:31:31.815364 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:31.812156    4400 generic.go:356] "Generic (PLEG): container finished" podID="c9569e45-00e1-4480-9335-6b41732856e9" containerID="800df96e28090adeb79cab13255b55959836dcf017de355d82ccf7df1e7a4aba" exitCode=0
Nov 01 03:31:31.828211 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:31.828088    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5c42938-7d76f878dc-knnh7"
Nov 01 03:31:31.844151 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:31.842849    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7911e5a0-6dbc75fc4d-grqqm"
Nov 01 03:31:31.848571 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:31.848522    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7911e5a0-6dbc75fc4d-grqqm" event={"ID":"c9569e45-00e1-4480-9335-6b41732856e9","Type":"ContainerDied","Data":"800df96e28090adeb79cab13255b55959836dcf017de355d82ccf7df1e7a4aba"}
Nov 01 03:31:31.852045 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:31.852021    4400 scope.go:122] "RemoveContainer" containerID="800df96e28090adeb79cab13255b55959836dcf017de355d82ccf7df1e7a4aba"
Nov 01 03:31:31.852203 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:31.851991    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5c42938-7d76f878dc-knnh7" event={"ID":"442765eb-e690-44fe-ba14-c627900cb8fc","Type":"ContainerDied","Data":"f3d4b11e532f677efa4ffdab7f78391c40f4724d744e642aefcdb324ce588b08"}
Nov 01 03:31:31.868088 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:31.868066    4400 scope.go:122] "RemoveContainer" containerID="f4aeded30bad1d316ee72cbcc2ca08ac0cb35be28fa3ee2d24aba98730440f81"
Nov 01 03:31:31.879519 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:31:31.879465    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:31:32.028462 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:32.026737    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9569e45-00e1-4480-9335-6b41732856e9-kube-api-access-dwl8v\" (UniqueName: \"kubernetes.io/projected/c9569e45-00e1-4480-9335-6b41732856e9-kube-api-access-dwl8v\") pod \"c9569e45-00e1-4480-9335-6b41732856e9\" (UID: \"c9569e45-00e1-4480-9335-6b41732856e9\") "
Nov 01 03:31:32.028462 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:32.026810    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9569e45-00e1-4480-9335-6b41732856e9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9569e45-00e1-4480-9335-6b41732856e9-maas-api-tls\") pod \"c9569e45-00e1-4480-9335-6b41732856e9\" (UID: \"c9569e45-00e1-4480-9335-6b41732856e9\") "
Nov 01 03:31:32.028462 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:32.026896    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9569e45-00e1-4480-9335-6b41732856e9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9569e45-00e1-4480-9335-6b41732856e9-openshift-service-ca\") pod \"c9569e45-00e1-4480-9335-6b41732856e9\" (UID: \"c9569e45-00e1-4480-9335-6b41732856e9\") "
Nov 01 03:31:32.028462 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:32.027003    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9569e45-00e1-4480-9335-6b41732856e9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9569e45-00e1-4480-9335-6b41732856e9-metrics-certs\") pod \"c9569e45-00e1-4480-9335-6b41732856e9\" (UID: \"c9569e45-00e1-4480-9335-6b41732856e9\") "
Nov 01 03:31:32.036955 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:32.034061    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9569e45-00e1-4480-9335-6b41732856e9-openshift-service-ca" pod "c9569e45-00e1-4480-9335-6b41732856e9" (UID: "c9569e45-00e1-4480-9335-6b41732856e9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:31:32.045044 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:32.042636    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d5c42938-7d76f878dc-knnh7"]
Nov 01 03:31:32.045044 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:32.044215    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9569e45-00e1-4480-9335-6b41732856e9-metrics-certs" pod "c9569e45-00e1-4480-9335-6b41732856e9" (UID: "c9569e45-00e1-4480-9335-6b41732856e9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:31:32.047097 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:32.047069    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9569e45-00e1-4480-9335-6b41732856e9-maas-api-tls" pod "c9569e45-00e1-4480-9335-6b41732856e9" (UID: "c9569e45-00e1-4480-9335-6b41732856e9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:31:32.048980 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:32.048948    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9569e45-00e1-4480-9335-6b41732856e9-kube-api-access-dwl8v" pod "c9569e45-00e1-4480-9335-6b41732856e9" (UID: "c9569e45-00e1-4480-9335-6b41732856e9"). InnerVolumeSpecName "kube-api-access-dwl8v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:31:32.068092 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:32.068066    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d5c42938-7d76f878dc-knnh7"]
Nov 01 03:31:32.130134 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:32.130104    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwl8v\" (UniqueName: \"kubernetes.io/projected/c9569e45-00e1-4480-9335-6b41732856e9-kube-api-access-dwl8v\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:31:32.130393 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:32.130372    4400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9569e45-00e1-4480-9335-6b41732856e9-maas-api-tls\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:31:32.130530 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:32.130515    4400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9569e45-00e1-4480-9335-6b41732856e9-openshift-service-ca\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:31:32.130631 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:32.130618    4400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9569e45-00e1-4480-9335-6b41732856e9-metrics-certs\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:31:32.840059 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:32.840018    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7911e5a0-6dbc75fc4d-grqqm" event={"ID":"c9569e45-00e1-4480-9335-6b41732856e9","Type":"ContainerDied","Data":"487462f7aeea5026ce2c7ba7ed717c6f472e729b2cb1d58a536db92abe20dab9"}
Nov 01 03:31:32.840471 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:32.840031    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7911e5a0-6dbc75fc4d-grqqm"
Nov 01 03:31:33.020726 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:33.020649    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7911e5a0-6dbc75fc4d-grqqm"]
Nov 01 03:31:33.072909 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:33.072864    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7911e5a0-6dbc75fc4d-grqqm"]
Nov 01 03:31:33.617265 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:33.617236    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="442765eb-e690-44fe-ba14-c627900cb8fc" path="/var/lib/kubelet/pods/442765eb-e690-44fe-ba14-c627900cb8fc/volumes"
Nov 01 03:31:33.617769 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:33.617746    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9569e45-00e1-4480-9335-6b41732856e9" path="/var/lib/kubelet/pods/c9569e45-00e1-4480-9335-6b41732856e9/volumes"
Nov 01 03:31:41.706824 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:41.705864    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d262c155-d97d78c75-8hcz2"]
Nov 01 03:31:41.707891 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:41.707868    4400 cpu_manager.go:414] "containerMap: removing container" podUID="71f320ef-9f70-41a4-887e-413230d65992" containerName="payload-processing"
Nov 01 03:31:41.708081 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:41.708062    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71f320ef-9f70-41a4-887e-413230d65992" containerName="payload-processing"
Nov 01 03:31:41.708238 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:41.708225    4400 cpu_manager.go:414] "containerMap: removing container" podUID="d7e99137-7930-47f4-9f70-06f8cbd811cd" containerName="payload-processing"
Nov 01 03:31:41.708456 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:41.708433    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7e99137-7930-47f4-9f70-06f8cbd811cd" containerName="payload-processing"
Nov 01 03:31:41.708704 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:41.708605    4400 cpu_manager.go:414] "containerMap: removing container" podUID="c9569e45-00e1-4480-9335-6b41732856e9" containerName="maas-api"
Nov 01 03:31:41.708704 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:41.708621    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9569e45-00e1-4480-9335-6b41732856e9" containerName="maas-api"
Nov 01 03:31:41.708704 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:41.708664    4400 cpu_manager.go:414] "containerMap: removing container" podUID="0c264853-8d3b-4e19-8872-224f7aee939c" containerName="payload-pre-processing"
Nov 01 03:31:41.709149 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:41.708675    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c264853-8d3b-4e19-8872-224f7aee939c" containerName="payload-pre-processing"
Nov 01 03:31:41.709149 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:41.708953    4400 cpu_manager.go:414] "containerMap: removing container" podUID="442765eb-e690-44fe-ba14-c627900cb8fc" containerName="maas-api"
Nov 01 03:31:41.709149 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:41.708962    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="442765eb-e690-44fe-ba14-c627900cb8fc" containerName="maas-api"
Nov 01 03:31:41.709149 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:41.709012    4400 cpu_manager.go:414] "containerMap: removing container" podUID="71e81e40-e46d-4981-a403-d9a74bdc5957" containerName="payload-processing"
Nov 01 03:31:41.709149 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:41.709020    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71e81e40-e46d-4981-a403-d9a74bdc5957" containerName="payload-processing"
Nov 01 03:31:41.709772 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:41.709554    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9569e45-00e1-4480-9335-6b41732856e9" containerName="maas-api"
Nov 01 03:31:41.709772 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:41.709591    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c264853-8d3b-4e19-8872-224f7aee939c" containerName="payload-pre-processing"
Nov 01 03:31:41.709772 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:41.709609    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="442765eb-e690-44fe-ba14-c627900cb8fc" containerName="maas-api"
Nov 01 03:31:41.709772 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:41.709624    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="71e81e40-e46d-4981-a403-d9a74bdc5957" containerName="payload-processing"
Nov 01 03:31:41.709772 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:41.709634    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="71f320ef-9f70-41a4-887e-413230d65992" containerName="payload-processing"
Nov 01 03:31:41.709772 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:41.709646    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7e99137-7930-47f4-9f70-06f8cbd811cd" containerName="payload-processing"
Nov 01 03:31:41.730725 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:41.723390    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d262c155-d97d78c75-8hcz2"]
Nov 01 03:31:41.730725 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:41.723511    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d262c155-d97d78c75-8hcz2"
Nov 01 03:31:41.810712 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:41.807071    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-d262c155-dockercfg-dhttk\""
Nov 01 03:31:41.876189 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:41.873223    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-d262c155\""
Nov 01 03:31:41.999761 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:41.956474    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ae8054b-cf08-42e1-acfc-f1461e647095-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-d262c155-d97d78c75-8hcz2\" (UID: \"1ae8054b-cf08-42e1-acfc-f1461e647095\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d262c155-d97d78c75-8hcz2"
Nov 01 03:31:41.999761 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:41.956560    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j94kl\" (UniqueName: \"kubernetes.io/projected/1ae8054b-cf08-42e1-acfc-f1461e647095-kube-api-access-j94kl\") pod \"payload-pre-processing-e2e-migrate-d262c155-d97d78c75-8hcz2\" (UID: \"1ae8054b-cf08-42e1-acfc-f1461e647095\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d262c155-d97d78c75-8hcz2"
Nov 01 03:31:42.059324 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:42.059286    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ae8054b-cf08-42e1-acfc-f1461e647095-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-d262c155-d97d78c75-8hcz2\" (UID: \"1ae8054b-cf08-42e1-acfc-f1461e647095\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d262c155-d97d78c75-8hcz2"
Nov 01 03:31:42.059535 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:42.059343    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j94kl\" (UniqueName: \"kubernetes.io/projected/1ae8054b-cf08-42e1-acfc-f1461e647095-kube-api-access-j94kl\") pod \"payload-pre-processing-e2e-migrate-d262c155-d97d78c75-8hcz2\" (UID: \"1ae8054b-cf08-42e1-acfc-f1461e647095\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d262c155-d97d78c75-8hcz2"
Nov 01 03:31:42.060087 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:42.060066    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ae8054b-cf08-42e1-acfc-f1461e647095-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-d262c155-d97d78c75-8hcz2\" (UID: \"1ae8054b-cf08-42e1-acfc-f1461e647095\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d262c155-d97d78c75-8hcz2"
Nov 01 03:31:42.122539 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:42.122504    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j94kl\" (UniqueName: \"kubernetes.io/projected/1ae8054b-cf08-42e1-acfc-f1461e647095-kube-api-access-j94kl\") pod \"payload-pre-processing-e2e-migrate-d262c155-d97d78c75-8hcz2\" (UID: \"1ae8054b-cf08-42e1-acfc-f1461e647095\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d262c155-d97d78c75-8hcz2"
Nov 01 03:31:42.299970 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:42.299889    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d262c155-d97d78c75-8hcz2"
Nov 01 03:31:42.687714 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:31:42.687578    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ae8054b_cf08_42e1_acfc_f1461e647095.slice/crio-d0cf3bd95acbdd19cbdee30714bd45c90b11092bc84ab7bbf4e4982aa9d4cd24 WatchSource:0}: Error finding container d0cf3bd95acbdd19cbdee30714bd45c90b11092bc84ab7bbf4e4982aa9d4cd24: Status 404 returned error can't find the container with id d0cf3bd95acbdd19cbdee30714bd45c90b11092bc84ab7bbf4e4982aa9d4cd24
Nov 01 03:31:42.689640 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:42.689607    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d262c155-d97d78c75-8hcz2"]
Nov 01 03:31:43.028938 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:43.028846    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d262c155-d97d78c75-8hcz2" event={"ID":"1ae8054b-cf08-42e1-acfc-f1461e647095","Type":"ContainerStarted","Data":"f4caea899c34d807b74d2798ef29c002df3194413df3ea2711a00b28eb64f2db"}
Nov 01 03:31:43.028938 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:43.028909    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d262c155-d97d78c75-8hcz2" event={"ID":"1ae8054b-cf08-42e1-acfc-f1461e647095","Type":"ContainerStarted","Data":"d0cf3bd95acbdd19cbdee30714bd45c90b11092bc84ab7bbf4e4982aa9d4cd24"}
Nov 01 03:31:43.029369 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:43.028944    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d262c155-d97d78c75-8hcz2"
Nov 01 03:31:43.178705 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:43.177483    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d262c155-d97d78c75-8hcz2" podStartSLOduration=2.177463329 podStartE2EDuration="2.177463329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:31:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:31:43.165508662 +0000 UTC m=+1509.680154527" watchObservedRunningTime="2026-11-01 03:31:43.177463329 +0000 UTC m=+1509.692109194"
Nov 01 03:31:43.222416 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:43.222384    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e6e7ea12-5fc4d966cf-fllxv"]
Nov 01 03:31:43.222660 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:43.222627    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e6e7ea12-5fc4d966cf-fllxv" podUID="1c79d2fd-b23e-486b-b49c-1c8e401a75a7" containerName="maas-api" containerID="cri-o://60941bada9101829cc831e399dca3c549b1c2be28cdfc1ea9d822326ae1b589f" gracePeriod=30
Nov 01 03:31:43.743379 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:43.729595    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e6e7ea12-5fc4d966cf-fllxv"
Nov 01 03:31:43.799710 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:43.788626    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c79d2fd-b23e-486b-b49c-1c8e401a75a7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c79d2fd-b23e-486b-b49c-1c8e401a75a7-metrics-certs\") pod \"1c79d2fd-b23e-486b-b49c-1c8e401a75a7\" (UID: \"1c79d2fd-b23e-486b-b49c-1c8e401a75a7\") "
Nov 01 03:31:43.799710 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:43.788784    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c79d2fd-b23e-486b-b49c-1c8e401a75a7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c79d2fd-b23e-486b-b49c-1c8e401a75a7-maas-api-tls\") pod \"1c79d2fd-b23e-486b-b49c-1c8e401a75a7\" (UID: \"1c79d2fd-b23e-486b-b49c-1c8e401a75a7\") "
Nov 01 03:31:43.799710 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:43.788842    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c79d2fd-b23e-486b-b49c-1c8e401a75a7-kube-api-access-h5kh9\" (UniqueName: \"kubernetes.io/projected/1c79d2fd-b23e-486b-b49c-1c8e401a75a7-kube-api-access-h5kh9\") pod \"1c79d2fd-b23e-486b-b49c-1c8e401a75a7\" (UID: \"1c79d2fd-b23e-486b-b49c-1c8e401a75a7\") "
Nov 01 03:31:43.799710 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:43.788957    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c79d2fd-b23e-486b-b49c-1c8e401a75a7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c79d2fd-b23e-486b-b49c-1c8e401a75a7-openshift-service-ca\") pod \"1c79d2fd-b23e-486b-b49c-1c8e401a75a7\" (UID: \"1c79d2fd-b23e-486b-b49c-1c8e401a75a7\") "
Nov 01 03:31:43.799710 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:43.790338    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c79d2fd-b23e-486b-b49c-1c8e401a75a7-openshift-service-ca" pod "1c79d2fd-b23e-486b-b49c-1c8e401a75a7" (UID: "1c79d2fd-b23e-486b-b49c-1c8e401a75a7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:31:43.802239 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:43.802184    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c79d2fd-b23e-486b-b49c-1c8e401a75a7-maas-api-tls" pod "1c79d2fd-b23e-486b-b49c-1c8e401a75a7" (UID: "1c79d2fd-b23e-486b-b49c-1c8e401a75a7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:31:43.815725 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:43.811805    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c79d2fd-b23e-486b-b49c-1c8e401a75a7-metrics-certs" pod "1c79d2fd-b23e-486b-b49c-1c8e401a75a7" (UID: "1c79d2fd-b23e-486b-b49c-1c8e401a75a7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:31:43.815725 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:43.811997    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c79d2fd-b23e-486b-b49c-1c8e401a75a7-kube-api-access-h5kh9" pod "1c79d2fd-b23e-486b-b49c-1c8e401a75a7" (UID: "1c79d2fd-b23e-486b-b49c-1c8e401a75a7"). InnerVolumeSpecName "kube-api-access-h5kh9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:31:43.897098 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:43.896711    4400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c79d2fd-b23e-486b-b49c-1c8e401a75a7-metrics-certs\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:31:43.897357 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:43.897336    4400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c79d2fd-b23e-486b-b49c-1c8e401a75a7-maas-api-tls\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:31:43.897473 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:43.897460    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5kh9\" (UniqueName: \"kubernetes.io/projected/1c79d2fd-b23e-486b-b49c-1c8e401a75a7-kube-api-access-h5kh9\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:31:43.897550 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:43.897541    4400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c79d2fd-b23e-486b-b49c-1c8e401a75a7-openshift-service-ca\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:31:44.037972 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:44.037941    4400 generic.go:356] "Generic (PLEG): container finished" podID="1c79d2fd-b23e-486b-b49c-1c8e401a75a7" containerID="60941bada9101829cc831e399dca3c549b1c2be28cdfc1ea9d822326ae1b589f" exitCode=0
Nov 01 03:31:44.038391 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:44.037991    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e6e7ea12-5fc4d966cf-fllxv"
Nov 01 03:31:44.038391 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:44.038021    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e6e7ea12-5fc4d966cf-fllxv" event={"ID":"1c79d2fd-b23e-486b-b49c-1c8e401a75a7","Type":"ContainerDied","Data":"60941bada9101829cc831e399dca3c549b1c2be28cdfc1ea9d822326ae1b589f"}
Nov 01 03:31:44.038391 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:44.038069    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e6e7ea12-5fc4d966cf-fllxv" event={"ID":"1c79d2fd-b23e-486b-b49c-1c8e401a75a7","Type":"ContainerDied","Data":"cd5adf3b3cea6d4f7e8d2fb63757a215529586c9efcf81b027be54060661981d"}
Nov 01 03:31:44.038391 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:44.038091    4400 scope.go:122] "RemoveContainer" containerID="60941bada9101829cc831e399dca3c549b1c2be28cdfc1ea9d822326ae1b589f"
Nov 01 03:31:44.054151 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:44.053743    4400 scope.go:122] "RemoveContainer" containerID="60941bada9101829cc831e399dca3c549b1c2be28cdfc1ea9d822326ae1b589f"
Nov 01 03:31:44.054151 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:31:44.054076    4400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60941bada9101829cc831e399dca3c549b1c2be28cdfc1ea9d822326ae1b589f\": container with ID starting with 60941bada9101829cc831e399dca3c549b1c2be28cdfc1ea9d822326ae1b589f not found: ID does not exist" containerID="60941bada9101829cc831e399dca3c549b1c2be28cdfc1ea9d822326ae1b589f"
Nov 01 03:31:44.054151 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:44.054102    4400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60941bada9101829cc831e399dca3c549b1c2be28cdfc1ea9d822326ae1b589f"} err="failed to get container status \"60941bada9101829cc831e399dca3c549b1c2be28cdfc1ea9d822326ae1b589f\": rpc error: code = NotFound desc = could not find container \"60941bada9101829cc831e399dca3c549b1c2be28cdfc1ea9d822326ae1b589f\": container with ID starting with 60941bada9101829cc831e399dca3c549b1c2be28cdfc1ea9d822326ae1b589f not found: ID does not exist"
Nov 01 03:31:44.285761 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:44.275743    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e6e7ea12-5fc4d966cf-fllxv"]
Nov 01 03:31:44.306708 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:44.291464    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e6e7ea12-5fc4d966cf-fllxv"]
Nov 01 03:31:45.616165 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:45.616132    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c79d2fd-b23e-486b-b49c-1c8e401a75a7" path="/var/lib/kubelet/pods/1c79d2fd-b23e-486b-b49c-1c8e401a75a7/volumes"
Nov 01 03:31:46.829404 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:46.829363    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3513c2f1-fc98d8f8c-rml8z"]
Nov 01 03:31:46.831942 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:46.831913    4400 cpu_manager.go:414] "containerMap: removing container" podUID="1c79d2fd-b23e-486b-b49c-1c8e401a75a7" containerName="maas-api"
Nov 01 03:31:46.832116 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:46.832100    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c79d2fd-b23e-486b-b49c-1c8e401a75a7" containerName="maas-api"
Nov 01 03:31:46.832751 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:46.832730    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c79d2fd-b23e-486b-b49c-1c8e401a75a7" containerName="maas-api"
Nov 01 03:31:46.849407 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:46.849373    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-3513c2f1-fc98d8f8c-rml8z"
Nov 01 03:31:46.862519 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:46.860138    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3513c2f1-fc98d8f8c-rml8z"]
Nov 01 03:31:46.985742 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:46.980890    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-3513c2f1-dockercfg-9xznf\""
Nov 01 03:31:46.985742 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:46.980998    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-3513c2f1\""
Nov 01 03:31:47.053715 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:47.024515    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85be6802-a287-49ec-8fb9-19f94703f0b0-plugins-config-volume\") pod \"payload-processing-e2e-derive-3513c2f1-fc98d8f8c-rml8z\" (UID: \"85be6802-a287-49ec-8fb9-19f94703f0b0\") " pod="openshift-ingress/payload-processing-e2e-derive-3513c2f1-fc98d8f8c-rml8z"
Nov 01 03:31:47.153764 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:47.136878    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg6wz\" (UniqueName: \"kubernetes.io/projected/85be6802-a287-49ec-8fb9-19f94703f0b0-kube-api-access-pg6wz\") pod \"payload-processing-e2e-derive-3513c2f1-fc98d8f8c-rml8z\" (UID: \"85be6802-a287-49ec-8fb9-19f94703f0b0\") " pod="openshift-ingress/payload-processing-e2e-derive-3513c2f1-fc98d8f8c-rml8z"
Nov 01 03:31:47.331711 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:47.290801    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3513c2f1-5845c474f5-cs2th"]
Nov 01 03:31:47.392711 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:47.369806    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85be6802-a287-49ec-8fb9-19f94703f0b0-plugins-config-volume\") pod \"payload-processing-e2e-derive-3513c2f1-fc98d8f8c-rml8z\" (UID: \"85be6802-a287-49ec-8fb9-19f94703f0b0\") " pod="openshift-ingress/payload-processing-e2e-derive-3513c2f1-fc98d8f8c-rml8z"
Nov 01 03:31:47.392711 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:47.379927    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg6wz\" (UniqueName: \"kubernetes.io/projected/85be6802-a287-49ec-8fb9-19f94703f0b0-kube-api-access-pg6wz\") pod \"payload-processing-e2e-derive-3513c2f1-fc98d8f8c-rml8z\" (UID: \"85be6802-a287-49ec-8fb9-19f94703f0b0\") " pod="openshift-ingress/payload-processing-e2e-derive-3513c2f1-fc98d8f8c-rml8z"
Nov 01 03:31:47.416008 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:47.415932    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85be6802-a287-49ec-8fb9-19f94703f0b0-plugins-config-volume\") pod \"payload-processing-e2e-derive-3513c2f1-fc98d8f8c-rml8z\" (UID: \"85be6802-a287-49ec-8fb9-19f94703f0b0\") " pod="openshift-ingress/payload-processing-e2e-derive-3513c2f1-fc98d8f8c-rml8z"
Nov 01 03:31:47.529625 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:47.529600    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-3513c2f1-5845c474f5-cs2th"
Nov 01 03:31:47.531129 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:47.529779    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg6wz\" (UniqueName: \"kubernetes.io/projected/85be6802-a287-49ec-8fb9-19f94703f0b0-kube-api-access-pg6wz\") pod \"payload-processing-e2e-derive-3513c2f1-fc98d8f8c-rml8z\" (UID: \"85be6802-a287-49ec-8fb9-19f94703f0b0\") " pod="openshift-ingress/payload-processing-e2e-derive-3513c2f1-fc98d8f8c-rml8z"
Nov 01 03:31:47.531129 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:47.529809    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3513c2f1-5845c474f5-cs2th"]
Nov 01 03:31:47.606054 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:47.606025    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c96z6\" (UniqueName: \"kubernetes.io/projected/3d09cd59-1b40-4a61-b821-e22c482f3d3a-kube-api-access-c96z6\") pod \"payload-pre-processing-e2e-derive-3513c2f1-5845c474f5-cs2th\" (UID: \"3d09cd59-1b40-4a61-b821-e22c482f3d3a\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3513c2f1-5845c474f5-cs2th"
Nov 01 03:31:47.606264 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:47.606095    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d09cd59-1b40-4a61-b821-e22c482f3d3a-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-3513c2f1-5845c474f5-cs2th\" (UID: \"3d09cd59-1b40-4a61-b821-e22c482f3d3a\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3513c2f1-5845c474f5-cs2th"
Nov 01 03:31:47.650883 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:47.650847    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-3513c2f1-fc98d8f8c-rml8z"
Nov 01 03:31:47.708870 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:47.708470    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c96z6\" (UniqueName: \"kubernetes.io/projected/3d09cd59-1b40-4a61-b821-e22c482f3d3a-kube-api-access-c96z6\") pod \"payload-pre-processing-e2e-derive-3513c2f1-5845c474f5-cs2th\" (UID: \"3d09cd59-1b40-4a61-b821-e22c482f3d3a\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3513c2f1-5845c474f5-cs2th"
Nov 01 03:31:47.709051 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:47.708934    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d09cd59-1b40-4a61-b821-e22c482f3d3a-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-3513c2f1-5845c474f5-cs2th\" (UID: \"3d09cd59-1b40-4a61-b821-e22c482f3d3a\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3513c2f1-5845c474f5-cs2th"
Nov 01 03:31:47.709809 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:47.709763    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d09cd59-1b40-4a61-b821-e22c482f3d3a-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-3513c2f1-5845c474f5-cs2th\" (UID: \"3d09cd59-1b40-4a61-b821-e22c482f3d3a\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3513c2f1-5845c474f5-cs2th"
Nov 01 03:31:47.771234 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:47.771194    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c96z6\" (UniqueName: \"kubernetes.io/projected/3d09cd59-1b40-4a61-b821-e22c482f3d3a-kube-api-access-c96z6\") pod \"payload-pre-processing-e2e-derive-3513c2f1-5845c474f5-cs2th\" (UID: \"3d09cd59-1b40-4a61-b821-e22c482f3d3a\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3513c2f1-5845c474f5-cs2th"
Nov 01 03:31:47.898856 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:47.898823    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-3513c2f1-5845c474f5-cs2th"
Nov 01 03:31:48.037273 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:31:48.037242    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85be6802_a287_49ec_8fb9_19f94703f0b0.slice/crio-c67def94ef6099c5e0b4d786be375421f5d08001fd3aa8be2677f91b2797c783 WatchSource:0}: Error finding container c67def94ef6099c5e0b4d786be375421f5d08001fd3aa8be2677f91b2797c783: Status 404 returned error can't find the container with id c67def94ef6099c5e0b4d786be375421f5d08001fd3aa8be2677f91b2797c783
Nov 01 03:31:48.039067 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:48.039045    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3513c2f1-fc98d8f8c-rml8z"]
Nov 01 03:31:48.283805 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:31:48.283778    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d09cd59_1b40_4a61_b821_e22c482f3d3a.slice/crio-a9b0a71a432e0b6cd2aa89ae1570f679aecac31917a2bf774c775fbd4a9c6812 WatchSource:0}: Error finding container a9b0a71a432e0b6cd2aa89ae1570f679aecac31917a2bf774c775fbd4a9c6812: Status 404 returned error can't find the container with id a9b0a71a432e0b6cd2aa89ae1570f679aecac31917a2bf774c775fbd4a9c6812
Nov 01 03:31:48.284498 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:48.284466    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3513c2f1-5845c474f5-cs2th"]
Nov 01 03:31:48.473907 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:48.473871    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-3513c2f1-5845c474f5-cs2th" event={"ID":"3d09cd59-1b40-4a61-b821-e22c482f3d3a","Type":"ContainerStarted","Data":"aef2c6c2752ab5a5775b03844788a8531223177eb8b3e4c5d59b0371f5c45b4f"}
Nov 01 03:31:48.474111 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:48.473928    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-3513c2f1-5845c474f5-cs2th" event={"ID":"3d09cd59-1b40-4a61-b821-e22c482f3d3a","Type":"ContainerStarted","Data":"a9b0a71a432e0b6cd2aa89ae1570f679aecac31917a2bf774c775fbd4a9c6812"}
Nov 01 03:31:48.474111 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:48.473960    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-3513c2f1-5845c474f5-cs2th"
Nov 01 03:31:48.475498 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:48.475462    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-3513c2f1-fc98d8f8c-rml8z" event={"ID":"85be6802-a287-49ec-8fb9-19f94703f0b0","Type":"ContainerStarted","Data":"34c6ac7a1b6f9b88dcb3b26ebb5520d75a8e5b7070c6588a4efe790a3bc87c60"}
Nov 01 03:31:48.475610 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:48.475518    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-3513c2f1-fc98d8f8c-rml8z" event={"ID":"85be6802-a287-49ec-8fb9-19f94703f0b0","Type":"ContainerStarted","Data":"c67def94ef6099c5e0b4d786be375421f5d08001fd3aa8be2677f91b2797c783"}
Nov 01 03:31:48.475610 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:48.475595    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-3513c2f1-fc98d8f8c-rml8z"
Nov 01 03:31:48.592666 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:48.592616    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-3513c2f1-5845c474f5-cs2th" podStartSLOduration=2.5926015529999997 podStartE2EDuration="2.592601553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:31:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:31:48.591763958 +0000 UTC m=+1515.106409822" watchObservedRunningTime="2026-11-01 03:31:48.592601553 +0000 UTC m=+1515.107247416"
Nov 01 03:31:48.722728 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:48.721207    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-3513c2f1-fc98d8f8c-rml8z" podStartSLOduration=2.7211593020000002 podStartE2EDuration="2.721159302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:31:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:31:48.719059113 +0000 UTC m=+1515.233704977" watchObservedRunningTime="2026-11-01 03:31:48.721159302 +0000 UTC m=+1515.235805166"
Nov 01 03:31:54.041507 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:54.041476    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d262c155-d97d78c75-8hcz2"
Nov 01 03:31:59.485747 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:59.485716    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-3513c2f1-fc98d8f8c-rml8z"
Nov 01 03:31:59.486333 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:31:59.485801    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-3513c2f1-5845c474f5-cs2th"
Nov 01 03:32:01.731421 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:32:01.731384    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:32:04.630640 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:32:04.630607    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 03:32:23.933200 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:32:23.933167    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d262c155-6f74d6b7c4-rld8l"]
Nov 01 03:32:23.937847 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:32:23.937827    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d262c155-6f74d6b7c4-rld8l"
Nov 01 03:32:23.997304 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:32:23.997271    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d262c155-6f74d6b7c4-rld8l"]
Nov 01 03:32:24.031486 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:32:24.031461    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6084708e-57aa-4b1a-8dfe-d50e45585616-openshift-service-ca\") pod \"maas-api-e2e-migrate-d262c155-6f74d6b7c4-rld8l\" (UID: \"6084708e-57aa-4b1a-8dfe-d50e45585616\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d262c155-6f74d6b7c4-rld8l"
Nov 01 03:32:24.031637 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:32:24.031616    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6084708e-57aa-4b1a-8dfe-d50e45585616-metrics-certs\") pod \"maas-api-e2e-migrate-d262c155-6f74d6b7c4-rld8l\" (UID: \"6084708e-57aa-4b1a-8dfe-d50e45585616\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d262c155-6f74d6b7c4-rld8l"
Nov 01 03:32:24.031770 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:32:24.031754    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6084708e-57aa-4b1a-8dfe-d50e45585616-maas-api-tls\") pod \"maas-api-e2e-migrate-d262c155-6f74d6b7c4-rld8l\" (UID: \"6084708e-57aa-4b1a-8dfe-d50e45585616\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d262c155-6f74d6b7c4-rld8l"
Nov 01 03:32:24.031831 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:32:24.031793    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf4gn\" (UniqueName: \"kubernetes.io/projected/6084708e-57aa-4b1a-8dfe-d50e45585616-kube-api-access-kf4gn\") pod \"maas-api-e2e-migrate-d262c155-6f74d6b7c4-rld8l\" (UID: \"6084708e-57aa-4b1a-8dfe-d50e45585616\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d262c155-6f74d6b7c4-rld8l"
Nov 01 03:32:24.042258 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:32:24.042205    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-d262c155-serving-cert\""
Nov 01 03:32:24.133161 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:32:24.133135    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6084708e-57aa-4b1a-8dfe-d50e45585616-openshift-service-ca\") pod \"maas-api-e2e-migrate-d262c155-6f74d6b7c4-rld8l\" (UID: \"6084708e-57aa-4b1a-8dfe-d50e45585616\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d262c155-6f74d6b7c4-rld8l"
Nov 01 03:32:24.133321 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:32:24.133244    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6084708e-57aa-4b1a-8dfe-d50e45585616-metrics-certs\") pod \"maas-api-e2e-migrate-d262c155-6f74d6b7c4-rld8l\" (UID: \"6084708e-57aa-4b1a-8dfe-d50e45585616\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d262c155-6f74d6b7c4-rld8l"
Nov 01 03:32:24.133321 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:32:24.133315    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6084708e-57aa-4b1a-8dfe-d50e45585616-maas-api-tls\") pod \"maas-api-e2e-migrate-d262c155-6f74d6b7c4-rld8l\" (UID: \"6084708e-57aa-4b1a-8dfe-d50e45585616\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d262c155-6f74d6b7c4-rld8l"
Nov 01 03:32:24.133398 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:32:24.133343    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf4gn\" (UniqueName: \"kubernetes.io/projected/6084708e-57aa-4b1a-8dfe-d50e45585616-kube-api-access-kf4gn\") pod \"maas-api-e2e-migrate-d262c155-6f74d6b7c4-rld8l\" (UID: \"6084708e-57aa-4b1a-8dfe-d50e45585616\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d262c155-6f74d6b7c4-rld8l"
Nov 01 03:32:24.133433 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:32:24.133415    4400 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-d262c155-serving-cert: secret "maas-api-e2e-migrate-d262c155-serving-cert" not found
Nov 01 03:32:24.133502 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:32:24.133489    4400 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6084708e-57aa-4b1a-8dfe-d50e45585616-maas-api-tls podName:6084708e-57aa-4b1a-8dfe-d50e45585616 nodeName:}" failed. No retries permitted until 2026-11-01 03:32:24.633468032 +0000 UTC m=+1551.148113873 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6084708e-57aa-4b1a-8dfe-d50e45585616-maas-api-tls") pod "maas-api-e2e-migrate-d262c155-6f74d6b7c4-rld8l" (UID: "6084708e-57aa-4b1a-8dfe-d50e45585616") : secret "maas-api-e2e-migrate-d262c155-serving-cert" not found
Nov 01 03:32:24.133853 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:32:24.133834    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6084708e-57aa-4b1a-8dfe-d50e45585616-openshift-service-ca\") pod \"maas-api-e2e-migrate-d262c155-6f74d6b7c4-rld8l\" (UID: \"6084708e-57aa-4b1a-8dfe-d50e45585616\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d262c155-6f74d6b7c4-rld8l"
Nov 01 03:32:24.135868 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:32:24.135846    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6084708e-57aa-4b1a-8dfe-d50e45585616-metrics-certs\") pod \"maas-api-e2e-migrate-d262c155-6f74d6b7c4-rld8l\" (UID: \"6084708e-57aa-4b1a-8dfe-d50e45585616\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d262c155-6f74d6b7c4-rld8l"
Nov 01 03:32:24.214757 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:32:24.208942    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf4gn\" (UniqueName: \"kubernetes.io/projected/6084708e-57aa-4b1a-8dfe-d50e45585616-kube-api-access-kf4gn\") pod \"maas-api-e2e-migrate-d262c155-6f74d6b7c4-rld8l\" (UID: \"6084708e-57aa-4b1a-8dfe-d50e45585616\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d262c155-6f74d6b7c4-rld8l"
Nov 01 03:32:24.658811 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:32:24.658780    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6084708e-57aa-4b1a-8dfe-d50e45585616-maas-api-tls\") pod \"maas-api-e2e-migrate-d262c155-6f74d6b7c4-rld8l\" (UID: \"6084708e-57aa-4b1a-8dfe-d50e45585616\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d262c155-6f74d6b7c4-rld8l"
Nov 01 03:32:24.661331 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:32:24.661306    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6084708e-57aa-4b1a-8dfe-d50e45585616-maas-api-tls\") pod \"maas-api-e2e-migrate-d262c155-6f74d6b7c4-rld8l\" (UID: \"6084708e-57aa-4b1a-8dfe-d50e45585616\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d262c155-6f74d6b7c4-rld8l"
Nov 01 03:32:24.855415 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:32:24.855371    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d262c155-6f74d6b7c4-rld8l"
Nov 01 03:32:25.240952 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:32:25.240920    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6084708e_57aa_4b1a_8dfe_d50e45585616.slice/crio-861b2c87e76ffae3eadadf3dedd4fdd83af288bf236fa26b09888e240bc74b61 WatchSource:0}: Error finding container 861b2c87e76ffae3eadadf3dedd4fdd83af288bf236fa26b09888e240bc74b61: Status 404 returned error can't find the container with id 861b2c87e76ffae3eadadf3dedd4fdd83af288bf236fa26b09888e240bc74b61
Nov 01 03:32:25.241286 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:32:25.241096    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d262c155-6f74d6b7c4-rld8l"]
Nov 01 03:32:25.815230 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:32:25.815194    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d262c155-6f74d6b7c4-rld8l" event={"ID":"6084708e-57aa-4b1a-8dfe-d50e45585616","Type":"ContainerStarted","Data":"861b2c87e76ffae3eadadf3dedd4fdd83af288bf236fa26b09888e240bc74b61"}
Nov 01 03:32:26.826057 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:32:26.826022    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d262c155-6f74d6b7c4-rld8l" event={"ID":"6084708e-57aa-4b1a-8dfe-d50e45585616","Type":"ContainerStarted","Data":"abf05737869c5974ff5cee0e23b268019329929adeab19f8b599fee5469d10b8"}
Nov 01 03:32:26.826486 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:32:26.826111    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d262c155-6f74d6b7c4-rld8l"
Nov 01 03:32:28.502154 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:32:28.502088    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d262c155-6f74d6b7c4-rld8l" podStartSLOduration=4.605034068 podStartE2EDuration="5.502065422s" totalImagesPullingTime="897.031354ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:32:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:32:26.96834775 +0000 UTC m=+1553.482993615" watchObservedRunningTime="2026-11-01 03:32:28.502065422 +0000 UTC m=+1555.016711286"
Nov 01 03:32:28.506630 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:32:28.506563    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3513c2f1-748b9f9d5b-vdt9x"]
Nov 01 03:32:28.511364 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:32:28.511346    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3513c2f1-748b9f9d5b-vdt9x"
Nov 01 03:32:28.568991 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:32:28.568960    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3513c2f1-748b9f9d5b-vdt9x"]
Nov 01 03:32:28.615092 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:32:28.615064    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-3513c2f1-serving-cert\""
Nov 01 03:32:28.636383 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:32:28.636348    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edf3bdc9-df09-43cd-a4f3-b5c2ebfaa1dd-metrics-certs\") pod \"maas-api-e2e-derive-3513c2f1-748b9f9d5b-vdt9x\" (UID: \"edf3bdc9-df09-43cd-a4f3-b5c2ebfaa1dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3513c2f1-748b9f9d5b-vdt9x"
Nov 01 03:32:28.636535 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:32:28.636480    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edf3bdc9-df09-43cd-a4f3-b5c2ebfaa1dd-maas-api-tls\") pod \"maas-api-e2e-derive-3513c2f1-748b9f9d5b-vdt9x\" (UID: \"edf3bdc9-df09-43cd-a4f3-b5c2ebfaa1dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3513c2f1-748b9f9d5b-vdt9x"
Nov 01 03:32:28.636729 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:32:28.636708    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nm8g\" (UniqueName: \"kubernetes.io/projected/edf3bdc9-df09-43cd-a4f3-b5c2ebfaa1dd-kube-api-access-9nm8g\") pod \"maas-api-e2e-derive-3513c2f1-748b9f9d5b-vdt9x\" (UID: \"edf3bdc9-df09-43cd-a4f3-b5c2ebfaa1dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3513c2f1-748b9f9d5b-vdt9x"
Nov 01 03:32:28.636857 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:32:28.636839    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edf3bdc9-df09-43cd-a4f3-b5c2ebfaa1dd-openshift-service-ca\") pod \"maas-api-e2e-derive-3513c2f1-748b9f9d5b-vdt9x\" (UID: \"edf3bdc9-df09-43cd-a4f3-b5c2ebfaa1dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3513c2f1-748b9f9d5b-vdt9x"
Nov 01 03:32:28.738021 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:32:28.737988    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edf3bdc9-df09-43cd-a4f3-b5c2ebfaa1dd-maas-api-tls\") pod \"maas-api-e2e-derive-3513c2f1-748b9f9d5b-vdt9x\" (UID: \"edf3bdc9-df09-43cd-a4f3-b5c2ebfaa1dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3513c2f1-748b9f9d5b-vdt9x"
Nov 01 03:32:28.738216 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:32:28.738137    4400 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-3513c2f1-serving-cert: secret "maas-api-e2e-derive-3513c2f1-serving-cert" not found
Nov 01 03:32:28.738216 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:32:28.738203    4400 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edf3bdc9-df09-43cd-a4f3-b5c2ebfaa1dd-maas-api-tls podName:edf3bdc9-df09-43cd-a4f3-b5c2ebfaa1dd nodeName:}" failed. No retries permitted until 2026-11-01 03:32:29.238186653 +0000 UTC m=+1555.752832493 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/edf3bdc9-df09-43cd-a4f3-b5c2ebfaa1dd-maas-api-tls") pod "maas-api-e2e-derive-3513c2f1-748b9f9d5b-vdt9x" (UID: "edf3bdc9-df09-43cd-a4f3-b5c2ebfaa1dd") : secret "maas-api-e2e-derive-3513c2f1-serving-cert" not found
Nov 01 03:32:28.738357 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:32:28.738143    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nm8g\" (UniqueName: \"kubernetes.io/projected/edf3bdc9-df09-43cd-a4f3-b5c2ebfaa1dd-kube-api-access-9nm8g\") pod \"maas-api-e2e-derive-3513c2f1-748b9f9d5b-vdt9x\" (UID: \"edf3bdc9-df09-43cd-a4f3-b5c2ebfaa1dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3513c2f1-748b9f9d5b-vdt9x"
Nov 01 03:32:28.738419 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:32:28.738378    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edf3bdc9-df09-43cd-a4f3-b5c2ebfaa1dd-openshift-service-ca\") pod \"maas-api-e2e-derive-3513c2f1-748b9f9d5b-vdt9x\" (UID: \"edf3bdc9-df09-43cd-a4f3-b5c2ebfaa1dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3513c2f1-748b9f9d5b-vdt9x"
Nov 01 03:32:28.738637 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:32:28.738605    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edf3bdc9-df09-43cd-a4f3-b5c2ebfaa1dd-metrics-certs\") pod \"maas-api-e2e-derive-3513c2f1-748b9f9d5b-vdt9x\" (UID: \"edf3bdc9-df09-43cd-a4f3-b5c2ebfaa1dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3513c2f1-748b9f9d5b-vdt9x"
Nov 01 03:32:28.739099 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:32:28.739076    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edf3bdc9-df09-43cd-a4f3-b5c2ebfaa1dd-openshift-service-ca\") pod \"maas-api-e2e-derive-3513c2f1-748b9f9d5b-vdt9x\" (UID: \"edf3bdc9-df09-43cd-a4f3-b5c2ebfaa1dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3513c2f1-748b9f9d5b-vdt9x"
Nov 01 03:32:28.741470 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:32:28.741447    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edf3bdc9-df09-43cd-a4f3-b5c2ebfaa1dd-metrics-certs\") pod \"maas-api-e2e-derive-3513c2f1-748b9f9d5b-vdt9x\" (UID: \"edf3bdc9-df09-43cd-a4f3-b5c2ebfaa1dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3513c2f1-748b9f9d5b-vdt9x"
Nov 01 03:32:28.833762 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:32:28.827935    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nm8g\" (UniqueName: \"kubernetes.io/projected/edf3bdc9-df09-43cd-a4f3-b5c2ebfaa1dd-kube-api-access-9nm8g\") pod \"maas-api-e2e-derive-3513c2f1-748b9f9d5b-vdt9x\" (UID: \"edf3bdc9-df09-43cd-a4f3-b5c2ebfaa1dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3513c2f1-748b9f9d5b-vdt9x"
Nov 01 03:32:29.252719 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:32:29.252605    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edf3bdc9-df09-43cd-a4f3-b5c2ebfaa1dd-maas-api-tls\") pod \"maas-api-e2e-derive-3513c2f1-748b9f9d5b-vdt9x\" (UID: \"edf3bdc9-df09-43cd-a4f3-b5c2ebfaa1dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3513c2f1-748b9f9d5b-vdt9x"
Nov 01 03:32:29.255213 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:32:29.255189    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edf3bdc9-df09-43cd-a4f3-b5c2ebfaa1dd-maas-api-tls\") pod \"maas-api-e2e-derive-3513c2f1-748b9f9d5b-vdt9x\" (UID: \"edf3bdc9-df09-43cd-a4f3-b5c2ebfaa1dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3513c2f1-748b9f9d5b-vdt9x"
Nov 01 03:32:29.426489 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:32:29.426457    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3513c2f1-748b9f9d5b-vdt9x"
Nov 01 03:32:29.812424 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:32:29.812394    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedf3bdc9_df09_43cd_a4f3_b5c2ebfaa1dd.slice/crio-db150b65aeaaee64c3170bb26b040c06fb2314872c771c7e8e2dc5c0e2b1e115 WatchSource:0}: Error finding container db150b65aeaaee64c3170bb26b040c06fb2314872c771c7e8e2dc5c0e2b1e115: Status 404 returned error can't find the container with id db150b65aeaaee64c3170bb26b040c06fb2314872c771c7e8e2dc5c0e2b1e115
Nov 01 03:32:29.812801 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:32:29.812656    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3513c2f1-748b9f9d5b-vdt9x"]
Nov 01 03:32:29.868796 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:32:29.868762    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3513c2f1-748b9f9d5b-vdt9x" event={"ID":"edf3bdc9-df09-43cd-a4f3-b5c2ebfaa1dd","Type":"ContainerStarted","Data":"db150b65aeaaee64c3170bb26b040c06fb2314872c771c7e8e2dc5c0e2b1e115"}
Nov 01 03:32:30.877898 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:32:30.877794    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3513c2f1-748b9f9d5b-vdt9x" event={"ID":"edf3bdc9-df09-43cd-a4f3-b5c2ebfaa1dd","Type":"ContainerStarted","Data":"a8a6e323a8bca02e826675847c59c084b26a009f7f884e80090dc2641b93c035"}
Nov 01 03:32:30.877898 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:32:30.877894    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3513c2f1-748b9f9d5b-vdt9x"
Nov 01 03:32:31.001223 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:32:31.001173    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3513c2f1-748b9f9d5b-vdt9x" podStartSLOduration=2.214625489 podStartE2EDuration="3.001160062s" totalImagesPullingTime="786.534573ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:32:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:32:30.997738712 +0000 UTC m=+1557.512384575" watchObservedRunningTime="2026-11-01 03:32:31.001160062 +0000 UTC m=+1557.515805926"
Nov 01 03:32:31.815601 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:32:31.815513    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:32:32.842284 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:32:32.842255    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d262c155-6f74d6b7c4-rld8l"
Nov 01 03:32:36.890358 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:32:36.890331    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3513c2f1-748b9f9d5b-vdt9x"
Nov 01 03:32:40.644976 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:32:40.644941    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 03:33:01.404546 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:01.404448    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-44379c05-f9696f457-b2k6w"]
Nov 01 03:33:01.525278 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:01.525241    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-44379c05-68dbc8565f-dg4s6"]
Nov 01 03:33:01.739608 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:33:01.739516    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:33:06.405260 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:06.405210    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-44379c05-f9696f457-b2k6w" podUID="68faceb9-2a06-4170-b200-37c4133ea373" containerName="payload-processing" containerID="cri-o://85c9599de4bde60eee361a62a4d65acc7c7e65f9469b22f0595e3b8c9bf3402f" gracePeriod=25
Nov 01 03:33:06.526577 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:06.526534    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-44379c05-68dbc8565f-dg4s6" podUID="19fdf16d-cb31-448c-a99a-2d81d14622ab" containerName="payload-pre-processing" containerID="cri-o://bcfc68868f6f2a246925b16ee77e45b8e2cf519bf534618d57382b6b4c6abf26" gracePeriod=25
Nov 01 03:33:06.692910 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:06.692884    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-44379c05-f9696f457-b2k6w"
Nov 01 03:33:06.707482 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:06.707451    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68faceb9-2a06-4170-b200-37c4133ea373-kube-api-access-c27gd\" (UniqueName: \"kubernetes.io/projected/68faceb9-2a06-4170-b200-37c4133ea373-kube-api-access-c27gd\") pod \"68faceb9-2a06-4170-b200-37c4133ea373\" (UID: \"68faceb9-2a06-4170-b200-37c4133ea373\") "
Nov 01 03:33:06.708647 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:06.707635    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68faceb9-2a06-4170-b200-37c4133ea373-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68faceb9-2a06-4170-b200-37c4133ea373-plugins-config-volume\") pod \"68faceb9-2a06-4170-b200-37c4133ea373\" (UID: \"68faceb9-2a06-4170-b200-37c4133ea373\") "
Nov 01 03:33:06.709131 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:06.709066    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68faceb9-2a06-4170-b200-37c4133ea373-plugins-config-volume" pod "68faceb9-2a06-4170-b200-37c4133ea373" (UID: "68faceb9-2a06-4170-b200-37c4133ea373"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:33:06.717370 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:06.717338    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68faceb9-2a06-4170-b200-37c4133ea373-kube-api-access-c27gd" pod "68faceb9-2a06-4170-b200-37c4133ea373" (UID: "68faceb9-2a06-4170-b200-37c4133ea373"). InnerVolumeSpecName "kube-api-access-c27gd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:33:06.795927 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:06.795895    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-44379c05-68dbc8565f-dg4s6"
Nov 01 03:33:06.809394 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:06.809365    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19fdf16d-cb31-448c-a99a-2d81d14622ab-kube-api-access-lv2bm\" (UniqueName: \"kubernetes.io/projected/19fdf16d-cb31-448c-a99a-2d81d14622ab-kube-api-access-lv2bm\") pod \"19fdf16d-cb31-448c-a99a-2d81d14622ab\" (UID: \"19fdf16d-cb31-448c-a99a-2d81d14622ab\") "
Nov 01 03:33:06.809547 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:06.809475    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19fdf16d-cb31-448c-a99a-2d81d14622ab-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19fdf16d-cb31-448c-a99a-2d81d14622ab-plugins-config-volume\") pod \"19fdf16d-cb31-448c-a99a-2d81d14622ab\" (UID: \"19fdf16d-cb31-448c-a99a-2d81d14622ab\") "
Nov 01 03:33:06.809801 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:06.809778    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19fdf16d-cb31-448c-a99a-2d81d14622ab-plugins-config-volume" pod "19fdf16d-cb31-448c-a99a-2d81d14622ab" (UID: "19fdf16d-cb31-448c-a99a-2d81d14622ab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:33:06.810056 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:06.810036    4400 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19fdf16d-cb31-448c-a99a-2d81d14622ab-plugins-config-volume\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:33:06.810118 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:06.810062    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c27gd\" (UniqueName: \"kubernetes.io/projected/68faceb9-2a06-4170-b200-37c4133ea373-kube-api-access-c27gd\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:33:06.810118 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:06.810078    4400 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68faceb9-2a06-4170-b200-37c4133ea373-plugins-config-volume\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:33:06.811649 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:06.811625    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19fdf16d-cb31-448c-a99a-2d81d14622ab-kube-api-access-lv2bm" pod "19fdf16d-cb31-448c-a99a-2d81d14622ab" (UID: "19fdf16d-cb31-448c-a99a-2d81d14622ab"). InnerVolumeSpecName "kube-api-access-lv2bm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:33:06.911583 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:06.911548    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lv2bm\" (UniqueName: \"kubernetes.io/projected/19fdf16d-cb31-448c-a99a-2d81d14622ab-kube-api-access-lv2bm\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:33:07.363196 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:07.363156    4400 generic.go:356] "Generic (PLEG): container finished" podID="19fdf16d-cb31-448c-a99a-2d81d14622ab" containerID="bcfc68868f6f2a246925b16ee77e45b8e2cf519bf534618d57382b6b4c6abf26" exitCode=0
Nov 01 03:33:07.363401 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:07.363210    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-44379c05-68dbc8565f-dg4s6"
Nov 01 03:33:07.363401 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:07.363234    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-44379c05-68dbc8565f-dg4s6" event={"ID":"19fdf16d-cb31-448c-a99a-2d81d14622ab","Type":"ContainerDied","Data":"bcfc68868f6f2a246925b16ee77e45b8e2cf519bf534618d57382b6b4c6abf26"}
Nov 01 03:33:07.363401 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:07.363330    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-44379c05-68dbc8565f-dg4s6" event={"ID":"19fdf16d-cb31-448c-a99a-2d81d14622ab","Type":"ContainerDied","Data":"84a6b56d70ab125a410e609ad87b15b44e3e698f778ffa62be666fc0127c04bb"}
Nov 01 03:33:07.363401 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:07.363358    4400 scope.go:122] "RemoveContainer" containerID="bcfc68868f6f2a246925b16ee77e45b8e2cf519bf534618d57382b6b4c6abf26"
Nov 01 03:33:07.365105 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:07.365069    4400 generic.go:356] "Generic (PLEG): container finished" podID="68faceb9-2a06-4170-b200-37c4133ea373" containerID="85c9599de4bde60eee361a62a4d65acc7c7e65f9469b22f0595e3b8c9bf3402f" exitCode=0
Nov 01 03:33:07.365232 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:07.365141    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-44379c05-f9696f457-b2k6w"
Nov 01 03:33:07.365232 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:07.365165    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-44379c05-f9696f457-b2k6w" event={"ID":"68faceb9-2a06-4170-b200-37c4133ea373","Type":"ContainerDied","Data":"85c9599de4bde60eee361a62a4d65acc7c7e65f9469b22f0595e3b8c9bf3402f"}
Nov 01 03:33:07.365232 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:07.365203    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-44379c05-f9696f457-b2k6w" event={"ID":"68faceb9-2a06-4170-b200-37c4133ea373","Type":"ContainerDied","Data":"33213d0dc291ab238d015d0584e011796454f9c87287ff00244a33eae7c077fa"}
Nov 01 03:33:07.376450 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:07.376431    4400 scope.go:122] "RemoveContainer" containerID="bcfc68868f6f2a246925b16ee77e45b8e2cf519bf534618d57382b6b4c6abf26"
Nov 01 03:33:07.376798 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:33:07.376775    4400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcfc68868f6f2a246925b16ee77e45b8e2cf519bf534618d57382b6b4c6abf26\": container with ID starting with bcfc68868f6f2a246925b16ee77e45b8e2cf519bf534618d57382b6b4c6abf26 not found: ID does not exist" containerID="bcfc68868f6f2a246925b16ee77e45b8e2cf519bf534618d57382b6b4c6abf26"
Nov 01 03:33:07.376876 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:07.376804    4400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcfc68868f6f2a246925b16ee77e45b8e2cf519bf534618d57382b6b4c6abf26"} err="failed to get container status \"bcfc68868f6f2a246925b16ee77e45b8e2cf519bf534618d57382b6b4c6abf26\": rpc error: code = NotFound desc = could not find container \"bcfc68868f6f2a246925b16ee77e45b8e2cf519bf534618d57382b6b4c6abf26\": container with ID starting with bcfc68868f6f2a246925b16ee77e45b8e2cf519bf534618d57382b6b4c6abf26 not found: ID does not exist"
Nov 01 03:33:07.376876 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:07.376822    4400 scope.go:122] "RemoveContainer" containerID="85c9599de4bde60eee361a62a4d65acc7c7e65f9469b22f0595e3b8c9bf3402f"
Nov 01 03:33:07.387430 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:07.387406    4400 scope.go:122] "RemoveContainer" containerID="85c9599de4bde60eee361a62a4d65acc7c7e65f9469b22f0595e3b8c9bf3402f"
Nov 01 03:33:07.387781 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:33:07.387750    4400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85c9599de4bde60eee361a62a4d65acc7c7e65f9469b22f0595e3b8c9bf3402f\": container with ID starting with 85c9599de4bde60eee361a62a4d65acc7c7e65f9469b22f0595e3b8c9bf3402f not found: ID does not exist" containerID="85c9599de4bde60eee361a62a4d65acc7c7e65f9469b22f0595e3b8c9bf3402f"
Nov 01 03:33:07.387861 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:07.387790    4400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85c9599de4bde60eee361a62a4d65acc7c7e65f9469b22f0595e3b8c9bf3402f"} err="failed to get container status \"85c9599de4bde60eee361a62a4d65acc7c7e65f9469b22f0595e3b8c9bf3402f\": rpc error: code = NotFound desc = could not find container \"85c9599de4bde60eee361a62a4d65acc7c7e65f9469b22f0595e3b8c9bf3402f\": container with ID starting with 85c9599de4bde60eee361a62a4d65acc7c7e65f9469b22f0595e3b8c9bf3402f not found: ID does not exist"
Nov 01 03:33:07.535250 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:07.535216    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-44379c05-f9696f457-b2k6w"]
Nov 01 03:33:07.585717 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:07.585658    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-44379c05-f9696f457-b2k6w"]
Nov 01 03:33:07.616634 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:07.616549    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68faceb9-2a06-4170-b200-37c4133ea373" path="/var/lib/kubelet/pods/68faceb9-2a06-4170-b200-37c4133ea373/volumes"
Nov 01 03:33:07.654427 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:07.654390    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-44379c05-68dbc8565f-dg4s6"]
Nov 01 03:33:07.658337 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:07.658299    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-44379c05-68dbc8565f-dg4s6"]
Nov 01 03:33:09.616949 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:09.616911    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19fdf16d-cb31-448c-a99a-2d81d14622ab" path="/var/lib/kubelet/pods/19fdf16d-cb31-448c-a99a-2d81d14622ab/volumes"
Nov 01 03:33:31.733081 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:33:31.733043    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:33:37.349849 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:37.349810    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d262c155-d97d78c75-8hcz2"]
Nov 01 03:33:38.602882 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:38.602843    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2058fb-5995f847c9-mv55z"]
Nov 01 03:33:38.603637 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:38.603623    4400 cpu_manager.go:414] "containerMap: removing container" podUID="19fdf16d-cb31-448c-a99a-2d81d14622ab" containerName="payload-pre-processing"
Nov 01 03:33:38.603722 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:38.603639    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19fdf16d-cb31-448c-a99a-2d81d14622ab" containerName="payload-pre-processing"
Nov 01 03:33:38.603722 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:38.603664    4400 cpu_manager.go:414] "containerMap: removing container" podUID="68faceb9-2a06-4170-b200-37c4133ea373" containerName="payload-processing"
Nov 01 03:33:38.603722 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:38.603669    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68faceb9-2a06-4170-b200-37c4133ea373" containerName="payload-processing"
Nov 01 03:33:38.603827 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:38.603792    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="19fdf16d-cb31-448c-a99a-2d81d14622ab" containerName="payload-pre-processing"
Nov 01 03:33:38.603827 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:38.603804    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="68faceb9-2a06-4170-b200-37c4133ea373" containerName="payload-processing"
Nov 01 03:33:38.608786 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:38.608768    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2058fb-5995f847c9-mv55z"
Nov 01 03:33:38.667813 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:38.667779    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2058fb-5995f847c9-mv55z"]
Nov 01 03:33:38.689145 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:38.689111    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6kg8\" (UniqueName: \"kubernetes.io/projected/1083a71d-f12c-474c-992e-31393098ea20-kube-api-access-v6kg8\") pod \"maas-api-e2e-shared-a-w4-2058fb-5995f847c9-mv55z\" (UID: \"1083a71d-f12c-474c-992e-31393098ea20\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2058fb-5995f847c9-mv55z"
Nov 01 03:33:38.689333 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:38.689276    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1083a71d-f12c-474c-992e-31393098ea20-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-2058fb-5995f847c9-mv55z\" (UID: \"1083a71d-f12c-474c-992e-31393098ea20\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2058fb-5995f847c9-mv55z"
Nov 01 03:33:38.689406 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:38.689332    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1083a71d-f12c-474c-992e-31393098ea20-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-2058fb-5995f847c9-mv55z\" (UID: \"1083a71d-f12c-474c-992e-31393098ea20\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2058fb-5995f847c9-mv55z"
Nov 01 03:33:38.689406 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:38.689381    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1083a71d-f12c-474c-992e-31393098ea20-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-2058fb-5995f847c9-mv55z\" (UID: \"1083a71d-f12c-474c-992e-31393098ea20\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2058fb-5995f847c9-mv55z"
Nov 01 03:33:38.800728 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:38.737871    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-2058fb-serving-cert\""
Nov 01 03:33:39.027706 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:38.991936    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1083a71d-f12c-474c-992e-31393098ea20-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-2058fb-5995f847c9-mv55z\" (UID: \"1083a71d-f12c-474c-992e-31393098ea20\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2058fb-5995f847c9-mv55z"
Nov 01 03:33:39.027989 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:39.027954    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1083a71d-f12c-474c-992e-31393098ea20-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-2058fb-5995f847c9-mv55z\" (UID: \"1083a71d-f12c-474c-992e-31393098ea20\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2058fb-5995f847c9-mv55z"
Nov 01 03:33:39.028231 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:39.028213    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1083a71d-f12c-474c-992e-31393098ea20-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-2058fb-5995f847c9-mv55z\" (UID: \"1083a71d-f12c-474c-992e-31393098ea20\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2058fb-5995f847c9-mv55z"
Nov 01 03:33:39.028870 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:39.028832    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6kg8\" (UniqueName: \"kubernetes.io/projected/1083a71d-f12c-474c-992e-31393098ea20-kube-api-access-v6kg8\") pod \"maas-api-e2e-shared-a-w4-2058fb-5995f847c9-mv55z\" (UID: \"1083a71d-f12c-474c-992e-31393098ea20\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2058fb-5995f847c9-mv55z"
Nov 01 03:33:39.035709 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:39.035667    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1083a71d-f12c-474c-992e-31393098ea20-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-2058fb-5995f847c9-mv55z\" (UID: \"1083a71d-f12c-474c-992e-31393098ea20\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2058fb-5995f847c9-mv55z"
Nov 01 03:33:39.036797 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:33:39.036777    4400 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2058fb-serving-cert: secret "maas-api-e2e-shared-a-w4-2058fb-serving-cert" not found
Nov 01 03:33:39.037007 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:33:39.036994    4400 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1083a71d-f12c-474c-992e-31393098ea20-maas-api-tls podName:1083a71d-f12c-474c-992e-31393098ea20 nodeName:}" failed. No retries permitted until 2026-11-01 03:33:39.536973333 +0000 UTC m=+1626.051619176 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1083a71d-f12c-474c-992e-31393098ea20-maas-api-tls") pod "maas-api-e2e-shared-a-w4-2058fb-5995f847c9-mv55z" (UID: "1083a71d-f12c-474c-992e-31393098ea20") : secret "maas-api-e2e-shared-a-w4-2058fb-serving-cert" not found
Nov 01 03:33:39.055205 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:39.050281    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1083a71d-f12c-474c-992e-31393098ea20-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-2058fb-5995f847c9-mv55z\" (UID: \"1083a71d-f12c-474c-992e-31393098ea20\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2058fb-5995f847c9-mv55z"
Nov 01 03:33:39.233709 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:39.218972    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6kg8\" (UniqueName: \"kubernetes.io/projected/1083a71d-f12c-474c-992e-31393098ea20-kube-api-access-v6kg8\") pod \"maas-api-e2e-shared-a-w4-2058fb-5995f847c9-mv55z\" (UID: \"1083a71d-f12c-474c-992e-31393098ea20\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2058fb-5995f847c9-mv55z"
Nov 01 03:33:39.576523 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:39.576479    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1083a71d-f12c-474c-992e-31393098ea20-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-2058fb-5995f847c9-mv55z\" (UID: \"1083a71d-f12c-474c-992e-31393098ea20\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2058fb-5995f847c9-mv55z"
Nov 01 03:33:39.579303 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:39.579277    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1083a71d-f12c-474c-992e-31393098ea20-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-2058fb-5995f847c9-mv55z\" (UID: \"1083a71d-f12c-474c-992e-31393098ea20\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2058fb-5995f847c9-mv55z"
Nov 01 03:33:39.824091 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:39.824062    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2058fb-5995f847c9-mv55z"
Nov 01 03:33:40.211566 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:40.211540    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2058fb-5995f847c9-mv55z"]
Nov 01 03:33:40.211736 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:33:40.211714    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1083a71d_f12c_474c_992e_31393098ea20.slice/crio-67b8f70e4f2b6f4a3a486f48c3b43e66f9c344cee4d5cfc0f3d56f413572b977 WatchSource:0}: Error finding container 67b8f70e4f2b6f4a3a486f48c3b43e66f9c344cee4d5cfc0f3d56f413572b977: Status 404 returned error can't find the container with id 67b8f70e4f2b6f4a3a486f48c3b43e66f9c344cee4d5cfc0f3d56f413572b977
Nov 01 03:33:40.213789 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:40.213764    4400 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 03:33:40.878072 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:40.878031    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2058fb-5995f847c9-mv55z" event={"ID":"1083a71d-f12c-474c-992e-31393098ea20","Type":"ContainerStarted","Data":"67b8f70e4f2b6f4a3a486f48c3b43e66f9c344cee4d5cfc0f3d56f413572b977"}
Nov 01 03:33:41.888838 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:41.888798    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2058fb-5995f847c9-mv55z" event={"ID":"1083a71d-f12c-474c-992e-31393098ea20","Type":"ContainerStarted","Data":"64ed14ec931c30ce0b95173c5180ab1056f59d97ee15e8461dc1acaa17a6f111"}
Nov 01 03:33:41.889280 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:41.888866    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2058fb-5995f847c9-mv55z"
Nov 01 03:33:42.013542 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:42.013487    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2058fb-5995f847c9-mv55z" podStartSLOduration=3.16560514 podStartE2EDuration="4.013473261s" totalImagesPullingTime="847.868121ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:33:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:33:42.010246729 +0000 UTC m=+1628.524892594" watchObservedRunningTime="2026-11-01 03:33:42.013473261 +0000 UTC m=+1628.528119166"
Nov 01 03:33:42.351006 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:42.350964    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d262c155-d97d78c75-8hcz2" podUID="1ae8054b-cf08-42e1-acfc-f1461e647095" containerName="payload-pre-processing" containerID="cri-o://f4caea899c34d807b74d2798ef29c002df3194413df3ea2711a00b28eb64f2db" gracePeriod=25
Nov 01 03:33:42.390783 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:42.390751    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3513c2f1-fc98d8f8c-rml8z"]
Nov 01 03:33:42.516817 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:42.516775    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3513c2f1-5845c474f5-cs2th"]
Nov 01 03:33:42.674739 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:42.674671    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d262c155-d97d78c75-8hcz2"
Nov 01 03:33:42.758425 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:42.758387    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ae8054b-cf08-42e1-acfc-f1461e647095-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ae8054b-cf08-42e1-acfc-f1461e647095-plugins-config-volume\") pod \"1ae8054b-cf08-42e1-acfc-f1461e647095\" (UID: \"1ae8054b-cf08-42e1-acfc-f1461e647095\") "
Nov 01 03:33:42.758425 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:42.758429    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ae8054b-cf08-42e1-acfc-f1461e647095-kube-api-access-j94kl\" (UniqueName: \"kubernetes.io/projected/1ae8054b-cf08-42e1-acfc-f1461e647095-kube-api-access-j94kl\") pod \"1ae8054b-cf08-42e1-acfc-f1461e647095\" (UID: \"1ae8054b-cf08-42e1-acfc-f1461e647095\") "
Nov 01 03:33:42.758918 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:42.758892    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ae8054b-cf08-42e1-acfc-f1461e647095-plugins-config-volume" pod "1ae8054b-cf08-42e1-acfc-f1461e647095" (UID: "1ae8054b-cf08-42e1-acfc-f1461e647095"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:33:42.759197 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:42.759181    4400 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ae8054b-cf08-42e1-acfc-f1461e647095-plugins-config-volume\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:33:42.760826 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:42.760798    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ae8054b-cf08-42e1-acfc-f1461e647095-kube-api-access-j94kl" pod "1ae8054b-cf08-42e1-acfc-f1461e647095" (UID: "1ae8054b-cf08-42e1-acfc-f1461e647095"). InnerVolumeSpecName "kube-api-access-j94kl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:33:42.867006 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:42.864816    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j94kl\" (UniqueName: \"kubernetes.io/projected/1ae8054b-cf08-42e1-acfc-f1461e647095-kube-api-access-j94kl\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:33:42.977122 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:42.969568    4400 generic.go:356] "Generic (PLEG): container finished" podID="1ae8054b-cf08-42e1-acfc-f1461e647095" containerID="f4caea899c34d807b74d2798ef29c002df3194413df3ea2711a00b28eb64f2db" exitCode=0
Nov 01 03:33:42.977122 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:42.969902    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d262c155-d97d78c75-8hcz2"
Nov 01 03:33:42.977122 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:42.970808    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d262c155-d97d78c75-8hcz2" event={"ID":"1ae8054b-cf08-42e1-acfc-f1461e647095","Type":"ContainerDied","Data":"f4caea899c34d807b74d2798ef29c002df3194413df3ea2711a00b28eb64f2db"}
Nov 01 03:33:42.977122 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:42.970899    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d262c155-d97d78c75-8hcz2" event={"ID":"1ae8054b-cf08-42e1-acfc-f1461e647095","Type":"ContainerDied","Data":"d0cf3bd95acbdd19cbdee30714bd45c90b11092bc84ab7bbf4e4982aa9d4cd24"}
Nov 01 03:33:42.977122 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:42.970931    4400 scope.go:122] "RemoveContainer" containerID="f4caea899c34d807b74d2798ef29c002df3194413df3ea2711a00b28eb64f2db"
Nov 01 03:33:43.169915 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:43.166652    4400 scope.go:122] "RemoveContainer" containerID="f4caea899c34d807b74d2798ef29c002df3194413df3ea2711a00b28eb64f2db"
Nov 01 03:33:43.169915 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:33:43.168376    4400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4caea899c34d807b74d2798ef29c002df3194413df3ea2711a00b28eb64f2db\": container with ID starting with f4caea899c34d807b74d2798ef29c002df3194413df3ea2711a00b28eb64f2db not found: ID does not exist" containerID="f4caea899c34d807b74d2798ef29c002df3194413df3ea2711a00b28eb64f2db"
Nov 01 03:33:43.169915 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:43.168441    4400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4caea899c34d807b74d2798ef29c002df3194413df3ea2711a00b28eb64f2db"} err="failed to get container status \"f4caea899c34d807b74d2798ef29c002df3194413df3ea2711a00b28eb64f2db\": rpc error: code = NotFound desc = could not find container \"f4caea899c34d807b74d2798ef29c002df3194413df3ea2711a00b28eb64f2db\": container with ID starting with f4caea899c34d807b74d2798ef29c002df3194413df3ea2711a00b28eb64f2db not found: ID does not exist"
Nov 01 03:33:43.230168 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:43.230089    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d262c155-d97d78c75-8hcz2"]
Nov 01 03:33:43.234340 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:43.234312    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d262c155-d97d78c75-8hcz2"]
Nov 01 03:33:43.619033 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:43.619003    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ae8054b-cf08-42e1-acfc-f1461e647095" path="/var/lib/kubelet/pods/1ae8054b-cf08-42e1-acfc-f1461e647095/volumes"
Nov 01 03:33:47.391761 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:47.391723    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-3513c2f1-fc98d8f8c-rml8z" podUID="85be6802-a287-49ec-8fb9-19f94703f0b0" containerName="payload-processing" containerID="cri-o://34c6ac7a1b6f9b88dcb3b26ebb5520d75a8e5b7070c6588a4efe790a3bc87c60" gracePeriod=25
Nov 01 03:33:47.517824 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:47.517786    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-3513c2f1-5845c474f5-cs2th" podUID="3d09cd59-1b40-4a61-b821-e22c482f3d3a" containerName="payload-pre-processing" containerID="cri-o://aef2c6c2752ab5a5775b03844788a8531223177eb8b3e4c5d59b0371f5c45b4f" gracePeriod=25
Nov 01 03:33:47.683335 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:47.683303    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-3513c2f1-fc98d8f8c-rml8z"
Nov 01 03:33:47.820424 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:47.820392    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-3513c2f1-5845c474f5-cs2th"
Nov 01 03:33:47.822609 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:47.822582    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85be6802-a287-49ec-8fb9-19f94703f0b0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85be6802-a287-49ec-8fb9-19f94703f0b0-plugins-config-volume\") pod \"85be6802-a287-49ec-8fb9-19f94703f0b0\" (UID: \"85be6802-a287-49ec-8fb9-19f94703f0b0\") "
Nov 01 03:33:47.822850 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:47.822835    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85be6802-a287-49ec-8fb9-19f94703f0b0-kube-api-access-pg6wz\" (UniqueName: \"kubernetes.io/projected/85be6802-a287-49ec-8fb9-19f94703f0b0-kube-api-access-pg6wz\") pod \"85be6802-a287-49ec-8fb9-19f94703f0b0\" (UID: \"85be6802-a287-49ec-8fb9-19f94703f0b0\") "
Nov 01 03:33:47.823066 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:47.823033    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85be6802-a287-49ec-8fb9-19f94703f0b0-plugins-config-volume" pod "85be6802-a287-49ec-8fb9-19f94703f0b0" (UID: "85be6802-a287-49ec-8fb9-19f94703f0b0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:33:47.823609 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:47.823581    4400 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85be6802-a287-49ec-8fb9-19f94703f0b0-plugins-config-volume\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:33:47.825353 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:47.825317    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85be6802-a287-49ec-8fb9-19f94703f0b0-kube-api-access-pg6wz" pod "85be6802-a287-49ec-8fb9-19f94703f0b0" (UID: "85be6802-a287-49ec-8fb9-19f94703f0b0"). InnerVolumeSpecName "kube-api-access-pg6wz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:33:47.924876 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:47.924800    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d09cd59-1b40-4a61-b821-e22c482f3d3a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d09cd59-1b40-4a61-b821-e22c482f3d3a-plugins-config-volume\") pod \"3d09cd59-1b40-4a61-b821-e22c482f3d3a\" (UID: \"3d09cd59-1b40-4a61-b821-e22c482f3d3a\") "
Nov 01 03:33:47.925037 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:47.924879    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d09cd59-1b40-4a61-b821-e22c482f3d3a-kube-api-access-c96z6\" (UniqueName: \"kubernetes.io/projected/3d09cd59-1b40-4a61-b821-e22c482f3d3a-kube-api-access-c96z6\") pod \"3d09cd59-1b40-4a61-b821-e22c482f3d3a\" (UID: \"3d09cd59-1b40-4a61-b821-e22c482f3d3a\") "
Nov 01 03:33:47.925164 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:47.925139    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d09cd59-1b40-4a61-b821-e22c482f3d3a-plugins-config-volume" pod "3d09cd59-1b40-4a61-b821-e22c482f3d3a" (UID: "3d09cd59-1b40-4a61-b821-e22c482f3d3a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:33:47.925495 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:47.925479    4400 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d09cd59-1b40-4a61-b821-e22c482f3d3a-plugins-config-volume\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:33:47.925558 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:47.925496    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pg6wz\" (UniqueName: \"kubernetes.io/projected/85be6802-a287-49ec-8fb9-19f94703f0b0-kube-api-access-pg6wz\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:33:47.927246 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:47.927220    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d09cd59-1b40-4a61-b821-e22c482f3d3a-kube-api-access-c96z6" pod "3d09cd59-1b40-4a61-b821-e22c482f3d3a" (UID: "3d09cd59-1b40-4a61-b821-e22c482f3d3a"). InnerVolumeSpecName "kube-api-access-c96z6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:33:47.976863 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:47.976833    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2058fb-5995f847c9-mv55z"
Nov 01 03:33:48.027020 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:48.026984    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c96z6\" (UniqueName: \"kubernetes.io/projected/3d09cd59-1b40-4a61-b821-e22c482f3d3a-kube-api-access-c96z6\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:33:48.137759 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:48.126836    4400 generic.go:356] "Generic (PLEG): container finished" podID="85be6802-a287-49ec-8fb9-19f94703f0b0" containerID="34c6ac7a1b6f9b88dcb3b26ebb5520d75a8e5b7070c6588a4efe790a3bc87c60" exitCode=0
Nov 01 03:33:48.137759 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:48.127054    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-3513c2f1-fc98d8f8c-rml8z" event={"ID":"85be6802-a287-49ec-8fb9-19f94703f0b0","Type":"ContainerDied","Data":"34c6ac7a1b6f9b88dcb3b26ebb5520d75a8e5b7070c6588a4efe790a3bc87c60"}
Nov 01 03:33:48.137759 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:48.127155    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-3513c2f1-fc98d8f8c-rml8z" event={"ID":"85be6802-a287-49ec-8fb9-19f94703f0b0","Type":"ContainerDied","Data":"c67def94ef6099c5e0b4d786be375421f5d08001fd3aa8be2677f91b2797c783"}
Nov 01 03:33:48.137759 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:48.127193    4400 scope.go:122] "RemoveContainer" containerID="34c6ac7a1b6f9b88dcb3b26ebb5520d75a8e5b7070c6588a4efe790a3bc87c60"
Nov 01 03:33:48.137759 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:48.127413    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-3513c2f1-fc98d8f8c-rml8z"
Nov 01 03:33:48.145763 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:48.140328    4400 generic.go:356] "Generic (PLEG): container finished" podID="3d09cd59-1b40-4a61-b821-e22c482f3d3a" containerID="aef2c6c2752ab5a5775b03844788a8531223177eb8b3e4c5d59b0371f5c45b4f" exitCode=0
Nov 01 03:33:48.145763 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:48.140440    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-3513c2f1-5845c474f5-cs2th" event={"ID":"3d09cd59-1b40-4a61-b821-e22c482f3d3a","Type":"ContainerDied","Data":"aef2c6c2752ab5a5775b03844788a8531223177eb8b3e4c5d59b0371f5c45b4f"}
Nov 01 03:33:48.145763 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:48.140549    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-3513c2f1-5845c474f5-cs2th" event={"ID":"3d09cd59-1b40-4a61-b821-e22c482f3d3a","Type":"ContainerDied","Data":"a9b0a71a432e0b6cd2aa89ae1570f679aecac31917a2bf774c775fbd4a9c6812"}
Nov 01 03:33:48.145763 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:48.140599    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-3513c2f1-5845c474f5-cs2th"
Nov 01 03:33:48.153323 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:48.153292    4400 scope.go:122] "RemoveContainer" containerID="34c6ac7a1b6f9b88dcb3b26ebb5520d75a8e5b7070c6588a4efe790a3bc87c60"
Nov 01 03:33:48.158467 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:33:48.158426    4400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34c6ac7a1b6f9b88dcb3b26ebb5520d75a8e5b7070c6588a4efe790a3bc87c60\": container with ID starting with 34c6ac7a1b6f9b88dcb3b26ebb5520d75a8e5b7070c6588a4efe790a3bc87c60 not found: ID does not exist" containerID="34c6ac7a1b6f9b88dcb3b26ebb5520d75a8e5b7070c6588a4efe790a3bc87c60"
Nov 01 03:33:48.158579 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:48.158478    4400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34c6ac7a1b6f9b88dcb3b26ebb5520d75a8e5b7070c6588a4efe790a3bc87c60"} err="failed to get container status \"34c6ac7a1b6f9b88dcb3b26ebb5520d75a8e5b7070c6588a4efe790a3bc87c60\": rpc error: code = NotFound desc = could not find container \"34c6ac7a1b6f9b88dcb3b26ebb5520d75a8e5b7070c6588a4efe790a3bc87c60\": container with ID starting with 34c6ac7a1b6f9b88dcb3b26ebb5520d75a8e5b7070c6588a4efe790a3bc87c60 not found: ID does not exist"
Nov 01 03:33:48.158579 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:48.158503    4400 scope.go:122] "RemoveContainer" containerID="aef2c6c2752ab5a5775b03844788a8531223177eb8b3e4c5d59b0371f5c45b4f"
Nov 01 03:33:48.174746 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:48.174674    4400 scope.go:122] "RemoveContainer" containerID="aef2c6c2752ab5a5775b03844788a8531223177eb8b3e4c5d59b0371f5c45b4f"
Nov 01 03:33:48.177149 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:33:48.175962    4400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aef2c6c2752ab5a5775b03844788a8531223177eb8b3e4c5d59b0371f5c45b4f\": container with ID starting with aef2c6c2752ab5a5775b03844788a8531223177eb8b3e4c5d59b0371f5c45b4f not found: ID does not exist" containerID="aef2c6c2752ab5a5775b03844788a8531223177eb8b3e4c5d59b0371f5c45b4f"
Nov 01 03:33:48.177149 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:48.176002    4400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aef2c6c2752ab5a5775b03844788a8531223177eb8b3e4c5d59b0371f5c45b4f"} err="failed to get container status \"aef2c6c2752ab5a5775b03844788a8531223177eb8b3e4c5d59b0371f5c45b4f\": rpc error: code = NotFound desc = could not find container \"aef2c6c2752ab5a5775b03844788a8531223177eb8b3e4c5d59b0371f5c45b4f\": container with ID starting with aef2c6c2752ab5a5775b03844788a8531223177eb8b3e4c5d59b0371f5c45b4f not found: ID does not exist"
Nov 01 03:33:48.457758 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:48.452748    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3513c2f1-fc98d8f8c-rml8z"]
Nov 01 03:33:48.544824 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:48.544788    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3513c2f1-fc98d8f8c-rml8z"]
Nov 01 03:33:48.548928 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:48.548890    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3513c2f1-5845c474f5-cs2th"]
Nov 01 03:33:48.562673 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:48.562634    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3513c2f1-5845c474f5-cs2th"]
Nov 01 03:33:49.616364 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:49.616333    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d09cd59-1b40-4a61-b821-e22c482f3d3a" path="/var/lib/kubelet/pods/3d09cd59-1b40-4a61-b821-e22c482f3d3a/volumes"
Nov 01 03:33:49.616787 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:49.616772    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85be6802-a287-49ec-8fb9-19f94703f0b0" path="/var/lib/kubelet/pods/85be6802-a287-49ec-8fb9-19f94703f0b0/volumes"
Nov 01 03:33:49.910438 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:49.910352    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 03:33:53.114314 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:53.114276    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-44379c05-686994b9f-q8vrd"]
Nov 01 03:33:53.114803 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:53.114508    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-44379c05-686994b9f-q8vrd" podUID="a54ae264-a89c-4a76-b513-9d85150cd488" containerName="maas-api" containerID="cri-o://7fbed071b17c876a467b67df3b47d089a885e20552beed5b72161a51c2c3a6f7" gracePeriod=30
Nov 01 03:33:53.518476 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:53.518428    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-44379c05-686994b9f-q8vrd"
Nov 01 03:33:53.596601 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:53.596559    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a54ae264-a89c-4a76-b513-9d85150cd488-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a54ae264-a89c-4a76-b513-9d85150cd488-openshift-service-ca\") pod \"a54ae264-a89c-4a76-b513-9d85150cd488\" (UID: \"a54ae264-a89c-4a76-b513-9d85150cd488\") "
Nov 01 03:33:53.597091 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:53.597064    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a54ae264-a89c-4a76-b513-9d85150cd488-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a54ae264-a89c-4a76-b513-9d85150cd488-metrics-certs\") pod \"a54ae264-a89c-4a76-b513-9d85150cd488\" (UID: \"a54ae264-a89c-4a76-b513-9d85150cd488\") "
Nov 01 03:33:53.597320 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:53.597300    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a54ae264-a89c-4a76-b513-9d85150cd488-kube-api-access-kjj9h\" (UniqueName: \"kubernetes.io/projected/a54ae264-a89c-4a76-b513-9d85150cd488-kube-api-access-kjj9h\") pod \"a54ae264-a89c-4a76-b513-9d85150cd488\" (UID: \"a54ae264-a89c-4a76-b513-9d85150cd488\") "
Nov 01 03:33:53.597570 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:53.597552    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a54ae264-a89c-4a76-b513-9d85150cd488-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a54ae264-a89c-4a76-b513-9d85150cd488-maas-api-tls\") pod \"a54ae264-a89c-4a76-b513-9d85150cd488\" (UID: \"a54ae264-a89c-4a76-b513-9d85150cd488\") "
Nov 01 03:33:53.612346 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:53.612110    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a54ae264-a89c-4a76-b513-9d85150cd488-maas-api-tls" pod "a54ae264-a89c-4a76-b513-9d85150cd488" (UID: "a54ae264-a89c-4a76-b513-9d85150cd488"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:33:53.612841 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:53.612661    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a54ae264-a89c-4a76-b513-9d85150cd488-openshift-service-ca" pod "a54ae264-a89c-4a76-b513-9d85150cd488" (UID: "a54ae264-a89c-4a76-b513-9d85150cd488"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:33:53.612841 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:53.612806    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a54ae264-a89c-4a76-b513-9d85150cd488-kube-api-access-kjj9h" pod "a54ae264-a89c-4a76-b513-9d85150cd488" (UID: "a54ae264-a89c-4a76-b513-9d85150cd488"). InnerVolumeSpecName "kube-api-access-kjj9h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:33:53.617557 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:53.617528    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a54ae264-a89c-4a76-b513-9d85150cd488-metrics-certs" pod "a54ae264-a89c-4a76-b513-9d85150cd488" (UID: "a54ae264-a89c-4a76-b513-9d85150cd488"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:33:53.702710 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:53.702352    4400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a54ae264-a89c-4a76-b513-9d85150cd488-metrics-certs\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:33:53.702710 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:53.702388    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjj9h\" (UniqueName: \"kubernetes.io/projected/a54ae264-a89c-4a76-b513-9d85150cd488-kube-api-access-kjj9h\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:33:53.702710 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:53.702402    4400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a54ae264-a89c-4a76-b513-9d85150cd488-maas-api-tls\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:33:53.702710 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:53.702417    4400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a54ae264-a89c-4a76-b513-9d85150cd488-openshift-service-ca\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:33:54.192615 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:54.192582    4400 generic.go:356] "Generic (PLEG): container finished" podID="a54ae264-a89c-4a76-b513-9d85150cd488" containerID="7fbed071b17c876a467b67df3b47d089a885e20552beed5b72161a51c2c3a6f7" exitCode=0
Nov 01 03:33:54.193147 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:54.192735    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-44379c05-686994b9f-q8vrd"
Nov 01 03:33:54.193147 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:54.192733    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-44379c05-686994b9f-q8vrd" event={"ID":"a54ae264-a89c-4a76-b513-9d85150cd488","Type":"ContainerDied","Data":"7fbed071b17c876a467b67df3b47d089a885e20552beed5b72161a51c2c3a6f7"}
Nov 01 03:33:54.193147 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:54.192877    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-44379c05-686994b9f-q8vrd" event={"ID":"a54ae264-a89c-4a76-b513-9d85150cd488","Type":"ContainerDied","Data":"fc8fb2187c39fd0d4b28005b1a93c3eae4a1067aa20fdd380d93da14d5083e6f"}
Nov 01 03:33:54.193147 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:54.192909    4400 scope.go:122] "RemoveContainer" containerID="7fbed071b17c876a467b67df3b47d089a885e20552beed5b72161a51c2c3a6f7"
Nov 01 03:33:54.204046 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:54.204026    4400 scope.go:122] "RemoveContainer" containerID="7fbed071b17c876a467b67df3b47d089a885e20552beed5b72161a51c2c3a6f7"
Nov 01 03:33:54.204378 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:33:54.204322    4400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fbed071b17c876a467b67df3b47d089a885e20552beed5b72161a51c2c3a6f7\": container with ID starting with 7fbed071b17c876a467b67df3b47d089a885e20552beed5b72161a51c2c3a6f7 not found: ID does not exist" containerID="7fbed071b17c876a467b67df3b47d089a885e20552beed5b72161a51c2c3a6f7"
Nov 01 03:33:54.204378 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:54.204363    4400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fbed071b17c876a467b67df3b47d089a885e20552beed5b72161a51c2c3a6f7"} err="failed to get container status \"7fbed071b17c876a467b67df3b47d089a885e20552beed5b72161a51c2c3a6f7\": rpc error: code = NotFound desc = could not find container \"7fbed071b17c876a467b67df3b47d089a885e20552beed5b72161a51c2c3a6f7\": container with ID starting with 7fbed071b17c876a467b67df3b47d089a885e20552beed5b72161a51c2c3a6f7 not found: ID does not exist"
Nov 01 03:33:54.451758 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:54.450669    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-44379c05-686994b9f-q8vrd"]
Nov 01 03:33:54.478710 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:54.472016    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-44379c05-686994b9f-q8vrd"]
Nov 01 03:33:55.616144 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:55.616112    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a54ae264-a89c-4a76-b513-9d85150cd488" path="/var/lib/kubelet/pods/a54ae264-a89c-4a76-b513-9d85150cd488/volumes"
Nov 01 03:33:56.888710 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:56.879228    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fdbdfefd-8468f4575f-vcdsp"]
Nov 01 03:33:56.888710 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:56.880780    4400 cpu_manager.go:414] "containerMap: removing container" podUID="3d09cd59-1b40-4a61-b821-e22c482f3d3a" containerName="payload-pre-processing"
Nov 01 03:33:56.888710 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:56.880803    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d09cd59-1b40-4a61-b821-e22c482f3d3a" containerName="payload-pre-processing"
Nov 01 03:33:56.888710 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:56.880856    4400 cpu_manager.go:414] "containerMap: removing container" podUID="a54ae264-a89c-4a76-b513-9d85150cd488" containerName="maas-api"
Nov 01 03:33:56.888710 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:56.880864    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a54ae264-a89c-4a76-b513-9d85150cd488" containerName="maas-api"
Nov 01 03:33:56.888710 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:56.880876    4400 cpu_manager.go:414] "containerMap: removing container" podUID="1ae8054b-cf08-42e1-acfc-f1461e647095" containerName="payload-pre-processing"
Nov 01 03:33:56.888710 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:56.880884    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ae8054b-cf08-42e1-acfc-f1461e647095" containerName="payload-pre-processing"
Nov 01 03:33:56.888710 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:56.880897    4400 cpu_manager.go:414] "containerMap: removing container" podUID="85be6802-a287-49ec-8fb9-19f94703f0b0" containerName="payload-processing"
Nov 01 03:33:56.888710 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:56.880904    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85be6802-a287-49ec-8fb9-19f94703f0b0" containerName="payload-processing"
Nov 01 03:33:56.888710 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:56.881081    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d09cd59-1b40-4a61-b821-e22c482f3d3a" containerName="payload-pre-processing"
Nov 01 03:33:56.888710 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:56.881101    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="a54ae264-a89c-4a76-b513-9d85150cd488" containerName="maas-api"
Nov 01 03:33:56.888710 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:56.881109    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ae8054b-cf08-42e1-acfc-f1461e647095" containerName="payload-pre-processing"
Nov 01 03:33:56.888710 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:56.881121    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="85be6802-a287-49ec-8fb9-19f94703f0b0" containerName="payload-processing"
Nov 01 03:33:56.919760 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:56.916301    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-fdbdfefd-8468f4575f-vcdsp"
Nov 01 03:33:57.010710 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:57.006152    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fdbdfefd-8468f4575f-vcdsp"]
Nov 01 03:33:57.043717 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:57.040966    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9383b0fa-b53c-451f-9d55-c439dee7cde0-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-fdbdfefd-8468f4575f-vcdsp\" (UID: \"9383b0fa-b53c-451f-9d55-c439dee7cde0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fdbdfefd-8468f4575f-vcdsp"
Nov 01 03:33:57.043717 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:57.041211    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smn92\" (UniqueName: \"kubernetes.io/projected/9383b0fa-b53c-451f-9d55-c439dee7cde0-kube-api-access-smn92\") pod \"payload-pre-processing-e2e-mt-fdbdfefd-8468f4575f-vcdsp\" (UID: \"9383b0fa-b53c-451f-9d55-c439dee7cde0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fdbdfefd-8468f4575f-vcdsp"
Nov 01 03:33:57.166765 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:57.166397    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9383b0fa-b53c-451f-9d55-c439dee7cde0-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-fdbdfefd-8468f4575f-vcdsp\" (UID: \"9383b0fa-b53c-451f-9d55-c439dee7cde0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fdbdfefd-8468f4575f-vcdsp"
Nov 01 03:33:57.166765 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:57.166631    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smn92\" (UniqueName: \"kubernetes.io/projected/9383b0fa-b53c-451f-9d55-c439dee7cde0-kube-api-access-smn92\") pod \"payload-pre-processing-e2e-mt-fdbdfefd-8468f4575f-vcdsp\" (UID: \"9383b0fa-b53c-451f-9d55-c439dee7cde0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fdbdfefd-8468f4575f-vcdsp"
Nov 01 03:33:57.238707 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:57.204917    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-fdbdfefd-dockercfg-lxkv8\""
Nov 01 03:33:57.238707 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:57.205005    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-fdbdfefd\""
Nov 01 03:33:57.238707 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:57.208975    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9383b0fa-b53c-451f-9d55-c439dee7cde0-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-fdbdfefd-8468f4575f-vcdsp\" (UID: \"9383b0fa-b53c-451f-9d55-c439dee7cde0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fdbdfefd-8468f4575f-vcdsp"
Nov 01 03:33:57.322634 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:57.322584    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smn92\" (UniqueName: \"kubernetes.io/projected/9383b0fa-b53c-451f-9d55-c439dee7cde0-kube-api-access-smn92\") pod \"payload-pre-processing-e2e-mt-fdbdfefd-8468f4575f-vcdsp\" (UID: \"9383b0fa-b53c-451f-9d55-c439dee7cde0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fdbdfefd-8468f4575f-vcdsp"
Nov 01 03:33:57.607229 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:57.607199    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-fdbdfefd-8468f4575f-vcdsp"
Nov 01 03:33:57.989331 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:33:57.989299    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9383b0fa_b53c_451f_9d55_c439dee7cde0.slice/crio-5f462a841fb13b0d84faf1789e8a20f95d1aff8644f669f67c3ea4b9ffde7509 WatchSource:0}: Error finding container 5f462a841fb13b0d84faf1789e8a20f95d1aff8644f669f67c3ea4b9ffde7509: Status 404 returned error can't find the container with id 5f462a841fb13b0d84faf1789e8a20f95d1aff8644f669f67c3ea4b9ffde7509
Nov 01 03:33:57.990983 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:57.990916    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fdbdfefd-8468f4575f-vcdsp"]
Nov 01 03:33:58.299164 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:58.299040    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-fdbdfefd-8468f4575f-vcdsp" event={"ID":"9383b0fa-b53c-451f-9d55-c439dee7cde0","Type":"ContainerStarted","Data":"704151a76b326ccdba0ca62b620c3e9540345fad4711a76f4a2eb12da14b2f2f"}
Nov 01 03:33:58.299164 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:58.299106    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-fdbdfefd-8468f4575f-vcdsp" event={"ID":"9383b0fa-b53c-451f-9d55-c439dee7cde0","Type":"ContainerStarted","Data":"5f462a841fb13b0d84faf1789e8a20f95d1aff8644f669f67c3ea4b9ffde7509"}
Nov 01 03:33:58.299406 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:58.299280    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-fdbdfefd-8468f4575f-vcdsp"
Nov 01 03:33:58.419081 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:33:58.419039    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-fdbdfefd-8468f4575f-vcdsp" podStartSLOduration=2.419024891 podStartE2EDuration="2.419024891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:33:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:33:58.416998793 +0000 UTC m=+1644.931644657" watchObservedRunningTime="2026-11-01 03:33:58.419024891 +0000 UTC m=+1644.933670754"
Nov 01 03:34:01.732532 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:34:01.732489    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:34:09.309382 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:09.309351    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-fdbdfefd-8468f4575f-vcdsp"
Nov 01 03:34:15.484853 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:15.484779    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d262c155-6f74d6b7c4-rld8l"]
Nov 01 03:34:15.485267 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:15.485002    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d262c155-6f74d6b7c4-rld8l" podUID="6084708e-57aa-4b1a-8dfe-d50e45585616" containerName="maas-api" containerID="cri-o://abf05737869c5974ff5cee0e23b268019329929adeab19f8b599fee5469d10b8" gracePeriod=30
Nov 01 03:34:16.064709 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:16.059346    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d262c155-6f74d6b7c4-rld8l"
Nov 01 03:34:16.215313 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:16.215220    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6084708e-57aa-4b1a-8dfe-d50e45585616-kube-api-access-kf4gn\" (UniqueName: \"kubernetes.io/projected/6084708e-57aa-4b1a-8dfe-d50e45585616-kube-api-access-kf4gn\") pod \"6084708e-57aa-4b1a-8dfe-d50e45585616\" (UID: \"6084708e-57aa-4b1a-8dfe-d50e45585616\") "
Nov 01 03:34:16.215313 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:16.215283    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6084708e-57aa-4b1a-8dfe-d50e45585616-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6084708e-57aa-4b1a-8dfe-d50e45585616-openshift-service-ca\") pod \"6084708e-57aa-4b1a-8dfe-d50e45585616\" (UID: \"6084708e-57aa-4b1a-8dfe-d50e45585616\") "
Nov 01 03:34:16.215546 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:16.215344    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6084708e-57aa-4b1a-8dfe-d50e45585616-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6084708e-57aa-4b1a-8dfe-d50e45585616-metrics-certs\") pod \"6084708e-57aa-4b1a-8dfe-d50e45585616\" (UID: \"6084708e-57aa-4b1a-8dfe-d50e45585616\") "
Nov 01 03:34:16.215546 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:16.215507    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6084708e-57aa-4b1a-8dfe-d50e45585616-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6084708e-57aa-4b1a-8dfe-d50e45585616-maas-api-tls\") pod \"6084708e-57aa-4b1a-8dfe-d50e45585616\" (UID: \"6084708e-57aa-4b1a-8dfe-d50e45585616\") "
Nov 01 03:34:16.215732 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:16.215662    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6084708e-57aa-4b1a-8dfe-d50e45585616-openshift-service-ca" pod "6084708e-57aa-4b1a-8dfe-d50e45585616" (UID: "6084708e-57aa-4b1a-8dfe-d50e45585616"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:34:16.216254 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:16.216217    4400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6084708e-57aa-4b1a-8dfe-d50e45585616-openshift-service-ca\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:16.218535 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:16.218492    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6084708e-57aa-4b1a-8dfe-d50e45585616-metrics-certs" pod "6084708e-57aa-4b1a-8dfe-d50e45585616" (UID: "6084708e-57aa-4b1a-8dfe-d50e45585616"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:34:16.218640 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:16.218533    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6084708e-57aa-4b1a-8dfe-d50e45585616-kube-api-access-kf4gn" pod "6084708e-57aa-4b1a-8dfe-d50e45585616" (UID: "6084708e-57aa-4b1a-8dfe-d50e45585616"). InnerVolumeSpecName "kube-api-access-kf4gn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:34:16.218913 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:16.218890    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6084708e-57aa-4b1a-8dfe-d50e45585616-maas-api-tls" pod "6084708e-57aa-4b1a-8dfe-d50e45585616" (UID: "6084708e-57aa-4b1a-8dfe-d50e45585616"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:34:16.317533 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:16.317504    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kf4gn\" (UniqueName: \"kubernetes.io/projected/6084708e-57aa-4b1a-8dfe-d50e45585616-kube-api-access-kf4gn\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:16.317533 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:16.317532    4400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6084708e-57aa-4b1a-8dfe-d50e45585616-metrics-certs\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:16.317777 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:16.317544    4400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6084708e-57aa-4b1a-8dfe-d50e45585616-maas-api-tls\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:16.445932 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:16.445902    4400 generic.go:356] "Generic (PLEG): container finished" podID="6084708e-57aa-4b1a-8dfe-d50e45585616" containerID="abf05737869c5974ff5cee0e23b268019329929adeab19f8b599fee5469d10b8" exitCode=0
Nov 01 03:34:16.446076 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:16.445944    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d262c155-6f74d6b7c4-rld8l"
Nov 01 03:34:16.446076 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:16.445983    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d262c155-6f74d6b7c4-rld8l" event={"ID":"6084708e-57aa-4b1a-8dfe-d50e45585616","Type":"ContainerDied","Data":"abf05737869c5974ff5cee0e23b268019329929adeab19f8b599fee5469d10b8"}
Nov 01 03:34:16.446076 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:16.446034    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d262c155-6f74d6b7c4-rld8l" event={"ID":"6084708e-57aa-4b1a-8dfe-d50e45585616","Type":"ContainerDied","Data":"861b2c87e76ffae3eadadf3dedd4fdd83af288bf236fa26b09888e240bc74b61"}
Nov 01 03:34:16.446076 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:16.446057    4400 scope.go:122] "RemoveContainer" containerID="abf05737869c5974ff5cee0e23b268019329929adeab19f8b599fee5469d10b8"
Nov 01 03:34:16.456917 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:16.456899    4400 scope.go:122] "RemoveContainer" containerID="abf05737869c5974ff5cee0e23b268019329929adeab19f8b599fee5469d10b8"
Nov 01 03:34:16.457189 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:34:16.457161    4400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abf05737869c5974ff5cee0e23b268019329929adeab19f8b599fee5469d10b8\": container with ID starting with abf05737869c5974ff5cee0e23b268019329929adeab19f8b599fee5469d10b8 not found: ID does not exist" containerID="abf05737869c5974ff5cee0e23b268019329929adeab19f8b599fee5469d10b8"
Nov 01 03:34:16.457293 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:16.457189    4400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abf05737869c5974ff5cee0e23b268019329929adeab19f8b599fee5469d10b8"} err="failed to get container status \"abf05737869c5974ff5cee0e23b268019329929adeab19f8b599fee5469d10b8\": rpc error: code = NotFound desc = could not find container \"abf05737869c5974ff5cee0e23b268019329929adeab19f8b599fee5469d10b8\": container with ID starting with abf05737869c5974ff5cee0e23b268019329929adeab19f8b599fee5469d10b8 not found: ID does not exist"
Nov 01 03:34:16.616643 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:16.616621    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d262c155-6f74d6b7c4-rld8l"]
Nov 01 03:34:16.668326 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:16.668297    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d262c155-6f74d6b7c4-rld8l"]
Nov 01 03:34:16.671968 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:16.671944    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3513c2f1-748b9f9d5b-vdt9x"]
Nov 01 03:34:16.672175 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:16.672155    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3513c2f1-748b9f9d5b-vdt9x" podUID="edf3bdc9-df09-43cd-a4f3-b5c2ebfaa1dd" containerName="maas-api" containerID="cri-o://a8a6e323a8bca02e826675847c59c084b26a009f7f884e80090dc2641b93c035" gracePeriod=30
Nov 01 03:34:17.192892 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:17.192849    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3513c2f1-748b9f9d5b-vdt9x"
Nov 01 03:34:17.313335 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:17.313286    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/edf3bdc9-df09-43cd-a4f3-b5c2ebfaa1dd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edf3bdc9-df09-43cd-a4f3-b5c2ebfaa1dd-maas-api-tls\") pod \"edf3bdc9-df09-43cd-a4f3-b5c2ebfaa1dd\" (UID: \"edf3bdc9-df09-43cd-a4f3-b5c2ebfaa1dd\") "
Nov 01 03:34:17.313533 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:17.313438    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edf3bdc9-df09-43cd-a4f3-b5c2ebfaa1dd-kube-api-access-9nm8g\" (UniqueName: \"kubernetes.io/projected/edf3bdc9-df09-43cd-a4f3-b5c2ebfaa1dd-kube-api-access-9nm8g\") pod \"edf3bdc9-df09-43cd-a4f3-b5c2ebfaa1dd\" (UID: \"edf3bdc9-df09-43cd-a4f3-b5c2ebfaa1dd\") "
Nov 01 03:34:17.313533 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:17.313524    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/edf3bdc9-df09-43cd-a4f3-b5c2ebfaa1dd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/edf3bdc9-df09-43cd-a4f3-b5c2ebfaa1dd-metrics-certs\") pod \"edf3bdc9-df09-43cd-a4f3-b5c2ebfaa1dd\" (UID: \"edf3bdc9-df09-43cd-a4f3-b5c2ebfaa1dd\") "
Nov 01 03:34:17.314300 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:17.313710    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/edf3bdc9-df09-43cd-a4f3-b5c2ebfaa1dd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edf3bdc9-df09-43cd-a4f3-b5c2ebfaa1dd-openshift-service-ca\") pod \"edf3bdc9-df09-43cd-a4f3-b5c2ebfaa1dd\" (UID: \"edf3bdc9-df09-43cd-a4f3-b5c2ebfaa1dd\") "
Nov 01 03:34:17.315295 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:17.315267    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edf3bdc9-df09-43cd-a4f3-b5c2ebfaa1dd-openshift-service-ca" pod "edf3bdc9-df09-43cd-a4f3-b5c2ebfaa1dd" (UID: "edf3bdc9-df09-43cd-a4f3-b5c2ebfaa1dd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:34:17.317468 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:17.317447    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edf3bdc9-df09-43cd-a4f3-b5c2ebfaa1dd-metrics-certs" pod "edf3bdc9-df09-43cd-a4f3-b5c2ebfaa1dd" (UID: "edf3bdc9-df09-43cd-a4f3-b5c2ebfaa1dd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:34:17.317645 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:17.317620    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edf3bdc9-df09-43cd-a4f3-b5c2ebfaa1dd-kube-api-access-9nm8g" pod "edf3bdc9-df09-43cd-a4f3-b5c2ebfaa1dd" (UID: "edf3bdc9-df09-43cd-a4f3-b5c2ebfaa1dd"). InnerVolumeSpecName "kube-api-access-9nm8g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:34:17.319392 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:17.319367    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edf3bdc9-df09-43cd-a4f3-b5c2ebfaa1dd-maas-api-tls" pod "edf3bdc9-df09-43cd-a4f3-b5c2ebfaa1dd" (UID: "edf3bdc9-df09-43cd-a4f3-b5c2ebfaa1dd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:34:17.416430 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:17.416358    4400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edf3bdc9-df09-43cd-a4f3-b5c2ebfaa1dd-maas-api-tls\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:17.416430 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:17.416384    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nm8g\" (UniqueName: \"kubernetes.io/projected/edf3bdc9-df09-43cd-a4f3-b5c2ebfaa1dd-kube-api-access-9nm8g\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:17.416430 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:17.416395    4400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edf3bdc9-df09-43cd-a4f3-b5c2ebfaa1dd-metrics-certs\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:17.416430 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:17.416404    4400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edf3bdc9-df09-43cd-a4f3-b5c2ebfaa1dd-openshift-service-ca\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:17.455424 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:17.455388    4400 generic.go:356] "Generic (PLEG): container finished" podID="edf3bdc9-df09-43cd-a4f3-b5c2ebfaa1dd" containerID="a8a6e323a8bca02e826675847c59c084b26a009f7f884e80090dc2641b93c035" exitCode=0
Nov 01 03:34:17.455572 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:17.455443    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3513c2f1-748b9f9d5b-vdt9x"
Nov 01 03:34:17.455572 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:17.455477    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3513c2f1-748b9f9d5b-vdt9x" event={"ID":"edf3bdc9-df09-43cd-a4f3-b5c2ebfaa1dd","Type":"ContainerDied","Data":"a8a6e323a8bca02e826675847c59c084b26a009f7f884e80090dc2641b93c035"}
Nov 01 03:34:17.455572 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:17.455535    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3513c2f1-748b9f9d5b-vdt9x" event={"ID":"edf3bdc9-df09-43cd-a4f3-b5c2ebfaa1dd","Type":"ContainerDied","Data":"db150b65aeaaee64c3170bb26b040c06fb2314872c771c7e8e2dc5c0e2b1e115"}
Nov 01 03:34:17.455572 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:17.455557    4400 scope.go:122] "RemoveContainer" containerID="a8a6e323a8bca02e826675847c59c084b26a009f7f884e80090dc2641b93c035"
Nov 01 03:34:17.466059 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:17.466043    4400 scope.go:122] "RemoveContainer" containerID="a8a6e323a8bca02e826675847c59c084b26a009f7f884e80090dc2641b93c035"
Nov 01 03:34:17.466365 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:34:17.466345    4400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8a6e323a8bca02e826675847c59c084b26a009f7f884e80090dc2641b93c035\": container with ID starting with a8a6e323a8bca02e826675847c59c084b26a009f7f884e80090dc2641b93c035 not found: ID does not exist" containerID="a8a6e323a8bca02e826675847c59c084b26a009f7f884e80090dc2641b93c035"
Nov 01 03:34:17.466433 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:17.466373    4400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8a6e323a8bca02e826675847c59c084b26a009f7f884e80090dc2641b93c035"} err="failed to get container status \"a8a6e323a8bca02e826675847c59c084b26a009f7f884e80090dc2641b93c035\": rpc error: code = NotFound desc = could not find container \"a8a6e323a8bca02e826675847c59c084b26a009f7f884e80090dc2641b93c035\": container with ID starting with a8a6e323a8bca02e826675847c59c084b26a009f7f884e80090dc2641b93c035 not found: ID does not exist"
Nov 01 03:34:17.709749 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:17.709636    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6084708e-57aa-4b1a-8dfe-d50e45585616" path="/var/lib/kubelet/pods/6084708e-57aa-4b1a-8dfe-d50e45585616/volumes"
Nov 01 03:34:17.711065 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:17.711040    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3513c2f1-748b9f9d5b-vdt9x"]
Nov 01 03:34:17.711257 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:17.711244    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3513c2f1-748b9f9d5b-vdt9x"]
Nov 01 03:34:19.634589 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:19.634553    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="edf3bdc9-df09-43cd-a4f3-b5c2ebfaa1dd" path="/var/lib/kubelet/pods/edf3bdc9-df09-43cd-a4f3-b5c2ebfaa1dd/volumes"
Nov 01 03:34:19.890870 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:19.890769    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3513c2f1-748b9f9d5b-vdt9x" podUID="edf3bdc9-df09-43cd-a4f3-b5c2ebfaa1dd" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.107:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:34:29.281004 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:29.280955    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-sfxhr"]
Nov 01 03:34:29.282950 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:29.282630    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-sfxhr" podUID="86018717-3f70-43de-843c-f9429c605a74" containerName="istio-proxy" containerID="cri-o://00d3725e28cb6802aa49e20c719b9cfdbf8f6f40d73845192eec92cb97b5cc91" gracePeriod=30
Nov 01 03:34:31.729261 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:34:31.729224    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:34:34.286095 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:34.286010    4400 patch_prober.go:29] interesting pod/e2e-worker-w1-b12199-openshift-default-774cb8c567-sfxhr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 03:34:34.286708 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:34.286657    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-sfxhr" podUID="86018717-3f70-43de-843c-f9429c605a74" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 03:34:34.932130 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:34.932092    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-sfxhr"
Nov 01 03:34:34.964769 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:34.964744    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86018717-3f70-43de-843c-f9429c605a74-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86018717-3f70-43de-843c-f9429c605a74-credential-socket\") pod \"86018717-3f70-43de-843c-f9429c605a74\" (UID: \"86018717-3f70-43de-843c-f9429c605a74\") "
Nov 01 03:34:34.964919 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:34.964787    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86018717-3f70-43de-843c-f9429c605a74-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86018717-3f70-43de-843c-f9429c605a74-workload-certs\") pod \"86018717-3f70-43de-843c-f9429c605a74\" (UID: \"86018717-3f70-43de-843c-f9429c605a74\") "
Nov 01 03:34:34.964919 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:34.964804    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86018717-3f70-43de-843c-f9429c605a74-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86018717-3f70-43de-843c-f9429c605a74-istio-data\") pod \"86018717-3f70-43de-843c-f9429c605a74\" (UID: \"86018717-3f70-43de-843c-f9429c605a74\") "
Nov 01 03:34:34.965016 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:34.964954    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86018717-3f70-43de-843c-f9429c605a74-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86018717-3f70-43de-843c-f9429c605a74-istio-envoy\") pod \"86018717-3f70-43de-843c-f9429c605a74\" (UID: \"86018717-3f70-43de-843c-f9429c605a74\") "
Nov 01 03:34:34.965016 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:34.964988    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86018717-3f70-43de-843c-f9429c605a74-credential-socket" pod "86018717-3f70-43de-843c-f9429c605a74" (UID: "86018717-3f70-43de-843c-f9429c605a74"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:34:34.965090 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:34.965013    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86018717-3f70-43de-843c-f9429c605a74-kube-api-access-6dx52\" (UniqueName: \"kubernetes.io/projected/86018717-3f70-43de-843c-f9429c605a74-kube-api-access-6dx52\") pod \"86018717-3f70-43de-843c-f9429c605a74\" (UID: \"86018717-3f70-43de-843c-f9429c605a74\") "
Nov 01 03:34:34.965090 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:34.965067    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86018717-3f70-43de-843c-f9429c605a74-workload-certs" pod "86018717-3f70-43de-843c-f9429c605a74" (UID: "86018717-3f70-43de-843c-f9429c605a74"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:34:34.965152 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:34.965075    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86018717-3f70-43de-843c-f9429c605a74-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86018717-3f70-43de-843c-f9429c605a74-workload-socket\") pod \"86018717-3f70-43de-843c-f9429c605a74\" (UID: \"86018717-3f70-43de-843c-f9429c605a74\") "
Nov 01 03:34:34.965183 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:34.965139    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86018717-3f70-43de-843c-f9429c605a74-istio-data" pod "86018717-3f70-43de-843c-f9429c605a74" (UID: "86018717-3f70-43de-843c-f9429c605a74"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:34:34.965216 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:34.965188    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86018717-3f70-43de-843c-f9429c605a74-istio-token\" (UniqueName: \"kubernetes.io/projected/86018717-3f70-43de-843c-f9429c605a74-istio-token\") pod \"86018717-3f70-43de-843c-f9429c605a74\" (UID: \"86018717-3f70-43de-843c-f9429c605a74\") "
Nov 01 03:34:34.965259 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:34.965243    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/86018717-3f70-43de-843c-f9429c605a74-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86018717-3f70-43de-843c-f9429c605a74-istio-podinfo\") pod \"86018717-3f70-43de-843c-f9429c605a74\" (UID: \"86018717-3f70-43de-843c-f9429c605a74\") "
Nov 01 03:34:34.965314 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:34.965263    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86018717-3f70-43de-843c-f9429c605a74-workload-socket" pod "86018717-3f70-43de-843c-f9429c605a74" (UID: "86018717-3f70-43de-843c-f9429c605a74"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:34:34.965369 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:34.965312    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86018717-3f70-43de-843c-f9429c605a74-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86018717-3f70-43de-843c-f9429c605a74-istiod-ca-cert\") pod \"86018717-3f70-43de-843c-f9429c605a74\" (UID: \"86018717-3f70-43de-843c-f9429c605a74\") "
Nov 01 03:34:34.965782 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:34.965674    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86018717-3f70-43de-843c-f9429c605a74-istiod-ca-cert" pod "86018717-3f70-43de-843c-f9429c605a74" (UID: "86018717-3f70-43de-843c-f9429c605a74"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:34:34.966333 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:34.966308    4400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86018717-3f70-43de-843c-f9429c605a74-istiod-ca-cert\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:34.966509 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:34.966482    4400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86018717-3f70-43de-843c-f9429c605a74-credential-socket\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:34.966618 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:34.966512    4400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86018717-3f70-43de-843c-f9429c605a74-workload-certs\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:34.966618 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:34.966526    4400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86018717-3f70-43de-843c-f9429c605a74-istio-data\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:34.966618 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:34.966540    4400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86018717-3f70-43de-843c-f9429c605a74-workload-socket\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:34.967279 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:34.967252    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86018717-3f70-43de-843c-f9429c605a74-istio-envoy" pod "86018717-3f70-43de-843c-f9429c605a74" (UID: "86018717-3f70-43de-843c-f9429c605a74"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:34:34.967408 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:34.967384    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86018717-3f70-43de-843c-f9429c605a74-kube-api-access-6dx52" pod "86018717-3f70-43de-843c-f9429c605a74" (UID: "86018717-3f70-43de-843c-f9429c605a74"). InnerVolumeSpecName "kube-api-access-6dx52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:34:34.967408 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:34.967400    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86018717-3f70-43de-843c-f9429c605a74-istio-token" pod "86018717-3f70-43de-843c-f9429c605a74" (UID: "86018717-3f70-43de-843c-f9429c605a74"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:34:34.967630 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:34.967613    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/86018717-3f70-43de-843c-f9429c605a74-istio-podinfo" pod "86018717-3f70-43de-843c-f9429c605a74" (UID: "86018717-3f70-43de-843c-f9429c605a74"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 03:34:34.991861 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:34.991830    4400 generic.go:356] "Generic (PLEG): container finished" podID="86018717-3f70-43de-843c-f9429c605a74" containerID="00d3725e28cb6802aa49e20c719b9cfdbf8f6f40d73845192eec92cb97b5cc91" exitCode=0
Nov 01 03:34:34.991958 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:34.991882    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-sfxhr"
Nov 01 03:34:34.991958 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:34.991916    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-sfxhr" event={"ID":"86018717-3f70-43de-843c-f9429c605a74","Type":"ContainerDied","Data":"00d3725e28cb6802aa49e20c719b9cfdbf8f6f40d73845192eec92cb97b5cc91"}
Nov 01 03:34:34.992035 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:34.991999    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-sfxhr" event={"ID":"86018717-3f70-43de-843c-f9429c605a74","Type":"ContainerDied","Data":"52fb57724b24a417530f0dee01df005e5e8852622721bdd8e9e7f5290fff42b1"}
Nov 01 03:34:34.992035 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:34.992031    4400 scope.go:122] "RemoveContainer" containerID="00d3725e28cb6802aa49e20c719b9cfdbf8f6f40d73845192eec92cb97b5cc91"
Nov 01 03:34:35.002561 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:35.002543    4400 scope.go:122] "RemoveContainer" containerID="00d3725e28cb6802aa49e20c719b9cfdbf8f6f40d73845192eec92cb97b5cc91"
Nov 01 03:34:35.002840 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:34:35.002822    4400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00d3725e28cb6802aa49e20c719b9cfdbf8f6f40d73845192eec92cb97b5cc91\": container with ID starting with 00d3725e28cb6802aa49e20c719b9cfdbf8f6f40d73845192eec92cb97b5cc91 not found: ID does not exist" containerID="00d3725e28cb6802aa49e20c719b9cfdbf8f6f40d73845192eec92cb97b5cc91"
Nov 01 03:34:35.002882 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:35.002848    4400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00d3725e28cb6802aa49e20c719b9cfdbf8f6f40d73845192eec92cb97b5cc91"} err="failed to get container status \"00d3725e28cb6802aa49e20c719b9cfdbf8f6f40d73845192eec92cb97b5cc91\": rpc error: code = NotFound desc = could not find container \"00d3725e28cb6802aa49e20c719b9cfdbf8f6f40d73845192eec92cb97b5cc91\": container with ID starting with 00d3725e28cb6802aa49e20c719b9cfdbf8f6f40d73845192eec92cb97b5cc91 not found: ID does not exist"
Nov 01 03:34:35.067863 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:35.067838    4400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86018717-3f70-43de-843c-f9429c605a74-istio-envoy\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:35.067948 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:35.067865    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dx52\" (UniqueName: \"kubernetes.io/projected/86018717-3f70-43de-843c-f9429c605a74-kube-api-access-6dx52\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:35.067948 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:35.067880    4400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86018717-3f70-43de-843c-f9429c605a74-istio-token\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:35.067948 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:35.067889    4400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86018717-3f70-43de-843c-f9429c605a74-istio-podinfo\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:35.228914 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:35.228834    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-sfxhr"]
Nov 01 03:34:35.244420 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:35.244393    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-sfxhr"]
Nov 01 03:34:35.670901 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:35.653340    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86018717-3f70-43de-843c-f9429c605a74" path="/var/lib/kubelet/pods/86018717-3f70-43de-843c-f9429c605a74/volumes"
Nov 01 03:34:45.573047 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:45.573017    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fdbdfefd-cb877c7cc-9g4k2"]
Nov 01 03:34:45.573818 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:45.573803    4400 cpu_manager.go:414] "containerMap: removing container" podUID="6084708e-57aa-4b1a-8dfe-d50e45585616" containerName="maas-api"
Nov 01 03:34:45.573885 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:45.573819    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6084708e-57aa-4b1a-8dfe-d50e45585616" containerName="maas-api"
Nov 01 03:34:45.573885 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:45.573830    4400 cpu_manager.go:414] "containerMap: removing container" podUID="86018717-3f70-43de-843c-f9429c605a74" containerName="istio-proxy"
Nov 01 03:34:45.573885 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:45.573836    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86018717-3f70-43de-843c-f9429c605a74" containerName="istio-proxy"
Nov 01 03:34:45.573885 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:45.573865    4400 cpu_manager.go:414] "containerMap: removing container" podUID="edf3bdc9-df09-43cd-a4f3-b5c2ebfaa1dd" containerName="maas-api"
Nov 01 03:34:45.573885 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:45.573870    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edf3bdc9-df09-43cd-a4f3-b5c2ebfaa1dd" containerName="maas-api"
Nov 01 03:34:45.574031 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:45.573990    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="6084708e-57aa-4b1a-8dfe-d50e45585616" containerName="maas-api"
Nov 01 03:34:45.574031 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:45.573997    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="86018717-3f70-43de-843c-f9429c605a74" containerName="istio-proxy"
Nov 01 03:34:45.574031 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:45.574007    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="edf3bdc9-df09-43cd-a4f3-b5c2ebfaa1dd" containerName="maas-api"
Nov 01 03:34:45.579137 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:45.579120    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fdbdfefd-cb877c7cc-9g4k2"
Nov 01 03:34:45.633135 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:45.633067    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-fdbdfefd-serving-cert\""
Nov 01 03:34:45.683895 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:45.683858    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c60732bb-7134-40b1-b5b1-48607d07bd68-maas-api-tls\") pod \"maas-api-e2e-mt-fdbdfefd-cb877c7cc-9g4k2\" (UID: \"c60732bb-7134-40b1-b5b1-48607d07bd68\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fdbdfefd-cb877c7cc-9g4k2"
Nov 01 03:34:45.684111 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:45.684002    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsghh\" (UniqueName: \"kubernetes.io/projected/c60732bb-7134-40b1-b5b1-48607d07bd68-kube-api-access-hsghh\") pod \"maas-api-e2e-mt-fdbdfefd-cb877c7cc-9g4k2\" (UID: \"c60732bb-7134-40b1-b5b1-48607d07bd68\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fdbdfefd-cb877c7cc-9g4k2"
Nov 01 03:34:45.684111 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:45.684082    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c60732bb-7134-40b1-b5b1-48607d07bd68-metrics-certs\") pod \"maas-api-e2e-mt-fdbdfefd-cb877c7cc-9g4k2\" (UID: \"c60732bb-7134-40b1-b5b1-48607d07bd68\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fdbdfefd-cb877c7cc-9g4k2"
Nov 01 03:34:45.684265 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:45.684134    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c60732bb-7134-40b1-b5b1-48607d07bd68-openshift-service-ca\") pod \"maas-api-e2e-mt-fdbdfefd-cb877c7cc-9g4k2\" (UID: \"c60732bb-7134-40b1-b5b1-48607d07bd68\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fdbdfefd-cb877c7cc-9g4k2"
Nov 01 03:34:45.688416 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:45.688393    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fdbdfefd-cb877c7cc-9g4k2"]
Nov 01 03:34:45.785920 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:45.785884    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c60732bb-7134-40b1-b5b1-48607d07bd68-openshift-service-ca\") pod \"maas-api-e2e-mt-fdbdfefd-cb877c7cc-9g4k2\" (UID: \"c60732bb-7134-40b1-b5b1-48607d07bd68\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fdbdfefd-cb877c7cc-9g4k2"
Nov 01 03:34:45.786199 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:45.786162    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c60732bb-7134-40b1-b5b1-48607d07bd68-maas-api-tls\") pod \"maas-api-e2e-mt-fdbdfefd-cb877c7cc-9g4k2\" (UID: \"c60732bb-7134-40b1-b5b1-48607d07bd68\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fdbdfefd-cb877c7cc-9g4k2"
Nov 01 03:34:45.786308 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:34:45.786234    4400 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-fdbdfefd-serving-cert: secret "maas-api-e2e-mt-fdbdfefd-serving-cert" not found
Nov 01 03:34:45.786308 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:45.786262    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsghh\" (UniqueName: \"kubernetes.io/projected/c60732bb-7134-40b1-b5b1-48607d07bd68-kube-api-access-hsghh\") pod \"maas-api-e2e-mt-fdbdfefd-cb877c7cc-9g4k2\" (UID: \"c60732bb-7134-40b1-b5b1-48607d07bd68\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fdbdfefd-cb877c7cc-9g4k2"
Nov 01 03:34:45.786308 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:34:45.786298    4400 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c60732bb-7134-40b1-b5b1-48607d07bd68-maas-api-tls podName:c60732bb-7134-40b1-b5b1-48607d07bd68 nodeName:}" failed. No retries permitted until 2026-11-01 03:34:46.286282777 +0000 UTC m=+1692.800928620 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c60732bb-7134-40b1-b5b1-48607d07bd68-maas-api-tls") pod "maas-api-e2e-mt-fdbdfefd-cb877c7cc-9g4k2" (UID: "c60732bb-7134-40b1-b5b1-48607d07bd68") : secret "maas-api-e2e-mt-fdbdfefd-serving-cert" not found
Nov 01 03:34:45.786466 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:45.786367    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c60732bb-7134-40b1-b5b1-48607d07bd68-metrics-certs\") pod \"maas-api-e2e-mt-fdbdfefd-cb877c7cc-9g4k2\" (UID: \"c60732bb-7134-40b1-b5b1-48607d07bd68\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fdbdfefd-cb877c7cc-9g4k2"
Nov 01 03:34:45.786623 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:45.786601    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c60732bb-7134-40b1-b5b1-48607d07bd68-openshift-service-ca\") pod \"maas-api-e2e-mt-fdbdfefd-cb877c7cc-9g4k2\" (UID: \"c60732bb-7134-40b1-b5b1-48607d07bd68\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fdbdfefd-cb877c7cc-9g4k2"
Nov 01 03:34:45.788782 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:45.788765    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c60732bb-7134-40b1-b5b1-48607d07bd68-metrics-certs\") pod \"maas-api-e2e-mt-fdbdfefd-cb877c7cc-9g4k2\" (UID: \"c60732bb-7134-40b1-b5b1-48607d07bd68\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fdbdfefd-cb877c7cc-9g4k2"
Nov 01 03:34:45.849381 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:45.849311    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsghh\" (UniqueName: \"kubernetes.io/projected/c60732bb-7134-40b1-b5b1-48607d07bd68-kube-api-access-hsghh\") pod \"maas-api-e2e-mt-fdbdfefd-cb877c7cc-9g4k2\" (UID: \"c60732bb-7134-40b1-b5b1-48607d07bd68\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fdbdfefd-cb877c7cc-9g4k2"
Nov 01 03:34:46.312043 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:46.311969    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c60732bb-7134-40b1-b5b1-48607d07bd68-maas-api-tls\") pod \"maas-api-e2e-mt-fdbdfefd-cb877c7cc-9g4k2\" (UID: \"c60732bb-7134-40b1-b5b1-48607d07bd68\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fdbdfefd-cb877c7cc-9g4k2"
Nov 01 03:34:46.314586 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:46.314566    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c60732bb-7134-40b1-b5b1-48607d07bd68-maas-api-tls\") pod \"maas-api-e2e-mt-fdbdfefd-cb877c7cc-9g4k2\" (UID: \"c60732bb-7134-40b1-b5b1-48607d07bd68\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fdbdfefd-cb877c7cc-9g4k2"
Nov 01 03:34:46.493418 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:46.493377    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fdbdfefd-cb877c7cc-9g4k2"
Nov 01 03:34:46.880828 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:34:46.880800    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc60732bb_7134_40b1_b5b1_48607d07bd68.slice/crio-1457f0544345f7f7da50e2737f0b0b34c2a8ffbffb3dbf224ac41648ccc36336 WatchSource:0}: Error finding container 1457f0544345f7f7da50e2737f0b0b34c2a8ffbffb3dbf224ac41648ccc36336: Status 404 returned error can't find the container with id 1457f0544345f7f7da50e2737f0b0b34c2a8ffbffb3dbf224ac41648ccc36336
Nov 01 03:34:46.881524 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:46.881498    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fdbdfefd-cb877c7cc-9g4k2"]
Nov 01 03:34:47.216267 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:47.216159    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fdbdfefd-cb877c7cc-9g4k2" event={"ID":"c60732bb-7134-40b1-b5b1-48607d07bd68","Type":"ContainerStarted","Data":"1457f0544345f7f7da50e2737f0b0b34c2a8ffbffb3dbf224ac41648ccc36336"}
Nov 01 03:34:48.227728 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:48.227694    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fdbdfefd-cb877c7cc-9g4k2" event={"ID":"c60732bb-7134-40b1-b5b1-48607d07bd68","Type":"ContainerStarted","Data":"ffaf63382a918ecc17d90d8b1efe9b211425c055ad17d1dcb6b13ae5b816dfc1"}
Nov 01 03:34:48.228120 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:48.227776    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fdbdfefd-cb877c7cc-9g4k2"
Nov 01 03:34:48.377837 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:48.377788    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fdbdfefd-cb877c7cc-9g4k2" podStartSLOduration=2.237348576 podStartE2EDuration="3.377773561s" totalImagesPullingTime="1.140424985s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:34:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:34:48.374456104 +0000 UTC m=+1694.889101967" watchObservedRunningTime="2026-11-01 03:34:48.377773561 +0000 UTC m=+1694.892419423"
Nov 01 03:34:48.488661 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:48.488628    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 03:34:54.316872 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:34:54.316838    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fdbdfefd-cb877c7cc-9g4k2"
Nov 01 03:35:01.730411 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:35:01.730379    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:35:16.502109 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:35:16.502079    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e8c77d-7cdd9d68dd-hmjst"]
Nov 01 03:35:16.506448 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:35:16.506432    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e8c77d-7cdd9d68dd-hmjst"
Nov 01 03:35:16.563441 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:35:16.563417    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-e8c77d-serving-cert\""
Nov 01 03:35:16.565055 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:35:16.565032    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e8c77d-7cdd9d68dd-hmjst"]
Nov 01 03:35:16.624302 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:35:16.624272    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad9a53c5-5164-42e6-8a89-69666c43dec0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-e8c77d-7cdd9d68dd-hmjst\" (UID: \"ad9a53c5-5164-42e6-8a89-69666c43dec0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e8c77d-7cdd9d68dd-hmjst"
Nov 01 03:35:16.624429 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:35:16.624336    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad9a53c5-5164-42e6-8a89-69666c43dec0-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-e8c77d-7cdd9d68dd-hmjst\" (UID: \"ad9a53c5-5164-42e6-8a89-69666c43dec0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e8c77d-7cdd9d68dd-hmjst"
Nov 01 03:35:16.624429 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:35:16.624403    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad9a53c5-5164-42e6-8a89-69666c43dec0-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-e8c77d-7cdd9d68dd-hmjst\" (UID: \"ad9a53c5-5164-42e6-8a89-69666c43dec0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e8c77d-7cdd9d68dd-hmjst"
Nov 01 03:35:16.624584 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:35:16.624551    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzc55\" (UniqueName: \"kubernetes.io/projected/ad9a53c5-5164-42e6-8a89-69666c43dec0-kube-api-access-hzc55\") pod \"maas-api-e2e-shared-a-w0-e8c77d-7cdd9d68dd-hmjst\" (UID: \"ad9a53c5-5164-42e6-8a89-69666c43dec0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e8c77d-7cdd9d68dd-hmjst"
Nov 01 03:35:16.726203 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:35:16.726174    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad9a53c5-5164-42e6-8a89-69666c43dec0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-e8c77d-7cdd9d68dd-hmjst\" (UID: \"ad9a53c5-5164-42e6-8a89-69666c43dec0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e8c77d-7cdd9d68dd-hmjst"
Nov 01 03:35:16.726364 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:35:16.726270    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad9a53c5-5164-42e6-8a89-69666c43dec0-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-e8c77d-7cdd9d68dd-hmjst\" (UID: \"ad9a53c5-5164-42e6-8a89-69666c43dec0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e8c77d-7cdd9d68dd-hmjst"
Nov 01 03:35:16.726364 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:35:16.726300    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad9a53c5-5164-42e6-8a89-69666c43dec0-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-e8c77d-7cdd9d68dd-hmjst\" (UID: \"ad9a53c5-5164-42e6-8a89-69666c43dec0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e8c77d-7cdd9d68dd-hmjst"
Nov 01 03:35:16.726489 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:35:16.726382    4400 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e8c77d-serving-cert: secret "maas-api-e2e-shared-a-w0-e8c77d-serving-cert" not found
Nov 01 03:35:16.726489 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:35:16.726409    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzc55\" (UniqueName: \"kubernetes.io/projected/ad9a53c5-5164-42e6-8a89-69666c43dec0-kube-api-access-hzc55\") pod \"maas-api-e2e-shared-a-w0-e8c77d-7cdd9d68dd-hmjst\" (UID: \"ad9a53c5-5164-42e6-8a89-69666c43dec0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e8c77d-7cdd9d68dd-hmjst"
Nov 01 03:35:16.726489 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:35:16.726448    4400 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad9a53c5-5164-42e6-8a89-69666c43dec0-maas-api-tls podName:ad9a53c5-5164-42e6-8a89-69666c43dec0 nodeName:}" failed. No retries permitted until 2026-11-01 03:35:17.226426126 +0000 UTC m=+1723.741071986 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ad9a53c5-5164-42e6-8a89-69666c43dec0-maas-api-tls") pod "maas-api-e2e-shared-a-w0-e8c77d-7cdd9d68dd-hmjst" (UID: "ad9a53c5-5164-42e6-8a89-69666c43dec0") : secret "maas-api-e2e-shared-a-w0-e8c77d-serving-cert" not found
Nov 01 03:35:16.726923 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:35:16.726899    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad9a53c5-5164-42e6-8a89-69666c43dec0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-e8c77d-7cdd9d68dd-hmjst\" (UID: \"ad9a53c5-5164-42e6-8a89-69666c43dec0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e8c77d-7cdd9d68dd-hmjst"
Nov 01 03:35:16.728980 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:35:16.728959    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad9a53c5-5164-42e6-8a89-69666c43dec0-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-e8c77d-7cdd9d68dd-hmjst\" (UID: \"ad9a53c5-5164-42e6-8a89-69666c43dec0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e8c77d-7cdd9d68dd-hmjst"
Nov 01 03:35:16.833541 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:35:16.833509    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzc55\" (UniqueName: \"kubernetes.io/projected/ad9a53c5-5164-42e6-8a89-69666c43dec0-kube-api-access-hzc55\") pod \"maas-api-e2e-shared-a-w0-e8c77d-7cdd9d68dd-hmjst\" (UID: \"ad9a53c5-5164-42e6-8a89-69666c43dec0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e8c77d-7cdd9d68dd-hmjst"
Nov 01 03:35:17.244774 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:35:17.244677    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad9a53c5-5164-42e6-8a89-69666c43dec0-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-e8c77d-7cdd9d68dd-hmjst\" (UID: \"ad9a53c5-5164-42e6-8a89-69666c43dec0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e8c77d-7cdd9d68dd-hmjst"
Nov 01 03:35:17.247202 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:35:17.247175    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad9a53c5-5164-42e6-8a89-69666c43dec0-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-e8c77d-7cdd9d68dd-hmjst\" (UID: \"ad9a53c5-5164-42e6-8a89-69666c43dec0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e8c77d-7cdd9d68dd-hmjst"
Nov 01 03:35:17.420089 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:35:17.420064    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e8c77d-7cdd9d68dd-hmjst"
Nov 01 03:35:17.806856 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:35:17.806822    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e8c77d-7cdd9d68dd-hmjst"]
Nov 01 03:35:17.812262 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:35:17.812237    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad9a53c5_5164_42e6_8a89_69666c43dec0.slice/crio-cc5f3dc72e8268365028c7aeb173cd882d61644408164caeb67927fdc1369486 WatchSource:0}: Error finding container cc5f3dc72e8268365028c7aeb173cd882d61644408164caeb67927fdc1369486: Status 404 returned error can't find the container with id cc5f3dc72e8268365028c7aeb173cd882d61644408164caeb67927fdc1369486
Nov 01 03:35:18.847257 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:35:18.847223    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e8c77d-7cdd9d68dd-hmjst" event={"ID":"ad9a53c5-5164-42e6-8a89-69666c43dec0","Type":"ContainerStarted","Data":"9718e281d20654848880ce6a6470394c9067c978005b2e1e2415da47a36e62f5"}
Nov 01 03:35:18.847671 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:35:18.847277    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e8c77d-7cdd9d68dd-hmjst" event={"ID":"ad9a53c5-5164-42e6-8a89-69666c43dec0","Type":"ContainerStarted","Data":"cc5f3dc72e8268365028c7aeb173cd882d61644408164caeb67927fdc1369486"}
Nov 01 03:35:18.847671 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:35:18.847315    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e8c77d-7cdd9d68dd-hmjst"
Nov 01 03:35:18.998363 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:35:18.998278    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e8c77d-7cdd9d68dd-hmjst" podStartSLOduration=2.129227355 podStartE2EDuration="2.998264848s" totalImagesPullingTime="869.037493ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:35:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:35:18.995504253 +0000 UTC m=+1725.510150115" watchObservedRunningTime="2026-11-01 03:35:18.998264848 +0000 UTC m=+1725.512910708"
Nov 01 03:35:24.868435 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:35:24.868405    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e8c77d-7cdd9d68dd-hmjst"
Nov 01 03:35:31.729259 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:35:31.729224    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:35:34.016347 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:35:34.016315    4400 scope.go:122] "RemoveContainer" containerID="68d624ce390342eae619c6ac68c815e625408abf528480dbd626b7d1d5765ad2"
Nov 01 03:36:01.729975 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:36:01.729939    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:36:31.847551 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:36:31.847509    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:36:32.405270 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:36:32.404541    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 03:37:00.737729 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:00.737663    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-8sx2k"]
Nov 01 03:37:00.738260 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:00.737915    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-8sx2k" podUID="b3532ce0-59fa-4822-bff3-2d0d356a3f13" containerName="istio-proxy" containerID="cri-o://660447dc25d90263f45c91c2528a9e508a36a7eccf1457163368f6d54492bd36" gracePeriod=30
Nov 01 03:37:01.443978 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:01.443940    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-9qpm8"]
Nov 01 03:37:01.444492 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:01.444459    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-9qpm8" podUID="204735ae-bff5-4cdc-9df5-f85158677fdc" containerName="istio-proxy" containerID="cri-o://d858e3bb3f335736e568ec1f96c85dc678c5a395bc9294a668d681221704d47d" gracePeriod=30
Nov 01 03:37:01.723851 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:01.723774    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hf2dc"]
Nov 01 03:37:01.724081 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:01.724056    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hf2dc" podUID="51938745-c46e-4917-b963-5d1f98b0298d" containerName="istio-proxy" containerID="cri-o://6cff9759bad6097d625173d1c685932d01aeea4f36bb735534b7df550af291a8" gracePeriod=30
Nov 01 03:37:01.735217 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:37:01.735184    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:37:02.162764 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:02.162727    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-l4xqq"]
Nov 01 03:37:02.163529 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:02.163495    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-l4xqq" podUID="73ae6733-594c-47d9-89e9-8b92fde0b8f2" containerName="istio-proxy" containerID="cri-o://9bc3adca49fba1b1c0c649e84f537f42d45a25e8ad129ecb62c4d63e2c63292e" gracePeriod=30
Nov 01 03:37:02.171218 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:02.167271    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-9xqhq"]
Nov 01 03:37:02.171218 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:02.167420    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-9xqhq" podUID="6c782468-958d-496b-afec-73b546553a57" containerName="istio-proxy" containerID="cri-o://12cee91983923b762f742ec2bd784e664907973dbfe2b4f377a7298d7a67a855" gracePeriod=30
Nov 01 03:37:04.836460 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:04.836425    4400 patch_prober.go:29] interesting pod/e2e-worker-w6-19faa5-openshift-default-59c44967-8sx2k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 03:37:04.836881 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:04.836485    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-8sx2k" podUID="b3532ce0-59fa-4822-bff3-2d0d356a3f13" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 03:37:06.029399 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:06.029373    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-8sx2k"
Nov 01 03:37:06.092303 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:06.092270    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3532ce0-59fa-4822-bff3-2d0d356a3f13-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3532ce0-59fa-4822-bff3-2d0d356a3f13-istiod-ca-cert\") pod \"b3532ce0-59fa-4822-bff3-2d0d356a3f13\" (UID: \"b3532ce0-59fa-4822-bff3-2d0d356a3f13\") "
Nov 01 03:37:06.092475 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:06.092418    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3532ce0-59fa-4822-bff3-2d0d356a3f13-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3532ce0-59fa-4822-bff3-2d0d356a3f13-credential-socket\") pod \"b3532ce0-59fa-4822-bff3-2d0d356a3f13\" (UID: \"b3532ce0-59fa-4822-bff3-2d0d356a3f13\") "
Nov 01 03:37:06.092563 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:06.092476    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b3532ce0-59fa-4822-bff3-2d0d356a3f13-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3532ce0-59fa-4822-bff3-2d0d356a3f13-istio-podinfo\") pod \"b3532ce0-59fa-4822-bff3-2d0d356a3f13\" (UID: \"b3532ce0-59fa-4822-bff3-2d0d356a3f13\") "
Nov 01 03:37:06.092563 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:06.092543    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3532ce0-59fa-4822-bff3-2d0d356a3f13-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3532ce0-59fa-4822-bff3-2d0d356a3f13-workload-certs\") pod \"b3532ce0-59fa-4822-bff3-2d0d356a3f13\" (UID: \"b3532ce0-59fa-4822-bff3-2d0d356a3f13\") "
Nov 01 03:37:06.092672 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:06.092622    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3532ce0-59fa-4822-bff3-2d0d356a3f13-kube-api-access-dt952\" (UniqueName: \"kubernetes.io/projected/b3532ce0-59fa-4822-bff3-2d0d356a3f13-kube-api-access-dt952\") pod \"b3532ce0-59fa-4822-bff3-2d0d356a3f13\" (UID: \"b3532ce0-59fa-4822-bff3-2d0d356a3f13\") "
Nov 01 03:37:06.092672 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:06.092665    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3532ce0-59fa-4822-bff3-2d0d356a3f13-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3532ce0-59fa-4822-bff3-2d0d356a3f13-istio-envoy\") pod \"b3532ce0-59fa-4822-bff3-2d0d356a3f13\" (UID: \"b3532ce0-59fa-4822-bff3-2d0d356a3f13\") "
Nov 01 03:37:06.092836 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:06.092726    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3532ce0-59fa-4822-bff3-2d0d356a3f13-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3532ce0-59fa-4822-bff3-2d0d356a3f13-istio-data\") pod \"b3532ce0-59fa-4822-bff3-2d0d356a3f13\" (UID: \"b3532ce0-59fa-4822-bff3-2d0d356a3f13\") "
Nov 01 03:37:06.092836 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:06.092784    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3532ce0-59fa-4822-bff3-2d0d356a3f13-istio-token\" (UniqueName: \"kubernetes.io/projected/b3532ce0-59fa-4822-bff3-2d0d356a3f13-istio-token\") pod \"b3532ce0-59fa-4822-bff3-2d0d356a3f13\" (UID: \"b3532ce0-59fa-4822-bff3-2d0d356a3f13\") "
Nov 01 03:37:06.092836 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:06.092809    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3532ce0-59fa-4822-bff3-2d0d356a3f13-credential-socket" pod "b3532ce0-59fa-4822-bff3-2d0d356a3f13" (UID: "b3532ce0-59fa-4822-bff3-2d0d356a3f13"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:37:06.092836 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:06.092826    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3532ce0-59fa-4822-bff3-2d0d356a3f13-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3532ce0-59fa-4822-bff3-2d0d356a3f13-workload-socket\") pod \"b3532ce0-59fa-4822-bff3-2d0d356a3f13\" (UID: \"b3532ce0-59fa-4822-bff3-2d0d356a3f13\") "
Nov 01 03:37:06.092836 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:06.092825    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3532ce0-59fa-4822-bff3-2d0d356a3f13-workload-certs" pod "b3532ce0-59fa-4822-bff3-2d0d356a3f13" (UID: "b3532ce0-59fa-4822-bff3-2d0d356a3f13"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:37:06.093081 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:06.092846    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3532ce0-59fa-4822-bff3-2d0d356a3f13-istiod-ca-cert" pod "b3532ce0-59fa-4822-bff3-2d0d356a3f13" (UID: "b3532ce0-59fa-4822-bff3-2d0d356a3f13"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:37:06.093150 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:06.093085    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3532ce0-59fa-4822-bff3-2d0d356a3f13-workload-socket" pod "b3532ce0-59fa-4822-bff3-2d0d356a3f13" (UID: "b3532ce0-59fa-4822-bff3-2d0d356a3f13"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:37:06.093150 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:06.093102    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3532ce0-59fa-4822-bff3-2d0d356a3f13-istio-data" pod "b3532ce0-59fa-4822-bff3-2d0d356a3f13" (UID: "b3532ce0-59fa-4822-bff3-2d0d356a3f13"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:37:06.094045 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:06.093915    4400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3532ce0-59fa-4822-bff3-2d0d356a3f13-credential-socket\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:37:06.094045 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:06.093943    4400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3532ce0-59fa-4822-bff3-2d0d356a3f13-workload-certs\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:37:06.094045 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:06.093956    4400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3532ce0-59fa-4822-bff3-2d0d356a3f13-istio-data\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:37:06.094045 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:06.093969    4400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3532ce0-59fa-4822-bff3-2d0d356a3f13-workload-socket\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:37:06.094045 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:06.093982    4400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3532ce0-59fa-4822-bff3-2d0d356a3f13-istiod-ca-cert\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:37:06.095078 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:06.095052    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3532ce0-59fa-4822-bff3-2d0d356a3f13-istio-envoy" pod "b3532ce0-59fa-4822-bff3-2d0d356a3f13" (UID: "b3532ce0-59fa-4822-bff3-2d0d356a3f13"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:37:06.095210 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:06.095186    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3532ce0-59fa-4822-bff3-2d0d356a3f13-kube-api-access-dt952" pod "b3532ce0-59fa-4822-bff3-2d0d356a3f13" (UID: "b3532ce0-59fa-4822-bff3-2d0d356a3f13"). InnerVolumeSpecName "kube-api-access-dt952". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:37:06.095253 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:06.095205    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b3532ce0-59fa-4822-bff3-2d0d356a3f13-istio-podinfo" pod "b3532ce0-59fa-4822-bff3-2d0d356a3f13" (UID: "b3532ce0-59fa-4822-bff3-2d0d356a3f13"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 03:37:06.095253 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:06.095239    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3532ce0-59fa-4822-bff3-2d0d356a3f13-istio-token" pod "b3532ce0-59fa-4822-bff3-2d0d356a3f13" (UID: "b3532ce0-59fa-4822-bff3-2d0d356a3f13"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:37:06.206712 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:06.201757    4400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3532ce0-59fa-4822-bff3-2d0d356a3f13-istio-podinfo\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:37:06.206712 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:06.201819    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dt952\" (UniqueName: \"kubernetes.io/projected/b3532ce0-59fa-4822-bff3-2d0d356a3f13-kube-api-access-dt952\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:37:06.206712 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:06.201836    4400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3532ce0-59fa-4822-bff3-2d0d356a3f13-istio-envoy\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:37:06.206712 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:06.201848    4400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3532ce0-59fa-4822-bff3-2d0d356a3f13-istio-token\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:37:06.322257 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:06.322225    4400 generic.go:356] "Generic (PLEG): container finished" podID="b3532ce0-59fa-4822-bff3-2d0d356a3f13" containerID="660447dc25d90263f45c91c2528a9e508a36a7eccf1457163368f6d54492bd36" exitCode=0
Nov 01 03:37:06.322407 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:06.322273    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-8sx2k"
Nov 01 03:37:06.322407 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:06.322314    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-8sx2k" event={"ID":"b3532ce0-59fa-4822-bff3-2d0d356a3f13","Type":"ContainerDied","Data":"660447dc25d90263f45c91c2528a9e508a36a7eccf1457163368f6d54492bd36"}
Nov 01 03:37:06.322407 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:06.322368    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-8sx2k" event={"ID":"b3532ce0-59fa-4822-bff3-2d0d356a3f13","Type":"ContainerDied","Data":"4ad24337d9a122978a7cb41fdafdb42c4066d619c8a90ee0a33e1dac690f8788"}
Nov 01 03:37:06.322407 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:06.322395    4400 scope.go:122] "RemoveContainer" containerID="660447dc25d90263f45c91c2528a9e508a36a7eccf1457163368f6d54492bd36"
Nov 01 03:37:06.333553 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:06.333537    4400 scope.go:122] "RemoveContainer" containerID="660447dc25d90263f45c91c2528a9e508a36a7eccf1457163368f6d54492bd36"
Nov 01 03:37:06.333831 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:37:06.333813    4400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"660447dc25d90263f45c91c2528a9e508a36a7eccf1457163368f6d54492bd36\": container with ID starting with 660447dc25d90263f45c91c2528a9e508a36a7eccf1457163368f6d54492bd36 not found: ID does not exist" containerID="660447dc25d90263f45c91c2528a9e508a36a7eccf1457163368f6d54492bd36"
Nov 01 03:37:06.333874 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:06.333839    4400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"660447dc25d90263f45c91c2528a9e508a36a7eccf1457163368f6d54492bd36"} err="failed to get container status \"660447dc25d90263f45c91c2528a9e508a36a7eccf1457163368f6d54492bd36\": rpc error: code = NotFound desc = could not find container \"660447dc25d90263f45c91c2528a9e508a36a7eccf1457163368f6d54492bd36\": container with ID starting with 660447dc25d90263f45c91c2528a9e508a36a7eccf1457163368f6d54492bd36 not found: ID does not exist"
Nov 01 03:37:06.589886 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:06.589770    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-8sx2k"]
Nov 01 03:37:06.601280 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:06.601240    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-8sx2k"]
Nov 01 03:37:07.306996 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.306964    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hf2dc"
Nov 01 03:37:07.312168 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.312143    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-9qpm8"
Nov 01 03:37:07.347495 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.333057    4400 generic.go:356] "Generic (PLEG): container finished" podID="204735ae-bff5-4cdc-9df5-f85158677fdc" containerID="d858e3bb3f335736e568ec1f96c85dc678c5a395bc9294a668d681221704d47d" exitCode=0
Nov 01 03:37:07.347495 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.333169    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-9qpm8" event={"ID":"204735ae-bff5-4cdc-9df5-f85158677fdc","Type":"ContainerDied","Data":"d858e3bb3f335736e568ec1f96c85dc678c5a395bc9294a668d681221704d47d"}
Nov 01 03:37:07.347495 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.333250    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-9qpm8" event={"ID":"204735ae-bff5-4cdc-9df5-f85158677fdc","Type":"ContainerDied","Data":"dcbcb0cce3820a1b009665ce9e8b9681673c0848579a96f4a17f5531bd27ee65"}
Nov 01 03:37:07.347495 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.333281    4400 scope.go:122] "RemoveContainer" containerID="d858e3bb3f335736e568ec1f96c85dc678c5a395bc9294a668d681221704d47d"
Nov 01 03:37:07.347495 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.333423    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-9qpm8"
Nov 01 03:37:07.348570 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.348032    4400 generic.go:356] "Generic (PLEG): container finished" podID="51938745-c46e-4917-b963-5d1f98b0298d" containerID="6cff9759bad6097d625173d1c685932d01aeea4f36bb735534b7df550af291a8" exitCode=0
Nov 01 03:37:07.348570 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.348244    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hf2dc" event={"ID":"51938745-c46e-4917-b963-5d1f98b0298d","Type":"ContainerDied","Data":"6cff9759bad6097d625173d1c685932d01aeea4f36bb735534b7df550af291a8"}
Nov 01 03:37:07.348570 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.348302    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hf2dc" event={"ID":"51938745-c46e-4917-b963-5d1f98b0298d","Type":"ContainerDied","Data":"8e7fd7e6acd965a5f3af5d0be21d77d43aa4bca947958ca3270e9b2ffb241459"}
Nov 01 03:37:07.348570 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.348393    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hf2dc"
Nov 01 03:37:07.356102 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.355899    4400 generic.go:356] "Generic (PLEG): container finished" podID="6c782468-958d-496b-afec-73b546553a57" containerID="12cee91983923b762f742ec2bd784e664907973dbfe2b4f377a7298d7a67a855" exitCode=0
Nov 01 03:37:07.356102 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.356019    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-9xqhq" event={"ID":"6c782468-958d-496b-afec-73b546553a57","Type":"ContainerDied","Data":"12cee91983923b762f742ec2bd784e664907973dbfe2b4f377a7298d7a67a855"}
Nov 01 03:37:07.358519 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.358359    4400 generic.go:356] "Generic (PLEG): container finished" podID="73ae6733-594c-47d9-89e9-8b92fde0b8f2" containerID="9bc3adca49fba1b1c0c649e84f537f42d45a25e8ad129ecb62c4d63e2c63292e" exitCode=0
Nov 01 03:37:07.358519 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.358459    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-l4xqq" event={"ID":"73ae6733-594c-47d9-89e9-8b92fde0b8f2","Type":"ContainerDied","Data":"9bc3adca49fba1b1c0c649e84f537f42d45a25e8ad129ecb62c4d63e2c63292e"}
Nov 01 03:37:07.362369 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.361667    4400 scope.go:122] "RemoveContainer" containerID="d858e3bb3f335736e568ec1f96c85dc678c5a395bc9294a668d681221704d47d"
Nov 01 03:37:07.362369 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.362344    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51938745-c46e-4917-b963-5d1f98b0298d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51938745-c46e-4917-b963-5d1f98b0298d-istio-data\") pod \"51938745-c46e-4917-b963-5d1f98b0298d\" (UID: \"51938745-c46e-4917-b963-5d1f98b0298d\") "
Nov 01 03:37:07.362548 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.362386    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/204735ae-bff5-4cdc-9df5-f85158677fdc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/204735ae-bff5-4cdc-9df5-f85158677fdc-istiod-ca-cert\") pod \"204735ae-bff5-4cdc-9df5-f85158677fdc\" (UID: \"204735ae-bff5-4cdc-9df5-f85158677fdc\") "
Nov 01 03:37:07.362548 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.362446    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/204735ae-bff5-4cdc-9df5-f85158677fdc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/204735ae-bff5-4cdc-9df5-f85158677fdc-workload-certs\") pod \"204735ae-bff5-4cdc-9df5-f85158677fdc\" (UID: \"204735ae-bff5-4cdc-9df5-f85158677fdc\") "
Nov 01 03:37:07.362548 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.362475    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/204735ae-bff5-4cdc-9df5-f85158677fdc-kube-api-access-wb5xw\" (UniqueName: \"kubernetes.io/projected/204735ae-bff5-4cdc-9df5-f85158677fdc-kube-api-access-wb5xw\") pod \"204735ae-bff5-4cdc-9df5-f85158677fdc\" (UID: \"204735ae-bff5-4cdc-9df5-f85158677fdc\") "
Nov 01 03:37:07.362548 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.362502    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51938745-c46e-4917-b963-5d1f98b0298d-kube-api-access-97r95\" (UniqueName: \"kubernetes.io/projected/51938745-c46e-4917-b963-5d1f98b0298d-kube-api-access-97r95\") pod \"51938745-c46e-4917-b963-5d1f98b0298d\" (UID: \"51938745-c46e-4917-b963-5d1f98b0298d\") "
Nov 01 03:37:07.362755 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.362561    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51938745-c46e-4917-b963-5d1f98b0298d-istio-token\" (UniqueName: \"kubernetes.io/projected/51938745-c46e-4917-b963-5d1f98b0298d-istio-token\") pod \"51938745-c46e-4917-b963-5d1f98b0298d\" (UID: \"51938745-c46e-4917-b963-5d1f98b0298d\") "
Nov 01 03:37:07.362755 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.362597    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/204735ae-bff5-4cdc-9df5-f85158677fdc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/204735ae-bff5-4cdc-9df5-f85158677fdc-istio-data\") pod \"204735ae-bff5-4cdc-9df5-f85158677fdc\" (UID: \"204735ae-bff5-4cdc-9df5-f85158677fdc\") "
Nov 01 03:37:07.362755 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.362670    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/51938745-c46e-4917-b963-5d1f98b0298d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51938745-c46e-4917-b963-5d1f98b0298d-istio-podinfo\") pod \"51938745-c46e-4917-b963-5d1f98b0298d\" (UID: \"51938745-c46e-4917-b963-5d1f98b0298d\") "
Nov 01 03:37:07.362755 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.362735    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/204735ae-bff5-4cdc-9df5-f85158677fdc-istio-token\" (UniqueName: \"kubernetes.io/projected/204735ae-bff5-4cdc-9df5-f85158677fdc-istio-token\") pod \"204735ae-bff5-4cdc-9df5-f85158677fdc\" (UID: \"204735ae-bff5-4cdc-9df5-f85158677fdc\") "
Nov 01 03:37:07.363779 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:37:07.363748    4400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d858e3bb3f335736e568ec1f96c85dc678c5a395bc9294a668d681221704d47d\": container with ID starting with d858e3bb3f335736e568ec1f96c85dc678c5a395bc9294a668d681221704d47d not found: ID does not exist" containerID="d858e3bb3f335736e568ec1f96c85dc678c5a395bc9294a668d681221704d47d"
Nov 01 03:37:07.363880 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.363786    4400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d858e3bb3f335736e568ec1f96c85dc678c5a395bc9294a668d681221704d47d"} err="failed to get container status \"d858e3bb3f335736e568ec1f96c85dc678c5a395bc9294a668d681221704d47d\": rpc error: code = NotFound desc = could not find container \"d858e3bb3f335736e568ec1f96c85dc678c5a395bc9294a668d681221704d47d\": container with ID starting with d858e3bb3f335736e568ec1f96c85dc678c5a395bc9294a668d681221704d47d not found: ID does not exist"
Nov 01 03:37:07.363880 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.363809    4400 scope.go:122] "RemoveContainer" containerID="6cff9759bad6097d625173d1c685932d01aeea4f36bb735534b7df550af291a8"
Nov 01 03:37:07.364217 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.364111    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/204735ae-bff5-4cdc-9df5-f85158677fdc-istio-data" pod "204735ae-bff5-4cdc-9df5-f85158677fdc" (UID: "204735ae-bff5-4cdc-9df5-f85158677fdc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:37:07.364444 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.364413    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/204735ae-bff5-4cdc-9df5-f85158677fdc-workload-certs" pod "204735ae-bff5-4cdc-9df5-f85158677fdc" (UID: "204735ae-bff5-4cdc-9df5-f85158677fdc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:37:07.365258 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.364872    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51938745-c46e-4917-b963-5d1f98b0298d-istio-data" pod "51938745-c46e-4917-b963-5d1f98b0298d" (UID: "51938745-c46e-4917-b963-5d1f98b0298d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:37:07.365258 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.365023    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/204735ae-bff5-4cdc-9df5-f85158677fdc-istiod-ca-cert" pod "204735ae-bff5-4cdc-9df5-f85158677fdc" (UID: "204735ae-bff5-4cdc-9df5-f85158677fdc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:37:07.367447 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.366249    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/204735ae-bff5-4cdc-9df5-f85158677fdc-kube-api-access-wb5xw" pod "204735ae-bff5-4cdc-9df5-f85158677fdc" (UID: "204735ae-bff5-4cdc-9df5-f85158677fdc"). InnerVolumeSpecName "kube-api-access-wb5xw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:37:07.367861 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.367837    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/204735ae-bff5-4cdc-9df5-f85158677fdc-istio-token" pod "204735ae-bff5-4cdc-9df5-f85158677fdc" (UID: "204735ae-bff5-4cdc-9df5-f85158677fdc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:37:07.369528 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.369502    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/51938745-c46e-4917-b963-5d1f98b0298d-istio-podinfo" pod "51938745-c46e-4917-b963-5d1f98b0298d" (UID: "51938745-c46e-4917-b963-5d1f98b0298d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 03:37:07.369631 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.369503    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51938745-c46e-4917-b963-5d1f98b0298d-kube-api-access-97r95" pod "51938745-c46e-4917-b963-5d1f98b0298d" (UID: "51938745-c46e-4917-b963-5d1f98b0298d"). InnerVolumeSpecName "kube-api-access-97r95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:37:07.369752 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.369724    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51938745-c46e-4917-b963-5d1f98b0298d-istio-token" pod "51938745-c46e-4917-b963-5d1f98b0298d" (UID: "51938745-c46e-4917-b963-5d1f98b0298d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:37:07.393457 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.393427    4400 scope.go:122] "RemoveContainer" containerID="6cff9759bad6097d625173d1c685932d01aeea4f36bb735534b7df550af291a8"
Nov 01 03:37:07.393869 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:37:07.393811    4400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cff9759bad6097d625173d1c685932d01aeea4f36bb735534b7df550af291a8\": container with ID starting with 6cff9759bad6097d625173d1c685932d01aeea4f36bb735534b7df550af291a8 not found: ID does not exist" containerID="6cff9759bad6097d625173d1c685932d01aeea4f36bb735534b7df550af291a8"
Nov 01 03:37:07.393869 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.393843    4400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cff9759bad6097d625173d1c685932d01aeea4f36bb735534b7df550af291a8"} err="failed to get container status \"6cff9759bad6097d625173d1c685932d01aeea4f36bb735534b7df550af291a8\": rpc error: code = NotFound desc = could not find container \"6cff9759bad6097d625173d1c685932d01aeea4f36bb735534b7df550af291a8\": container with ID starting with 6cff9759bad6097d625173d1c685932d01aeea4f36bb735534b7df550af291a8 not found: ID does not exist"
Nov 01 03:37:07.463853 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.463825    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51938745-c46e-4917-b963-5d1f98b0298d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51938745-c46e-4917-b963-5d1f98b0298d-workload-socket\") pod \"51938745-c46e-4917-b963-5d1f98b0298d\" (UID: \"51938745-c46e-4917-b963-5d1f98b0298d\") "
Nov 01 03:37:07.464019 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.463869    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/204735ae-bff5-4cdc-9df5-f85158677fdc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/204735ae-bff5-4cdc-9df5-f85158677fdc-credential-socket\") pod \"204735ae-bff5-4cdc-9df5-f85158677fdc\" (UID: \"204735ae-bff5-4cdc-9df5-f85158677fdc\") "
Nov 01 03:37:07.464019 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.463896    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51938745-c46e-4917-b963-5d1f98b0298d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51938745-c46e-4917-b963-5d1f98b0298d-istiod-ca-cert\") pod \"51938745-c46e-4917-b963-5d1f98b0298d\" (UID: \"51938745-c46e-4917-b963-5d1f98b0298d\") "
Nov 01 03:37:07.464019 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.463924    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/204735ae-bff5-4cdc-9df5-f85158677fdc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/204735ae-bff5-4cdc-9df5-f85158677fdc-istio-envoy\") pod \"204735ae-bff5-4cdc-9df5-f85158677fdc\" (UID: \"204735ae-bff5-4cdc-9df5-f85158677fdc\") "
Nov 01 03:37:07.464019 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.463963    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/204735ae-bff5-4cdc-9df5-f85158677fdc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/204735ae-bff5-4cdc-9df5-f85158677fdc-istio-podinfo\") pod \"204735ae-bff5-4cdc-9df5-f85158677fdc\" (UID: \"204735ae-bff5-4cdc-9df5-f85158677fdc\") "
Nov 01 03:37:07.464237 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.464056    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/204735ae-bff5-4cdc-9df5-f85158677fdc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/204735ae-bff5-4cdc-9df5-f85158677fdc-workload-socket\") pod \"204735ae-bff5-4cdc-9df5-f85158677fdc\" (UID: \"204735ae-bff5-4cdc-9df5-f85158677fdc\") "
Nov 01 03:37:07.464237 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.464086    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51938745-c46e-4917-b963-5d1f98b0298d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51938745-c46e-4917-b963-5d1f98b0298d-workload-certs\") pod \"51938745-c46e-4917-b963-5d1f98b0298d\" (UID: \"51938745-c46e-4917-b963-5d1f98b0298d\") "
Nov 01 03:37:07.464237 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.464106    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/204735ae-bff5-4cdc-9df5-f85158677fdc-credential-socket" pod "204735ae-bff5-4cdc-9df5-f85158677fdc" (UID: "204735ae-bff5-4cdc-9df5-f85158677fdc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:37:07.464237 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.464114    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51938745-c46e-4917-b963-5d1f98b0298d-workload-socket" pod "51938745-c46e-4917-b963-5d1f98b0298d" (UID: "51938745-c46e-4917-b963-5d1f98b0298d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:37:07.464237 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.464166    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51938745-c46e-4917-b963-5d1f98b0298d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51938745-c46e-4917-b963-5d1f98b0298d-credential-socket\") pod \"51938745-c46e-4917-b963-5d1f98b0298d\" (UID: \"51938745-c46e-4917-b963-5d1f98b0298d\") "
Nov 01 03:37:07.464237 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.464197    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51938745-c46e-4917-b963-5d1f98b0298d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51938745-c46e-4917-b963-5d1f98b0298d-istio-envoy\") pod \"51938745-c46e-4917-b963-5d1f98b0298d\" (UID: \"51938745-c46e-4917-b963-5d1f98b0298d\") "
Nov 01 03:37:07.464863 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.464271    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51938745-c46e-4917-b963-5d1f98b0298d-istiod-ca-cert" pod "51938745-c46e-4917-b963-5d1f98b0298d" (UID: "51938745-c46e-4917-b963-5d1f98b0298d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:37:07.464863 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.464631    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51938745-c46e-4917-b963-5d1f98b0298d-credential-socket" pod "51938745-c46e-4917-b963-5d1f98b0298d" (UID: "51938745-c46e-4917-b963-5d1f98b0298d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:37:07.464863 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.464652    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51938745-c46e-4917-b963-5d1f98b0298d-workload-certs" pod "51938745-c46e-4917-b963-5d1f98b0298d" (UID: "51938745-c46e-4917-b963-5d1f98b0298d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:37:07.464863 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.464671    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/204735ae-bff5-4cdc-9df5-f85158677fdc-workload-socket" pod "204735ae-bff5-4cdc-9df5-f85158677fdc" (UID: "204735ae-bff5-4cdc-9df5-f85158677fdc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:37:07.465397 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.465281    4400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/204735ae-bff5-4cdc-9df5-f85158677fdc-workload-socket\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:37:07.465397 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.465304    4400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51938745-c46e-4917-b963-5d1f98b0298d-workload-certs\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:37:07.465397 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.465317    4400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51938745-c46e-4917-b963-5d1f98b0298d-credential-socket\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:37:07.465397 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.465329    4400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51938745-c46e-4917-b963-5d1f98b0298d-istio-data\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:37:07.465397 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.465341    4400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/204735ae-bff5-4cdc-9df5-f85158677fdc-istiod-ca-cert\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:37:07.465397 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.465353    4400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/204735ae-bff5-4cdc-9df5-f85158677fdc-workload-certs\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:37:07.465397 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.465366    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wb5xw\" (UniqueName: \"kubernetes.io/projected/204735ae-bff5-4cdc-9df5-f85158677fdc-kube-api-access-wb5xw\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:37:07.465397 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.465379    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97r95\" (UniqueName: \"kubernetes.io/projected/51938745-c46e-4917-b963-5d1f98b0298d-kube-api-access-97r95\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:37:07.465397 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.465391    4400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51938745-c46e-4917-b963-5d1f98b0298d-istio-token\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:37:07.465769 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.465403    4400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/204735ae-bff5-4cdc-9df5-f85158677fdc-istio-data\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:37:07.465769 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.465415    4400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51938745-c46e-4917-b963-5d1f98b0298d-istio-podinfo\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:37:07.465769 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.465428    4400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/204735ae-bff5-4cdc-9df5-f85158677fdc-istio-token\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:37:07.465769 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.465441    4400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51938745-c46e-4917-b963-5d1f98b0298d-workload-socket\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:37:07.465769 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.465454    4400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/204735ae-bff5-4cdc-9df5-f85158677fdc-credential-socket\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:37:07.465769 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.465466    4400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51938745-c46e-4917-b963-5d1f98b0298d-istiod-ca-cert\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:37:07.470185 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.470162    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/204735ae-bff5-4cdc-9df5-f85158677fdc-istio-podinfo" pod "204735ae-bff5-4cdc-9df5-f85158677fdc" (UID: "204735ae-bff5-4cdc-9df5-f85158677fdc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 03:37:07.473261 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.473237    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51938745-c46e-4917-b963-5d1f98b0298d-istio-envoy" pod "51938745-c46e-4917-b963-5d1f98b0298d" (UID: "51938745-c46e-4917-b963-5d1f98b0298d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:37:07.476157 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.476138    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-l4xqq"
Nov 01 03:37:07.488498 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.488472    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/204735ae-bff5-4cdc-9df5-f85158677fdc-istio-envoy" pod "204735ae-bff5-4cdc-9df5-f85158677fdc" (UID: "204735ae-bff5-4cdc-9df5-f85158677fdc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:37:07.506475 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.506458    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-9xqhq"
Nov 01 03:37:07.566872 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.566841    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c782468-958d-496b-afec-73b546553a57-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c782468-958d-496b-afec-73b546553a57-istio-envoy\") pod \"6c782468-958d-496b-afec-73b546553a57\" (UID: \"6c782468-958d-496b-afec-73b546553a57\") "
Nov 01 03:37:07.567063 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.566904    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c782468-958d-496b-afec-73b546553a57-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c782468-958d-496b-afec-73b546553a57-istio-data\") pod \"6c782468-958d-496b-afec-73b546553a57\" (UID: \"6c782468-958d-496b-afec-73b546553a57\") "
Nov 01 03:37:07.567063 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.566922    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73ae6733-594c-47d9-89e9-8b92fde0b8f2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73ae6733-594c-47d9-89e9-8b92fde0b8f2-credential-socket\") pod \"73ae6733-594c-47d9-89e9-8b92fde0b8f2\" (UID: \"73ae6733-594c-47d9-89e9-8b92fde0b8f2\") "
Nov 01 03:37:07.567063 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.566946    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c782468-958d-496b-afec-73b546553a57-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c782468-958d-496b-afec-73b546553a57-istiod-ca-cert\") pod \"6c782468-958d-496b-afec-73b546553a57\" (UID: \"6c782468-958d-496b-afec-73b546553a57\") "
Nov 01 03:37:07.567063 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.566973    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73ae6733-594c-47d9-89e9-8b92fde0b8f2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73ae6733-594c-47d9-89e9-8b92fde0b8f2-workload-socket\") pod \"73ae6733-594c-47d9-89e9-8b92fde0b8f2\" (UID: \"73ae6733-594c-47d9-89e9-8b92fde0b8f2\") "
Nov 01 03:37:07.567063 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.567001    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73ae6733-594c-47d9-89e9-8b92fde0b8f2-istio-token\" (UniqueName: \"kubernetes.io/projected/73ae6733-594c-47d9-89e9-8b92fde0b8f2-istio-token\") pod \"73ae6733-594c-47d9-89e9-8b92fde0b8f2\" (UID: \"73ae6733-594c-47d9-89e9-8b92fde0b8f2\") "
Nov 01 03:37:07.567063 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.567028    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c782468-958d-496b-afec-73b546553a57-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c782468-958d-496b-afec-73b546553a57-workload-socket\") pod \"6c782468-958d-496b-afec-73b546553a57\" (UID: \"6c782468-958d-496b-afec-73b546553a57\") "
Nov 01 03:37:07.567063 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.567061    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73ae6733-594c-47d9-89e9-8b92fde0b8f2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73ae6733-594c-47d9-89e9-8b92fde0b8f2-istiod-ca-cert\") pod \"73ae6733-594c-47d9-89e9-8b92fde0b8f2\" (UID: \"73ae6733-594c-47d9-89e9-8b92fde0b8f2\") "
Nov 01 03:37:07.567424 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.567111    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73ae6733-594c-47d9-89e9-8b92fde0b8f2-kube-api-access-w42cj\" (UniqueName: \"kubernetes.io/projected/73ae6733-594c-47d9-89e9-8b92fde0b8f2-kube-api-access-w42cj\") pod \"73ae6733-594c-47d9-89e9-8b92fde0b8f2\" (UID: \"73ae6733-594c-47d9-89e9-8b92fde0b8f2\") "
Nov 01 03:37:07.567424 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.567173    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73ae6733-594c-47d9-89e9-8b92fde0b8f2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73ae6733-594c-47d9-89e9-8b92fde0b8f2-istio-envoy\") pod \"73ae6733-594c-47d9-89e9-8b92fde0b8f2\" (UID: \"73ae6733-594c-47d9-89e9-8b92fde0b8f2\") "
Nov 01 03:37:07.567424 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.567214    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73ae6733-594c-47d9-89e9-8b92fde0b8f2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73ae6733-594c-47d9-89e9-8b92fde0b8f2-istio-data\") pod \"73ae6733-594c-47d9-89e9-8b92fde0b8f2\" (UID: \"73ae6733-594c-47d9-89e9-8b92fde0b8f2\") "
Nov 01 03:37:07.567424 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.567222    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c782468-958d-496b-afec-73b546553a57-istio-data" pod "6c782468-958d-496b-afec-73b546553a57" (UID: "6c782468-958d-496b-afec-73b546553a57"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:37:07.567424 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.567237    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73ae6733-594c-47d9-89e9-8b92fde0b8f2-credential-socket" pod "73ae6733-594c-47d9-89e9-8b92fde0b8f2" (UID: "73ae6733-594c-47d9-89e9-8b92fde0b8f2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:37:07.567424 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.567288    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c782468-958d-496b-afec-73b546553a57-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c782468-958d-496b-afec-73b546553a57-credential-socket\") pod \"6c782468-958d-496b-afec-73b546553a57\" (UID: \"6c782468-958d-496b-afec-73b546553a57\") "
Nov 01 03:37:07.567424 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.567303    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c782468-958d-496b-afec-73b546553a57-workload-socket" pod "6c782468-958d-496b-afec-73b546553a57" (UID: "6c782468-958d-496b-afec-73b546553a57"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:37:07.567424 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.567342    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73ae6733-594c-47d9-89e9-8b92fde0b8f2-workload-socket" pod "73ae6733-594c-47d9-89e9-8b92fde0b8f2" (UID: "73ae6733-594c-47d9-89e9-8b92fde0b8f2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:37:07.567424 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.567348    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c782468-958d-496b-afec-73b546553a57-istio-token\" (UniqueName: \"kubernetes.io/projected/6c782468-958d-496b-afec-73b546553a57-istio-token\") pod \"6c782468-958d-496b-afec-73b546553a57\" (UID: \"6c782468-958d-496b-afec-73b546553a57\") "
Nov 01 03:37:07.567424 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.567380    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c782468-958d-496b-afec-73b546553a57-istiod-ca-cert" pod "6c782468-958d-496b-afec-73b546553a57" (UID: "6c782468-958d-496b-afec-73b546553a57"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:37:07.567424 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.567405    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73ae6733-594c-47d9-89e9-8b92fde0b8f2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73ae6733-594c-47d9-89e9-8b92fde0b8f2-workload-certs\") pod \"73ae6733-594c-47d9-89e9-8b92fde0b8f2\" (UID: \"73ae6733-594c-47d9-89e9-8b92fde0b8f2\") "
Nov 01 03:37:07.568094 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.567491    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6c782468-958d-496b-afec-73b546553a57-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c782468-958d-496b-afec-73b546553a57-istio-podinfo\") pod \"6c782468-958d-496b-afec-73b546553a57\" (UID: \"6c782468-958d-496b-afec-73b546553a57\") "
Nov 01 03:37:07.568094 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.567517    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/73ae6733-594c-47d9-89e9-8b92fde0b8f2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73ae6733-594c-47d9-89e9-8b92fde0b8f2-istio-podinfo\") pod \"73ae6733-594c-47d9-89e9-8b92fde0b8f2\" (UID: \"73ae6733-594c-47d9-89e9-8b92fde0b8f2\") "
Nov 01 03:37:07.568094 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.567573    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73ae6733-594c-47d9-89e9-8b92fde0b8f2-istiod-ca-cert" pod "73ae6733-594c-47d9-89e9-8b92fde0b8f2" (UID: "73ae6733-594c-47d9-89e9-8b92fde0b8f2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:37:07.568094 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.567617    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c782468-958d-496b-afec-73b546553a57-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c782468-958d-496b-afec-73b546553a57-workload-certs\") pod \"6c782468-958d-496b-afec-73b546553a57\" (UID: \"6c782468-958d-496b-afec-73b546553a57\") "
Nov 01 03:37:07.568094 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.567647    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c782468-958d-496b-afec-73b546553a57-kube-api-access-cz2ff\" (UniqueName: \"kubernetes.io/projected/6c782468-958d-496b-afec-73b546553a57-kube-api-access-cz2ff\") pod \"6c782468-958d-496b-afec-73b546553a57\" (UID: \"6c782468-958d-496b-afec-73b546553a57\") "
Nov 01 03:37:07.568094 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.567648    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73ae6733-594c-47d9-89e9-8b92fde0b8f2-istio-data" pod "73ae6733-594c-47d9-89e9-8b92fde0b8f2" (UID: "73ae6733-594c-47d9-89e9-8b92fde0b8f2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:37:07.568094 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.567853    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c782468-958d-496b-afec-73b546553a57-credential-socket" pod "6c782468-958d-496b-afec-73b546553a57" (UID: "6c782468-958d-496b-afec-73b546553a57"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:37:07.568831 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.568759    4400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51938745-c46e-4917-b963-5d1f98b0298d-istio-envoy\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:37:07.568831 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.568781    4400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c782468-958d-496b-afec-73b546553a57-istio-data\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:37:07.568831 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.568799    4400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73ae6733-594c-47d9-89e9-8b92fde0b8f2-credential-socket\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:37:07.568831 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.568812    4400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c782468-958d-496b-afec-73b546553a57-istiod-ca-cert\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:37:07.568831 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.568827    4400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73ae6733-594c-47d9-89e9-8b92fde0b8f2-workload-socket\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:37:07.569169 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.568840    4400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c782468-958d-496b-afec-73b546553a57-workload-socket\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:37:07.569169 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.568855    4400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73ae6733-594c-47d9-89e9-8b92fde0b8f2-istiod-ca-cert\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:37:07.569169 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.568868    4400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73ae6733-594c-47d9-89e9-8b92fde0b8f2-istio-data\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:37:07.569169 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.568881    4400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c782468-958d-496b-afec-73b546553a57-credential-socket\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:37:07.569169 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.568894    4400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/204735ae-bff5-4cdc-9df5-f85158677fdc-istio-envoy\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:37:07.569169 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.568907    4400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/204735ae-bff5-4cdc-9df5-f85158677fdc-istio-podinfo\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:37:07.570137 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.570113    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73ae6733-594c-47d9-89e9-8b92fde0b8f2-kube-api-access-w42cj" pod "73ae6733-594c-47d9-89e9-8b92fde0b8f2" (UID: "73ae6733-594c-47d9-89e9-8b92fde0b8f2"). InnerVolumeSpecName "kube-api-access-w42cj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:37:07.570230 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.570206    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73ae6733-594c-47d9-89e9-8b92fde0b8f2-istio-token" pod "73ae6733-594c-47d9-89e9-8b92fde0b8f2" (UID: "73ae6733-594c-47d9-89e9-8b92fde0b8f2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:37:07.570275 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.570237    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6c782468-958d-496b-afec-73b546553a57-istio-podinfo" pod "6c782468-958d-496b-afec-73b546553a57" (UID: "6c782468-958d-496b-afec-73b546553a57"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 03:37:07.570310 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.570277    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73ae6733-594c-47d9-89e9-8b92fde0b8f2-istio-envoy" pod "73ae6733-594c-47d9-89e9-8b92fde0b8f2" (UID: "73ae6733-594c-47d9-89e9-8b92fde0b8f2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:37:07.570364 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.570342    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c782468-958d-496b-afec-73b546553a57-workload-certs" pod "6c782468-958d-496b-afec-73b546553a57" (UID: "6c782468-958d-496b-afec-73b546553a57"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:37:07.570422 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.570408    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73ae6733-594c-47d9-89e9-8b92fde0b8f2-workload-certs" pod "73ae6733-594c-47d9-89e9-8b92fde0b8f2" (UID: "73ae6733-594c-47d9-89e9-8b92fde0b8f2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:37:07.570468 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.570426    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c782468-958d-496b-afec-73b546553a57-istio-envoy" pod "6c782468-958d-496b-afec-73b546553a57" (UID: "6c782468-958d-496b-afec-73b546553a57"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:37:07.570594 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.570577    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c782468-958d-496b-afec-73b546553a57-istio-token" pod "6c782468-958d-496b-afec-73b546553a57" (UID: "6c782468-958d-496b-afec-73b546553a57"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:37:07.571055 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.571025    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/73ae6733-594c-47d9-89e9-8b92fde0b8f2-istio-podinfo" pod "73ae6733-594c-47d9-89e9-8b92fde0b8f2" (UID: "73ae6733-594c-47d9-89e9-8b92fde0b8f2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 03:37:07.571663 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.571648    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c782468-958d-496b-afec-73b546553a57-kube-api-access-cz2ff" pod "6c782468-958d-496b-afec-73b546553a57" (UID: "6c782468-958d-496b-afec-73b546553a57"). InnerVolumeSpecName "kube-api-access-cz2ff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:37:07.616222 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.616186    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3532ce0-59fa-4822-bff3-2d0d356a3f13" path="/var/lib/kubelet/pods/b3532ce0-59fa-4822-bff3-2d0d356a3f13/volumes"
Nov 01 03:37:07.670590 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.670520    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w42cj\" (UniqueName: \"kubernetes.io/projected/73ae6733-594c-47d9-89e9-8b92fde0b8f2-kube-api-access-w42cj\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:37:07.670590 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.670553    4400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73ae6733-594c-47d9-89e9-8b92fde0b8f2-istio-envoy\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:37:07.670590 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.670568    4400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c782468-958d-496b-afec-73b546553a57-istio-token\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:37:07.670590 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.670581    4400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73ae6733-594c-47d9-89e9-8b92fde0b8f2-workload-certs\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:37:07.670899 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.670594    4400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c782468-958d-496b-afec-73b546553a57-istio-podinfo\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:37:07.670899 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.670607    4400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73ae6733-594c-47d9-89e9-8b92fde0b8f2-istio-podinfo\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:37:07.670899 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.670620    4400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c782468-958d-496b-afec-73b546553a57-workload-certs\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:37:07.670899 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.670635    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cz2ff\" (UniqueName: \"kubernetes.io/projected/6c782468-958d-496b-afec-73b546553a57-kube-api-access-cz2ff\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:37:07.670899 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.670651    4400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c782468-958d-496b-afec-73b546553a57-istio-envoy\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:37:07.670899 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:07.670664    4400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73ae6733-594c-47d9-89e9-8b92fde0b8f2-istio-token\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:37:08.043280 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:08.043204    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-9qpm8"]
Nov 01 03:37:08.051428 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:08.051401    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-9qpm8"]
Nov 01 03:37:08.174930 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:08.174898    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hf2dc"]
Nov 01 03:37:08.181362 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:08.181333    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hf2dc"]
Nov 01 03:37:08.368506 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:08.368479    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-9xqhq"
Nov 01 03:37:08.368966 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:08.368507    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-9xqhq" event={"ID":"6c782468-958d-496b-afec-73b546553a57","Type":"ContainerDied","Data":"f88d191b9d8e216a4b881510d23731cd3ccc5cac212696d55e0720eb16a8eb6a"}
Nov 01 03:37:08.368966 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:08.368579    4400 scope.go:122] "RemoveContainer" containerID="12cee91983923b762f742ec2bd784e664907973dbfe2b4f377a7298d7a67a855"
Nov 01 03:37:08.370077 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:08.370056    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-l4xqq"
Nov 01 03:37:08.370162 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:08.370078    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-l4xqq" event={"ID":"73ae6733-594c-47d9-89e9-8b92fde0b8f2","Type":"ContainerDied","Data":"55ed45e1c6ffe60ef8329cf602f92e4d61932e535b3b379490b54bff7c9c3be8"}
Nov 01 03:37:08.378848 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:08.378834    4400 scope.go:122] "RemoveContainer" containerID="9bc3adca49fba1b1c0c649e84f537f42d45a25e8ad129ecb62c4d63e2c63292e"
Nov 01 03:37:08.565840 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:08.565814    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-9xqhq"]
Nov 01 03:37:08.569578 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:08.569560    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-9xqhq"]
Nov 01 03:37:08.694830 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:08.694774    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-l4xqq"]
Nov 01 03:37:08.700144 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:08.700116    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-l4xqq"]
Nov 01 03:37:09.615183 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:09.615150    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="204735ae-bff5-4cdc-9df5-f85158677fdc" path="/var/lib/kubelet/pods/204735ae-bff5-4cdc-9df5-f85158677fdc/volumes"
Nov 01 03:37:09.615595 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:09.615582    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51938745-c46e-4917-b963-5d1f98b0298d" path="/var/lib/kubelet/pods/51938745-c46e-4917-b963-5d1f98b0298d/volumes"
Nov 01 03:37:09.616037 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:09.616022    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c782468-958d-496b-afec-73b546553a57" path="/var/lib/kubelet/pods/6c782468-958d-496b-afec-73b546553a57/volumes"
Nov 01 03:37:09.616798 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:09.616781    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73ae6733-594c-47d9-89e9-8b92fde0b8f2" path="/var/lib/kubelet/pods/73ae6733-594c-47d9-89e9-8b92fde0b8f2/volumes"
Nov 01 03:37:16.098801 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:16.096916    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7d7329-fcbf4bdc7-r8q86"]
Nov 01 03:37:16.098801 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:16.098035    4400 cpu_manager.go:414] "containerMap: removing container" podUID="51938745-c46e-4917-b963-5d1f98b0298d" containerName="istio-proxy"
Nov 01 03:37:16.098801 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:16.098057    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51938745-c46e-4917-b963-5d1f98b0298d" containerName="istio-proxy"
Nov 01 03:37:16.098801 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:16.098088    4400 cpu_manager.go:414] "containerMap: removing container" podUID="b3532ce0-59fa-4822-bff3-2d0d356a3f13" containerName="istio-proxy"
Nov 01 03:37:16.098801 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:16.098097    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3532ce0-59fa-4822-bff3-2d0d356a3f13" containerName="istio-proxy"
Nov 01 03:37:16.098801 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:16.098112    4400 cpu_manager.go:414] "containerMap: removing container" podUID="73ae6733-594c-47d9-89e9-8b92fde0b8f2" containerName="istio-proxy"
Nov 01 03:37:16.098801 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:16.098120    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73ae6733-594c-47d9-89e9-8b92fde0b8f2" containerName="istio-proxy"
Nov 01 03:37:16.098801 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:16.098137    4400 cpu_manager.go:414] "containerMap: removing container" podUID="204735ae-bff5-4cdc-9df5-f85158677fdc" containerName="istio-proxy"
Nov 01 03:37:16.098801 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:16.098145    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="204735ae-bff5-4cdc-9df5-f85158677fdc" containerName="istio-proxy"
Nov 01 03:37:16.098801 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:16.098171    4400 cpu_manager.go:414] "containerMap: removing container" podUID="6c782468-958d-496b-afec-73b546553a57" containerName="istio-proxy"
Nov 01 03:37:16.098801 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:16.098179    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c782468-958d-496b-afec-73b546553a57" containerName="istio-proxy"
Nov 01 03:37:16.098801 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:16.098351    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c782468-958d-496b-afec-73b546553a57" containerName="istio-proxy"
Nov 01 03:37:16.098801 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:16.098373    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="51938745-c46e-4917-b963-5d1f98b0298d" containerName="istio-proxy"
Nov 01 03:37:16.098801 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:16.098384    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3532ce0-59fa-4822-bff3-2d0d356a3f13" containerName="istio-proxy"
Nov 01 03:37:16.098801 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:16.098395    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="73ae6733-594c-47d9-89e9-8b92fde0b8f2" containerName="istio-proxy"
Nov 01 03:37:16.098801 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:16.098407    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="204735ae-bff5-4cdc-9df5-f85158677fdc" containerName="istio-proxy"
Nov 01 03:37:16.116983 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:16.116953    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-7d7329-fcbf4bdc7-r8q86"
Nov 01 03:37:16.173289 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:16.168957    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7d7329-fcbf4bdc7-r8q86"]
Nov 01 03:37:16.203018 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:16.202988    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-7d7329\""
Nov 01 03:37:16.203352 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:16.203338    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-7d7329-dockercfg-bkh86\""
Nov 01 03:37:16.206483 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:16.204250    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb66c491-eeb8-40f2-b553-669555c43b95-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-7d7329-fcbf4bdc7-r8q86\" (UID: \"cb66c491-eeb8-40f2-b553-669555c43b95\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7d7329-fcbf4bdc7-r8q86"
Nov 01 03:37:16.206983 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:16.206917    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9mjn\" (UniqueName: \"kubernetes.io/projected/cb66c491-eeb8-40f2-b553-669555c43b95-kube-api-access-q9mjn\") pod \"payload-processing-e2e-inf-a-7d7329-fcbf4bdc7-r8q86\" (UID: \"cb66c491-eeb8-40f2-b553-669555c43b95\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7d7329-fcbf4bdc7-r8q86"
Nov 01 03:37:16.303722 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:16.282426    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7d7329-87fbf9b5f-chwdx"]
Nov 01 03:37:16.303722 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:16.291603    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7d7329-87fbf9b5f-chwdx"
Nov 01 03:37:16.309107 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:16.308467    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb66c491-eeb8-40f2-b553-669555c43b95-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-7d7329-fcbf4bdc7-r8q86\" (UID: \"cb66c491-eeb8-40f2-b553-669555c43b95\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7d7329-fcbf4bdc7-r8q86"
Nov 01 03:37:16.309107 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:16.308746    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9mjn\" (UniqueName: \"kubernetes.io/projected/cb66c491-eeb8-40f2-b553-669555c43b95-kube-api-access-q9mjn\") pod \"payload-processing-e2e-inf-a-7d7329-fcbf4bdc7-r8q86\" (UID: \"cb66c491-eeb8-40f2-b553-669555c43b95\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7d7329-fcbf4bdc7-r8q86"
Nov 01 03:37:16.309654 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:16.309631    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb66c491-eeb8-40f2-b553-669555c43b95-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-7d7329-fcbf4bdc7-r8q86\" (UID: \"cb66c491-eeb8-40f2-b553-669555c43b95\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7d7329-fcbf4bdc7-r8q86"
Nov 01 03:37:16.342339 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:16.342316    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7d7329-87fbf9b5f-chwdx"]
Nov 01 03:37:16.373614 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:16.373530    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9mjn\" (UniqueName: \"kubernetes.io/projected/cb66c491-eeb8-40f2-b553-669555c43b95-kube-api-access-q9mjn\") pod \"payload-processing-e2e-inf-a-7d7329-fcbf4bdc7-r8q86\" (UID: \"cb66c491-eeb8-40f2-b553-669555c43b95\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7d7329-fcbf4bdc7-r8q86"
Nov 01 03:37:16.453785 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:16.436702    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwfnh\" (UniqueName: \"kubernetes.io/projected/0a527c89-9499-4a79-8b3f-176e206c3cde-kube-api-access-kwfnh\") pod \"payload-pre-processing-e2e-inf-a-7d7329-87fbf9b5f-chwdx\" (UID: \"0a527c89-9499-4a79-8b3f-176e206c3cde\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7d7329-87fbf9b5f-chwdx"
Nov 01 03:37:16.453785 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:16.436905    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a527c89-9499-4a79-8b3f-176e206c3cde-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-7d7329-87fbf9b5f-chwdx\" (UID: \"0a527c89-9499-4a79-8b3f-176e206c3cde\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7d7329-87fbf9b5f-chwdx"
Nov 01 03:37:16.520552 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:16.505906    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-7d7329-fcbf4bdc7-r8q86"
Nov 01 03:37:16.541713 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:16.538703    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a527c89-9499-4a79-8b3f-176e206c3cde-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-7d7329-87fbf9b5f-chwdx\" (UID: \"0a527c89-9499-4a79-8b3f-176e206c3cde\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7d7329-87fbf9b5f-chwdx"
Nov 01 03:37:16.541713 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:16.538942    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwfnh\" (UniqueName: \"kubernetes.io/projected/0a527c89-9499-4a79-8b3f-176e206c3cde-kube-api-access-kwfnh\") pod \"payload-pre-processing-e2e-inf-a-7d7329-87fbf9b5f-chwdx\" (UID: \"0a527c89-9499-4a79-8b3f-176e206c3cde\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7d7329-87fbf9b5f-chwdx"
Nov 01 03:37:16.555712 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:16.543214    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a527c89-9499-4a79-8b3f-176e206c3cde-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-7d7329-87fbf9b5f-chwdx\" (UID: \"0a527c89-9499-4a79-8b3f-176e206c3cde\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7d7329-87fbf9b5f-chwdx"
Nov 01 03:37:16.621126 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:16.621096    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwfnh\" (UniqueName: \"kubernetes.io/projected/0a527c89-9499-4a79-8b3f-176e206c3cde-kube-api-access-kwfnh\") pod \"payload-pre-processing-e2e-inf-a-7d7329-87fbf9b5f-chwdx\" (UID: \"0a527c89-9499-4a79-8b3f-176e206c3cde\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7d7329-87fbf9b5f-chwdx"
Nov 01 03:37:16.915223 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:16.915152    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7d7329-87fbf9b5f-chwdx"
Nov 01 03:37:16.959230 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:37:16.959197    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb66c491_eeb8_40f2_b553_669555c43b95.slice/crio-59883414818ab4cb893541edb1740923af7681639951a882fde2e2bbda9e870c WatchSource:0}: Error finding container 59883414818ab4cb893541edb1740923af7681639951a882fde2e2bbda9e870c: Status 404 returned error can't find the container with id 59883414818ab4cb893541edb1740923af7681639951a882fde2e2bbda9e870c
Nov 01 03:37:16.959940 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:16.959709    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7d7329-fcbf4bdc7-r8q86"]
Nov 01 03:37:17.334773 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:37:17.334747    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a527c89_9499_4a79_8b3f_176e206c3cde.slice/crio-ecdd49c4048a5bca8905c712d210d74cb778582efe17d0c6379ecb2f3e69188b WatchSource:0}: Error finding container ecdd49c4048a5bca8905c712d210d74cb778582efe17d0c6379ecb2f3e69188b: Status 404 returned error can't find the container with id ecdd49c4048a5bca8905c712d210d74cb778582efe17d0c6379ecb2f3e69188b
Nov 01 03:37:17.336151 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:17.336122    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7d7329-87fbf9b5f-chwdx"]
Nov 01 03:37:17.565551 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:17.565510    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7d7329-87fbf9b5f-chwdx" event={"ID":"0a527c89-9499-4a79-8b3f-176e206c3cde","Type":"ContainerStarted","Data":"4be5291c7f7cd4bc8ecff93420d13605b0a0e60bc7be86519908a7c536f7dc29"}
Nov 01 03:37:17.565760 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:17.565715    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7d7329-87fbf9b5f-chwdx"
Nov 01 03:37:17.565827 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:17.565760    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7d7329-87fbf9b5f-chwdx" event={"ID":"0a527c89-9499-4a79-8b3f-176e206c3cde","Type":"ContainerStarted","Data":"ecdd49c4048a5bca8905c712d210d74cb778582efe17d0c6379ecb2f3e69188b"}
Nov 01 03:37:17.567381 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:17.567361    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-7d7329-fcbf4bdc7-r8q86" event={"ID":"cb66c491-eeb8-40f2-b553-669555c43b95","Type":"ContainerStarted","Data":"29004d2109dab3557c825638be974c869cdd800fd635d4f11e590241512d834b"}
Nov 01 03:37:17.567492 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:17.567393    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-7d7329-fcbf4bdc7-r8q86" event={"ID":"cb66c491-eeb8-40f2-b553-669555c43b95","Type":"ContainerStarted","Data":"59883414818ab4cb893541edb1740923af7681639951a882fde2e2bbda9e870c"}
Nov 01 03:37:17.567492 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:17.567467    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-7d7329-fcbf4bdc7-r8q86"
Nov 01 03:37:17.701769 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:17.699220    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7d7329-87fbf9b5f-chwdx" podStartSLOduration=1.6992026569999998 podStartE2EDuration="1.699202657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:37:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:37:17.695095938 +0000 UTC m=+1844.209741799" watchObservedRunningTime="2026-11-01 03:37:17.699202657 +0000 UTC m=+1844.213848521"
Nov 01 03:37:17.825675 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:17.825623    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-7d7329-fcbf4bdc7-r8q86" podStartSLOduration=1.825609561 podStartE2EDuration="1.825609561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:37:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:37:17.824200299 +0000 UTC m=+1844.338846159" watchObservedRunningTime="2026-11-01 03:37:17.825609561 +0000 UTC m=+1844.340255422"
Nov 01 03:37:28.578029 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:28.577907    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-7d7329-fcbf4bdc7-r8q86"
Nov 01 03:37:28.578029 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:28.577980    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7d7329-87fbf9b5f-chwdx"
Nov 01 03:37:31.730330 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:37:31.730296    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:37:50.649041 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:50.649009    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-7cd000-795dd66c4d-l77dz"]
Nov 01 03:37:50.654704 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:50.654666    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-7cd000-795dd66c4d-l77dz"
Nov 01 03:37:50.709795 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:50.709773    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w6-7cd000-serving-cert\""
Nov 01 03:37:50.754725 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:50.754696    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-7cd000-795dd66c4d-l77dz"]
Nov 01 03:37:50.780123 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:50.780098    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2550fb6-41b8-4237-b186-237684022a2a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-7cd000-795dd66c4d-l77dz\" (UID: \"d2550fb6-41b8-4237-b186-237684022a2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-7cd000-795dd66c4d-l77dz"
Nov 01 03:37:50.780231 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:50.780128    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gm7k\" (UniqueName: \"kubernetes.io/projected/d2550fb6-41b8-4237-b186-237684022a2a-kube-api-access-9gm7k\") pod \"maas-api-e2e-shared-a-w6-7cd000-795dd66c4d-l77dz\" (UID: \"d2550fb6-41b8-4237-b186-237684022a2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-7cd000-795dd66c4d-l77dz"
Nov 01 03:37:50.780269 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:50.780246    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2550fb6-41b8-4237-b186-237684022a2a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-7cd000-795dd66c4d-l77dz\" (UID: \"d2550fb6-41b8-4237-b186-237684022a2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-7cd000-795dd66c4d-l77dz"
Nov 01 03:37:50.780302 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:50.780280    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2550fb6-41b8-4237-b186-237684022a2a-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-7cd000-795dd66c4d-l77dz\" (UID: \"d2550fb6-41b8-4237-b186-237684022a2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-7cd000-795dd66c4d-l77dz"
Nov 01 03:37:50.902924 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:50.902836    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2550fb6-41b8-4237-b186-237684022a2a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-7cd000-795dd66c4d-l77dz\" (UID: \"d2550fb6-41b8-4237-b186-237684022a2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-7cd000-795dd66c4d-l77dz"
Nov 01 03:37:50.903233 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:50.903202    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gm7k\" (UniqueName: \"kubernetes.io/projected/d2550fb6-41b8-4237-b186-237684022a2a-kube-api-access-9gm7k\") pod \"maas-api-e2e-shared-a-w6-7cd000-795dd66c4d-l77dz\" (UID: \"d2550fb6-41b8-4237-b186-237684022a2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-7cd000-795dd66c4d-l77dz"
Nov 01 03:37:50.903488 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:50.903469    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2550fb6-41b8-4237-b186-237684022a2a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-7cd000-795dd66c4d-l77dz\" (UID: \"d2550fb6-41b8-4237-b186-237684022a2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-7cd000-795dd66c4d-l77dz"
Nov 01 03:37:50.903613 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:50.903600    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2550fb6-41b8-4237-b186-237684022a2a-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-7cd000-795dd66c4d-l77dz\" (UID: \"d2550fb6-41b8-4237-b186-237684022a2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-7cd000-795dd66c4d-l77dz"
Nov 01 03:37:50.914827 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:50.905457    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2550fb6-41b8-4237-b186-237684022a2a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-7cd000-795dd66c4d-l77dz\" (UID: \"d2550fb6-41b8-4237-b186-237684022a2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-7cd000-795dd66c4d-l77dz"
Nov 01 03:37:50.915188 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:37:50.915160    4400 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-7cd000-serving-cert: secret "maas-api-e2e-shared-a-w6-7cd000-serving-cert" not found
Nov 01 03:37:50.915367 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:37:50.915354    4400 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2550fb6-41b8-4237-b186-237684022a2a-maas-api-tls podName:d2550fb6-41b8-4237-b186-237684022a2a nodeName:}" failed. No retries permitted until 2026-11-01 03:37:51.415332081 +0000 UTC m=+1877.929977934 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d2550fb6-41b8-4237-b186-237684022a2a-maas-api-tls") pod "maas-api-e2e-shared-a-w6-7cd000-795dd66c4d-l77dz" (UID: "d2550fb6-41b8-4237-b186-237684022a2a") : secret "maas-api-e2e-shared-a-w6-7cd000-serving-cert" not found
Nov 01 03:37:50.921139 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:50.921112    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2550fb6-41b8-4237-b186-237684022a2a-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-7cd000-795dd66c4d-l77dz\" (UID: \"d2550fb6-41b8-4237-b186-237684022a2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-7cd000-795dd66c4d-l77dz"
Nov 01 03:37:50.969875 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:50.969844    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gm7k\" (UniqueName: \"kubernetes.io/projected/d2550fb6-41b8-4237-b186-237684022a2a-kube-api-access-9gm7k\") pod \"maas-api-e2e-shared-a-w6-7cd000-795dd66c4d-l77dz\" (UID: \"d2550fb6-41b8-4237-b186-237684022a2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-7cd000-795dd66c4d-l77dz"
Nov 01 03:37:51.515856 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:51.515826    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2550fb6-41b8-4237-b186-237684022a2a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-7cd000-795dd66c4d-l77dz\" (UID: \"d2550fb6-41b8-4237-b186-237684022a2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-7cd000-795dd66c4d-l77dz"
Nov 01 03:37:51.518092 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:51.518073    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2550fb6-41b8-4237-b186-237684022a2a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-7cd000-795dd66c4d-l77dz\" (UID: \"d2550fb6-41b8-4237-b186-237684022a2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-7cd000-795dd66c4d-l77dz"
Nov 01 03:37:51.568195 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:51.568173    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-7cd000-795dd66c4d-l77dz"
Nov 01 03:37:51.956500 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:37:51.956464    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2550fb6_41b8_4237_b186_237684022a2a.slice/crio-92313d051f53f4b188a03d4fa2fb3a0e36c711cdb646bff46087d4a76417531e WatchSource:0}: Error finding container 92313d051f53f4b188a03d4fa2fb3a0e36c711cdb646bff46087d4a76417531e: Status 404 returned error can't find the container with id 92313d051f53f4b188a03d4fa2fb3a0e36c711cdb646bff46087d4a76417531e
Nov 01 03:37:51.956997 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:51.956974    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-7cd000-795dd66c4d-l77dz"]
Nov 01 03:37:52.821969 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:52.821931    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-7cd000-795dd66c4d-l77dz" event={"ID":"d2550fb6-41b8-4237-b186-237684022a2a","Type":"ContainerStarted","Data":"92313d051f53f4b188a03d4fa2fb3a0e36c711cdb646bff46087d4a76417531e"}
Nov 01 03:37:53.898746 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:53.898710    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-7cd000-795dd66c4d-l77dz" event={"ID":"d2550fb6-41b8-4237-b186-237684022a2a","Type":"ContainerStarted","Data":"e69c737a5c1aa8d30d17e262e0bc9134c8d8d72c805cdc0ff43b1fbe82a9d7cf"}
Nov 01 03:37:53.899175 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:53.898926    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-7cd000-795dd66c4d-l77dz"
Nov 01 03:37:53.964746 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:53.964718    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 03:37:54.024249 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:54.024206    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-7cd000-795dd66c4d-l77dz" podStartSLOduration=3.124887269 podStartE2EDuration="4.024192968s" totalImagesPullingTime="899.305699ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:37:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:37:54.022107153 +0000 UTC m=+1880.536753015" watchObservedRunningTime="2026-11-01 03:37:54.024192968 +0000 UTC m=+1880.538838828"
Nov 01 03:37:59.911474 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:37:59.911446    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-7cd000-795dd66c4d-l77dz"
Nov 01 03:38:00.520389 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:00.520359    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-438641-856ccc5949-tcnlc"]
Nov 01 03:38:00.524759 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:00.524738    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-438641-856ccc5949-tcnlc"
Nov 01 03:38:00.579518 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:00.579497    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-438641-serving-cert\""
Nov 01 03:38:00.584134 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:00.584116    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-438641-856ccc5949-tcnlc"]
Nov 01 03:38:00.610849 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:00.610826    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bdca032-ad0e-4346-8543-6eba340faf44-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-438641-856ccc5949-tcnlc\" (UID: \"6bdca032-ad0e-4346-8543-6eba340faf44\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-438641-856ccc5949-tcnlc"
Nov 01 03:38:00.610939 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:00.610860    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bdca032-ad0e-4346-8543-6eba340faf44-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-438641-856ccc5949-tcnlc\" (UID: \"6bdca032-ad0e-4346-8543-6eba340faf44\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-438641-856ccc5949-tcnlc"
Nov 01 03:38:00.611008 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:00.610995    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bdca032-ad0e-4346-8543-6eba340faf44-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-438641-856ccc5949-tcnlc\" (UID: \"6bdca032-ad0e-4346-8543-6eba340faf44\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-438641-856ccc5949-tcnlc"
Nov 01 03:38:00.611083 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:00.611068    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tmdd\" (UniqueName: \"kubernetes.io/projected/6bdca032-ad0e-4346-8543-6eba340faf44-kube-api-access-6tmdd\") pod \"maas-api-e2e-shared-b-w0-438641-856ccc5949-tcnlc\" (UID: \"6bdca032-ad0e-4346-8543-6eba340faf44\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-438641-856ccc5949-tcnlc"
Nov 01 03:38:00.711807 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:00.711783    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bdca032-ad0e-4346-8543-6eba340faf44-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-438641-856ccc5949-tcnlc\" (UID: \"6bdca032-ad0e-4346-8543-6eba340faf44\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-438641-856ccc5949-tcnlc"
Nov 01 03:38:00.711922 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:00.711821    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bdca032-ad0e-4346-8543-6eba340faf44-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-438641-856ccc5949-tcnlc\" (UID: \"6bdca032-ad0e-4346-8543-6eba340faf44\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-438641-856ccc5949-tcnlc"
Nov 01 03:38:00.711961 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:38:00.711932    4400 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-438641-serving-cert: secret "maas-api-e2e-shared-b-w0-438641-serving-cert" not found
Nov 01 03:38:00.712005 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:38:00.711994    4400 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bdca032-ad0e-4346-8543-6eba340faf44-maas-api-tls podName:6bdca032-ad0e-4346-8543-6eba340faf44 nodeName:}" failed. No retries permitted until 2026-11-01 03:38:01.211972809 +0000 UTC m=+1887.726618661 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6bdca032-ad0e-4346-8543-6eba340faf44-maas-api-tls") pod "maas-api-e2e-shared-b-w0-438641-856ccc5949-tcnlc" (UID: "6bdca032-ad0e-4346-8543-6eba340faf44") : secret "maas-api-e2e-shared-b-w0-438641-serving-cert" not found
Nov 01 03:38:00.712058 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:00.712035    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bdca032-ad0e-4346-8543-6eba340faf44-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-438641-856ccc5949-tcnlc\" (UID: \"6bdca032-ad0e-4346-8543-6eba340faf44\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-438641-856ccc5949-tcnlc"
Nov 01 03:38:00.712204 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:00.712181    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tmdd\" (UniqueName: \"kubernetes.io/projected/6bdca032-ad0e-4346-8543-6eba340faf44-kube-api-access-6tmdd\") pod \"maas-api-e2e-shared-b-w0-438641-856ccc5949-tcnlc\" (UID: \"6bdca032-ad0e-4346-8543-6eba340faf44\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-438641-856ccc5949-tcnlc"
Nov 01 03:38:00.712554 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:00.712533    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bdca032-ad0e-4346-8543-6eba340faf44-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-438641-856ccc5949-tcnlc\" (UID: \"6bdca032-ad0e-4346-8543-6eba340faf44\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-438641-856ccc5949-tcnlc"
Nov 01 03:38:00.714549 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:00.714525    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bdca032-ad0e-4346-8543-6eba340faf44-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-438641-856ccc5949-tcnlc\" (UID: \"6bdca032-ad0e-4346-8543-6eba340faf44\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-438641-856ccc5949-tcnlc"
Nov 01 03:38:00.793666 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:00.793550    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tmdd\" (UniqueName: \"kubernetes.io/projected/6bdca032-ad0e-4346-8543-6eba340faf44-kube-api-access-6tmdd\") pod \"maas-api-e2e-shared-b-w0-438641-856ccc5949-tcnlc\" (UID: \"6bdca032-ad0e-4346-8543-6eba340faf44\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-438641-856ccc5949-tcnlc"
Nov 01 03:38:01.237511 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:01.237465    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bdca032-ad0e-4346-8543-6eba340faf44-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-438641-856ccc5949-tcnlc\" (UID: \"6bdca032-ad0e-4346-8543-6eba340faf44\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-438641-856ccc5949-tcnlc"
Nov 01 03:38:01.240002 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:01.239975    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bdca032-ad0e-4346-8543-6eba340faf44-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-438641-856ccc5949-tcnlc\" (UID: \"6bdca032-ad0e-4346-8543-6eba340faf44\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-438641-856ccc5949-tcnlc"
Nov 01 03:38:01.438983 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:01.438960    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-438641-856ccc5949-tcnlc"
Nov 01 03:38:01.730089 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:38:01.730056    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:38:01.825291 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:38:01.825264    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bdca032_ad0e_4346_8543_6eba340faf44.slice/crio-d2bc6b5762622a412a39cfc797d82b79c69b13214be8b82034467bdab48208bc WatchSource:0}: Error finding container d2bc6b5762622a412a39cfc797d82b79c69b13214be8b82034467bdab48208bc: Status 404 returned error can't find the container with id d2bc6b5762622a412a39cfc797d82b79c69b13214be8b82034467bdab48208bc
Nov 01 03:38:01.825401 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:01.825317    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-438641-856ccc5949-tcnlc"]
Nov 01 03:38:01.983049 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:01.983016    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-438641-856ccc5949-tcnlc" event={"ID":"6bdca032-ad0e-4346-8543-6eba340faf44","Type":"ContainerStarted","Data":"d2bc6b5762622a412a39cfc797d82b79c69b13214be8b82034467bdab48208bc"}
Nov 01 03:38:02.991596 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:02.991562    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-438641-856ccc5949-tcnlc" event={"ID":"6bdca032-ad0e-4346-8543-6eba340faf44","Type":"ContainerStarted","Data":"afdecec6d3ca60021733802b774b97804c6f2f66a005c4f1827435b1cd5b05ec"}
Nov 01 03:38:02.992066 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:02.991621    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-438641-856ccc5949-tcnlc"
Nov 01 03:38:03.131302 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:03.131254    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-438641-856ccc5949-tcnlc" podStartSLOduration=2.181751505 podStartE2EDuration="3.131241642s" totalImagesPullingTime="949.490137ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:38:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:38:03.126843621 +0000 UTC m=+1889.641489483" watchObservedRunningTime="2026-11-01 03:38:03.131241642 +0000 UTC m=+1889.645887502"
Nov 01 03:38:05.407829 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:05.407798    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-634332-854c684b9f-8qg4v"]
Nov 01 03:38:05.412439 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:05.412422    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-634332-854c684b9f-8qg4v"
Nov 01 03:38:05.466521 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:05.466496    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-634332-serving-cert\""
Nov 01 03:38:05.470902 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:05.470884    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-634332-854c684b9f-8qg4v"]
Nov 01 03:38:05.517194 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:05.517172    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b174e47-5aad-474b-a5cf-4fe09a417fe3-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-634332-854c684b9f-8qg4v\" (UID: \"9b174e47-5aad-474b-a5cf-4fe09a417fe3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-634332-854c684b9f-8qg4v"
Nov 01 03:38:05.517307 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:05.517226    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bk44\" (UniqueName: \"kubernetes.io/projected/9b174e47-5aad-474b-a5cf-4fe09a417fe3-kube-api-access-8bk44\") pod \"maas-api-e2e-shared-b-w1-634332-854c684b9f-8qg4v\" (UID: \"9b174e47-5aad-474b-a5cf-4fe09a417fe3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-634332-854c684b9f-8qg4v"
Nov 01 03:38:05.517307 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:05.517247    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b174e47-5aad-474b-a5cf-4fe09a417fe3-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-634332-854c684b9f-8qg4v\" (UID: \"9b174e47-5aad-474b-a5cf-4fe09a417fe3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-634332-854c684b9f-8qg4v"
Nov 01 03:38:05.517490 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:05.517334    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b174e47-5aad-474b-a5cf-4fe09a417fe3-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-634332-854c684b9f-8qg4v\" (UID: \"9b174e47-5aad-474b-a5cf-4fe09a417fe3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-634332-854c684b9f-8qg4v"
Nov 01 03:38:05.618876 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:05.618851    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b174e47-5aad-474b-a5cf-4fe09a417fe3-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-634332-854c684b9f-8qg4v\" (UID: \"9b174e47-5aad-474b-a5cf-4fe09a417fe3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-634332-854c684b9f-8qg4v"
Nov 01 03:38:05.619022 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:05.618936    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bk44\" (UniqueName: \"kubernetes.io/projected/9b174e47-5aad-474b-a5cf-4fe09a417fe3-kube-api-access-8bk44\") pod \"maas-api-e2e-shared-b-w1-634332-854c684b9f-8qg4v\" (UID: \"9b174e47-5aad-474b-a5cf-4fe09a417fe3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-634332-854c684b9f-8qg4v"
Nov 01 03:38:05.619022 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:05.618965    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b174e47-5aad-474b-a5cf-4fe09a417fe3-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-634332-854c684b9f-8qg4v\" (UID: \"9b174e47-5aad-474b-a5cf-4fe09a417fe3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-634332-854c684b9f-8qg4v"
Nov 01 03:38:05.619022 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:05.618988    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b174e47-5aad-474b-a5cf-4fe09a417fe3-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-634332-854c684b9f-8qg4v\" (UID: \"9b174e47-5aad-474b-a5cf-4fe09a417fe3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-634332-854c684b9f-8qg4v"
Nov 01 03:38:05.619185 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:38:05.619076    4400 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-634332-serving-cert: secret "maas-api-e2e-shared-b-w1-634332-serving-cert" not found
Nov 01 03:38:05.619185 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:38:05.619132    4400 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b174e47-5aad-474b-a5cf-4fe09a417fe3-maas-api-tls podName:9b174e47-5aad-474b-a5cf-4fe09a417fe3 nodeName:}" failed. No retries permitted until 2026-11-01 03:38:06.119119079 +0000 UTC m=+1892.633764923 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9b174e47-5aad-474b-a5cf-4fe09a417fe3-maas-api-tls") pod "maas-api-e2e-shared-b-w1-634332-854c684b9f-8qg4v" (UID: "9b174e47-5aad-474b-a5cf-4fe09a417fe3") : secret "maas-api-e2e-shared-b-w1-634332-serving-cert" not found
Nov 01 03:38:05.619610 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:05.619581    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b174e47-5aad-474b-a5cf-4fe09a417fe3-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-634332-854c684b9f-8qg4v\" (UID: \"9b174e47-5aad-474b-a5cf-4fe09a417fe3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-634332-854c684b9f-8qg4v"
Nov 01 03:38:05.621338 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:05.621320    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b174e47-5aad-474b-a5cf-4fe09a417fe3-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-634332-854c684b9f-8qg4v\" (UID: \"9b174e47-5aad-474b-a5cf-4fe09a417fe3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-634332-854c684b9f-8qg4v"
Nov 01 03:38:05.695369 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:05.695296    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bk44\" (UniqueName: \"kubernetes.io/projected/9b174e47-5aad-474b-a5cf-4fe09a417fe3-kube-api-access-8bk44\") pod \"maas-api-e2e-shared-b-w1-634332-854c684b9f-8qg4v\" (UID: \"9b174e47-5aad-474b-a5cf-4fe09a417fe3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-634332-854c684b9f-8qg4v"
Nov 01 03:38:06.133302 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:06.133273    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b174e47-5aad-474b-a5cf-4fe09a417fe3-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-634332-854c684b9f-8qg4v\" (UID: \"9b174e47-5aad-474b-a5cf-4fe09a417fe3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-634332-854c684b9f-8qg4v"
Nov 01 03:38:06.135923 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:06.135886    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b174e47-5aad-474b-a5cf-4fe09a417fe3-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-634332-854c684b9f-8qg4v\" (UID: \"9b174e47-5aad-474b-a5cf-4fe09a417fe3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-634332-854c684b9f-8qg4v"
Nov 01 03:38:06.328356 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:06.328329    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-634332-854c684b9f-8qg4v"
Nov 01 03:38:06.713043 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:38:06.713014    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b174e47_5aad_474b_a5cf_4fe09a417fe3.slice/crio-8dc19fb363b2556a3642e1c6a0ab5f20f2fc8ef13623b55f9996744da2ba8363 WatchSource:0}: Error finding container 8dc19fb363b2556a3642e1c6a0ab5f20f2fc8ef13623b55f9996744da2ba8363: Status 404 returned error can't find the container with id 8dc19fb363b2556a3642e1c6a0ab5f20f2fc8ef13623b55f9996744da2ba8363
Nov 01 03:38:06.713043 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:06.713018    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-634332-854c684b9f-8qg4v"]
Nov 01 03:38:07.019392 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:07.019326    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-634332-854c684b9f-8qg4v" event={"ID":"9b174e47-5aad-474b-a5cf-4fe09a417fe3","Type":"ContainerStarted","Data":"8dc19fb363b2556a3642e1c6a0ab5f20f2fc8ef13623b55f9996744da2ba8363"}
Nov 01 03:38:08.027217 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:08.027186    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-634332-854c684b9f-8qg4v" event={"ID":"9b174e47-5aad-474b-a5cf-4fe09a417fe3","Type":"ContainerStarted","Data":"0865bc9bdd2810bcef05e3b89a22bea7e239ba9bd4463b224262ce5ad5cc86a9"}
Nov 01 03:38:08.027669 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:08.027327    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-634332-854c684b9f-8qg4v"
Nov 01 03:38:08.188950 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:08.188906    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-634332-854c684b9f-8qg4v" podStartSLOduration=2.385400507 podStartE2EDuration="3.188893299s" totalImagesPullingTime="803.492792ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:38:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:38:08.185369773 +0000 UTC m=+1894.700015636" watchObservedRunningTime="2026-11-01 03:38:08.188893299 +0000 UTC m=+1894.703539184"
Nov 01 03:38:09.003708 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:09.003652    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-438641-856ccc5949-tcnlc"
Nov 01 03:38:14.038595 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:14.038568    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-634332-854c684b9f-8qg4v"
Nov 01 03:38:31.774430 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:38:31.774384    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:38:35.081148 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:35.081113    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7d7329-85768dff6d-dgz5b"]
Nov 01 03:38:35.086318 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:35.086297    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7d7329-85768dff6d-dgz5b"
Nov 01 03:38:35.144722 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:35.144677    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7d7329-85768dff6d-dgz5b"]
Nov 01 03:38:35.189907 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:35.189886    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-7d7329-serving-cert\""
Nov 01 03:38:35.258978 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:35.258917    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/884eee93-f740-4ccd-8ec2-12b26f39e775-metrics-certs\") pod \"maas-api-e2e-inf-a-7d7329-85768dff6d-dgz5b\" (UID: \"884eee93-f740-4ccd-8ec2-12b26f39e775\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7d7329-85768dff6d-dgz5b"
Nov 01 03:38:35.259222 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:35.259199    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/884eee93-f740-4ccd-8ec2-12b26f39e775-maas-api-tls\") pod \"maas-api-e2e-inf-a-7d7329-85768dff6d-dgz5b\" (UID: \"884eee93-f740-4ccd-8ec2-12b26f39e775\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7d7329-85768dff6d-dgz5b"
Nov 01 03:38:35.259476 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:35.259449    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/884eee93-f740-4ccd-8ec2-12b26f39e775-openshift-service-ca\") pod \"maas-api-e2e-inf-a-7d7329-85768dff6d-dgz5b\" (UID: \"884eee93-f740-4ccd-8ec2-12b26f39e775\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7d7329-85768dff6d-dgz5b"
Nov 01 03:38:35.259639 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:35.259616    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvkvj\" (UniqueName: \"kubernetes.io/projected/884eee93-f740-4ccd-8ec2-12b26f39e775-kube-api-access-dvkvj\") pod \"maas-api-e2e-inf-a-7d7329-85768dff6d-dgz5b\" (UID: \"884eee93-f740-4ccd-8ec2-12b26f39e775\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7d7329-85768dff6d-dgz5b"
Nov 01 03:38:35.367342 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:35.367250    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/884eee93-f740-4ccd-8ec2-12b26f39e775-metrics-certs\") pod \"maas-api-e2e-inf-a-7d7329-85768dff6d-dgz5b\" (UID: \"884eee93-f740-4ccd-8ec2-12b26f39e775\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7d7329-85768dff6d-dgz5b"
Nov 01 03:38:35.368193 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:35.368166    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/884eee93-f740-4ccd-8ec2-12b26f39e775-maas-api-tls\") pod \"maas-api-e2e-inf-a-7d7329-85768dff6d-dgz5b\" (UID: \"884eee93-f740-4ccd-8ec2-12b26f39e775\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7d7329-85768dff6d-dgz5b"
Nov 01 03:38:35.368548 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:35.368510    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/884eee93-f740-4ccd-8ec2-12b26f39e775-openshift-service-ca\") pod \"maas-api-e2e-inf-a-7d7329-85768dff6d-dgz5b\" (UID: \"884eee93-f740-4ccd-8ec2-12b26f39e775\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7d7329-85768dff6d-dgz5b"
Nov 01 03:38:35.419709 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:35.368829    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvkvj\" (UniqueName: \"kubernetes.io/projected/884eee93-f740-4ccd-8ec2-12b26f39e775-kube-api-access-dvkvj\") pod \"maas-api-e2e-inf-a-7d7329-85768dff6d-dgz5b\" (UID: \"884eee93-f740-4ccd-8ec2-12b26f39e775\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7d7329-85768dff6d-dgz5b"
Nov 01 03:38:35.420130 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:35.380993    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/884eee93-f740-4ccd-8ec2-12b26f39e775-openshift-service-ca\") pod \"maas-api-e2e-inf-a-7d7329-85768dff6d-dgz5b\" (UID: \"884eee93-f740-4ccd-8ec2-12b26f39e775\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7d7329-85768dff6d-dgz5b"
Nov 01 03:38:35.420283 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:35.396241    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/884eee93-f740-4ccd-8ec2-12b26f39e775-metrics-certs\") pod \"maas-api-e2e-inf-a-7d7329-85768dff6d-dgz5b\" (UID: \"884eee93-f740-4ccd-8ec2-12b26f39e775\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7d7329-85768dff6d-dgz5b"
Nov 01 03:38:35.420283 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:35.403307    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/884eee93-f740-4ccd-8ec2-12b26f39e775-maas-api-tls\") pod \"maas-api-e2e-inf-a-7d7329-85768dff6d-dgz5b\" (UID: \"884eee93-f740-4ccd-8ec2-12b26f39e775\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7d7329-85768dff6d-dgz5b"
Nov 01 03:38:35.447656 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:35.447626    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvkvj\" (UniqueName: \"kubernetes.io/projected/884eee93-f740-4ccd-8ec2-12b26f39e775-kube-api-access-dvkvj\") pod \"maas-api-e2e-inf-a-7d7329-85768dff6d-dgz5b\" (UID: \"884eee93-f740-4ccd-8ec2-12b26f39e775\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7d7329-85768dff6d-dgz5b"
Nov 01 03:38:35.702110 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:35.702033    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7d7329-85768dff6d-dgz5b"
Nov 01 03:38:36.115722 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:36.101609    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7d7329-85768dff6d-dgz5b"]
Nov 01 03:38:36.543275 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:36.542204    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7d7329-85768dff6d-dgz5b" event={"ID":"884eee93-f740-4ccd-8ec2-12b26f39e775","Type":"ContainerStarted","Data":"7196ea88f312c6ac390ec5d7829ecb0d9592db4d06179f8e84871588699bc186"}
Nov 01 03:38:37.551336 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:37.551303    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7d7329-85768dff6d-dgz5b" event={"ID":"884eee93-f740-4ccd-8ec2-12b26f39e775","Type":"ContainerStarted","Data":"1623be535d0b12eb73c3310b90d9a437238a020ed1ab4bb29ceb96b6542357ae"}
Nov 01 03:38:37.551763 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:37.551392    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7d7329-85768dff6d-dgz5b"
Nov 01 03:38:37.673088 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:37.673040    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7d7329-85768dff6d-dgz5b" podStartSLOduration=1.855870041 podStartE2EDuration="2.67302836s" totalImagesPullingTime="817.158319ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:38:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:38:37.671545089 +0000 UTC m=+1924.186190949" watchObservedRunningTime="2026-11-01 03:38:37.67302836 +0000 UTC m=+1924.187674221"
Nov 01 03:38:43.563946 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:43.563917    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7d7329-85768dff6d-dgz5b"
Nov 01 03:38:50.505843 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:50.505766    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-k2w66"]
Nov 01 03:38:50.510631 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:50.510612    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-k2w66"
Nov 01 03:38:50.570376 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:50.570348    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54027127-b211-45d7-a049-e2c6cf7ee964-istiod-ca-cert\") pod \"e2e-shared-b-w0-438641-openshift-default-974f6f5d5-k2w66\" (UID: \"54027127-b211-45d7-a049-e2c6cf7ee964\") " pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-k2w66"
Nov 01 03:38:50.570489 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:50.570391    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54027127-b211-45d7-a049-e2c6cf7ee964-workload-socket\") pod \"e2e-shared-b-w0-438641-openshift-default-974f6f5d5-k2w66\" (UID: \"54027127-b211-45d7-a049-e2c6cf7ee964\") " pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-k2w66"
Nov 01 03:38:50.570489 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:50.570414    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54027127-b211-45d7-a049-e2c6cf7ee964-credential-socket\") pod \"e2e-shared-b-w0-438641-openshift-default-974f6f5d5-k2w66\" (UID: \"54027127-b211-45d7-a049-e2c6cf7ee964\") " pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-k2w66"
Nov 01 03:38:50.570588 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:50.570534    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54027127-b211-45d7-a049-e2c6cf7ee964-istio-token\") pod \"e2e-shared-b-w0-438641-openshift-default-974f6f5d5-k2w66\" (UID: \"54027127-b211-45d7-a049-e2c6cf7ee964\") " pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-k2w66"
Nov 01 03:38:50.570641 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:50.570629    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54027127-b211-45d7-a049-e2c6cf7ee964-workload-certs\") pod \"e2e-shared-b-w0-438641-openshift-default-974f6f5d5-k2w66\" (UID: \"54027127-b211-45d7-a049-e2c6cf7ee964\") " pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-k2w66"
Nov 01 03:38:50.570744 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:50.570721    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4pbx\" (UniqueName: \"kubernetes.io/projected/54027127-b211-45d7-a049-e2c6cf7ee964-kube-api-access-b4pbx\") pod \"e2e-shared-b-w0-438641-openshift-default-974f6f5d5-k2w66\" (UID: \"54027127-b211-45d7-a049-e2c6cf7ee964\") " pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-k2w66"
Nov 01 03:38:50.570796 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:50.570787    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54027127-b211-45d7-a049-e2c6cf7ee964-istio-podinfo\") pod \"e2e-shared-b-w0-438641-openshift-default-974f6f5d5-k2w66\" (UID: \"54027127-b211-45d7-a049-e2c6cf7ee964\") " pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-k2w66"
Nov 01 03:38:50.570838 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:50.570827    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54027127-b211-45d7-a049-e2c6cf7ee964-istio-data\") pod \"e2e-shared-b-w0-438641-openshift-default-974f6f5d5-k2w66\" (UID: \"54027127-b211-45d7-a049-e2c6cf7ee964\") " pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-k2w66"
Nov 01 03:38:50.570885 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:50.570851    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54027127-b211-45d7-a049-e2c6cf7ee964-istio-envoy\") pod \"e2e-shared-b-w0-438641-openshift-default-974f6f5d5-k2w66\" (UID: \"54027127-b211-45d7-a049-e2c6cf7ee964\") " pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-k2w66"
Nov 01 03:38:50.645030 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:50.644999    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-438641-openshift-default-dockercfg-4hlvf\""
Nov 01 03:38:50.671987 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:50.671962    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54027127-b211-45d7-a049-e2c6cf7ee964-workload-certs\") pod \"e2e-shared-b-w0-438641-openshift-default-974f6f5d5-k2w66\" (UID: \"54027127-b211-45d7-a049-e2c6cf7ee964\") " pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-k2w66"
Nov 01 03:38:50.672136 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:50.672004    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4pbx\" (UniqueName: \"kubernetes.io/projected/54027127-b211-45d7-a049-e2c6cf7ee964-kube-api-access-b4pbx\") pod \"e2e-shared-b-w0-438641-openshift-default-974f6f5d5-k2w66\" (UID: \"54027127-b211-45d7-a049-e2c6cf7ee964\") " pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-k2w66"
Nov 01 03:38:50.672136 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:50.672031    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54027127-b211-45d7-a049-e2c6cf7ee964-istio-podinfo\") pod \"e2e-shared-b-w0-438641-openshift-default-974f6f5d5-k2w66\" (UID: \"54027127-b211-45d7-a049-e2c6cf7ee964\") " pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-k2w66"
Nov 01 03:38:50.672136 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:50.672071    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54027127-b211-45d7-a049-e2c6cf7ee964-istio-data\") pod \"e2e-shared-b-w0-438641-openshift-default-974f6f5d5-k2w66\" (UID: \"54027127-b211-45d7-a049-e2c6cf7ee964\") " pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-k2w66"
Nov 01 03:38:50.672136 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:50.672099    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54027127-b211-45d7-a049-e2c6cf7ee964-istio-envoy\") pod \"e2e-shared-b-w0-438641-openshift-default-974f6f5d5-k2w66\" (UID: \"54027127-b211-45d7-a049-e2c6cf7ee964\") " pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-k2w66"
Nov 01 03:38:50.672346 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:50.672238    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54027127-b211-45d7-a049-e2c6cf7ee964-istiod-ca-cert\") pod \"e2e-shared-b-w0-438641-openshift-default-974f6f5d5-k2w66\" (UID: \"54027127-b211-45d7-a049-e2c6cf7ee964\") " pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-k2w66"
Nov 01 03:38:50.672346 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:50.672281    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54027127-b211-45d7-a049-e2c6cf7ee964-workload-socket\") pod \"e2e-shared-b-w0-438641-openshift-default-974f6f5d5-k2w66\" (UID: \"54027127-b211-45d7-a049-e2c6cf7ee964\") " pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-k2w66"
Nov 01 03:38:50.672346 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:50.672308    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54027127-b211-45d7-a049-e2c6cf7ee964-credential-socket\") pod \"e2e-shared-b-w0-438641-openshift-default-974f6f5d5-k2w66\" (UID: \"54027127-b211-45d7-a049-e2c6cf7ee964\") " pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-k2w66"
Nov 01 03:38:50.672346 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:50.672337    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54027127-b211-45d7-a049-e2c6cf7ee964-istio-token\") pod \"e2e-shared-b-w0-438641-openshift-default-974f6f5d5-k2w66\" (UID: \"54027127-b211-45d7-a049-e2c6cf7ee964\") " pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-k2w66"
Nov 01 03:38:50.672569 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:50.672436    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54027127-b211-45d7-a049-e2c6cf7ee964-workload-certs\") pod \"e2e-shared-b-w0-438641-openshift-default-974f6f5d5-k2w66\" (UID: \"54027127-b211-45d7-a049-e2c6cf7ee964\") " pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-k2w66"
Nov 01 03:38:50.672736 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:50.672711    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54027127-b211-45d7-a049-e2c6cf7ee964-credential-socket\") pod \"e2e-shared-b-w0-438641-openshift-default-974f6f5d5-k2w66\" (UID: \"54027127-b211-45d7-a049-e2c6cf7ee964\") " pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-k2w66"
Nov 01 03:38:50.672829 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:50.672793    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54027127-b211-45d7-a049-e2c6cf7ee964-istio-data\") pod \"e2e-shared-b-w0-438641-openshift-default-974f6f5d5-k2w66\" (UID: \"54027127-b211-45d7-a049-e2c6cf7ee964\") " pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-k2w66"
Nov 01 03:38:50.672976 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:50.672950    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54027127-b211-45d7-a049-e2c6cf7ee964-workload-socket\") pod \"e2e-shared-b-w0-438641-openshift-default-974f6f5d5-k2w66\" (UID: \"54027127-b211-45d7-a049-e2c6cf7ee964\") " pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-k2w66"
Nov 01 03:38:50.673166 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:50.673140    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54027127-b211-45d7-a049-e2c6cf7ee964-istiod-ca-cert\") pod \"e2e-shared-b-w0-438641-openshift-default-974f6f5d5-k2w66\" (UID: \"54027127-b211-45d7-a049-e2c6cf7ee964\") " pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-k2w66"
Nov 01 03:38:50.675238 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:50.675217    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54027127-b211-45d7-a049-e2c6cf7ee964-istio-envoy\") pod \"e2e-shared-b-w0-438641-openshift-default-974f6f5d5-k2w66\" (UID: \"54027127-b211-45d7-a049-e2c6cf7ee964\") " pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-k2w66"
Nov 01 03:38:50.675543 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:50.675526    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54027127-b211-45d7-a049-e2c6cf7ee964-istio-podinfo\") pod \"e2e-shared-b-w0-438641-openshift-default-974f6f5d5-k2w66\" (UID: \"54027127-b211-45d7-a049-e2c6cf7ee964\") " pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-k2w66"
Nov 01 03:38:50.738557 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:50.738528    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54027127-b211-45d7-a049-e2c6cf7ee964-istio-token\") pod \"e2e-shared-b-w0-438641-openshift-default-974f6f5d5-k2w66\" (UID: \"54027127-b211-45d7-a049-e2c6cf7ee964\") " pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-k2w66"
Nov 01 03:38:50.738924 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:50.738906    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4pbx\" (UniqueName: \"kubernetes.io/projected/54027127-b211-45d7-a049-e2c6cf7ee964-kube-api-access-b4pbx\") pod \"e2e-shared-b-w0-438641-openshift-default-974f6f5d5-k2w66\" (UID: \"54027127-b211-45d7-a049-e2c6cf7ee964\") " pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-k2w66"
Nov 01 03:38:50.745427 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:50.745405    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-k2w66"]
Nov 01 03:38:50.825240 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:50.825204    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-k2w66"
Nov 01 03:38:51.329248 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:38:51.329214    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54027127_b211_45d7_a049_e2c6cf7ee964.slice/crio-f2e1d82cbedc06db81d9174eedd595dccd0e91a8eb0736e04e5fdc8b66ef5a78 WatchSource:0}: Error finding container f2e1d82cbedc06db81d9174eedd595dccd0e91a8eb0736e04e5fdc8b66ef5a78: Status 404 returned error can't find the container with id f2e1d82cbedc06db81d9174eedd595dccd0e91a8eb0736e04e5fdc8b66ef5a78
Nov 01 03:38:51.329677 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:51.329653    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-k2w66"]
Nov 01 03:38:51.331825 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:51.331792    4400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 03:38:51.331922 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:51.331864    4400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 03:38:51.331922 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:51.331907    4400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 03:38:51.646887 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:51.646804    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-k2w66" event={"ID":"54027127-b211-45d7-a049-e2c6cf7ee964","Type":"ContainerStarted","Data":"3202fb001781b99e88ecc888ea3c59abb79506c67f2232937b12cd123da6392e"}
Nov 01 03:38:51.646887 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:51.646858    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-k2w66" event={"ID":"54027127-b211-45d7-a049-e2c6cf7ee964","Type":"ContainerStarted","Data":"f2e1d82cbedc06db81d9174eedd595dccd0e91a8eb0736e04e5fdc8b66ef5a78"}
Nov 01 03:38:51.773346 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:51.773291    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-k2w66" podStartSLOduration=1.773273863 podStartE2EDuration="1.773273863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:38:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:38:51.770217735 +0000 UTC m=+1938.284863597" watchObservedRunningTime="2026-11-01 03:38:51.773273863 +0000 UTC m=+1938.287919725"
Nov 01 03:38:51.826039 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:51.826010    4400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-k2w66"
Nov 01 03:38:52.828174 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:52.828143    4400 patch_prober.go:29] interesting pod/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-k2w66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Nov 01 03:38:52.828585 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:52.828191    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-k2w66" podUID="54027127-b211-45d7-a049-e2c6cf7ee964" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Nov 01 03:38:53.830708 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:53.830022    4400 patch_prober.go:29] interesting pod/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-k2w66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Nov 01 03:38:53.830708 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:53.830087    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-k2w66" podUID="54027127-b211-45d7-a049-e2c6cf7ee964" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Nov 01 03:38:54.827104 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:54.827067    4400 patch_prober.go:29] interesting pod/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-k2w66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Nov 01 03:38:54.827321 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:54.827126    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-k2w66" podUID="54027127-b211-45d7-a049-e2c6cf7ee964" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Nov 01 03:38:55.827021 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:55.826981    4400 patch_prober.go:29] interesting pod/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-k2w66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Nov 01 03:38:55.827500 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:55.827042    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-k2w66" podUID="54027127-b211-45d7-a049-e2c6cf7ee964" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Nov 01 03:38:56.829032 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:56.828996    4400 patch_prober.go:29] interesting pod/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-k2w66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Nov 01 03:38:56.829630 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:56.829589    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-k2w66" podUID="54027127-b211-45d7-a049-e2c6cf7ee964" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Nov 01 03:38:57.596433 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:57.596404    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7072-954f75"]
Nov 01 03:38:57.600940 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:57.600923    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7072-954f75"
Nov 01 03:38:57.656332 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:57.656304    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7072-954f75"]
Nov 01 03:38:57.692280 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:57.692248    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qc5vk\" (UniqueName: \"kubernetes.io/projected/5225f061-3c3d-47b2-8d9d-ff286b3cc977-kube-api-access-qc5vk\") pod \"test-curl-7072-954f75\" (UID: \"5225f061-3c3d-47b2-8d9d-ff286b3cc977\") " pod="opendatahub/test-curl-7072-954f75"
Nov 01 03:38:57.702433 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:57.702413    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-h2kzj\""
Nov 01 03:38:57.793250 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:57.793219    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc5vk\" (UniqueName: \"kubernetes.io/projected/5225f061-3c3d-47b2-8d9d-ff286b3cc977-kube-api-access-qc5vk\") pod \"test-curl-7072-954f75\" (UID: \"5225f061-3c3d-47b2-8d9d-ff286b3cc977\") " pod="opendatahub/test-curl-7072-954f75"
Nov 01 03:38:57.830660 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:57.830636    4400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-k2w66"
Nov 01 03:38:57.831070 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:57.831029    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-k2w66"
Nov 01 03:38:57.831760 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:57.831744    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-k2w66"
Nov 01 03:38:57.855836 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:57.855786    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qc5vk\" (UniqueName: \"kubernetes.io/projected/5225f061-3c3d-47b2-8d9d-ff286b3cc977-kube-api-access-qc5vk\") pod \"test-curl-7072-954f75\" (UID: \"5225f061-3c3d-47b2-8d9d-ff286b3cc977\") " pod="opendatahub/test-curl-7072-954f75"
Nov 01 03:38:57.914511 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:57.914490    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7072-954f75"
Nov 01 03:38:58.300715 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:38:58.300669    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5225f061_3c3d_47b2_8d9d_ff286b3cc977.slice/crio-f46018bf87098285cb126a366dd7754bab8da880693297c1bb77b981b7e78e18 WatchSource:0}: Error finding container f46018bf87098285cb126a366dd7754bab8da880693297c1bb77b981b7e78e18: Status 404 returned error can't find the container with id f46018bf87098285cb126a366dd7754bab8da880693297c1bb77b981b7e78e18
Nov 01 03:38:58.302090 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:58.302064    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7072-954f75"]
Nov 01 03:38:58.697607 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:58.697575    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7072-954f75" event={"ID":"5225f061-3c3d-47b2-8d9d-ff286b3cc977","Type":"ContainerStarted","Data":"021d624dc824cc3fa283dbb986b196d410390dd5fa1d2770d7c6d390b3f89049"}
Nov 01 03:38:58.697785 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:38:58.697622    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7072-954f75" event={"ID":"5225f061-3c3d-47b2-8d9d-ff286b3cc977","Type":"ContainerStarted","Data":"f46018bf87098285cb126a366dd7754bab8da880693297c1bb77b981b7e78e18"}
Nov 01 03:39:00.344007 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:00.343952    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7072-954f75" podStartSLOduration=3.343934695 podStartE2EDuration="3.343934695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:38:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:38:58.82855057 +0000 UTC m=+1945.343196448" watchObservedRunningTime="2026-11-01 03:39:00.343934695 +0000 UTC m=+1946.858580561"
Nov 01 03:39:00.346798 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:00.346768    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7072-954f75"]
Nov 01 03:39:00.352138 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:00.352113    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7072-954f75"]
Nov 01 03:39:00.709580 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:00.709505    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7072-954f75" podUID="5225f061-3c3d-47b2-8d9d-ff286b3cc977" containerName="test-curl-7072-954f75" containerID="cri-o://021d624dc824cc3fa283dbb986b196d410390dd5fa1d2770d7c6d390b3f89049" gracePeriod=2
Nov 01 03:39:00.764463 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:39:00.764429    4400 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7072-954f75\" is forbidden: User \"system:node:ip-10-0-133-54.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-54.us-west-1.compute.internal' and this object" podUID="5225f061-3c3d-47b2-8d9d-ff286b3cc977" pod="opendatahub/test-curl-7072-954f75"
Nov 01 03:39:01.730341 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:39:01.730305    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:39:02.957106 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:02.957083    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7072-954f75"
Nov 01 03:39:03.085572 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:03.085544    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5225f061-3c3d-47b2-8d9d-ff286b3cc977-kube-api-access-qc5vk\" (UniqueName: \"kubernetes.io/projected/5225f061-3c3d-47b2-8d9d-ff286b3cc977-kube-api-access-qc5vk\") pod \"5225f061-3c3d-47b2-8d9d-ff286b3cc977\" (UID: \"5225f061-3c3d-47b2-8d9d-ff286b3cc977\") "
Nov 01 03:39:03.087705 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:03.087663    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5225f061-3c3d-47b2-8d9d-ff286b3cc977-kube-api-access-qc5vk" pod "5225f061-3c3d-47b2-8d9d-ff286b3cc977" (UID: "5225f061-3c3d-47b2-8d9d-ff286b3cc977"). InnerVolumeSpecName "kube-api-access-qc5vk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:39:03.187408 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:03.187377    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qc5vk\" (UniqueName: \"kubernetes.io/projected/5225f061-3c3d-47b2-8d9d-ff286b3cc977-kube-api-access-qc5vk\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:39:03.616153 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:03.616124    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5225f061-3c3d-47b2-8d9d-ff286b3cc977" path="/var/lib/kubelet/pods/5225f061-3c3d-47b2-8d9d-ff286b3cc977/volumes"
Nov 01 03:39:03.730288 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:03.730261    4400 generic.go:356] "Generic (PLEG): container finished" podID="5225f061-3c3d-47b2-8d9d-ff286b3cc977" containerID="021d624dc824cc3fa283dbb986b196d410390dd5fa1d2770d7c6d390b3f89049" exitCode=137
Nov 01 03:39:03.730416 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:03.730293    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7072-954f75"
Nov 01 03:39:03.730416 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:03.730354    4400 scope.go:122] "RemoveContainer" containerID="021d624dc824cc3fa283dbb986b196d410390dd5fa1d2770d7c6d390b3f89049"
Nov 01 03:39:03.742279 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:03.742265    4400 scope.go:122] "RemoveContainer" containerID="021d624dc824cc3fa283dbb986b196d410390dd5fa1d2770d7c6d390b3f89049"
Nov 01 03:39:03.742544 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:39:03.742522    4400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"021d624dc824cc3fa283dbb986b196d410390dd5fa1d2770d7c6d390b3f89049\": container with ID starting with 021d624dc824cc3fa283dbb986b196d410390dd5fa1d2770d7c6d390b3f89049 not found: ID does not exist" containerID="021d624dc824cc3fa283dbb986b196d410390dd5fa1d2770d7c6d390b3f89049"
Nov 01 03:39:03.742618 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:03.742552    4400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"021d624dc824cc3fa283dbb986b196d410390dd5fa1d2770d7c6d390b3f89049"} err="failed to get container status \"021d624dc824cc3fa283dbb986b196d410390dd5fa1d2770d7c6d390b3f89049\": rpc error: code = NotFound desc = could not find container \"021d624dc824cc3fa283dbb986b196d410390dd5fa1d2770d7c6d390b3f89049\": container with ID starting with 021d624dc824cc3fa283dbb986b196d410390dd5fa1d2770d7c6d390b3f89049 not found: ID does not exist"
Nov 01 03:39:08.342615 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:08.342578    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 03:39:13.151226 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:13.151197    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fdbdfefd-8468f4575f-vcdsp"]
Nov 01 03:39:18.151883 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:18.151816    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-fdbdfefd-8468f4575f-vcdsp" podUID="9383b0fa-b53c-451f-9d55-c439dee7cde0" containerName="payload-pre-processing" containerID="cri-o://704151a76b326ccdba0ca62b620c3e9540345fad4711a76f4a2eb12da14b2f2f" gracePeriod=25
Nov 01 03:39:19.307408 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:19.307371    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-fdbdfefd-8468f4575f-vcdsp" podUID="9383b0fa-b53c-451f-9d55-c439dee7cde0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 03:39:29.307362 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:29.307328    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-fdbdfefd-8468f4575f-vcdsp" podUID="9383b0fa-b53c-451f-9d55-c439dee7cde0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 03:39:31.730972 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:39:31.730937    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:39:38.489867 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:38.489844    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-fdbdfefd-8468f4575f-vcdsp"
Nov 01 03:39:38.525171 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:38.525151    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9383b0fa-b53c-451f-9d55-c439dee7cde0-kube-api-access-smn92\" (UniqueName: \"kubernetes.io/projected/9383b0fa-b53c-451f-9d55-c439dee7cde0-kube-api-access-smn92\") pod \"9383b0fa-b53c-451f-9d55-c439dee7cde0\" (UID: \"9383b0fa-b53c-451f-9d55-c439dee7cde0\") "
Nov 01 03:39:38.525282 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:38.525191    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9383b0fa-b53c-451f-9d55-c439dee7cde0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9383b0fa-b53c-451f-9d55-c439dee7cde0-plugins-config-volume\") pod \"9383b0fa-b53c-451f-9d55-c439dee7cde0\" (UID: \"9383b0fa-b53c-451f-9d55-c439dee7cde0\") "
Nov 01 03:39:38.525584 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:38.525563    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9383b0fa-b53c-451f-9d55-c439dee7cde0-plugins-config-volume" pod "9383b0fa-b53c-451f-9d55-c439dee7cde0" (UID: "9383b0fa-b53c-451f-9d55-c439dee7cde0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:39:38.525760 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:38.525747    4400 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9383b0fa-b53c-451f-9d55-c439dee7cde0-plugins-config-volume\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:39:38.527266 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:38.527248    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9383b0fa-b53c-451f-9d55-c439dee7cde0-kube-api-access-smn92" pod "9383b0fa-b53c-451f-9d55-c439dee7cde0" (UID: "9383b0fa-b53c-451f-9d55-c439dee7cde0"). InnerVolumeSpecName "kube-api-access-smn92". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:39:38.627287 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:38.627255    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smn92\" (UniqueName: \"kubernetes.io/projected/9383b0fa-b53c-451f-9d55-c439dee7cde0-kube-api-access-smn92\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:39:39.074833 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:39.074801    4400 generic.go:356] "Generic (PLEG): container finished" podID="9383b0fa-b53c-451f-9d55-c439dee7cde0" containerID="704151a76b326ccdba0ca62b620c3e9540345fad4711a76f4a2eb12da14b2f2f" exitCode=0
Nov 01 03:39:39.075012 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:39.074853    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-fdbdfefd-8468f4575f-vcdsp"
Nov 01 03:39:39.075012 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:39.074883    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-fdbdfefd-8468f4575f-vcdsp" event={"ID":"9383b0fa-b53c-451f-9d55-c439dee7cde0","Type":"ContainerDied","Data":"704151a76b326ccdba0ca62b620c3e9540345fad4711a76f4a2eb12da14b2f2f"}
Nov 01 03:39:39.075012 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:39.074948    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-fdbdfefd-8468f4575f-vcdsp" event={"ID":"9383b0fa-b53c-451f-9d55-c439dee7cde0","Type":"ContainerDied","Data":"5f462a841fb13b0d84faf1789e8a20f95d1aff8644f669f67c3ea4b9ffde7509"}
Nov 01 03:39:39.075012 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:39.074977    4400 scope.go:122] "RemoveContainer" containerID="704151a76b326ccdba0ca62b620c3e9540345fad4711a76f4a2eb12da14b2f2f"
Nov 01 03:39:39.085643 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:39.085629    4400 scope.go:122] "RemoveContainer" containerID="704151a76b326ccdba0ca62b620c3e9540345fad4711a76f4a2eb12da14b2f2f"
Nov 01 03:39:39.085920 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:39:39.085902    4400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"704151a76b326ccdba0ca62b620c3e9540345fad4711a76f4a2eb12da14b2f2f\": container with ID starting with 704151a76b326ccdba0ca62b620c3e9540345fad4711a76f4a2eb12da14b2f2f not found: ID does not exist" containerID="704151a76b326ccdba0ca62b620c3e9540345fad4711a76f4a2eb12da14b2f2f"
Nov 01 03:39:39.085974 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:39.085927    4400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"704151a76b326ccdba0ca62b620c3e9540345fad4711a76f4a2eb12da14b2f2f"} err="failed to get container status \"704151a76b326ccdba0ca62b620c3e9540345fad4711a76f4a2eb12da14b2f2f\": rpc error: code = NotFound desc = could not find container \"704151a76b326ccdba0ca62b620c3e9540345fad4711a76f4a2eb12da14b2f2f\": container with ID starting with 704151a76b326ccdba0ca62b620c3e9540345fad4711a76f4a2eb12da14b2f2f not found: ID does not exist"
Nov 01 03:39:39.244075 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:39.244051    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fdbdfefd-8468f4575f-vcdsp"]
Nov 01 03:39:39.295234 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:39.295205    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fdbdfefd-8468f4575f-vcdsp"]
Nov 01 03:39:39.615533 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:39.615506    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9383b0fa-b53c-451f-9d55-c439dee7cde0" path="/var/lib/kubelet/pods/9383b0fa-b53c-451f-9d55-c439dee7cde0/volumes"
Nov 01 03:39:48.787305 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:48.787270    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-mp6pq"]
Nov 01 03:39:48.787745 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:48.787491    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-mp6pq" podUID="bd8db90d-adb2-4601-bd83-5f8dafbc02ce" containerName="istio-proxy" containerID="cri-o://d46c3b33c85ec5ae8946575290fa3b1536711ce6d58cf5e3094b59917a754555" gracePeriod=30
Nov 01 03:39:51.388972 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:51.388924    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-211380-598d8b59f-gs5lv"]
Nov 01 03:39:51.398911 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:51.398580    4400 cpu_manager.go:414] "containerMap: removing container" podUID="9383b0fa-b53c-451f-9d55-c439dee7cde0" containerName="payload-pre-processing"
Nov 01 03:39:51.399127 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:51.399108    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9383b0fa-b53c-451f-9d55-c439dee7cde0" containerName="payload-pre-processing"
Nov 01 03:39:51.399256 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:51.399246    4400 cpu_manager.go:414] "containerMap: removing container" podUID="5225f061-3c3d-47b2-8d9d-ff286b3cc977" containerName="test-curl-7072-954f75"
Nov 01 03:39:51.399323 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:51.399313    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5225f061-3c3d-47b2-8d9d-ff286b3cc977" containerName="test-curl-7072-954f75"
Nov 01 03:39:51.399660 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:51.399646    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="5225f061-3c3d-47b2-8d9d-ff286b3cc977" containerName="test-curl-7072-954f75"
Nov 01 03:39:51.399795 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:51.399785    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="9383b0fa-b53c-451f-9d55-c439dee7cde0" containerName="payload-pre-processing"
Nov 01 03:39:51.410633 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:51.410010    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-211380-598d8b59f-gs5lv"]
Nov 01 03:39:51.410633 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:51.410081    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-211380-598d8b59f-gs5lv"
Nov 01 03:39:51.552469 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:51.478067    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w6-211380-dockercfg-cflzk\""
Nov 01 03:39:51.564709 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:51.555524    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w6-211380\""
Nov 01 03:39:51.585712 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:51.566071    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc0b9a54-e2e5-44e2-bb9f-62949ff3db59-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-211380-598d8b59f-gs5lv\" (UID: \"bc0b9a54-e2e5-44e2-bb9f-62949ff3db59\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-211380-598d8b59f-gs5lv"
Nov 01 03:39:51.661856 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:51.651158    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgsx7\" (UniqueName: \"kubernetes.io/projected/bc0b9a54-e2e5-44e2-bb9f-62949ff3db59-kube-api-access-fgsx7\") pod \"payload-pre-processing-e2e-shared-b-w6-211380-598d8b59f-gs5lv\" (UID: \"bc0b9a54-e2e5-44e2-bb9f-62949ff3db59\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-211380-598d8b59f-gs5lv"
Nov 01 03:39:51.716266 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:51.716226    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-211380-7d964dd845-2blzj"]
Nov 01 03:39:51.727201 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:51.727166    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-211380-7d964dd845-2blzj"]
Nov 01 03:39:51.727587 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:51.727498    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-211380-7d964dd845-2blzj"
Nov 01 03:39:51.752281 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:51.752258    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc0b9a54-e2e5-44e2-bb9f-62949ff3db59-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-211380-598d8b59f-gs5lv\" (UID: \"bc0b9a54-e2e5-44e2-bb9f-62949ff3db59\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-211380-598d8b59f-gs5lv"
Nov 01 03:39:51.752406 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:51.752382    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgsx7\" (UniqueName: \"kubernetes.io/projected/bc0b9a54-e2e5-44e2-bb9f-62949ff3db59-kube-api-access-fgsx7\") pod \"payload-pre-processing-e2e-shared-b-w6-211380-598d8b59f-gs5lv\" (UID: \"bc0b9a54-e2e5-44e2-bb9f-62949ff3db59\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-211380-598d8b59f-gs5lv"
Nov 01 03:39:51.753103 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:51.753072    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc0b9a54-e2e5-44e2-bb9f-62949ff3db59-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-211380-598d8b59f-gs5lv\" (UID: \"bc0b9a54-e2e5-44e2-bb9f-62949ff3db59\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-211380-598d8b59f-gs5lv"
Nov 01 03:39:51.820178 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:51.820149    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgsx7\" (UniqueName: \"kubernetes.io/projected/bc0b9a54-e2e5-44e2-bb9f-62949ff3db59-kube-api-access-fgsx7\") pod \"payload-pre-processing-e2e-shared-b-w6-211380-598d8b59f-gs5lv\" (UID: \"bc0b9a54-e2e5-44e2-bb9f-62949ff3db59\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-211380-598d8b59f-gs5lv"
Nov 01 03:39:51.853996 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:51.853964    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a5c6392-1b3d-4823-a5fa-f91005df6759-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-211380-7d964dd845-2blzj\" (UID: \"0a5c6392-1b3d-4823-a5fa-f91005df6759\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-211380-7d964dd845-2blzj"
Nov 01 03:39:51.854127 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:51.854110    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xppfh\" (UniqueName: \"kubernetes.io/projected/0a5c6392-1b3d-4823-a5fa-f91005df6759-kube-api-access-xppfh\") pod \"payload-processing-e2e-shared-b-w6-211380-7d964dd845-2blzj\" (UID: \"0a5c6392-1b3d-4823-a5fa-f91005df6759\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-211380-7d964dd845-2blzj"
Nov 01 03:39:51.955741 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:51.955675    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a5c6392-1b3d-4823-a5fa-f91005df6759-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-211380-7d964dd845-2blzj\" (UID: \"0a5c6392-1b3d-4823-a5fa-f91005df6759\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-211380-7d964dd845-2blzj"
Nov 01 03:39:51.955831 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:51.955770    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xppfh\" (UniqueName: \"kubernetes.io/projected/0a5c6392-1b3d-4823-a5fa-f91005df6759-kube-api-access-xppfh\") pod \"payload-processing-e2e-shared-b-w6-211380-7d964dd845-2blzj\" (UID: \"0a5c6392-1b3d-4823-a5fa-f91005df6759\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-211380-7d964dd845-2blzj"
Nov 01 03:39:51.956279 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:51.956261    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a5c6392-1b3d-4823-a5fa-f91005df6759-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-211380-7d964dd845-2blzj\" (UID: \"0a5c6392-1b3d-4823-a5fa-f91005df6759\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-211380-7d964dd845-2blzj"
Nov 01 03:39:52.023577 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:52.023551    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-211380-598d8b59f-gs5lv"
Nov 01 03:39:52.030834 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:52.030814    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xppfh\" (UniqueName: \"kubernetes.io/projected/0a5c6392-1b3d-4823-a5fa-f91005df6759-kube-api-access-xppfh\") pod \"payload-processing-e2e-shared-b-w6-211380-7d964dd845-2blzj\" (UID: \"0a5c6392-1b3d-4823-a5fa-f91005df6759\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-211380-7d964dd845-2blzj"
Nov 01 03:39:52.047911 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:52.047889    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-211380-7d964dd845-2blzj"
Nov 01 03:39:52.426357 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:52.426324    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-211380-598d8b59f-gs5lv"]
Nov 01 03:39:52.451559 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:39:52.451528    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc0b9a54_e2e5_44e2_bb9f_62949ff3db59.slice/crio-b5e464fb8bb89ceed7eaf1c1fc7310b219d6c8f514e987af2498e4c8130e9427 WatchSource:0}: Error finding container b5e464fb8bb89ceed7eaf1c1fc7310b219d6c8f514e987af2498e4c8130e9427: Status 404 returned error can't find the container with id b5e464fb8bb89ceed7eaf1c1fc7310b219d6c8f514e987af2498e4c8130e9427
Nov 01 03:39:52.470737 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:52.470669    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-211380-7d964dd845-2blzj"]
Nov 01 03:39:52.474065 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:39:52.474041    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a5c6392_1b3d_4823_a5fa_f91005df6759.slice/crio-0c910f73bf0234d1487d8c7b65bcb996f458848583e5adc2be2938583cc0f4dc WatchSource:0}: Error finding container 0c910f73bf0234d1487d8c7b65bcb996f458848583e5adc2be2938583cc0f4dc: Status 404 returned error can't find the container with id 0c910f73bf0234d1487d8c7b65bcb996f458848583e5adc2be2938583cc0f4dc
Nov 01 03:39:52.838043 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:52.838005    4400 patch_prober.go:29] interesting pod/e2e-worker-w4-e8c789-openshift-default-774d997df-mp6pq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 03:39:52.838223 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:52.838081    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-mp6pq" podUID="bd8db90d-adb2-4601-bd83-5f8dafbc02ce" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 03:39:53.356369 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:53.356330    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-211380-7d964dd845-2blzj" event={"ID":"0a5c6392-1b3d-4823-a5fa-f91005df6759","Type":"ContainerStarted","Data":"745e0f7dbdb631f410a4d8da5d2a8b2505a246b25322f6411cf68b85030848bf"}
Nov 01 03:39:53.356571 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:53.356395    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-211380-7d964dd845-2blzj" event={"ID":"0a5c6392-1b3d-4823-a5fa-f91005df6759","Type":"ContainerStarted","Data":"0c910f73bf0234d1487d8c7b65bcb996f458848583e5adc2be2938583cc0f4dc"}
Nov 01 03:39:53.356571 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:53.356425    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-211380-7d964dd845-2blzj"
Nov 01 03:39:53.358205 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:53.358177    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-211380-598d8b59f-gs5lv" event={"ID":"bc0b9a54-e2e5-44e2-bb9f-62949ff3db59","Type":"ContainerStarted","Data":"5596f24ad7026543a059004bcd039a00dc996e096fe2bf55de0c7f51b6242e5f"}
Nov 01 03:39:53.358347 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:53.358222    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-211380-598d8b59f-gs5lv" event={"ID":"bc0b9a54-e2e5-44e2-bb9f-62949ff3db59","Type":"ContainerStarted","Data":"b5e464fb8bb89ceed7eaf1c1fc7310b219d6c8f514e987af2498e4c8130e9427"}
Nov 01 03:39:53.358347 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:53.358288    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-211380-598d8b59f-gs5lv"
Nov 01 03:39:53.477966 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:53.477925    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-211380-7d964dd845-2blzj" podStartSLOduration=2.4779130560000002 podStartE2EDuration="2.477913056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:39:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:39:53.475806133 +0000 UTC m=+1999.990451993" watchObservedRunningTime="2026-11-01 03:39:53.477913056 +0000 UTC m=+1999.992558916"
Nov 01 03:39:53.596066 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:53.596021    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-211380-598d8b59f-gs5lv" podStartSLOduration=2.596008105 podStartE2EDuration="2.596008105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:39:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:39:53.59373027 +0000 UTC m=+2000.108376130" watchObservedRunningTime="2026-11-01 03:39:53.596008105 +0000 UTC m=+2000.110653965"
Nov 01 03:39:54.239783 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:54.239760    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-mp6pq"
Nov 01 03:39:54.264279 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:54.264253    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd8db90d-adb2-4601-bd83-5f8dafbc02ce-kube-api-access-ddxgr\" (UniqueName: \"kubernetes.io/projected/bd8db90d-adb2-4601-bd83-5f8dafbc02ce-kube-api-access-ddxgr\") pod \"bd8db90d-adb2-4601-bd83-5f8dafbc02ce\" (UID: \"bd8db90d-adb2-4601-bd83-5f8dafbc02ce\") "
Nov 01 03:39:54.264379 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:54.264348    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bd8db90d-adb2-4601-bd83-5f8dafbc02ce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd8db90d-adb2-4601-bd83-5f8dafbc02ce-istio-podinfo\") pod \"bd8db90d-adb2-4601-bd83-5f8dafbc02ce\" (UID: \"bd8db90d-adb2-4601-bd83-5f8dafbc02ce\") "
Nov 01 03:39:54.264379 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:54.264371    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd8db90d-adb2-4601-bd83-5f8dafbc02ce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd8db90d-adb2-4601-bd83-5f8dafbc02ce-workload-certs\") pod \"bd8db90d-adb2-4601-bd83-5f8dafbc02ce\" (UID: \"bd8db90d-adb2-4601-bd83-5f8dafbc02ce\") "
Nov 01 03:39:54.264453 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:54.264390    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd8db90d-adb2-4601-bd83-5f8dafbc02ce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd8db90d-adb2-4601-bd83-5f8dafbc02ce-credential-socket\") pod \"bd8db90d-adb2-4601-bd83-5f8dafbc02ce\" (UID: \"bd8db90d-adb2-4601-bd83-5f8dafbc02ce\") "
Nov 01 03:39:54.264496 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:54.264451    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd8db90d-adb2-4601-bd83-5f8dafbc02ce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd8db90d-adb2-4601-bd83-5f8dafbc02ce-istio-data\") pod \"bd8db90d-adb2-4601-bd83-5f8dafbc02ce\" (UID: \"bd8db90d-adb2-4601-bd83-5f8dafbc02ce\") "
Nov 01 03:39:54.264594 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:54.264577    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd8db90d-adb2-4601-bd83-5f8dafbc02ce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd8db90d-adb2-4601-bd83-5f8dafbc02ce-workload-socket\") pod \"bd8db90d-adb2-4601-bd83-5f8dafbc02ce\" (UID: \"bd8db90d-adb2-4601-bd83-5f8dafbc02ce\") "
Nov 01 03:39:54.264743 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:54.264713    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd8db90d-adb2-4601-bd83-5f8dafbc02ce-workload-certs" pod "bd8db90d-adb2-4601-bd83-5f8dafbc02ce" (UID: "bd8db90d-adb2-4601-bd83-5f8dafbc02ce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:39:54.264835 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:54.264778    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd8db90d-adb2-4601-bd83-5f8dafbc02ce-istio-token\" (UniqueName: \"kubernetes.io/projected/bd8db90d-adb2-4601-bd83-5f8dafbc02ce-istio-token\") pod \"bd8db90d-adb2-4601-bd83-5f8dafbc02ce\" (UID: \"bd8db90d-adb2-4601-bd83-5f8dafbc02ce\") "
Nov 01 03:39:54.264835 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:54.264795    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd8db90d-adb2-4601-bd83-5f8dafbc02ce-credential-socket" pod "bd8db90d-adb2-4601-bd83-5f8dafbc02ce" (UID: "bd8db90d-adb2-4601-bd83-5f8dafbc02ce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:39:54.264943 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:54.264845    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd8db90d-adb2-4601-bd83-5f8dafbc02ce-istio-data" pod "bd8db90d-adb2-4601-bd83-5f8dafbc02ce" (UID: "bd8db90d-adb2-4601-bd83-5f8dafbc02ce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:39:54.264943 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:54.264867    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd8db90d-adb2-4601-bd83-5f8dafbc02ce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd8db90d-adb2-4601-bd83-5f8dafbc02ce-istio-envoy\") pod \"bd8db90d-adb2-4601-bd83-5f8dafbc02ce\" (UID: \"bd8db90d-adb2-4601-bd83-5f8dafbc02ce\") "
Nov 01 03:39:54.265044 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:54.264937    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd8db90d-adb2-4601-bd83-5f8dafbc02ce-workload-socket" pod "bd8db90d-adb2-4601-bd83-5f8dafbc02ce" (UID: "bd8db90d-adb2-4601-bd83-5f8dafbc02ce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:39:54.265044 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:54.265023    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd8db90d-adb2-4601-bd83-5f8dafbc02ce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd8db90d-adb2-4601-bd83-5f8dafbc02ce-istiod-ca-cert\") pod \"bd8db90d-adb2-4601-bd83-5f8dafbc02ce\" (UID: \"bd8db90d-adb2-4601-bd83-5f8dafbc02ce\") "
Nov 01 03:39:54.265595 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:54.265542    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd8db90d-adb2-4601-bd83-5f8dafbc02ce-istiod-ca-cert" pod "bd8db90d-adb2-4601-bd83-5f8dafbc02ce" (UID: "bd8db90d-adb2-4601-bd83-5f8dafbc02ce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:39:54.265838 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:54.265818    4400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd8db90d-adb2-4601-bd83-5f8dafbc02ce-istiod-ca-cert\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:39:54.265912 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:54.265844    4400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd8db90d-adb2-4601-bd83-5f8dafbc02ce-workload-certs\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:39:54.265912 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:54.265857    4400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd8db90d-adb2-4601-bd83-5f8dafbc02ce-credential-socket\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:39:54.265912 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:54.265870    4400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd8db90d-adb2-4601-bd83-5f8dafbc02ce-istio-data\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:39:54.265912 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:54.265881    4400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd8db90d-adb2-4601-bd83-5f8dafbc02ce-workload-socket\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:39:54.266602 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:54.266575    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd8db90d-adb2-4601-bd83-5f8dafbc02ce-kube-api-access-ddxgr" pod "bd8db90d-adb2-4601-bd83-5f8dafbc02ce" (UID: "bd8db90d-adb2-4601-bd83-5f8dafbc02ce"). InnerVolumeSpecName "kube-api-access-ddxgr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:39:54.266728 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:54.266673    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bd8db90d-adb2-4601-bd83-5f8dafbc02ce-istio-podinfo" pod "bd8db90d-adb2-4601-bd83-5f8dafbc02ce" (UID: "bd8db90d-adb2-4601-bd83-5f8dafbc02ce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 03:39:54.267278 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:54.267258    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd8db90d-adb2-4601-bd83-5f8dafbc02ce-istio-envoy" pod "bd8db90d-adb2-4601-bd83-5f8dafbc02ce" (UID: "bd8db90d-adb2-4601-bd83-5f8dafbc02ce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:39:54.267443 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:54.267426    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd8db90d-adb2-4601-bd83-5f8dafbc02ce-istio-token" pod "bd8db90d-adb2-4601-bd83-5f8dafbc02ce" (UID: "bd8db90d-adb2-4601-bd83-5f8dafbc02ce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:39:54.367033 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:54.367000    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddxgr\" (UniqueName: \"kubernetes.io/projected/bd8db90d-adb2-4601-bd83-5f8dafbc02ce-kube-api-access-ddxgr\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:39:54.367033 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:54.367030    4400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd8db90d-adb2-4601-bd83-5f8dafbc02ce-istio-podinfo\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:39:54.367204 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:54.367043    4400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd8db90d-adb2-4601-bd83-5f8dafbc02ce-istio-token\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:39:54.367204 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:54.367056    4400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd8db90d-adb2-4601-bd83-5f8dafbc02ce-istio-envoy\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:39:54.367307 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:54.367265    4400 generic.go:356] "Generic (PLEG): container finished" podID="bd8db90d-adb2-4601-bd83-5f8dafbc02ce" containerID="d46c3b33c85ec5ae8946575290fa3b1536711ce6d58cf5e3094b59917a754555" exitCode=0
Nov 01 03:39:54.367356 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:54.367321    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-mp6pq"
Nov 01 03:39:54.367417 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:54.367393    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-mp6pq" event={"ID":"bd8db90d-adb2-4601-bd83-5f8dafbc02ce","Type":"ContainerDied","Data":"d46c3b33c85ec5ae8946575290fa3b1536711ce6d58cf5e3094b59917a754555"}
Nov 01 03:39:54.367465 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:54.367449    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-mp6pq" event={"ID":"bd8db90d-adb2-4601-bd83-5f8dafbc02ce","Type":"ContainerDied","Data":"7cf07daba7798e43432ef535f2de1cdda937afaff472c9862240318bcf4bcce3"}
Nov 01 03:39:54.367512 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:54.367469    4400 scope.go:122] "RemoveContainer" containerID="d46c3b33c85ec5ae8946575290fa3b1536711ce6d58cf5e3094b59917a754555"
Nov 01 03:39:54.379427 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:54.379412    4400 scope.go:122] "RemoveContainer" containerID="d46c3b33c85ec5ae8946575290fa3b1536711ce6d58cf5e3094b59917a754555"
Nov 01 03:39:54.379696 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:39:54.379667    4400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d46c3b33c85ec5ae8946575290fa3b1536711ce6d58cf5e3094b59917a754555\": container with ID starting with d46c3b33c85ec5ae8946575290fa3b1536711ce6d58cf5e3094b59917a754555 not found: ID does not exist" containerID="d46c3b33c85ec5ae8946575290fa3b1536711ce6d58cf5e3094b59917a754555"
Nov 01 03:39:54.379755 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:54.379729    4400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d46c3b33c85ec5ae8946575290fa3b1536711ce6d58cf5e3094b59917a754555"} err="failed to get container status \"d46c3b33c85ec5ae8946575290fa3b1536711ce6d58cf5e3094b59917a754555\": rpc error: code = NotFound desc = could not find container \"d46c3b33c85ec5ae8946575290fa3b1536711ce6d58cf5e3094b59917a754555\": container with ID starting with d46c3b33c85ec5ae8946575290fa3b1536711ce6d58cf5e3094b59917a754555 not found: ID does not exist"
Nov 01 03:39:54.600174 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:54.600143    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-mp6pq"]
Nov 01 03:39:54.606789 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:54.606764    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-mp6pq"]
Nov 01 03:39:55.625272 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:39:55.617442    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd8db90d-adb2-4601-bd83-5f8dafbc02ce" path="/var/lib/kubelet/pods/bd8db90d-adb2-4601-bd83-5f8dafbc02ce/volumes"
Nov 01 03:40:01.730336 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:40:01.730298    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:40:04.370572 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:40:04.370543    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-211380-7d964dd845-2blzj"
Nov 01 03:40:04.371082 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:40:04.370828    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-211380-598d8b59f-gs5lv"
Nov 01 03:40:09.925902 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:40:09.925873    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-da4f30-5d8cbd8644-kztdd"]
Nov 01 03:40:09.926655 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:40:09.926640    4400 cpu_manager.go:414] "containerMap: removing container" podUID="bd8db90d-adb2-4601-bd83-5f8dafbc02ce" containerName="istio-proxy"
Nov 01 03:40:09.926734 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:40:09.926656    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd8db90d-adb2-4601-bd83-5f8dafbc02ce" containerName="istio-proxy"
Nov 01 03:40:09.926792 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:40:09.926782    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd8db90d-adb2-4601-bd83-5f8dafbc02ce" containerName="istio-proxy"
Nov 01 03:40:09.930727 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:40:09.930711    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da4f30-5d8cbd8644-kztdd"
Nov 01 03:40:09.984604 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:40:09.984578    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-da4f30-serving-cert\""
Nov 01 03:40:09.995348 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:40:09.995325    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/110908f1-9498-4f67-95b3-75b3e9a45571-openshift-service-ca\") pod \"maas-api-e2e-api-b-da4f30-5d8cbd8644-kztdd\" (UID: \"110908f1-9498-4f67-95b3-75b3e9a45571\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da4f30-5d8cbd8644-kztdd"
Nov 01 03:40:09.995436 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:40:09.995361    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/110908f1-9498-4f67-95b3-75b3e9a45571-maas-api-tls\") pod \"maas-api-e2e-api-b-da4f30-5d8cbd8644-kztdd\" (UID: \"110908f1-9498-4f67-95b3-75b3e9a45571\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da4f30-5d8cbd8644-kztdd"
Nov 01 03:40:09.995492 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:40:09.995464    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/110908f1-9498-4f67-95b3-75b3e9a45571-metrics-certs\") pod \"maas-api-e2e-api-b-da4f30-5d8cbd8644-kztdd\" (UID: \"110908f1-9498-4f67-95b3-75b3e9a45571\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da4f30-5d8cbd8644-kztdd"
Nov 01 03:40:09.995635 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:40:09.995618    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lv2kx\" (UniqueName: \"kubernetes.io/projected/110908f1-9498-4f67-95b3-75b3e9a45571-kube-api-access-lv2kx\") pod \"maas-api-e2e-api-b-da4f30-5d8cbd8644-kztdd\" (UID: \"110908f1-9498-4f67-95b3-75b3e9a45571\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da4f30-5d8cbd8644-kztdd"
Nov 01 03:40:10.042252 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:40:10.042222    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-da4f30-5d8cbd8644-kztdd"]
Nov 01 03:40:10.097124 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:40:10.097100    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lv2kx\" (UniqueName: \"kubernetes.io/projected/110908f1-9498-4f67-95b3-75b3e9a45571-kube-api-access-lv2kx\") pod \"maas-api-e2e-api-b-da4f30-5d8cbd8644-kztdd\" (UID: \"110908f1-9498-4f67-95b3-75b3e9a45571\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da4f30-5d8cbd8644-kztdd"
Nov 01 03:40:10.097274 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:40:10.097245    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/110908f1-9498-4f67-95b3-75b3e9a45571-openshift-service-ca\") pod \"maas-api-e2e-api-b-da4f30-5d8cbd8644-kztdd\" (UID: \"110908f1-9498-4f67-95b3-75b3e9a45571\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da4f30-5d8cbd8644-kztdd"
Nov 01 03:40:10.097274 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:40:10.097271    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/110908f1-9498-4f67-95b3-75b3e9a45571-maas-api-tls\") pod \"maas-api-e2e-api-b-da4f30-5d8cbd8644-kztdd\" (UID: \"110908f1-9498-4f67-95b3-75b3e9a45571\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da4f30-5d8cbd8644-kztdd"
Nov 01 03:40:10.097388 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:40:10.097313    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/110908f1-9498-4f67-95b3-75b3e9a45571-metrics-certs\") pod \"maas-api-e2e-api-b-da4f30-5d8cbd8644-kztdd\" (UID: \"110908f1-9498-4f67-95b3-75b3e9a45571\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da4f30-5d8cbd8644-kztdd"
Nov 01 03:40:10.097443 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:40:10.097409    4400 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-da4f30-serving-cert: secret "maas-api-e2e-api-b-da4f30-serving-cert" not found
Nov 01 03:40:10.097516 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:40:10.097504    4400 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/110908f1-9498-4f67-95b3-75b3e9a45571-maas-api-tls podName:110908f1-9498-4f67-95b3-75b3e9a45571 nodeName:}" failed. No retries permitted until 2026-11-01 03:40:10.597489429 +0000 UTC m=+2017.112135285 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/110908f1-9498-4f67-95b3-75b3e9a45571-maas-api-tls") pod "maas-api-e2e-api-b-da4f30-5d8cbd8644-kztdd" (UID: "110908f1-9498-4f67-95b3-75b3e9a45571") : secret "maas-api-e2e-api-b-da4f30-serving-cert" not found
Nov 01 03:40:10.097969 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:40:10.097942    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/110908f1-9498-4f67-95b3-75b3e9a45571-openshift-service-ca\") pod \"maas-api-e2e-api-b-da4f30-5d8cbd8644-kztdd\" (UID: \"110908f1-9498-4f67-95b3-75b3e9a45571\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da4f30-5d8cbd8644-kztdd"
Nov 01 03:40:10.099866 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:40:10.099846    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/110908f1-9498-4f67-95b3-75b3e9a45571-metrics-certs\") pod \"maas-api-e2e-api-b-da4f30-5d8cbd8644-kztdd\" (UID: \"110908f1-9498-4f67-95b3-75b3e9a45571\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da4f30-5d8cbd8644-kztdd"
Nov 01 03:40:10.159980 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:40:10.159958    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lv2kx\" (UniqueName: \"kubernetes.io/projected/110908f1-9498-4f67-95b3-75b3e9a45571-kube-api-access-lv2kx\") pod \"maas-api-e2e-api-b-da4f30-5d8cbd8644-kztdd\" (UID: \"110908f1-9498-4f67-95b3-75b3e9a45571\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da4f30-5d8cbd8644-kztdd"
Nov 01 03:40:10.635334 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:40:10.635304    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/110908f1-9498-4f67-95b3-75b3e9a45571-maas-api-tls\") pod \"maas-api-e2e-api-b-da4f30-5d8cbd8644-kztdd\" (UID: \"110908f1-9498-4f67-95b3-75b3e9a45571\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da4f30-5d8cbd8644-kztdd"
Nov 01 03:40:10.637795 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:40:10.637759    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/110908f1-9498-4f67-95b3-75b3e9a45571-maas-api-tls\") pod \"maas-api-e2e-api-b-da4f30-5d8cbd8644-kztdd\" (UID: \"110908f1-9498-4f67-95b3-75b3e9a45571\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da4f30-5d8cbd8644-kztdd"
Nov 01 03:40:10.845248 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:40:10.845222    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da4f30-5d8cbd8644-kztdd"
Nov 01 03:40:11.225702 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:40:11.225655    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod110908f1_9498_4f67_95b3_75b3e9a45571.slice/crio-c22184a0f4286a21cfdd1d04f772bf69047fdd78da2fb860ba29409c931ab365 WatchSource:0}: Error finding container c22184a0f4286a21cfdd1d04f772bf69047fdd78da2fb860ba29409c931ab365: Status 404 returned error can't find the container with id c22184a0f4286a21cfdd1d04f772bf69047fdd78da2fb860ba29409c931ab365
Nov 01 03:40:11.226339 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:40:11.226321    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-da4f30-5d8cbd8644-kztdd"]
Nov 01 03:40:11.227231 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:40:11.227205    4400 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 03:40:11.574890 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:40:11.574862    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da4f30-5d8cbd8644-kztdd" event={"ID":"110908f1-9498-4f67-95b3-75b3e9a45571","Type":"ContainerStarted","Data":"c22184a0f4286a21cfdd1d04f772bf69047fdd78da2fb860ba29409c931ab365"}
Nov 01 03:40:12.584056 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:40:12.584026    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da4f30-5d8cbd8644-kztdd" event={"ID":"110908f1-9498-4f67-95b3-75b3e9a45571","Type":"ContainerStarted","Data":"d096d99affa4629c52253654eda4c14d54af895c25ef01d71a104696fea57890"}
Nov 01 03:40:12.584547 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:40:12.584098    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da4f30-5d8cbd8644-kztdd"
Nov 01 03:40:12.710891 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:40:12.710846    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da4f30-5d8cbd8644-kztdd" podStartSLOduration=2.716173466 podStartE2EDuration="3.710833865s" totalImagesPullingTime="994.660399ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:40:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:40:12.706966401 +0000 UTC m=+2019.221612262" watchObservedRunningTime="2026-11-01 03:40:12.710833865 +0000 UTC m=+2019.225479775"
Nov 01 03:40:18.597360 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:40:18.597289    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da4f30-5d8cbd8644-kztdd"
Nov 01 03:40:23.196122 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:40:23.196090    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7d7329-fcbf4bdc7-r8q86"]
Nov 01 03:40:23.308226 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:40:23.308184    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7d7329-87fbf9b5f-chwdx"]
Nov 01 03:40:28.197172 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:40:28.197136    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-7d7329-fcbf4bdc7-r8q86" podUID="cb66c491-eeb8-40f2-b553-669555c43b95" containerName="payload-processing" containerID="cri-o://29004d2109dab3557c825638be974c869cdd800fd635d4f11e590241512d834b" gracePeriod=25
Nov 01 03:40:28.308781 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:40:28.308753    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7d7329-87fbf9b5f-chwdx" podUID="0a527c89-9499-4a79-8b3f-176e206c3cde" containerName="payload-pre-processing" containerID="cri-o://4be5291c7f7cd4bc8ecff93420d13605b0a0e60bc7be86519908a7c536f7dc29" gracePeriod=25
Nov 01 03:40:28.488101 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:40:28.488072    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-7d7329-fcbf4bdc7-r8q86"
Nov 01 03:40:28.581631 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:40:28.581608    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb66c491-eeb8-40f2-b553-669555c43b95-kube-api-access-q9mjn\" (UniqueName: \"kubernetes.io/projected/cb66c491-eeb8-40f2-b553-669555c43b95-kube-api-access-q9mjn\") pod \"cb66c491-eeb8-40f2-b553-669555c43b95\" (UID: \"cb66c491-eeb8-40f2-b553-669555c43b95\") "
Nov 01 03:40:28.581833 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:40:28.581820    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb66c491-eeb8-40f2-b553-669555c43b95-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb66c491-eeb8-40f2-b553-669555c43b95-plugins-config-volume\") pod \"cb66c491-eeb8-40f2-b553-669555c43b95\" (UID: \"cb66c491-eeb8-40f2-b553-669555c43b95\") "
Nov 01 03:40:28.582149 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:40:28.582132    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb66c491-eeb8-40f2-b553-669555c43b95-plugins-config-volume" pod "cb66c491-eeb8-40f2-b553-669555c43b95" (UID: "cb66c491-eeb8-40f2-b553-669555c43b95"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:40:28.582402 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:40:28.582384    4400 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb66c491-eeb8-40f2-b553-669555c43b95-plugins-config-volume\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:40:28.583909 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:40:28.583890    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb66c491-eeb8-40f2-b553-669555c43b95-kube-api-access-q9mjn" pod "cb66c491-eeb8-40f2-b553-669555c43b95" (UID: "cb66c491-eeb8-40f2-b553-669555c43b95"). InnerVolumeSpecName "kube-api-access-q9mjn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:40:28.588487 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:40:28.588468    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7d7329-87fbf9b5f-chwdx"
Nov 01 03:40:28.683017 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:40:28.682992    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a527c89-9499-4a79-8b3f-176e206c3cde-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a527c89-9499-4a79-8b3f-176e206c3cde-plugins-config-volume\") pod \"0a527c89-9499-4a79-8b3f-176e206c3cde\" (UID: \"0a527c89-9499-4a79-8b3f-176e206c3cde\") "
Nov 01 03:40:28.683209 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:40:28.683198    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a527c89-9499-4a79-8b3f-176e206c3cde-kube-api-access-kwfnh\" (UniqueName: \"kubernetes.io/projected/0a527c89-9499-4a79-8b3f-176e206c3cde-kube-api-access-kwfnh\") pod \"0a527c89-9499-4a79-8b3f-176e206c3cde\" (UID: \"0a527c89-9499-4a79-8b3f-176e206c3cde\") "
Nov 01 03:40:28.683327 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:40:28.683307    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a527c89-9499-4a79-8b3f-176e206c3cde-plugins-config-volume" pod "0a527c89-9499-4a79-8b3f-176e206c3cde" (UID: "0a527c89-9499-4a79-8b3f-176e206c3cde"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:40:28.683721 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:40:28.683698    4400 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a527c89-9499-4a79-8b3f-176e206c3cde-plugins-config-volume\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:40:28.683829 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:40:28.683725    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9mjn\" (UniqueName: \"kubernetes.io/projected/cb66c491-eeb8-40f2-b553-669555c43b95-kube-api-access-q9mjn\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:40:28.685524 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:40:28.685499    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a527c89-9499-4a79-8b3f-176e206c3cde-kube-api-access-kwfnh" pod "0a527c89-9499-4a79-8b3f-176e206c3cde" (UID: "0a527c89-9499-4a79-8b3f-176e206c3cde"). InnerVolumeSpecName "kube-api-access-kwfnh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:40:28.723511 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:40:28.723454    4400 generic.go:356] "Generic (PLEG): container finished" podID="cb66c491-eeb8-40f2-b553-669555c43b95" containerID="29004d2109dab3557c825638be974c869cdd800fd635d4f11e590241512d834b" exitCode=0
Nov 01 03:40:28.723511 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:40:28.723500    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-7d7329-fcbf4bdc7-r8q86"
Nov 01 03:40:28.723630 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:40:28.723535    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-7d7329-fcbf4bdc7-r8q86" event={"ID":"cb66c491-eeb8-40f2-b553-669555c43b95","Type":"ContainerDied","Data":"29004d2109dab3557c825638be974c869cdd800fd635d4f11e590241512d834b"}
Nov 01 03:40:28.723630 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:40:28.723584    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-7d7329-fcbf4bdc7-r8q86" event={"ID":"cb66c491-eeb8-40f2-b553-669555c43b95","Type":"ContainerDied","Data":"59883414818ab4cb893541edb1740923af7681639951a882fde2e2bbda9e870c"}
Nov 01 03:40:28.723630 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:40:28.723604    4400 scope.go:122] "RemoveContainer" containerID="29004d2109dab3557c825638be974c869cdd800fd635d4f11e590241512d834b"
Nov 01 03:40:28.726908 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:40:28.726865    4400 generic.go:356] "Generic (PLEG): container finished" podID="0a527c89-9499-4a79-8b3f-176e206c3cde" containerID="4be5291c7f7cd4bc8ecff93420d13605b0a0e60bc7be86519908a7c536f7dc29" exitCode=0
Nov 01 03:40:28.727020 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:40:28.726921    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7d7329-87fbf9b5f-chwdx" event={"ID":"0a527c89-9499-4a79-8b3f-176e206c3cde","Type":"ContainerDied","Data":"4be5291c7f7cd4bc8ecff93420d13605b0a0e60bc7be86519908a7c536f7dc29"}
Nov 01 03:40:28.727020 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:40:28.726957    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7d7329-87fbf9b5f-chwdx" event={"ID":"0a527c89-9499-4a79-8b3f-176e206c3cde","Type":"ContainerDied","Data":"ecdd49c4048a5bca8905c712d210d74cb778582efe17d0c6379ecb2f3e69188b"}
Nov 01 03:40:28.727020 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:40:28.727006    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7d7329-87fbf9b5f-chwdx"
Nov 01 03:40:28.739087 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:40:28.739071    4400 scope.go:122] "RemoveContainer" containerID="29004d2109dab3557c825638be974c869cdd800fd635d4f11e590241512d834b"
Nov 01 03:40:28.739387 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:40:28.739364    4400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29004d2109dab3557c825638be974c869cdd800fd635d4f11e590241512d834b\": container with ID starting with 29004d2109dab3557c825638be974c869cdd800fd635d4f11e590241512d834b not found: ID does not exist" containerID="29004d2109dab3557c825638be974c869cdd800fd635d4f11e590241512d834b"
Nov 01 03:40:28.739480 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:40:28.739392    4400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29004d2109dab3557c825638be974c869cdd800fd635d4f11e590241512d834b"} err="failed to get container status \"29004d2109dab3557c825638be974c869cdd800fd635d4f11e590241512d834b\": rpc error: code = NotFound desc = could not find container \"29004d2109dab3557c825638be974c869cdd800fd635d4f11e590241512d834b\": container with ID starting with 29004d2109dab3557c825638be974c869cdd800fd635d4f11e590241512d834b not found: ID does not exist"
Nov 01 03:40:28.739480 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:40:28.739410    4400 scope.go:122] "RemoveContainer" containerID="4be5291c7f7cd4bc8ecff93420d13605b0a0e60bc7be86519908a7c536f7dc29"
Nov 01 03:40:28.752436 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:40:28.752402    4400 scope.go:122] "RemoveContainer" containerID="4be5291c7f7cd4bc8ecff93420d13605b0a0e60bc7be86519908a7c536f7dc29"
Nov 01 03:40:28.752834 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:40:28.752811    4400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4be5291c7f7cd4bc8ecff93420d13605b0a0e60bc7be86519908a7c536f7dc29\": container with ID starting with 4be5291c7f7cd4bc8ecff93420d13605b0a0e60bc7be86519908a7c536f7dc29 not found: ID does not exist" containerID="4be5291c7f7cd4bc8ecff93420d13605b0a0e60bc7be86519908a7c536f7dc29"
Nov 01 03:40:28.752973 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:40:28.752944    4400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4be5291c7f7cd4bc8ecff93420d13605b0a0e60bc7be86519908a7c536f7dc29"} err="failed to get container status \"4be5291c7f7cd4bc8ecff93420d13605b0a0e60bc7be86519908a7c536f7dc29\": rpc error: code = NotFound desc = could not find container \"4be5291c7f7cd4bc8ecff93420d13605b0a0e60bc7be86519908a7c536f7dc29\": container with ID starting with 4be5291c7f7cd4bc8ecff93420d13605b0a0e60bc7be86519908a7c536f7dc29 not found: ID does not exist"
Nov 01 03:40:28.784593 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:40:28.784574    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwfnh\" (UniqueName: \"kubernetes.io/projected/0a527c89-9499-4a79-8b3f-176e206c3cde-kube-api-access-kwfnh\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:40:28.894505 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:40:28.894480    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7d7329-fcbf4bdc7-r8q86"]
Nov 01 03:40:28.898967 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:40:28.898950    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7d7329-fcbf4bdc7-r8q86"]
Nov 01 03:40:29.009849 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:40:29.009798    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7d7329-87fbf9b5f-chwdx"]
Nov 01 03:40:29.016543 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:40:29.016507    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7d7329-87fbf9b5f-chwdx"]
Nov 01 03:40:29.575303 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:40:29.575273    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7d7329-87fbf9b5f-chwdx" podUID="0a527c89-9499-4a79-8b3f-176e206c3cde" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.113:50052\" within 1s: context deadline exceeded"
Nov 01 03:40:29.575649 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:40:29.575382    4400 logging.go:55] [core] [Channel #5108 SubChannel #5109]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.113:50052", ServerName: "10.132.0.113:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.113:50052: operation was canceled"
Nov 01 03:40:29.616346 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:40:29.616321    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a527c89-9499-4a79-8b3f-176e206c3cde" path="/var/lib/kubelet/pods/0a527c89-9499-4a79-8b3f-176e206c3cde/volumes"
Nov 01 03:40:29.616786 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:40:29.616772    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb66c491-eeb8-40f2-b553-669555c43b95" path="/var/lib/kubelet/pods/cb66c491-eeb8-40f2-b553-669555c43b95/volumes"
Nov 01 03:40:31.730825 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:40:31.730792    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:40:38.945727 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:40:38.945691    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e8c77d-7cdd9d68dd-hmjst"]
Nov 01 03:40:38.946271 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:40:38.945932    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e8c77d-7cdd9d68dd-hmjst" podUID="ad9a53c5-5164-42e6-8a89-69666c43dec0" containerName="maas-api" containerID="cri-o://9718e281d20654848880ce6a6470394c9067c978005b2e1e2415da47a36e62f5" gracePeriod=30
Nov 01 03:40:39.330713 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:40:39.330674    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e8c77d-7cdd9d68dd-hmjst"
Nov 01 03:40:39.417711 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:40:39.391715    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad9a53c5-5164-42e6-8a89-69666c43dec0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad9a53c5-5164-42e6-8a89-69666c43dec0-maas-api-tls\") pod \"ad9a53c5-5164-42e6-8a89-69666c43dec0\" (UID: \"ad9a53c5-5164-42e6-8a89-69666c43dec0\") "
Nov 01 03:40:39.445715 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:40:39.411102    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad9a53c5-5164-42e6-8a89-69666c43dec0-maas-api-tls" pod "ad9a53c5-5164-42e6-8a89-69666c43dec0" (UID: "ad9a53c5-5164-42e6-8a89-69666c43dec0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:40:39.445715 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:40:39.433530    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad9a53c5-5164-42e6-8a89-69666c43dec0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad9a53c5-5164-42e6-8a89-69666c43dec0-metrics-certs\") pod \"ad9a53c5-5164-42e6-8a89-69666c43dec0\" (UID: \"ad9a53c5-5164-42e6-8a89-69666c43dec0\") "
Nov 01 03:40:39.445715 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:40:39.433572    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad9a53c5-5164-42e6-8a89-69666c43dec0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad9a53c5-5164-42e6-8a89-69666c43dec0-openshift-service-ca\") pod \"ad9a53c5-5164-42e6-8a89-69666c43dec0\" (UID: \"ad9a53c5-5164-42e6-8a89-69666c43dec0\") "
Nov 01 03:40:39.445715 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:40:39.433709    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad9a53c5-5164-42e6-8a89-69666c43dec0-kube-api-access-hzc55\" (UniqueName: \"kubernetes.io/projected/ad9a53c5-5164-42e6-8a89-69666c43dec0-kube-api-access-hzc55\") pod \"ad9a53c5-5164-42e6-8a89-69666c43dec0\" (UID: \"ad9a53c5-5164-42e6-8a89-69666c43dec0\") "
Nov 01 03:40:39.445715 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:40:39.434604    4400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad9a53c5-5164-42e6-8a89-69666c43dec0-maas-api-tls\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:40:39.481709 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:40:39.453815    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad9a53c5-5164-42e6-8a89-69666c43dec0-kube-api-access-hzc55" pod "ad9a53c5-5164-42e6-8a89-69666c43dec0" (UID: "ad9a53c5-5164-42e6-8a89-69666c43dec0"). InnerVolumeSpecName "kube-api-access-hzc55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:40:39.481709 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:40:39.456349    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad9a53c5-5164-42e6-8a89-69666c43dec0-metrics-certs" pod "ad9a53c5-5164-42e6-8a89-69666c43dec0" (UID: "ad9a53c5-5164-42e6-8a89-69666c43dec0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:40:39.506301 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:40:39.506268    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad9a53c5-5164-42e6-8a89-69666c43dec0-openshift-service-ca" pod "ad9a53c5-5164-42e6-8a89-69666c43dec0" (UID: "ad9a53c5-5164-42e6-8a89-69666c43dec0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:40:39.537068 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:40:39.536756    4400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad9a53c5-5164-42e6-8a89-69666c43dec0-metrics-certs\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:40:39.537068 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:40:39.536792    4400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad9a53c5-5164-42e6-8a89-69666c43dec0-openshift-service-ca\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:40:39.537068 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:40:39.536808    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzc55\" (UniqueName: \"kubernetes.io/projected/ad9a53c5-5164-42e6-8a89-69666c43dec0-kube-api-access-hzc55\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:40:39.806793 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:40:39.806764    4400 generic.go:356] "Generic (PLEG): container finished" podID="ad9a53c5-5164-42e6-8a89-69666c43dec0" containerID="9718e281d20654848880ce6a6470394c9067c978005b2e1e2415da47a36e62f5" exitCode=0
Nov 01 03:40:39.806931 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:40:39.806824    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e8c77d-7cdd9d68dd-hmjst"
Nov 01 03:40:39.806931 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:40:39.806846    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e8c77d-7cdd9d68dd-hmjst" event={"ID":"ad9a53c5-5164-42e6-8a89-69666c43dec0","Type":"ContainerDied","Data":"9718e281d20654848880ce6a6470394c9067c978005b2e1e2415da47a36e62f5"}
Nov 01 03:40:39.806931 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:40:39.806897    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e8c77d-7cdd9d68dd-hmjst" event={"ID":"ad9a53c5-5164-42e6-8a89-69666c43dec0","Type":"ContainerDied","Data":"cc5f3dc72e8268365028c7aeb173cd882d61644408164caeb67927fdc1369486"}
Nov 01 03:40:39.806931 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:40:39.806919    4400 scope.go:122] "RemoveContainer" containerID="9718e281d20654848880ce6a6470394c9067c978005b2e1e2415da47a36e62f5"
Nov 01 03:40:39.817071 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:40:39.817052    4400 scope.go:122] "RemoveContainer" containerID="9718e281d20654848880ce6a6470394c9067c978005b2e1e2415da47a36e62f5"
Nov 01 03:40:39.817332 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:40:39.817313    4400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9718e281d20654848880ce6a6470394c9067c978005b2e1e2415da47a36e62f5\": container with ID starting with 9718e281d20654848880ce6a6470394c9067c978005b2e1e2415da47a36e62f5 not found: ID does not exist" containerID="9718e281d20654848880ce6a6470394c9067c978005b2e1e2415da47a36e62f5"
Nov 01 03:40:39.817383 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:40:39.817338    4400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9718e281d20654848880ce6a6470394c9067c978005b2e1e2415da47a36e62f5"} err="failed to get container status \"9718e281d20654848880ce6a6470394c9067c978005b2e1e2415da47a36e62f5\": rpc error: code = NotFound desc = could not find container \"9718e281d20654848880ce6a6470394c9067c978005b2e1e2415da47a36e62f5\": container with ID starting with 9718e281d20654848880ce6a6470394c9067c978005b2e1e2415da47a36e62f5 not found: ID does not exist"
Nov 01 03:40:40.019705 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:40:40.013612    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e8c77d-7cdd9d68dd-hmjst"]
Nov 01 03:40:40.047658 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:40:40.047581    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e8c77d-7cdd9d68dd-hmjst"]
Nov 01 03:40:41.615357 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:40:41.615325    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad9a53c5-5164-42e6-8a89-69666c43dec0" path="/var/lib/kubelet/pods/ad9a53c5-5164-42e6-8a89-69666c43dec0/volumes"
Nov 01 03:40:45.624080 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:40:45.624053    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 03:41:01.730416 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:41:01.730379    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:41:02.881935 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:02.881906    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-211380-5b58f48f46-7r8nn"]
Nov 01 03:41:02.882635 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:02.882620    4400 cpu_manager.go:414] "containerMap: removing container" podUID="ad9a53c5-5164-42e6-8a89-69666c43dec0" containerName="maas-api"
Nov 01 03:41:02.882676 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:02.882636    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad9a53c5-5164-42e6-8a89-69666c43dec0" containerName="maas-api"
Nov 01 03:41:02.882676 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:02.882665    4400 cpu_manager.go:414] "containerMap: removing container" podUID="0a527c89-9499-4a79-8b3f-176e206c3cde" containerName="payload-pre-processing"
Nov 01 03:41:02.882676 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:02.882670    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a527c89-9499-4a79-8b3f-176e206c3cde" containerName="payload-pre-processing"
Nov 01 03:41:02.882803 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:02.882706    4400 cpu_manager.go:414] "containerMap: removing container" podUID="cb66c491-eeb8-40f2-b553-669555c43b95" containerName="payload-processing"
Nov 01 03:41:02.882803 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:02.882711    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb66c491-eeb8-40f2-b553-669555c43b95" containerName="payload-processing"
Nov 01 03:41:02.882860 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:02.882813    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb66c491-eeb8-40f2-b553-669555c43b95" containerName="payload-processing"
Nov 01 03:41:02.882860 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:02.882828    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad9a53c5-5164-42e6-8a89-69666c43dec0" containerName="maas-api"
Nov 01 03:41:02.882860 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:02.882846    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a527c89-9499-4a79-8b3f-176e206c3cde" containerName="payload-pre-processing"
Nov 01 03:41:02.886702 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:02.886670    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-211380-5b58f48f46-7r8nn"
Nov 01 03:41:02.940475 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:02.940448    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w6-211380-serving-cert\""
Nov 01 03:41:02.976228 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:02.976200    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21b19cc7-731a-45fb-b864-da76b9b41be0-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-211380-5b58f48f46-7r8nn\" (UID: \"21b19cc7-731a-45fb-b864-da76b9b41be0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-211380-5b58f48f46-7r8nn"
Nov 01 03:41:02.976400 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:02.976379    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkztn\" (UniqueName: \"kubernetes.io/projected/21b19cc7-731a-45fb-b864-da76b9b41be0-kube-api-access-pkztn\") pod \"maas-api-e2e-shared-b-w6-211380-5b58f48f46-7r8nn\" (UID: \"21b19cc7-731a-45fb-b864-da76b9b41be0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-211380-5b58f48f46-7r8nn"
Nov 01 03:41:02.976559 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:02.976543    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21b19cc7-731a-45fb-b864-da76b9b41be0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-211380-5b58f48f46-7r8nn\" (UID: \"21b19cc7-731a-45fb-b864-da76b9b41be0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-211380-5b58f48f46-7r8nn"
Nov 01 03:41:02.976649 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:02.976586    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21b19cc7-731a-45fb-b864-da76b9b41be0-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-211380-5b58f48f46-7r8nn\" (UID: \"21b19cc7-731a-45fb-b864-da76b9b41be0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-211380-5b58f48f46-7r8nn"
Nov 01 03:41:02.995754 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:02.995718    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-211380-5b58f48f46-7r8nn"]
Nov 01 03:41:03.077313 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:03.077283    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21b19cc7-731a-45fb-b864-da76b9b41be0-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-211380-5b58f48f46-7r8nn\" (UID: \"21b19cc7-731a-45fb-b864-da76b9b41be0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-211380-5b58f48f46-7r8nn"
Nov 01 03:41:03.077454 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:03.077361    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21b19cc7-731a-45fb-b864-da76b9b41be0-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-211380-5b58f48f46-7r8nn\" (UID: \"21b19cc7-731a-45fb-b864-da76b9b41be0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-211380-5b58f48f46-7r8nn"
Nov 01 03:41:03.077454 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:41:03.077415    4400 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-211380-serving-cert: secret "maas-api-e2e-shared-b-w6-211380-serving-cert" not found
Nov 01 03:41:03.077594 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:03.077452    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkztn\" (UniqueName: \"kubernetes.io/projected/21b19cc7-731a-45fb-b864-da76b9b41be0-kube-api-access-pkztn\") pod \"maas-api-e2e-shared-b-w6-211380-5b58f48f46-7r8nn\" (UID: \"21b19cc7-731a-45fb-b864-da76b9b41be0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-211380-5b58f48f46-7r8nn"
Nov 01 03:41:03.077594 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:41:03.077480    4400 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21b19cc7-731a-45fb-b864-da76b9b41be0-maas-api-tls podName:21b19cc7-731a-45fb-b864-da76b9b41be0 nodeName:}" failed. No retries permitted until 2026-11-01 03:41:03.577464496 +0000 UTC m=+2070.092110340 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/21b19cc7-731a-45fb-b864-da76b9b41be0-maas-api-tls") pod "maas-api-e2e-shared-b-w6-211380-5b58f48f46-7r8nn" (UID: "21b19cc7-731a-45fb-b864-da76b9b41be0") : secret "maas-api-e2e-shared-b-w6-211380-serving-cert" not found
Nov 01 03:41:03.077668 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:03.077597    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21b19cc7-731a-45fb-b864-da76b9b41be0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-211380-5b58f48f46-7r8nn\" (UID: \"21b19cc7-731a-45fb-b864-da76b9b41be0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-211380-5b58f48f46-7r8nn"
Nov 01 03:41:03.078177 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:03.078159    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21b19cc7-731a-45fb-b864-da76b9b41be0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-211380-5b58f48f46-7r8nn\" (UID: \"21b19cc7-731a-45fb-b864-da76b9b41be0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-211380-5b58f48f46-7r8nn"
Nov 01 03:41:03.079703 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:03.079672    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21b19cc7-731a-45fb-b864-da76b9b41be0-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-211380-5b58f48f46-7r8nn\" (UID: \"21b19cc7-731a-45fb-b864-da76b9b41be0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-211380-5b58f48f46-7r8nn"
Nov 01 03:41:03.152376 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:03.151092    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkztn\" (UniqueName: \"kubernetes.io/projected/21b19cc7-731a-45fb-b864-da76b9b41be0-kube-api-access-pkztn\") pod \"maas-api-e2e-shared-b-w6-211380-5b58f48f46-7r8nn\" (UID: \"21b19cc7-731a-45fb-b864-da76b9b41be0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-211380-5b58f48f46-7r8nn"
Nov 01 03:41:03.589252 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:03.589209    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21b19cc7-731a-45fb-b864-da76b9b41be0-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-211380-5b58f48f46-7r8nn\" (UID: \"21b19cc7-731a-45fb-b864-da76b9b41be0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-211380-5b58f48f46-7r8nn"
Nov 01 03:41:03.662710 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:03.655777    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21b19cc7-731a-45fb-b864-da76b9b41be0-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-211380-5b58f48f46-7r8nn\" (UID: \"21b19cc7-731a-45fb-b864-da76b9b41be0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-211380-5b58f48f46-7r8nn"
Nov 01 03:41:03.813709 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:03.808595    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-211380-5b58f48f46-7r8nn"
Nov 01 03:41:04.359547 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:04.359507    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-211380-5b58f48f46-7r8nn"]
Nov 01 03:41:04.361946 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:41:04.361918    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21b19cc7_731a_45fb_b864_da76b9b41be0.slice/crio-8d6d4e6b0fd6c347263f9ce4308d48eac8a1750266489ae7f0b4b8249b721822 WatchSource:0}: Error finding container 8d6d4e6b0fd6c347263f9ce4308d48eac8a1750266489ae7f0b4b8249b721822: Status 404 returned error can't find the container with id 8d6d4e6b0fd6c347263f9ce4308d48eac8a1750266489ae7f0b4b8249b721822
Nov 01 03:41:05.056611 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:05.056493    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-211380-5b58f48f46-7r8nn" event={"ID":"21b19cc7-731a-45fb-b864-da76b9b41be0","Type":"ContainerStarted","Data":"8d6d4e6b0fd6c347263f9ce4308d48eac8a1750266489ae7f0b4b8249b721822"}
Nov 01 03:41:06.065463 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:06.065432    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-211380-5b58f48f46-7r8nn" event={"ID":"21b19cc7-731a-45fb-b864-da76b9b41be0","Type":"ContainerStarted","Data":"8d161bda216feb0fc151394f7b0bdb6eca47125ba1651f0101f06762bfc19049"}
Nov 01 03:41:06.065911 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:06.065579    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-211380-5b58f48f46-7r8nn"
Nov 01 03:41:06.187633 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:06.187587    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-211380-5b58f48f46-7r8nn" podStartSLOduration=3.363023994 podStartE2EDuration="4.187574562s" totalImagesPullingTime="824.550568ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:41:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:41:06.185886941 +0000 UTC m=+2072.700532801" watchObservedRunningTime="2026-11-01 03:41:06.187574562 +0000 UTC m=+2072.702220422"
Nov 01 03:41:07.352638 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:07.352608    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-634332-854c684b9f-8qg4v"]
Nov 01 03:41:07.353079 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:07.352863    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-634332-854c684b9f-8qg4v" podUID="9b174e47-5aad-474b-a5cf-4fe09a417fe3" containerName="maas-api" containerID="cri-o://0865bc9bdd2810bcef05e3b89a22bea7e239ba9bd4463b224262ce5ad5cc86a9" gracePeriod=30
Nov 01 03:41:07.772360 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:07.772052    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-634332-854c684b9f-8qg4v"
Nov 01 03:41:07.808993 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:07.808958    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b174e47-5aad-474b-a5cf-4fe09a417fe3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b174e47-5aad-474b-a5cf-4fe09a417fe3-metrics-certs\") pod \"9b174e47-5aad-474b-a5cf-4fe09a417fe3\" (UID: \"9b174e47-5aad-474b-a5cf-4fe09a417fe3\") "
Nov 01 03:41:07.809662 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:07.809640    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b174e47-5aad-474b-a5cf-4fe09a417fe3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b174e47-5aad-474b-a5cf-4fe09a417fe3-openshift-service-ca\") pod \"9b174e47-5aad-474b-a5cf-4fe09a417fe3\" (UID: \"9b174e47-5aad-474b-a5cf-4fe09a417fe3\") "
Nov 01 03:41:07.810311 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:07.810295    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b174e47-5aad-474b-a5cf-4fe09a417fe3-kube-api-access-8bk44\" (UniqueName: \"kubernetes.io/projected/9b174e47-5aad-474b-a5cf-4fe09a417fe3-kube-api-access-8bk44\") pod \"9b174e47-5aad-474b-a5cf-4fe09a417fe3\" (UID: \"9b174e47-5aad-474b-a5cf-4fe09a417fe3\") "
Nov 01 03:41:07.810888 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:07.810867    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b174e47-5aad-474b-a5cf-4fe09a417fe3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b174e47-5aad-474b-a5cf-4fe09a417fe3-maas-api-tls\") pod \"9b174e47-5aad-474b-a5cf-4fe09a417fe3\" (UID: \"9b174e47-5aad-474b-a5cf-4fe09a417fe3\") "
Nov 01 03:41:07.811419 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:07.810176    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b174e47-5aad-474b-a5cf-4fe09a417fe3-openshift-service-ca" pod "9b174e47-5aad-474b-a5cf-4fe09a417fe3" (UID: "9b174e47-5aad-474b-a5cf-4fe09a417fe3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:41:07.812087 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:07.812062    4400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b174e47-5aad-474b-a5cf-4fe09a417fe3-openshift-service-ca\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:41:07.872190 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:07.871469    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b174e47-5aad-474b-a5cf-4fe09a417fe3-maas-api-tls" pod "9b174e47-5aad-474b-a5cf-4fe09a417fe3" (UID: "9b174e47-5aad-474b-a5cf-4fe09a417fe3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:41:07.872995 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:07.872950    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b174e47-5aad-474b-a5cf-4fe09a417fe3-metrics-certs" pod "9b174e47-5aad-474b-a5cf-4fe09a417fe3" (UID: "9b174e47-5aad-474b-a5cf-4fe09a417fe3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:41:07.890957 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:07.890917    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b174e47-5aad-474b-a5cf-4fe09a417fe3-kube-api-access-8bk44" pod "9b174e47-5aad-474b-a5cf-4fe09a417fe3" (UID: "9b174e47-5aad-474b-a5cf-4fe09a417fe3"). InnerVolumeSpecName "kube-api-access-8bk44". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:41:07.913470 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:07.913441    4400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b174e47-5aad-474b-a5cf-4fe09a417fe3-maas-api-tls\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:41:07.913470 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:07.913471    4400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b174e47-5aad-474b-a5cf-4fe09a417fe3-metrics-certs\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:41:07.913670 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:07.913486    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bk44\" (UniqueName: \"kubernetes.io/projected/9b174e47-5aad-474b-a5cf-4fe09a417fe3-kube-api-access-8bk44\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:41:08.113835 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:08.113806    4400 generic.go:356] "Generic (PLEG): container finished" podID="9b174e47-5aad-474b-a5cf-4fe09a417fe3" containerID="0865bc9bdd2810bcef05e3b89a22bea7e239ba9bd4463b224262ce5ad5cc86a9" exitCode=0
Nov 01 03:41:08.114019 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:08.113861    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-634332-854c684b9f-8qg4v"
Nov 01 03:41:08.114019 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:08.113892    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-634332-854c684b9f-8qg4v" event={"ID":"9b174e47-5aad-474b-a5cf-4fe09a417fe3","Type":"ContainerDied","Data":"0865bc9bdd2810bcef05e3b89a22bea7e239ba9bd4463b224262ce5ad5cc86a9"}
Nov 01 03:41:08.114019 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:08.113952    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-634332-854c684b9f-8qg4v" event={"ID":"9b174e47-5aad-474b-a5cf-4fe09a417fe3","Type":"ContainerDied","Data":"8dc19fb363b2556a3642e1c6a0ab5f20f2fc8ef13623b55f9996744da2ba8363"}
Nov 01 03:41:08.114019 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:08.113974    4400 scope.go:122] "RemoveContainer" containerID="0865bc9bdd2810bcef05e3b89a22bea7e239ba9bd4463b224262ce5ad5cc86a9"
Nov 01 03:41:08.126070 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:08.126055    4400 scope.go:122] "RemoveContainer" containerID="0865bc9bdd2810bcef05e3b89a22bea7e239ba9bd4463b224262ce5ad5cc86a9"
Nov 01 03:41:08.126373 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:41:08.126352    4400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0865bc9bdd2810bcef05e3b89a22bea7e239ba9bd4463b224262ce5ad5cc86a9\": container with ID starting with 0865bc9bdd2810bcef05e3b89a22bea7e239ba9bd4463b224262ce5ad5cc86a9 not found: ID does not exist" containerID="0865bc9bdd2810bcef05e3b89a22bea7e239ba9bd4463b224262ce5ad5cc86a9"
Nov 01 03:41:08.126434 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:08.126380    4400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0865bc9bdd2810bcef05e3b89a22bea7e239ba9bd4463b224262ce5ad5cc86a9"} err="failed to get container status \"0865bc9bdd2810bcef05e3b89a22bea7e239ba9bd4463b224262ce5ad5cc86a9\": rpc error: code = NotFound desc = could not find container \"0865bc9bdd2810bcef05e3b89a22bea7e239ba9bd4463b224262ce5ad5cc86a9\": container with ID starting with 0865bc9bdd2810bcef05e3b89a22bea7e239ba9bd4463b224262ce5ad5cc86a9 not found: ID does not exist"
Nov 01 03:41:08.301731 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:08.293336    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-634332-854c684b9f-8qg4v"]
Nov 01 03:41:08.301731 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:08.297539    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-634332-854c684b9f-8qg4v"]
Nov 01 03:41:09.616294 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:09.616255    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b174e47-5aad-474b-a5cf-4fe09a417fe3" path="/var/lib/kubelet/pods/9b174e47-5aad-474b-a5cf-4fe09a417fe3/volumes"
Nov 01 03:41:12.112256 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:12.112227    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-211380-5b58f48f46-7r8nn"
Nov 01 03:41:13.704721 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:13.704675    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-22e11b-75d4c95497-n67kh"]
Nov 01 03:41:13.705438 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:13.705423    4400 cpu_manager.go:414] "containerMap: removing container" podUID="9b174e47-5aad-474b-a5cf-4fe09a417fe3" containerName="maas-api"
Nov 01 03:41:13.705480 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:13.705443    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b174e47-5aad-474b-a5cf-4fe09a417fe3" containerName="maas-api"
Nov 01 03:41:13.705744 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:13.705716    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b174e47-5aad-474b-a5cf-4fe09a417fe3" containerName="maas-api"
Nov 01 03:41:13.710945 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:13.710926    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-22e11b-75d4c95497-n67kh"
Nov 01 03:41:13.768624 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:13.768601    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-22e11b-75d4c95497-n67kh"]
Nov 01 03:41:13.807566 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:13.807546    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxj4d\" (UniqueName: \"kubernetes.io/projected/a617524f-b187-4692-aea7-89193c7bcad0-kube-api-access-pxj4d\") pod \"maas-api-e2e-inf-a-22e11b-75d4c95497-n67kh\" (UID: \"a617524f-b187-4692-aea7-89193c7bcad0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-22e11b-75d4c95497-n67kh"
Nov 01 03:41:13.807711 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:13.807589    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a617524f-b187-4692-aea7-89193c7bcad0-maas-api-tls\") pod \"maas-api-e2e-inf-a-22e11b-75d4c95497-n67kh\" (UID: \"a617524f-b187-4692-aea7-89193c7bcad0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-22e11b-75d4c95497-n67kh"
Nov 01 03:41:13.807711 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:13.807660    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a617524f-b187-4692-aea7-89193c7bcad0-metrics-certs\") pod \"maas-api-e2e-inf-a-22e11b-75d4c95497-n67kh\" (UID: \"a617524f-b187-4692-aea7-89193c7bcad0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-22e11b-75d4c95497-n67kh"
Nov 01 03:41:13.807838 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:13.807778    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a617524f-b187-4692-aea7-89193c7bcad0-openshift-service-ca\") pod \"maas-api-e2e-inf-a-22e11b-75d4c95497-n67kh\" (UID: \"a617524f-b187-4692-aea7-89193c7bcad0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-22e11b-75d4c95497-n67kh"
Nov 01 03:41:13.814774 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:13.814755    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-22e11b-serving-cert\""
Nov 01 03:41:13.908650 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:13.908621    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a617524f-b187-4692-aea7-89193c7bcad0-maas-api-tls\") pod \"maas-api-e2e-inf-a-22e11b-75d4c95497-n67kh\" (UID: \"a617524f-b187-4692-aea7-89193c7bcad0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-22e11b-75d4c95497-n67kh"
Nov 01 03:41:13.908807 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:13.908699    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a617524f-b187-4692-aea7-89193c7bcad0-metrics-certs\") pod \"maas-api-e2e-inf-a-22e11b-75d4c95497-n67kh\" (UID: \"a617524f-b187-4692-aea7-89193c7bcad0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-22e11b-75d4c95497-n67kh"
Nov 01 03:41:13.908807 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:13.908750    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a617524f-b187-4692-aea7-89193c7bcad0-openshift-service-ca\") pod \"maas-api-e2e-inf-a-22e11b-75d4c95497-n67kh\" (UID: \"a617524f-b187-4692-aea7-89193c7bcad0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-22e11b-75d4c95497-n67kh"
Nov 01 03:41:13.908807 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:41:13.908761    4400 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-22e11b-serving-cert: secret "maas-api-e2e-inf-a-22e11b-serving-cert" not found
Nov 01 03:41:13.908920 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:41:13.908825    4400 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a617524f-b187-4692-aea7-89193c7bcad0-maas-api-tls podName:a617524f-b187-4692-aea7-89193c7bcad0 nodeName:}" failed. No retries permitted until 2026-11-01 03:41:14.408806427 +0000 UTC m=+2080.923452287 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a617524f-b187-4692-aea7-89193c7bcad0-maas-api-tls") pod "maas-api-e2e-inf-a-22e11b-75d4c95497-n67kh" (UID: "a617524f-b187-4692-aea7-89193c7bcad0") : secret "maas-api-e2e-inf-a-22e11b-serving-cert" not found
Nov 01 03:41:13.909080 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:13.909063    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxj4d\" (UniqueName: \"kubernetes.io/projected/a617524f-b187-4692-aea7-89193c7bcad0-kube-api-access-pxj4d\") pod \"maas-api-e2e-inf-a-22e11b-75d4c95497-n67kh\" (UID: \"a617524f-b187-4692-aea7-89193c7bcad0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-22e11b-75d4c95497-n67kh"
Nov 01 03:41:13.909459 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:13.909435    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a617524f-b187-4692-aea7-89193c7bcad0-openshift-service-ca\") pod \"maas-api-e2e-inf-a-22e11b-75d4c95497-n67kh\" (UID: \"a617524f-b187-4692-aea7-89193c7bcad0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-22e11b-75d4c95497-n67kh"
Nov 01 03:41:13.911186 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:13.911170    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a617524f-b187-4692-aea7-89193c7bcad0-metrics-certs\") pod \"maas-api-e2e-inf-a-22e11b-75d4c95497-n67kh\" (UID: \"a617524f-b187-4692-aea7-89193c7bcad0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-22e11b-75d4c95497-n67kh"
Nov 01 03:41:13.991749 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:13.987975    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxj4d\" (UniqueName: \"kubernetes.io/projected/a617524f-b187-4692-aea7-89193c7bcad0-kube-api-access-pxj4d\") pod \"maas-api-e2e-inf-a-22e11b-75d4c95497-n67kh\" (UID: \"a617524f-b187-4692-aea7-89193c7bcad0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-22e11b-75d4c95497-n67kh"
Nov 01 03:41:14.444320 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:14.444286    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a617524f-b187-4692-aea7-89193c7bcad0-maas-api-tls\") pod \"maas-api-e2e-inf-a-22e11b-75d4c95497-n67kh\" (UID: \"a617524f-b187-4692-aea7-89193c7bcad0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-22e11b-75d4c95497-n67kh"
Nov 01 03:41:14.446760 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:14.446738    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a617524f-b187-4692-aea7-89193c7bcad0-maas-api-tls\") pod \"maas-api-e2e-inf-a-22e11b-75d4c95497-n67kh\" (UID: \"a617524f-b187-4692-aea7-89193c7bcad0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-22e11b-75d4c95497-n67kh"
Nov 01 03:41:14.625198 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:14.625173    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-22e11b-75d4c95497-n67kh"
Nov 01 03:41:15.041160 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:15.029896    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-22e11b-75d4c95497-n67kh"]
Nov 01 03:41:15.173869 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:15.173834    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-22e11b-75d4c95497-n67kh" event={"ID":"a617524f-b187-4692-aea7-89193c7bcad0","Type":"ContainerStarted","Data":"b781d0a044c5bc91ce40916c7a7ff76b902708b9625f39364069451368202c53"}
Nov 01 03:41:17.189011 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:17.188977    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-22e11b-75d4c95497-n67kh" event={"ID":"a617524f-b187-4692-aea7-89193c7bcad0","Type":"ContainerStarted","Data":"3839dc1d5020709f743a71b0fbfaae307a80d41d65c222176b02c52c64d428c7"}
Nov 01 03:41:17.189409 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:17.189043    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-22e11b-75d4c95497-n67kh"
Nov 01 03:41:17.312319 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:17.312270    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-22e11b-75d4c95497-n67kh" podStartSLOduration=3.212630303 podStartE2EDuration="4.312246746s" totalImagesPullingTime="1.099616443s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:41:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:41:17.309957198 +0000 UTC m=+2083.824603058" watchObservedRunningTime="2026-11-01 03:41:17.312246746 +0000 UTC m=+2083.826892606"
Nov 01 03:41:23.202193 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:23.202162    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-22e11b-75d4c95497-n67kh"
Nov 01 03:41:31.807916 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:41:31.807872    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:41:34.095153 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:34.095123    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fdbdfefd-cb877c7cc-9g4k2"]
Nov 01 03:41:34.095627 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:34.095346    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fdbdfefd-cb877c7cc-9g4k2" podUID="c60732bb-7134-40b1-b5b1-48607d07bd68" containerName="maas-api" containerID="cri-o://ffaf63382a918ecc17d90d8b1efe9b211425c055ad17d1dcb6b13ae5b816dfc1" gracePeriod=30
Nov 01 03:41:34.460877 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:34.460350    4400 generic.go:356] "Generic (PLEG): container finished" podID="c60732bb-7134-40b1-b5b1-48607d07bd68" containerID="ffaf63382a918ecc17d90d8b1efe9b211425c055ad17d1dcb6b13ae5b816dfc1" exitCode=0
Nov 01 03:41:34.460877 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:34.460630    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fdbdfefd-cb877c7cc-9g4k2" event={"ID":"c60732bb-7134-40b1-b5b1-48607d07bd68","Type":"ContainerDied","Data":"ffaf63382a918ecc17d90d8b1efe9b211425c055ad17d1dcb6b13ae5b816dfc1"}
Nov 01 03:41:34.482528 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:34.482500    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fdbdfefd-cb877c7cc-9g4k2"
Nov 01 03:41:34.598712 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:34.597439    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c60732bb-7134-40b1-b5b1-48607d07bd68-kube-api-access-hsghh\" (UniqueName: \"kubernetes.io/projected/c60732bb-7134-40b1-b5b1-48607d07bd68-kube-api-access-hsghh\") pod \"c60732bb-7134-40b1-b5b1-48607d07bd68\" (UID: \"c60732bb-7134-40b1-b5b1-48607d07bd68\") "
Nov 01 03:41:34.610758 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:34.600319    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c60732bb-7134-40b1-b5b1-48607d07bd68-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c60732bb-7134-40b1-b5b1-48607d07bd68-maas-api-tls\") pod \"c60732bb-7134-40b1-b5b1-48607d07bd68\" (UID: \"c60732bb-7134-40b1-b5b1-48607d07bd68\") "
Nov 01 03:41:34.610758 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:34.600389    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c60732bb-7134-40b1-b5b1-48607d07bd68-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c60732bb-7134-40b1-b5b1-48607d07bd68-metrics-certs\") pod \"c60732bb-7134-40b1-b5b1-48607d07bd68\" (UID: \"c60732bb-7134-40b1-b5b1-48607d07bd68\") "
Nov 01 03:41:34.610758 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:34.600433    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c60732bb-7134-40b1-b5b1-48607d07bd68-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c60732bb-7134-40b1-b5b1-48607d07bd68-openshift-service-ca\") pod \"c60732bb-7134-40b1-b5b1-48607d07bd68\" (UID: \"c60732bb-7134-40b1-b5b1-48607d07bd68\") "
Nov 01 03:41:34.630921 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:34.626024    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c60732bb-7134-40b1-b5b1-48607d07bd68-openshift-service-ca" pod "c60732bb-7134-40b1-b5b1-48607d07bd68" (UID: "c60732bb-7134-40b1-b5b1-48607d07bd68"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:41:34.654708 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:34.653205    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c60732bb-7134-40b1-b5b1-48607d07bd68-kube-api-access-hsghh" pod "c60732bb-7134-40b1-b5b1-48607d07bd68" (UID: "c60732bb-7134-40b1-b5b1-48607d07bd68"). InnerVolumeSpecName "kube-api-access-hsghh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:41:34.697582 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:34.692566    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c60732bb-7134-40b1-b5b1-48607d07bd68-metrics-certs" pod "c60732bb-7134-40b1-b5b1-48607d07bd68" (UID: "c60732bb-7134-40b1-b5b1-48607d07bd68"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:41:34.697582 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:34.692782    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c60732bb-7134-40b1-b5b1-48607d07bd68-maas-api-tls" pod "c60732bb-7134-40b1-b5b1-48607d07bd68" (UID: "c60732bb-7134-40b1-b5b1-48607d07bd68"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:41:34.705765 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:34.703845    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsghh\" (UniqueName: \"kubernetes.io/projected/c60732bb-7134-40b1-b5b1-48607d07bd68-kube-api-access-hsghh\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:41:34.705765 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:34.703882    4400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c60732bb-7134-40b1-b5b1-48607d07bd68-maas-api-tls\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:41:34.705765 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:34.703898    4400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c60732bb-7134-40b1-b5b1-48607d07bd68-metrics-certs\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:41:34.705765 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:34.703914    4400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c60732bb-7134-40b1-b5b1-48607d07bd68-openshift-service-ca\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:41:35.245699 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:35.245077    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7d7329-85768dff6d-dgz5b"]
Nov 01 03:41:35.245699 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:35.245302    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7d7329-85768dff6d-dgz5b" podUID="884eee93-f740-4ccd-8ec2-12b26f39e775" containerName="maas-api" containerID="cri-o://1623be535d0b12eb73c3310b90d9a437238a020ed1ab4bb29ceb96b6542357ae" gracePeriod=30
Nov 01 03:41:35.582793 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:35.581705    4400 generic.go:356] "Generic (PLEG): container finished" podID="884eee93-f740-4ccd-8ec2-12b26f39e775" containerID="1623be535d0b12eb73c3310b90d9a437238a020ed1ab4bb29ceb96b6542357ae" exitCode=0
Nov 01 03:41:35.582793 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:35.581842    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7d7329-85768dff6d-dgz5b" event={"ID":"884eee93-f740-4ccd-8ec2-12b26f39e775","Type":"ContainerDied","Data":"1623be535d0b12eb73c3310b90d9a437238a020ed1ab4bb29ceb96b6542357ae"}
Nov 01 03:41:35.585552 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:35.585530    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fdbdfefd-cb877c7cc-9g4k2"
Nov 01 03:41:35.585742 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:35.585578    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fdbdfefd-cb877c7cc-9g4k2" event={"ID":"c60732bb-7134-40b1-b5b1-48607d07bd68","Type":"ContainerDied","Data":"1457f0544345f7f7da50e2737f0b0b34c2a8ffbffb3dbf224ac41648ccc36336"}
Nov 01 03:41:35.585742 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:35.585650    4400 scope.go:122] "RemoveContainer" containerID="ffaf63382a918ecc17d90d8b1efe9b211425c055ad17d1dcb6b13ae5b816dfc1"
Nov 01 03:41:35.623986 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:35.623947    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7d7329-85768dff6d-dgz5b"
Nov 01 03:41:35.763705 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:35.762768    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/884eee93-f740-4ccd-8ec2-12b26f39e775-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/884eee93-f740-4ccd-8ec2-12b26f39e775-maas-api-tls\") pod \"884eee93-f740-4ccd-8ec2-12b26f39e775\" (UID: \"884eee93-f740-4ccd-8ec2-12b26f39e775\") "
Nov 01 03:41:35.763705 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:35.763018    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/884eee93-f740-4ccd-8ec2-12b26f39e775-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/884eee93-f740-4ccd-8ec2-12b26f39e775-openshift-service-ca\") pod \"884eee93-f740-4ccd-8ec2-12b26f39e775\" (UID: \"884eee93-f740-4ccd-8ec2-12b26f39e775\") "
Nov 01 03:41:35.763705 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:35.763089    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/884eee93-f740-4ccd-8ec2-12b26f39e775-kube-api-access-dvkvj\" (UniqueName: \"kubernetes.io/projected/884eee93-f740-4ccd-8ec2-12b26f39e775-kube-api-access-dvkvj\") pod \"884eee93-f740-4ccd-8ec2-12b26f39e775\" (UID: \"884eee93-f740-4ccd-8ec2-12b26f39e775\") "
Nov 01 03:41:35.763705 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:35.763238    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/884eee93-f740-4ccd-8ec2-12b26f39e775-metrics-certs\" (UniqueName: \"kubernetes.io/secret/884eee93-f740-4ccd-8ec2-12b26f39e775-metrics-certs\") pod \"884eee93-f740-4ccd-8ec2-12b26f39e775\" (UID: \"884eee93-f740-4ccd-8ec2-12b26f39e775\") "
Nov 01 03:41:35.775283 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:35.765951    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/884eee93-f740-4ccd-8ec2-12b26f39e775-openshift-service-ca" pod "884eee93-f740-4ccd-8ec2-12b26f39e775" (UID: "884eee93-f740-4ccd-8ec2-12b26f39e775"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:41:35.777218 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:35.777145    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/884eee93-f740-4ccd-8ec2-12b26f39e775-maas-api-tls" pod "884eee93-f740-4ccd-8ec2-12b26f39e775" (UID: "884eee93-f740-4ccd-8ec2-12b26f39e775"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:41:35.779539 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:35.779504    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/884eee93-f740-4ccd-8ec2-12b26f39e775-kube-api-access-dvkvj" pod "884eee93-f740-4ccd-8ec2-12b26f39e775" (UID: "884eee93-f740-4ccd-8ec2-12b26f39e775"). InnerVolumeSpecName "kube-api-access-dvkvj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:41:35.781927 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:35.781886    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/884eee93-f740-4ccd-8ec2-12b26f39e775-metrics-certs" pod "884eee93-f740-4ccd-8ec2-12b26f39e775" (UID: "884eee93-f740-4ccd-8ec2-12b26f39e775"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:41:35.792323 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:35.785987    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fdbdfefd-cb877c7cc-9g4k2"]
Nov 01 03:41:35.833280 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:35.833207    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fdbdfefd-cb877c7cc-9g4k2"]
Nov 01 03:41:35.865492 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:35.865468    4400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/884eee93-f740-4ccd-8ec2-12b26f39e775-openshift-service-ca\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:41:35.865492 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:35.865492    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvkvj\" (UniqueName: \"kubernetes.io/projected/884eee93-f740-4ccd-8ec2-12b26f39e775-kube-api-access-dvkvj\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:41:35.865649 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:35.865503    4400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/884eee93-f740-4ccd-8ec2-12b26f39e775-metrics-certs\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:41:35.865649 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:35.865513    4400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/884eee93-f740-4ccd-8ec2-12b26f39e775-maas-api-tls\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:41:36.593008 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:36.592982    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7d7329-85768dff6d-dgz5b"
Nov 01 03:41:36.593451 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:36.593028    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7d7329-85768dff6d-dgz5b" event={"ID":"884eee93-f740-4ccd-8ec2-12b26f39e775","Type":"ContainerDied","Data":"7196ea88f312c6ac390ec5d7829ecb0d9592db4d06179f8e84871588699bc186"}
Nov 01 03:41:36.593451 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:36.593103    4400 scope.go:122] "RemoveContainer" containerID="1623be535d0b12eb73c3310b90d9a437238a020ed1ab4bb29ceb96b6542357ae"
Nov 01 03:41:36.763917 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:36.763889    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7d7329-85768dff6d-dgz5b"]
Nov 01 03:41:36.814458 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:36.814431    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7d7329-85768dff6d-dgz5b"]
Nov 01 03:41:37.325297 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:37.325259    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fdbdfefd-cb877c7cc-9g4k2" podUID="c60732bb-7134-40b1-b5b1-48607d07bd68" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.110:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:41:37.616322 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:37.616248    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="884eee93-f740-4ccd-8ec2-12b26f39e775" path="/var/lib/kubelet/pods/884eee93-f740-4ccd-8ec2-12b26f39e775/volumes"
Nov 01 03:41:37.616763 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:37.616747    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c60732bb-7134-40b1-b5b1-48607d07bd68" path="/var/lib/kubelet/pods/c60732bb-7134-40b1-b5b1-48607d07bd68/volumes"
Nov 01 03:41:43.551443 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:41:43.551407    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 03:42:01.840708 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:42:01.832912    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:42:02.836694 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:02.836659    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-57cp9"]
Nov 01 03:42:02.836961 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:02.836936    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-57cp9" podUID="543cb5b7-afc2-4042-af65-69bf92772c80" containerName="istio-proxy" containerID="cri-o://78f6bd63b99b9fa7a6a4d06f1f1dbbb3a0ba7d1968322da13ca109fa9bc48e90" gracePeriod=30
Nov 01 03:42:02.887422 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:02.887394    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-hnd5d"]
Nov 01 03:42:02.887848 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:02.887589    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-hnd5d" podUID="a03a2ca3-9463-4838-8c2f-a37293da9e6a" containerName="istio-proxy" containerID="cri-o://5239558be482e883d1a22ca33d40634a37aed71201d84fcf5751ba3a42045bfb" gracePeriod=30
Nov 01 03:42:02.977384 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:02.977348    4400 patch_prober.go:29] interesting pod/e2e-worker-w1-b12199-openshift-default-774cb8c567-hnd5d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 03:42:02.977567 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:02.977401    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-hnd5d" podUID="a03a2ca3-9463-4838-8c2f-a37293da9e6a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 03:42:03.509623 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:03.509592    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lv9hr"]
Nov 01 03:42:03.509898 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:03.509847    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lv9hr" podUID="ed1e2049-23c6-43e6-811c-fb0c5a9541b5" containerName="istio-proxy" containerID="cri-o://899f076ad230e0c666313f4c60979453677c213c000b89605566eaca2d7c7449" gracePeriod=30
Nov 01 03:42:03.513841 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:03.513805    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-2sbzb"]
Nov 01 03:42:03.514017 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:03.513997    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-2sbzb" podUID="863e73b9-dddf-41c3-9a3d-eca8d7e2b727" containerName="istio-proxy" containerID="cri-o://c2165e9ef486f13dc4f334eb78392a42cf94525247353124496a23de97ac76e1" gracePeriod=30
Nov 01 03:42:03.825629 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:03.823505    4400 patch_prober.go:29] interesting pod/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-2sbzb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 03:42:03.825629 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:03.823558    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-2sbzb" podUID="863e73b9-dddf-41c3-9a3d-eca8d7e2b727" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 03:42:03.831437 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:03.831412    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hsfpq"]
Nov 01 03:42:03.831862 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:03.831830    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hsfpq" podUID="3cb9c447-7466-4f2e-9449-d153b9b8ec2c" containerName="istio-proxy" containerID="cri-o://490c384a9a528bcb5a9456833fe3ada4aca2f2907dcaf15580536eb4ff4eb40c" gracePeriod=30
Nov 01 03:42:03.837988 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:03.837963    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-mk69d"]
Nov 01 03:42:03.838165 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:03.838146    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-mk69d" podUID="4988f434-dc04-41bd-8b3f-6e386bca1334" containerName="istio-proxy" containerID="cri-o://c8de3e02ecb3cc679bfa7ff7557f3c7baedf4eb48cdf77aaa4f24aca38b9302f" gracePeriod=30
Nov 01 03:42:04.102942 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:04.102873    4400 patch_prober.go:29] interesting pod/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lv9hr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 03:42:04.102942 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:04.102928    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lv9hr" podUID="ed1e2049-23c6-43e6-811c-fb0c5a9541b5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 03:42:06.759293 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:06.759260    4400 patch_prober.go:29] interesting pod/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hsfpq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 03:42:06.759670 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:06.759337    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hsfpq" podUID="3cb9c447-7466-4f2e-9449-d153b9b8ec2c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 03:42:07.185632 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:07.185596    4400 patch_prober.go:29] interesting pod/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-mk69d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 03:42:07.185823 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:07.185670    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-mk69d" podUID="4988f434-dc04-41bd-8b3f-6e386bca1334" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 03:42:08.187250 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:08.187205    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-57cp9"
Nov 01 03:42:08.219098 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:08.219075    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-hnd5d"
Nov 01 03:42:08.285765 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:08.285737    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/543cb5b7-afc2-4042-af65-69bf92772c80-istio-token\" (UniqueName: \"kubernetes.io/projected/543cb5b7-afc2-4042-af65-69bf92772c80-istio-token\") pod \"543cb5b7-afc2-4042-af65-69bf92772c80\" (UID: \"543cb5b7-afc2-4042-af65-69bf92772c80\") "
Nov 01 03:42:08.285896 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:08.285791    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a03a2ca3-9463-4838-8c2f-a37293da9e6a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a03a2ca3-9463-4838-8c2f-a37293da9e6a-istio-data\") pod \"a03a2ca3-9463-4838-8c2f-a37293da9e6a\" (UID: \"a03a2ca3-9463-4838-8c2f-a37293da9e6a\") "
Nov 01 03:42:08.285896 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:08.285823    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a03a2ca3-9463-4838-8c2f-a37293da9e6a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a03a2ca3-9463-4838-8c2f-a37293da9e6a-workload-certs\") pod \"a03a2ca3-9463-4838-8c2f-a37293da9e6a\" (UID: \"a03a2ca3-9463-4838-8c2f-a37293da9e6a\") "
Nov 01 03:42:08.286012 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:08.285892    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a03a2ca3-9463-4838-8c2f-a37293da9e6a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a03a2ca3-9463-4838-8c2f-a37293da9e6a-istiod-ca-cert\") pod \"a03a2ca3-9463-4838-8c2f-a37293da9e6a\" (UID: \"a03a2ca3-9463-4838-8c2f-a37293da9e6a\") "
Nov 01 03:42:08.286012 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:08.285923    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a03a2ca3-9463-4838-8c2f-a37293da9e6a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a03a2ca3-9463-4838-8c2f-a37293da9e6a-credential-socket\") pod \"a03a2ca3-9463-4838-8c2f-a37293da9e6a\" (UID: \"a03a2ca3-9463-4838-8c2f-a37293da9e6a\") "
Nov 01 03:42:08.286012 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:08.285952    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a03a2ca3-9463-4838-8c2f-a37293da9e6a-istio-token\" (UniqueName: \"kubernetes.io/projected/a03a2ca3-9463-4838-8c2f-a37293da9e6a-istio-token\") pod \"a03a2ca3-9463-4838-8c2f-a37293da9e6a\" (UID: \"a03a2ca3-9463-4838-8c2f-a37293da9e6a\") "
Nov 01 03:42:08.286160 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:08.286111    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/543cb5b7-afc2-4042-af65-69bf92772c80-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/543cb5b7-afc2-4042-af65-69bf92772c80-workload-certs\") pod \"543cb5b7-afc2-4042-af65-69bf92772c80\" (UID: \"543cb5b7-afc2-4042-af65-69bf92772c80\") "
Nov 01 03:42:08.286160 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:08.286123    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a03a2ca3-9463-4838-8c2f-a37293da9e6a-istio-data" pod "a03a2ca3-9463-4838-8c2f-a37293da9e6a" (UID: "a03a2ca3-9463-4838-8c2f-a37293da9e6a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:42:08.286260 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:08.286190    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/543cb5b7-afc2-4042-af65-69bf92772c80-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/543cb5b7-afc2-4042-af65-69bf92772c80-istio-data\") pod \"543cb5b7-afc2-4042-af65-69bf92772c80\" (UID: \"543cb5b7-afc2-4042-af65-69bf92772c80\") "
Nov 01 03:42:08.286260 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:08.286226    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/543cb5b7-afc2-4042-af65-69bf92772c80-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/543cb5b7-afc2-4042-af65-69bf92772c80-istio-envoy\") pod \"543cb5b7-afc2-4042-af65-69bf92772c80\" (UID: \"543cb5b7-afc2-4042-af65-69bf92772c80\") "
Nov 01 03:42:08.286260 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:08.286251    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a03a2ca3-9463-4838-8c2f-a37293da9e6a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a03a2ca3-9463-4838-8c2f-a37293da9e6a-workload-socket\") pod \"a03a2ca3-9463-4838-8c2f-a37293da9e6a\" (UID: \"a03a2ca3-9463-4838-8c2f-a37293da9e6a\") "
Nov 01 03:42:08.286407 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:08.286324    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a03a2ca3-9463-4838-8c2f-a37293da9e6a-kube-api-access-77cr5\" (UniqueName: \"kubernetes.io/projected/a03a2ca3-9463-4838-8c2f-a37293da9e6a-kube-api-access-77cr5\") pod \"a03a2ca3-9463-4838-8c2f-a37293da9e6a\" (UID: \"a03a2ca3-9463-4838-8c2f-a37293da9e6a\") "
Nov 01 03:42:08.286407 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:08.286369    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/543cb5b7-afc2-4042-af65-69bf92772c80-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/543cb5b7-afc2-4042-af65-69bf92772c80-workload-socket\") pod \"543cb5b7-afc2-4042-af65-69bf92772c80\" (UID: \"543cb5b7-afc2-4042-af65-69bf92772c80\") "
Nov 01 03:42:08.286512 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:08.286404    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a03a2ca3-9463-4838-8c2f-a37293da9e6a-workload-certs" pod "a03a2ca3-9463-4838-8c2f-a37293da9e6a" (UID: "a03a2ca3-9463-4838-8c2f-a37293da9e6a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:42:08.286512 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:08.286423    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a03a2ca3-9463-4838-8c2f-a37293da9e6a-credential-socket" pod "a03a2ca3-9463-4838-8c2f-a37293da9e6a" (UID: "a03a2ca3-9463-4838-8c2f-a37293da9e6a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:42:08.286512 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:08.286456    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a03a2ca3-9463-4838-8c2f-a37293da9e6a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a03a2ca3-9463-4838-8c2f-a37293da9e6a-istio-envoy\") pod \"a03a2ca3-9463-4838-8c2f-a37293da9e6a\" (UID: \"a03a2ca3-9463-4838-8c2f-a37293da9e6a\") "
Nov 01 03:42:08.286512 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:08.286488    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/543cb5b7-afc2-4042-af65-69bf92772c80-workload-certs" pod "543cb5b7-afc2-4042-af65-69bf92772c80" (UID: "543cb5b7-afc2-4042-af65-69bf92772c80"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:42:08.286512 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:08.286497    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a03a2ca3-9463-4838-8c2f-a37293da9e6a-istiod-ca-cert" pod "a03a2ca3-9463-4838-8c2f-a37293da9e6a" (UID: "a03a2ca3-9463-4838-8c2f-a37293da9e6a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:42:08.286798 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:08.286517    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/543cb5b7-afc2-4042-af65-69bf92772c80-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/543cb5b7-afc2-4042-af65-69bf92772c80-istio-podinfo\") pod \"543cb5b7-afc2-4042-af65-69bf92772c80\" (UID: \"543cb5b7-afc2-4042-af65-69bf92772c80\") "
Nov 01 03:42:08.286798 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:08.286573    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a03a2ca3-9463-4838-8c2f-a37293da9e6a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a03a2ca3-9463-4838-8c2f-a37293da9e6a-istio-podinfo\") pod \"a03a2ca3-9463-4838-8c2f-a37293da9e6a\" (UID: \"a03a2ca3-9463-4838-8c2f-a37293da9e6a\") "
Nov 01 03:42:08.286798 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:08.286669    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a03a2ca3-9463-4838-8c2f-a37293da9e6a-workload-socket" pod "a03a2ca3-9463-4838-8c2f-a37293da9e6a" (UID: "a03a2ca3-9463-4838-8c2f-a37293da9e6a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:42:08.286798 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:08.286671    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/543cb5b7-afc2-4042-af65-69bf92772c80-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/543cb5b7-afc2-4042-af65-69bf92772c80-istiod-ca-cert\") pod \"543cb5b7-afc2-4042-af65-69bf92772c80\" (UID: \"543cb5b7-afc2-4042-af65-69bf92772c80\") "
Nov 01 03:42:08.287001 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:08.286822    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/543cb5b7-afc2-4042-af65-69bf92772c80-kube-api-access-q8wr2\" (UniqueName: \"kubernetes.io/projected/543cb5b7-afc2-4042-af65-69bf92772c80-kube-api-access-q8wr2\") pod \"543cb5b7-afc2-4042-af65-69bf92772c80\" (UID: \"543cb5b7-afc2-4042-af65-69bf92772c80\") "
Nov 01 03:42:08.287001 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:08.286906    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/543cb5b7-afc2-4042-af65-69bf92772c80-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/543cb5b7-afc2-4042-af65-69bf92772c80-credential-socket\") pod \"543cb5b7-afc2-4042-af65-69bf92772c80\" (UID: \"543cb5b7-afc2-4042-af65-69bf92772c80\") "
Nov 01 03:42:08.287877 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:08.286715    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/543cb5b7-afc2-4042-af65-69bf92772c80-istio-data" pod "543cb5b7-afc2-4042-af65-69bf92772c80" (UID: "543cb5b7-afc2-4042-af65-69bf92772c80"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:42:08.287877 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:08.287655    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/543cb5b7-afc2-4042-af65-69bf92772c80-workload-socket" pod "543cb5b7-afc2-4042-af65-69bf92772c80" (UID: "543cb5b7-afc2-4042-af65-69bf92772c80"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:42:08.287877 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:08.287144    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/543cb5b7-afc2-4042-af65-69bf92772c80-istiod-ca-cert" pod "543cb5b7-afc2-4042-af65-69bf92772c80" (UID: "543cb5b7-afc2-4042-af65-69bf92772c80"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:42:08.287877 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:08.287869    4400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a03a2ca3-9463-4838-8c2f-a37293da9e6a-istiod-ca-cert\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:08.288160 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:08.287890    4400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a03a2ca3-9463-4838-8c2f-a37293da9e6a-credential-socket\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:08.288160 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:08.287905    4400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/543cb5b7-afc2-4042-af65-69bf92772c80-workload-certs\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:08.288160 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:08.287920    4400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/543cb5b7-afc2-4042-af65-69bf92772c80-istio-data\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:08.288160 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:08.287934    4400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a03a2ca3-9463-4838-8c2f-a37293da9e6a-workload-socket\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:08.288160 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:08.287946    4400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/543cb5b7-afc2-4042-af65-69bf92772c80-workload-socket\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:08.288160 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:08.287959    4400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/543cb5b7-afc2-4042-af65-69bf92772c80-istiod-ca-cert\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:08.288160 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:08.287971    4400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a03a2ca3-9463-4838-8c2f-a37293da9e6a-istio-data\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:08.288160 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:08.287984    4400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a03a2ca3-9463-4838-8c2f-a37293da9e6a-workload-certs\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:08.288880 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:08.288851    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/543cb5b7-afc2-4042-af65-69bf92772c80-istio-token" pod "543cb5b7-afc2-4042-af65-69bf92772c80" (UID: "543cb5b7-afc2-4042-af65-69bf92772c80"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:42:08.289011 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:08.288973    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a03a2ca3-9463-4838-8c2f-a37293da9e6a-istio-token" pod "a03a2ca3-9463-4838-8c2f-a37293da9e6a" (UID: "a03a2ca3-9463-4838-8c2f-a37293da9e6a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:42:08.289120 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:08.289018    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/543cb5b7-afc2-4042-af65-69bf92772c80-credential-socket" pod "543cb5b7-afc2-4042-af65-69bf92772c80" (UID: "543cb5b7-afc2-4042-af65-69bf92772c80"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:42:08.289176 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:08.289156    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a03a2ca3-9463-4838-8c2f-a37293da9e6a-istio-envoy" pod "a03a2ca3-9463-4838-8c2f-a37293da9e6a" (UID: "a03a2ca3-9463-4838-8c2f-a37293da9e6a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:42:08.289236 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:08.289219    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/543cb5b7-afc2-4042-af65-69bf92772c80-istio-envoy" pod "543cb5b7-afc2-4042-af65-69bf92772c80" (UID: "543cb5b7-afc2-4042-af65-69bf92772c80"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:42:08.289236 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:08.289229    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a03a2ca3-9463-4838-8c2f-a37293da9e6a-kube-api-access-77cr5" pod "a03a2ca3-9463-4838-8c2f-a37293da9e6a" (UID: "a03a2ca3-9463-4838-8c2f-a37293da9e6a"). InnerVolumeSpecName "kube-api-access-77cr5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:42:08.289620 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:08.289598    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/543cb5b7-afc2-4042-af65-69bf92772c80-istio-podinfo" pod "543cb5b7-afc2-4042-af65-69bf92772c80" (UID: "543cb5b7-afc2-4042-af65-69bf92772c80"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 03:42:08.291094 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:08.291078    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a03a2ca3-9463-4838-8c2f-a37293da9e6a-istio-podinfo" pod "a03a2ca3-9463-4838-8c2f-a37293da9e6a" (UID: "a03a2ca3-9463-4838-8c2f-a37293da9e6a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 03:42:08.291162 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:08.291091    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/543cb5b7-afc2-4042-af65-69bf92772c80-kube-api-access-q8wr2" pod "543cb5b7-afc2-4042-af65-69bf92772c80" (UID: "543cb5b7-afc2-4042-af65-69bf92772c80"). InnerVolumeSpecName "kube-api-access-q8wr2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:42:08.389531 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:08.389472    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77cr5\" (UniqueName: \"kubernetes.io/projected/a03a2ca3-9463-4838-8c2f-a37293da9e6a-kube-api-access-77cr5\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:08.389531 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:08.389498    4400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a03a2ca3-9463-4838-8c2f-a37293da9e6a-istio-envoy\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:08.389531 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:08.389507    4400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/543cb5b7-afc2-4042-af65-69bf92772c80-istio-podinfo\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:08.389531 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:08.389515    4400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a03a2ca3-9463-4838-8c2f-a37293da9e6a-istio-podinfo\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:08.389531 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:08.389523    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8wr2\" (UniqueName: \"kubernetes.io/projected/543cb5b7-afc2-4042-af65-69bf92772c80-kube-api-access-q8wr2\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:08.389531 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:08.389532    4400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/543cb5b7-afc2-4042-af65-69bf92772c80-credential-socket\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:08.389807 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:08.389541    4400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/543cb5b7-afc2-4042-af65-69bf92772c80-istio-token\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:08.389807 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:08.389551    4400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a03a2ca3-9463-4838-8c2f-a37293da9e6a-istio-token\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:08.389807 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:08.389559    4400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/543cb5b7-afc2-4042-af65-69bf92772c80-istio-envoy\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:08.782156 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:08.782128    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-2sbzb"
Nov 01 03:42:08.909485 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:08.909102    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/863e73b9-dddf-41c3-9a3d-eca8d7e2b727-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/863e73b9-dddf-41c3-9a3d-eca8d7e2b727-istio-envoy\") pod \"863e73b9-dddf-41c3-9a3d-eca8d7e2b727\" (UID: \"863e73b9-dddf-41c3-9a3d-eca8d7e2b727\") "
Nov 01 03:42:08.909485 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:08.909194    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/863e73b9-dddf-41c3-9a3d-eca8d7e2b727-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/863e73b9-dddf-41c3-9a3d-eca8d7e2b727-istio-podinfo\") pod \"863e73b9-dddf-41c3-9a3d-eca8d7e2b727\" (UID: \"863e73b9-dddf-41c3-9a3d-eca8d7e2b727\") "
Nov 01 03:42:08.909485 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:08.909232    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/863e73b9-dddf-41c3-9a3d-eca8d7e2b727-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/863e73b9-dddf-41c3-9a3d-eca8d7e2b727-istiod-ca-cert\") pod \"863e73b9-dddf-41c3-9a3d-eca8d7e2b727\" (UID: \"863e73b9-dddf-41c3-9a3d-eca8d7e2b727\") "
Nov 01 03:42:08.909485 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:08.909349    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/863e73b9-dddf-41c3-9a3d-eca8d7e2b727-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/863e73b9-dddf-41c3-9a3d-eca8d7e2b727-istio-data\") pod \"863e73b9-dddf-41c3-9a3d-eca8d7e2b727\" (UID: \"863e73b9-dddf-41c3-9a3d-eca8d7e2b727\") "
Nov 01 03:42:08.909485 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:08.909400    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/863e73b9-dddf-41c3-9a3d-eca8d7e2b727-kube-api-access-27g8r\" (UniqueName: \"kubernetes.io/projected/863e73b9-dddf-41c3-9a3d-eca8d7e2b727-kube-api-access-27g8r\") pod \"863e73b9-dddf-41c3-9a3d-eca8d7e2b727\" (UID: \"863e73b9-dddf-41c3-9a3d-eca8d7e2b727\") "
Nov 01 03:42:08.909799 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:08.909494    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/863e73b9-dddf-41c3-9a3d-eca8d7e2b727-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/863e73b9-dddf-41c3-9a3d-eca8d7e2b727-credential-socket\") pod \"863e73b9-dddf-41c3-9a3d-eca8d7e2b727\" (UID: \"863e73b9-dddf-41c3-9a3d-eca8d7e2b727\") "
Nov 01 03:42:08.909799 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:08.909524    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/863e73b9-dddf-41c3-9a3d-eca8d7e2b727-istio-token\" (UniqueName: \"kubernetes.io/projected/863e73b9-dddf-41c3-9a3d-eca8d7e2b727-istio-token\") pod \"863e73b9-dddf-41c3-9a3d-eca8d7e2b727\" (UID: \"863e73b9-dddf-41c3-9a3d-eca8d7e2b727\") "
Nov 01 03:42:08.909799 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:08.909658    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/863e73b9-dddf-41c3-9a3d-eca8d7e2b727-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/863e73b9-dddf-41c3-9a3d-eca8d7e2b727-workload-socket\") pod \"863e73b9-dddf-41c3-9a3d-eca8d7e2b727\" (UID: \"863e73b9-dddf-41c3-9a3d-eca8d7e2b727\") "
Nov 01 03:42:08.909799 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:08.909708    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/863e73b9-dddf-41c3-9a3d-eca8d7e2b727-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/863e73b9-dddf-41c3-9a3d-eca8d7e2b727-workload-certs\") pod \"863e73b9-dddf-41c3-9a3d-eca8d7e2b727\" (UID: \"863e73b9-dddf-41c3-9a3d-eca8d7e2b727\") "
Nov 01 03:42:08.910507 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:08.910154    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/863e73b9-dddf-41c3-9a3d-eca8d7e2b727-credential-socket" pod "863e73b9-dddf-41c3-9a3d-eca8d7e2b727" (UID: "863e73b9-dddf-41c3-9a3d-eca8d7e2b727"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:42:08.911356 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:08.910965    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/863e73b9-dddf-41c3-9a3d-eca8d7e2b727-workload-socket" pod "863e73b9-dddf-41c3-9a3d-eca8d7e2b727" (UID: "863e73b9-dddf-41c3-9a3d-eca8d7e2b727"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:42:08.911356 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:08.911176    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/863e73b9-dddf-41c3-9a3d-eca8d7e2b727-istio-data" pod "863e73b9-dddf-41c3-9a3d-eca8d7e2b727" (UID: "863e73b9-dddf-41c3-9a3d-eca8d7e2b727"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:42:08.911356 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:08.911300    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/863e73b9-dddf-41c3-9a3d-eca8d7e2b727-istiod-ca-cert" pod "863e73b9-dddf-41c3-9a3d-eca8d7e2b727" (UID: "863e73b9-dddf-41c3-9a3d-eca8d7e2b727"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:42:08.911985 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:08.911702    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/863e73b9-dddf-41c3-9a3d-eca8d7e2b727-workload-certs" pod "863e73b9-dddf-41c3-9a3d-eca8d7e2b727" (UID: "863e73b9-dddf-41c3-9a3d-eca8d7e2b727"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:42:08.911985 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:08.911823    4400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/863e73b9-dddf-41c3-9a3d-eca8d7e2b727-istio-data\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:08.911985 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:08.911844    4400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/863e73b9-dddf-41c3-9a3d-eca8d7e2b727-credential-socket\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:08.911985 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:08.911858    4400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/863e73b9-dddf-41c3-9a3d-eca8d7e2b727-workload-socket\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:08.911985 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:08.911871    4400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/863e73b9-dddf-41c3-9a3d-eca8d7e2b727-workload-certs\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:08.911985 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:08.911884    4400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/863e73b9-dddf-41c3-9a3d-eca8d7e2b727-istiod-ca-cert\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:08.914665 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:08.914638    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/863e73b9-dddf-41c3-9a3d-eca8d7e2b727-istio-envoy" pod "863e73b9-dddf-41c3-9a3d-eca8d7e2b727" (UID: "863e73b9-dddf-41c3-9a3d-eca8d7e2b727"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:42:08.915107 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:08.915085    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/863e73b9-dddf-41c3-9a3d-eca8d7e2b727-istio-podinfo" pod "863e73b9-dddf-41c3-9a3d-eca8d7e2b727" (UID: "863e73b9-dddf-41c3-9a3d-eca8d7e2b727"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 03:42:08.916012 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:08.915988    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/863e73b9-dddf-41c3-9a3d-eca8d7e2b727-istio-token" pod "863e73b9-dddf-41c3-9a3d-eca8d7e2b727" (UID: "863e73b9-dddf-41c3-9a3d-eca8d7e2b727"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:42:08.916105 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:08.916025    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/863e73b9-dddf-41c3-9a3d-eca8d7e2b727-kube-api-access-27g8r" pod "863e73b9-dddf-41c3-9a3d-eca8d7e2b727" (UID: "863e73b9-dddf-41c3-9a3d-eca8d7e2b727"). InnerVolumeSpecName "kube-api-access-27g8r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:42:08.964091 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:08.964063    4400 generic.go:356] "Generic (PLEG): container finished" podID="a03a2ca3-9463-4838-8c2f-a37293da9e6a" containerID="5239558be482e883d1a22ca33d40634a37aed71201d84fcf5751ba3a42045bfb" exitCode=0
Nov 01 03:42:08.964202 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:08.964126    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-hnd5d"
Nov 01 03:42:08.964202 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:08.964152    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-hnd5d" event={"ID":"a03a2ca3-9463-4838-8c2f-a37293da9e6a","Type":"ContainerDied","Data":"5239558be482e883d1a22ca33d40634a37aed71201d84fcf5751ba3a42045bfb"}
Nov 01 03:42:08.964275 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:08.964220    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-hnd5d" event={"ID":"a03a2ca3-9463-4838-8c2f-a37293da9e6a","Type":"ContainerDied","Data":"7740dd97850cdfcc1d3efc7ad6bcdda355ad1b5a716758f628f399b8bd2414b8"}
Nov 01 03:42:08.964275 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:08.964254    4400 scope.go:122] "RemoveContainer" containerID="5239558be482e883d1a22ca33d40634a37aed71201d84fcf5751ba3a42045bfb"
Nov 01 03:42:08.966748 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:08.966722    4400 generic.go:356] "Generic (PLEG): container finished" podID="ed1e2049-23c6-43e6-811c-fb0c5a9541b5" containerID="899f076ad230e0c666313f4c60979453677c213c000b89605566eaca2d7c7449" exitCode=0
Nov 01 03:42:08.966897 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:08.966878    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lv9hr" event={"ID":"ed1e2049-23c6-43e6-811c-fb0c5a9541b5","Type":"ContainerDied","Data":"899f076ad230e0c666313f4c60979453677c213c000b89605566eaca2d7c7449"}
Nov 01 03:42:08.970116 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:08.970039    4400 generic.go:356] "Generic (PLEG): container finished" podID="543cb5b7-afc2-4042-af65-69bf92772c80" containerID="78f6bd63b99b9fa7a6a4d06f1f1dbbb3a0ba7d1968322da13ca109fa9bc48e90" exitCode=0
Nov 01 03:42:08.970233 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:08.970126    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-57cp9" event={"ID":"543cb5b7-afc2-4042-af65-69bf92772c80","Type":"ContainerDied","Data":"78f6bd63b99b9fa7a6a4d06f1f1dbbb3a0ba7d1968322da13ca109fa9bc48e90"}
Nov 01 03:42:08.970233 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:08.970169    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-57cp9" event={"ID":"543cb5b7-afc2-4042-af65-69bf92772c80","Type":"ContainerDied","Data":"df171d0e3fa955992cf36d59fbf241249dae4b152b5493dca2ef8e17c451ea14"}
Nov 01 03:42:08.970370 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:08.970252    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-57cp9"
Nov 01 03:42:08.979584 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:08.979556    4400 generic.go:356] "Generic (PLEG): container finished" podID="4988f434-dc04-41bd-8b3f-6e386bca1334" containerID="c8de3e02ecb3cc679bfa7ff7557f3c7baedf4eb48cdf77aaa4f24aca38b9302f" exitCode=0
Nov 01 03:42:08.979715 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:08.979651    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-mk69d" event={"ID":"4988f434-dc04-41bd-8b3f-6e386bca1334","Type":"ContainerDied","Data":"c8de3e02ecb3cc679bfa7ff7557f3c7baedf4eb48cdf77aaa4f24aca38b9302f"}
Nov 01 03:42:08.981427 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:08.981410    4400 generic.go:356] "Generic (PLEG): container finished" podID="3cb9c447-7466-4f2e-9449-d153b9b8ec2c" containerID="490c384a9a528bcb5a9456833fe3ada4aca2f2907dcaf15580536eb4ff4eb40c" exitCode=0
Nov 01 03:42:08.981509 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:08.981472    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hsfpq" event={"ID":"3cb9c447-7466-4f2e-9449-d153b9b8ec2c","Type":"ContainerDied","Data":"490c384a9a528bcb5a9456833fe3ada4aca2f2907dcaf15580536eb4ff4eb40c"}
Nov 01 03:42:08.983176 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:08.983154    4400 generic.go:356] "Generic (PLEG): container finished" podID="863e73b9-dddf-41c3-9a3d-eca8d7e2b727" containerID="c2165e9ef486f13dc4f334eb78392a42cf94525247353124496a23de97ac76e1" exitCode=0
Nov 01 03:42:08.983265 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:08.983185    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-2sbzb"
Nov 01 03:42:08.983265 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:08.983216    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-2sbzb" event={"ID":"863e73b9-dddf-41c3-9a3d-eca8d7e2b727","Type":"ContainerDied","Data":"c2165e9ef486f13dc4f334eb78392a42cf94525247353124496a23de97ac76e1"}
Nov 01 03:42:08.983265 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:08.983261    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-2sbzb" event={"ID":"863e73b9-dddf-41c3-9a3d-eca8d7e2b727","Type":"ContainerDied","Data":"ddf44a4558e4529230be0a3f2b0b2b17d7dcfb9d93c5c92ef04ac52fb6a9df28"}
Nov 01 03:42:09.002226 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.002152    4400 scope.go:122] "RemoveContainer" containerID="5239558be482e883d1a22ca33d40634a37aed71201d84fcf5751ba3a42045bfb"
Nov 01 03:42:09.002761 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:42:09.002672    4400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5239558be482e883d1a22ca33d40634a37aed71201d84fcf5751ba3a42045bfb\": container with ID starting with 5239558be482e883d1a22ca33d40634a37aed71201d84fcf5751ba3a42045bfb not found: ID does not exist" containerID="5239558be482e883d1a22ca33d40634a37aed71201d84fcf5751ba3a42045bfb"
Nov 01 03:42:09.002887 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.002773    4400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5239558be482e883d1a22ca33d40634a37aed71201d84fcf5751ba3a42045bfb"} err="failed to get container status \"5239558be482e883d1a22ca33d40634a37aed71201d84fcf5751ba3a42045bfb\": rpc error: code = NotFound desc = could not find container \"5239558be482e883d1a22ca33d40634a37aed71201d84fcf5751ba3a42045bfb\": container with ID starting with 5239558be482e883d1a22ca33d40634a37aed71201d84fcf5751ba3a42045bfb not found: ID does not exist"
Nov 01 03:42:09.002887 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.002818    4400 scope.go:122] "RemoveContainer" containerID="78f6bd63b99b9fa7a6a4d06f1f1dbbb3a0ba7d1968322da13ca109fa9bc48e90"
Nov 01 03:42:09.013348 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.013264    4400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/863e73b9-dddf-41c3-9a3d-eca8d7e2b727-istio-podinfo\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:09.013348 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.013294    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27g8r\" (UniqueName: \"kubernetes.io/projected/863e73b9-dddf-41c3-9a3d-eca8d7e2b727-kube-api-access-27g8r\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:09.013348 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.013311    4400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/863e73b9-dddf-41c3-9a3d-eca8d7e2b727-istio-token\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:09.013603 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.013575    4400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/863e73b9-dddf-41c3-9a3d-eca8d7e2b727-istio-envoy\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:09.030641 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.020478    4400 scope.go:122] "RemoveContainer" containerID="78f6bd63b99b9fa7a6a4d06f1f1dbbb3a0ba7d1968322da13ca109fa9bc48e90"
Nov 01 03:42:09.030641 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:42:09.021107    4400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78f6bd63b99b9fa7a6a4d06f1f1dbbb3a0ba7d1968322da13ca109fa9bc48e90\": container with ID starting with 78f6bd63b99b9fa7a6a4d06f1f1dbbb3a0ba7d1968322da13ca109fa9bc48e90 not found: ID does not exist" containerID="78f6bd63b99b9fa7a6a4d06f1f1dbbb3a0ba7d1968322da13ca109fa9bc48e90"
Nov 01 03:42:09.030641 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.021139    4400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78f6bd63b99b9fa7a6a4d06f1f1dbbb3a0ba7d1968322da13ca109fa9bc48e90"} err="failed to get container status \"78f6bd63b99b9fa7a6a4d06f1f1dbbb3a0ba7d1968322da13ca109fa9bc48e90\": rpc error: code = NotFound desc = could not find container \"78f6bd63b99b9fa7a6a4d06f1f1dbbb3a0ba7d1968322da13ca109fa9bc48e90\": container with ID starting with 78f6bd63b99b9fa7a6a4d06f1f1dbbb3a0ba7d1968322da13ca109fa9bc48e90 not found: ID does not exist"
Nov 01 03:42:09.030641 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.021162    4400 scope.go:122] "RemoveContainer" containerID="c2165e9ef486f13dc4f334eb78392a42cf94525247353124496a23de97ac76e1"
Nov 01 03:42:09.043553 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.043129    4400 scope.go:122] "RemoveContainer" containerID="c2165e9ef486f13dc4f334eb78392a42cf94525247353124496a23de97ac76e1"
Nov 01 03:42:09.043976 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:42:09.043901    4400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2165e9ef486f13dc4f334eb78392a42cf94525247353124496a23de97ac76e1\": container with ID starting with c2165e9ef486f13dc4f334eb78392a42cf94525247353124496a23de97ac76e1 not found: ID does not exist" containerID="c2165e9ef486f13dc4f334eb78392a42cf94525247353124496a23de97ac76e1"
Nov 01 03:42:09.043976 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.043940    4400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2165e9ef486f13dc4f334eb78392a42cf94525247353124496a23de97ac76e1"} err="failed to get container status \"c2165e9ef486f13dc4f334eb78392a42cf94525247353124496a23de97ac76e1\": rpc error: code = NotFound desc = could not find container \"c2165e9ef486f13dc4f334eb78392a42cf94525247353124496a23de97ac76e1\": container with ID starting with c2165e9ef486f13dc4f334eb78392a42cf94525247353124496a23de97ac76e1 not found: ID does not exist"
Nov 01 03:42:09.137773 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.136650    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-2sbzb"]
Nov 01 03:42:09.140724 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.140388    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-2sbzb"]
Nov 01 03:42:09.161421 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.161398    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-mk69d"
Nov 01 03:42:09.179727 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.179702    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hsfpq"
Nov 01 03:42:09.216030 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.215934    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4988f434-dc04-41bd-8b3f-6e386bca1334-istio-token\" (UniqueName: \"kubernetes.io/projected/4988f434-dc04-41bd-8b3f-6e386bca1334-istio-token\") pod \"4988f434-dc04-41bd-8b3f-6e386bca1334\" (UID: \"4988f434-dc04-41bd-8b3f-6e386bca1334\") "
Nov 01 03:42:09.216629 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.216097    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4988f434-dc04-41bd-8b3f-6e386bca1334-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4988f434-dc04-41bd-8b3f-6e386bca1334-credential-socket\") pod \"4988f434-dc04-41bd-8b3f-6e386bca1334\" (UID: \"4988f434-dc04-41bd-8b3f-6e386bca1334\") "
Nov 01 03:42:09.216629 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.216208    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4988f434-dc04-41bd-8b3f-6e386bca1334-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4988f434-dc04-41bd-8b3f-6e386bca1334-workload-socket\") pod \"4988f434-dc04-41bd-8b3f-6e386bca1334\" (UID: \"4988f434-dc04-41bd-8b3f-6e386bca1334\") "
Nov 01 03:42:09.216629 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.216256    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4988f434-dc04-41bd-8b3f-6e386bca1334-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4988f434-dc04-41bd-8b3f-6e386bca1334-workload-certs\") pod \"4988f434-dc04-41bd-8b3f-6e386bca1334\" (UID: \"4988f434-dc04-41bd-8b3f-6e386bca1334\") "
Nov 01 03:42:09.216629 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.216286    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4988f434-dc04-41bd-8b3f-6e386bca1334-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4988f434-dc04-41bd-8b3f-6e386bca1334-istiod-ca-cert\") pod \"4988f434-dc04-41bd-8b3f-6e386bca1334\" (UID: \"4988f434-dc04-41bd-8b3f-6e386bca1334\") "
Nov 01 03:42:09.216629 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.216412    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4988f434-dc04-41bd-8b3f-6e386bca1334-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4988f434-dc04-41bd-8b3f-6e386bca1334-istio-data\") pod \"4988f434-dc04-41bd-8b3f-6e386bca1334\" (UID: \"4988f434-dc04-41bd-8b3f-6e386bca1334\") "
Nov 01 03:42:09.216629 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.216445    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4988f434-dc04-41bd-8b3f-6e386bca1334-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4988f434-dc04-41bd-8b3f-6e386bca1334-istio-envoy\") pod \"4988f434-dc04-41bd-8b3f-6e386bca1334\" (UID: \"4988f434-dc04-41bd-8b3f-6e386bca1334\") "
Nov 01 03:42:09.216629 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.216528    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4988f434-dc04-41bd-8b3f-6e386bca1334-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4988f434-dc04-41bd-8b3f-6e386bca1334-istio-podinfo\") pod \"4988f434-dc04-41bd-8b3f-6e386bca1334\" (UID: \"4988f434-dc04-41bd-8b3f-6e386bca1334\") "
Nov 01 03:42:09.216629 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.216567    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4988f434-dc04-41bd-8b3f-6e386bca1334-kube-api-access-lvrpv\" (UniqueName: \"kubernetes.io/projected/4988f434-dc04-41bd-8b3f-6e386bca1334-kube-api-access-lvrpv\") pod \"4988f434-dc04-41bd-8b3f-6e386bca1334\" (UID: \"4988f434-dc04-41bd-8b3f-6e386bca1334\") "
Nov 01 03:42:09.216629 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.216613    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4988f434-dc04-41bd-8b3f-6e386bca1334-workload-socket" pod "4988f434-dc04-41bd-8b3f-6e386bca1334" (UID: "4988f434-dc04-41bd-8b3f-6e386bca1334"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:42:09.218062 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.216805    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4988f434-dc04-41bd-8b3f-6e386bca1334-credential-socket" pod "4988f434-dc04-41bd-8b3f-6e386bca1334" (UID: "4988f434-dc04-41bd-8b3f-6e386bca1334"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:42:09.218062 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.217428    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4988f434-dc04-41bd-8b3f-6e386bca1334-workload-certs" pod "4988f434-dc04-41bd-8b3f-6e386bca1334" (UID: "4988f434-dc04-41bd-8b3f-6e386bca1334"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:42:09.218062 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.217735    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4988f434-dc04-41bd-8b3f-6e386bca1334-istiod-ca-cert" pod "4988f434-dc04-41bd-8b3f-6e386bca1334" (UID: "4988f434-dc04-41bd-8b3f-6e386bca1334"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:42:09.218062 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.218049    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4988f434-dc04-41bd-8b3f-6e386bca1334-istio-data" pod "4988f434-dc04-41bd-8b3f-6e386bca1334" (UID: "4988f434-dc04-41bd-8b3f-6e386bca1334"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:42:09.223035 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.223015    4400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4988f434-dc04-41bd-8b3f-6e386bca1334-credential-socket\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:09.223196 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.223182    4400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4988f434-dc04-41bd-8b3f-6e386bca1334-workload-socket\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:09.223286 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.223276    4400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4988f434-dc04-41bd-8b3f-6e386bca1334-workload-certs\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:09.223362 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.223352    4400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4988f434-dc04-41bd-8b3f-6e386bca1334-istiod-ca-cert\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:09.223438 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.223428    4400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4988f434-dc04-41bd-8b3f-6e386bca1334-istio-data\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:09.224058 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.224034    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4988f434-dc04-41bd-8b3f-6e386bca1334-istio-envoy" pod "4988f434-dc04-41bd-8b3f-6e386bca1334" (UID: "4988f434-dc04-41bd-8b3f-6e386bca1334"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:42:09.224125 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.224051    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4988f434-dc04-41bd-8b3f-6e386bca1334-istio-podinfo" pod "4988f434-dc04-41bd-8b3f-6e386bca1334" (UID: "4988f434-dc04-41bd-8b3f-6e386bca1334"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 03:42:09.224197 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.224180    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4988f434-dc04-41bd-8b3f-6e386bca1334-istio-token" pod "4988f434-dc04-41bd-8b3f-6e386bca1334" (UID: "4988f434-dc04-41bd-8b3f-6e386bca1334"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:42:09.225764 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.225743    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4988f434-dc04-41bd-8b3f-6e386bca1334-kube-api-access-lvrpv" pod "4988f434-dc04-41bd-8b3f-6e386bca1334" (UID: "4988f434-dc04-41bd-8b3f-6e386bca1334"). InnerVolumeSpecName "kube-api-access-lvrpv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:42:09.252653 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.252601    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-57cp9"]
Nov 01 03:42:09.256778 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.256757    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-57cp9"]
Nov 01 03:42:09.261172 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.261150    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lv9hr"
Nov 01 03:42:09.325623 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.325480    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cb9c447-7466-4f2e-9449-d153b9b8ec2c-istio-token\" (UniqueName: \"kubernetes.io/projected/3cb9c447-7466-4f2e-9449-d153b9b8ec2c-istio-token\") pod \"3cb9c447-7466-4f2e-9449-d153b9b8ec2c\" (UID: \"3cb9c447-7466-4f2e-9449-d153b9b8ec2c\") "
Nov 01 03:42:09.325623 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.325543    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed1e2049-23c6-43e6-811c-fb0c5a9541b5-istio-token\" (UniqueName: \"kubernetes.io/projected/ed1e2049-23c6-43e6-811c-fb0c5a9541b5-istio-token\") pod \"ed1e2049-23c6-43e6-811c-fb0c5a9541b5\" (UID: \"ed1e2049-23c6-43e6-811c-fb0c5a9541b5\") "
Nov 01 03:42:09.325623 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.325595    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3cb9c447-7466-4f2e-9449-d153b9b8ec2c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cb9c447-7466-4f2e-9449-d153b9b8ec2c-istiod-ca-cert\") pod \"3cb9c447-7466-4f2e-9449-d153b9b8ec2c\" (UID: \"3cb9c447-7466-4f2e-9449-d153b9b8ec2c\") "
Nov 01 03:42:09.325623 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.325631    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cb9c447-7466-4f2e-9449-d153b9b8ec2c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cb9c447-7466-4f2e-9449-d153b9b8ec2c-workload-certs\") pod \"3cb9c447-7466-4f2e-9449-d153b9b8ec2c\" (UID: \"3cb9c447-7466-4f2e-9449-d153b9b8ec2c\") "
Nov 01 03:42:09.326008 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.325659    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cb9c447-7466-4f2e-9449-d153b9b8ec2c-kube-api-access-dz6ct\" (UniqueName: \"kubernetes.io/projected/3cb9c447-7466-4f2e-9449-d153b9b8ec2c-kube-api-access-dz6ct\") pod \"3cb9c447-7466-4f2e-9449-d153b9b8ec2c\" (UID: \"3cb9c447-7466-4f2e-9449-d153b9b8ec2c\") "
Nov 01 03:42:09.326008 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.325729    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed1e2049-23c6-43e6-811c-fb0c5a9541b5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed1e2049-23c6-43e6-811c-fb0c5a9541b5-istiod-ca-cert\") pod \"ed1e2049-23c6-43e6-811c-fb0c5a9541b5\" (UID: \"ed1e2049-23c6-43e6-811c-fb0c5a9541b5\") "
Nov 01 03:42:09.326008 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.325764    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cb9c447-7466-4f2e-9449-d153b9b8ec2c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cb9c447-7466-4f2e-9449-d153b9b8ec2c-istio-data\") pod \"3cb9c447-7466-4f2e-9449-d153b9b8ec2c\" (UID: \"3cb9c447-7466-4f2e-9449-d153b9b8ec2c\") "
Nov 01 03:42:09.326008 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.325832    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed1e2049-23c6-43e6-811c-fb0c5a9541b5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed1e2049-23c6-43e6-811c-fb0c5a9541b5-istio-envoy\") pod \"ed1e2049-23c6-43e6-811c-fb0c5a9541b5\" (UID: \"ed1e2049-23c6-43e6-811c-fb0c5a9541b5\") "
Nov 01 03:42:09.326008 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.325870    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cb9c447-7466-4f2e-9449-d153b9b8ec2c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cb9c447-7466-4f2e-9449-d153b9b8ec2c-credential-socket\") pod \"3cb9c447-7466-4f2e-9449-d153b9b8ec2c\" (UID: \"3cb9c447-7466-4f2e-9449-d153b9b8ec2c\") "
Nov 01 03:42:09.326008 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.325964    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed1e2049-23c6-43e6-811c-fb0c5a9541b5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed1e2049-23c6-43e6-811c-fb0c5a9541b5-workload-socket\") pod \"ed1e2049-23c6-43e6-811c-fb0c5a9541b5\" (UID: \"ed1e2049-23c6-43e6-811c-fb0c5a9541b5\") "
Nov 01 03:42:09.326008 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.326008    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed1e2049-23c6-43e6-811c-fb0c5a9541b5-kube-api-access-x2s7p\" (UniqueName: \"kubernetes.io/projected/ed1e2049-23c6-43e6-811c-fb0c5a9541b5-kube-api-access-x2s7p\") pod \"ed1e2049-23c6-43e6-811c-fb0c5a9541b5\" (UID: \"ed1e2049-23c6-43e6-811c-fb0c5a9541b5\") "
Nov 01 03:42:09.326558 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.326037    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed1e2049-23c6-43e6-811c-fb0c5a9541b5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed1e2049-23c6-43e6-811c-fb0c5a9541b5-credential-socket\") pod \"ed1e2049-23c6-43e6-811c-fb0c5a9541b5\" (UID: \"ed1e2049-23c6-43e6-811c-fb0c5a9541b5\") "
Nov 01 03:42:09.326558 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.326078    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed1e2049-23c6-43e6-811c-fb0c5a9541b5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed1e2049-23c6-43e6-811c-fb0c5a9541b5-istio-data\") pod \"ed1e2049-23c6-43e6-811c-fb0c5a9541b5\" (UID: \"ed1e2049-23c6-43e6-811c-fb0c5a9541b5\") "
Nov 01 03:42:09.326558 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.326181    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cb9c447-7466-4f2e-9449-d153b9b8ec2c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cb9c447-7466-4f2e-9449-d153b9b8ec2c-istio-envoy\") pod \"3cb9c447-7466-4f2e-9449-d153b9b8ec2c\" (UID: \"3cb9c447-7466-4f2e-9449-d153b9b8ec2c\") "
Nov 01 03:42:09.326558 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.326266    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cb9c447-7466-4f2e-9449-d153b9b8ec2c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cb9c447-7466-4f2e-9449-d153b9b8ec2c-workload-socket\") pod \"3cb9c447-7466-4f2e-9449-d153b9b8ec2c\" (UID: \"3cb9c447-7466-4f2e-9449-d153b9b8ec2c\") "
Nov 01 03:42:09.326558 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.326329    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ed1e2049-23c6-43e6-811c-fb0c5a9541b5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed1e2049-23c6-43e6-811c-fb0c5a9541b5-istio-podinfo\") pod \"ed1e2049-23c6-43e6-811c-fb0c5a9541b5\" (UID: \"ed1e2049-23c6-43e6-811c-fb0c5a9541b5\") "
Nov 01 03:42:09.326558 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.326461    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3cb9c447-7466-4f2e-9449-d153b9b8ec2c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cb9c447-7466-4f2e-9449-d153b9b8ec2c-istio-podinfo\") pod \"3cb9c447-7466-4f2e-9449-d153b9b8ec2c\" (UID: \"3cb9c447-7466-4f2e-9449-d153b9b8ec2c\") "
Nov 01 03:42:09.326558 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.326498    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed1e2049-23c6-43e6-811c-fb0c5a9541b5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed1e2049-23c6-43e6-811c-fb0c5a9541b5-workload-certs\") pod \"ed1e2049-23c6-43e6-811c-fb0c5a9541b5\" (UID: \"ed1e2049-23c6-43e6-811c-fb0c5a9541b5\") "
Nov 01 03:42:09.326913 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.326714    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cb9c447-7466-4f2e-9449-d153b9b8ec2c-credential-socket" pod "3cb9c447-7466-4f2e-9449-d153b9b8ec2c" (UID: "3cb9c447-7466-4f2e-9449-d153b9b8ec2c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:42:09.326913 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.326764    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed1e2049-23c6-43e6-811c-fb0c5a9541b5-istiod-ca-cert" pod "ed1e2049-23c6-43e6-811c-fb0c5a9541b5" (UID: "ed1e2049-23c6-43e6-811c-fb0c5a9541b5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:42:09.326913 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.326884    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed1e2049-23c6-43e6-811c-fb0c5a9541b5-istio-data" pod "ed1e2049-23c6-43e6-811c-fb0c5a9541b5" (UID: "ed1e2049-23c6-43e6-811c-fb0c5a9541b5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:42:09.326913 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.326904    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cb9c447-7466-4f2e-9449-d153b9b8ec2c-istiod-ca-cert" pod "3cb9c447-7466-4f2e-9449-d153b9b8ec2c" (UID: "3cb9c447-7466-4f2e-9449-d153b9b8ec2c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:42:09.327115 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.326919    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed1e2049-23c6-43e6-811c-fb0c5a9541b5-workload-socket" pod "ed1e2049-23c6-43e6-811c-fb0c5a9541b5" (UID: "ed1e2049-23c6-43e6-811c-fb0c5a9541b5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:42:09.327115 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.327094    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cb9c447-7466-4f2e-9449-d153b9b8ec2c-workload-certs" pod "3cb9c447-7466-4f2e-9449-d153b9b8ec2c" (UID: "3cb9c447-7466-4f2e-9449-d153b9b8ec2c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:42:09.327221 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.327184    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cb9c447-7466-4f2e-9449-d153b9b8ec2c-workload-socket" pod "3cb9c447-7466-4f2e-9449-d153b9b8ec2c" (UID: "3cb9c447-7466-4f2e-9449-d153b9b8ec2c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:42:09.327956 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.327933    4400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cb9c447-7466-4f2e-9449-d153b9b8ec2c-workload-socket\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:09.327956 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.327960    4400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4988f434-dc04-41bd-8b3f-6e386bca1334-istio-envoy\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:09.327956 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.327974    4400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4988f434-dc04-41bd-8b3f-6e386bca1334-istio-podinfo\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:09.327956 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.327991    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvrpv\" (UniqueName: \"kubernetes.io/projected/4988f434-dc04-41bd-8b3f-6e386bca1334-kube-api-access-lvrpv\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:09.327956 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.328007    4400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cb9c447-7466-4f2e-9449-d153b9b8ec2c-istiod-ca-cert\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:09.327956 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.328021    4400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4988f434-dc04-41bd-8b3f-6e386bca1334-istio-token\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:09.327956 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.328035    4400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cb9c447-7466-4f2e-9449-d153b9b8ec2c-workload-certs\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:09.327956 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.328049    4400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed1e2049-23c6-43e6-811c-fb0c5a9541b5-istiod-ca-cert\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:09.327956 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.328063    4400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cb9c447-7466-4f2e-9449-d153b9b8ec2c-credential-socket\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:09.327956 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.328076    4400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed1e2049-23c6-43e6-811c-fb0c5a9541b5-workload-socket\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:09.328657 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.328090    4400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed1e2049-23c6-43e6-811c-fb0c5a9541b5-istio-data\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:09.328657 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.328322    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed1e2049-23c6-43e6-811c-fb0c5a9541b5-credential-socket" pod "ed1e2049-23c6-43e6-811c-fb0c5a9541b5" (UID: "ed1e2049-23c6-43e6-811c-fb0c5a9541b5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:42:09.329973 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.329601    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed1e2049-23c6-43e6-811c-fb0c5a9541b5-istio-envoy" pod "ed1e2049-23c6-43e6-811c-fb0c5a9541b5" (UID: "ed1e2049-23c6-43e6-811c-fb0c5a9541b5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:42:09.329973 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.329705    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ed1e2049-23c6-43e6-811c-fb0c5a9541b5-istio-podinfo" pod "ed1e2049-23c6-43e6-811c-fb0c5a9541b5" (UID: "ed1e2049-23c6-43e6-811c-fb0c5a9541b5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 03:42:09.329973 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.329779    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cb9c447-7466-4f2e-9449-d153b9b8ec2c-istio-token" pod "3cb9c447-7466-4f2e-9449-d153b9b8ec2c" (UID: "3cb9c447-7466-4f2e-9449-d153b9b8ec2c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:42:09.329973 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.329929    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed1e2049-23c6-43e6-811c-fb0c5a9541b5-workload-certs" pod "ed1e2049-23c6-43e6-811c-fb0c5a9541b5" (UID: "ed1e2049-23c6-43e6-811c-fb0c5a9541b5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:42:09.330435 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.330056    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed1e2049-23c6-43e6-811c-fb0c5a9541b5-kube-api-access-x2s7p" pod "ed1e2049-23c6-43e6-811c-fb0c5a9541b5" (UID: "ed1e2049-23c6-43e6-811c-fb0c5a9541b5"). InnerVolumeSpecName "kube-api-access-x2s7p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:42:09.330435 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.330331    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cb9c447-7466-4f2e-9449-d153b9b8ec2c-kube-api-access-dz6ct" pod "3cb9c447-7466-4f2e-9449-d153b9b8ec2c" (UID: "3cb9c447-7466-4f2e-9449-d153b9b8ec2c"). InnerVolumeSpecName "kube-api-access-dz6ct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:42:09.330711 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.330654    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cb9c447-7466-4f2e-9449-d153b9b8ec2c-istio-data" pod "3cb9c447-7466-4f2e-9449-d153b9b8ec2c" (UID: "3cb9c447-7466-4f2e-9449-d153b9b8ec2c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:42:09.330906 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.330885    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed1e2049-23c6-43e6-811c-fb0c5a9541b5-istio-token" pod "ed1e2049-23c6-43e6-811c-fb0c5a9541b5" (UID: "ed1e2049-23c6-43e6-811c-fb0c5a9541b5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:42:09.331380 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.331358    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cb9c447-7466-4f2e-9449-d153b9b8ec2c-istio-envoy" pod "3cb9c447-7466-4f2e-9449-d153b9b8ec2c" (UID: "3cb9c447-7466-4f2e-9449-d153b9b8ec2c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:42:09.331478 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.331461    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3cb9c447-7466-4f2e-9449-d153b9b8ec2c-istio-podinfo" pod "3cb9c447-7466-4f2e-9449-d153b9b8ec2c" (UID: "3cb9c447-7466-4f2e-9449-d153b9b8ec2c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 03:42:09.369879 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.369858    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-hnd5d"]
Nov 01 03:42:09.373526 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.373506    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-hnd5d"]
Nov 01 03:42:09.429251 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.429226    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dz6ct\" (UniqueName: \"kubernetes.io/projected/3cb9c447-7466-4f2e-9449-d153b9b8ec2c-kube-api-access-dz6ct\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:09.429346 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.429257    4400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cb9c447-7466-4f2e-9449-d153b9b8ec2c-istio-data\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:09.429346 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.429271    4400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed1e2049-23c6-43e6-811c-fb0c5a9541b5-istio-envoy\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:09.429346 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.429280    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2s7p\" (UniqueName: \"kubernetes.io/projected/ed1e2049-23c6-43e6-811c-fb0c5a9541b5-kube-api-access-x2s7p\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:09.429346 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.429289    4400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed1e2049-23c6-43e6-811c-fb0c5a9541b5-credential-socket\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:09.429346 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.429300    4400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cb9c447-7466-4f2e-9449-d153b9b8ec2c-istio-envoy\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:09.429346 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.429308    4400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed1e2049-23c6-43e6-811c-fb0c5a9541b5-istio-podinfo\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:09.429346 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.429317    4400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cb9c447-7466-4f2e-9449-d153b9b8ec2c-istio-podinfo\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:09.429346 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.429325    4400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed1e2049-23c6-43e6-811c-fb0c5a9541b5-workload-certs\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:09.429346 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.429332    4400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cb9c447-7466-4f2e-9449-d153b9b8ec2c-istio-token\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:09.429346 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.429340    4400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed1e2049-23c6-43e6-811c-fb0c5a9541b5-istio-token\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:09.617006 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.616976    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="543cb5b7-afc2-4042-af65-69bf92772c80" path="/var/lib/kubelet/pods/543cb5b7-afc2-4042-af65-69bf92772c80/volumes"
Nov 01 03:42:09.617549 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.617529    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="863e73b9-dddf-41c3-9a3d-eca8d7e2b727" path="/var/lib/kubelet/pods/863e73b9-dddf-41c3-9a3d-eca8d7e2b727/volumes"
Nov 01 03:42:09.618042 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.618023    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a03a2ca3-9463-4838-8c2f-a37293da9e6a" path="/var/lib/kubelet/pods/a03a2ca3-9463-4838-8c2f-a37293da9e6a/volumes"
Nov 01 03:42:09.992603 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.992577    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lv9hr"
Nov 01 03:42:09.992800 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.992601    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lv9hr" event={"ID":"ed1e2049-23c6-43e6-811c-fb0c5a9541b5","Type":"ContainerDied","Data":"3cff5ab83bf57f5ee546aad90aa96b33422e1ebddc7f1d147db05696788c3752"}
Nov 01 03:42:09.992800 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.992666    4400 scope.go:122] "RemoveContainer" containerID="899f076ad230e0c666313f4c60979453677c213c000b89605566eaca2d7c7449"
Nov 01 03:42:09.996061 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.995999    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-mk69d"
Nov 01 03:42:09.996186 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.995993    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-mk69d" event={"ID":"4988f434-dc04-41bd-8b3f-6e386bca1334","Type":"ContainerDied","Data":"32a1e9425bc4ad14558c06fcced5c07acec8b57c841ffa4801bf4053bfcd262e"}
Nov 01 03:42:09.998349 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.997741    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hsfpq"
Nov 01 03:42:09.998349 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:09.997916    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hsfpq" event={"ID":"3cb9c447-7466-4f2e-9449-d153b9b8ec2c","Type":"ContainerDied","Data":"42e311494b0cbc676556b8ec9ba25d8bad9b8bf1ade2142eb04e147c1fb862a0"}
Nov 01 03:42:10.004478 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:10.004284    4400 scope.go:122] "RemoveContainer" containerID="c8de3e02ecb3cc679bfa7ff7557f3c7baedf4eb48cdf77aaa4f24aca38b9302f"
Nov 01 03:42:10.013540 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:10.013520    4400 scope.go:122] "RemoveContainer" containerID="490c384a9a528bcb5a9456833fe3ada4aca2f2907dcaf15580536eb4ff4eb40c"
Nov 01 03:42:10.163128 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:10.163099    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hsfpq"]
Nov 01 03:42:10.167206 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:10.167158    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-hsfpq"]
Nov 01 03:42:10.279070 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:10.279044    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-mk69d"]
Nov 01 03:42:10.285518 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:10.285490    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-mk69d"]
Nov 01 03:42:10.397467 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:10.397437    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lv9hr"]
Nov 01 03:42:10.403177 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:10.403156    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lv9hr"]
Nov 01 03:42:11.616420 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:11.616386    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cb9c447-7466-4f2e-9449-d153b9b8ec2c" path="/var/lib/kubelet/pods/3cb9c447-7466-4f2e-9449-d153b9b8ec2c/volumes"
Nov 01 03:42:11.617080 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:11.617058    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4988f434-dc04-41bd-8b3f-6e386bca1334" path="/var/lib/kubelet/pods/4988f434-dc04-41bd-8b3f-6e386bca1334/volumes"
Nov 01 03:42:11.617646 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:11.617625    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed1e2049-23c6-43e6-811c-fb0c5a9541b5" path="/var/lib/kubelet/pods/ed1e2049-23c6-43e6-811c-fb0c5a9541b5/volumes"
Nov 01 03:42:17.304825 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:17.304788    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2058fb-5995f847c9-mv55z"]
Nov 01 03:42:17.305375 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:17.304983    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2058fb-5995f847c9-mv55z" podUID="1083a71d-f12c-474c-992e-31393098ea20" containerName="maas-api" containerID="cri-o://64ed14ec931c30ce0b95173c5180ab1056f59d97ee15e8461dc1acaa17a6f111" gracePeriod=30
Nov 01 03:42:17.626198 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:17.626169    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2058fb-5995f847c9-mv55z"
Nov 01 03:42:17.640991 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:17.640965    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1083a71d-f12c-474c-992e-31393098ea20-kube-api-access-v6kg8\" (UniqueName: \"kubernetes.io/projected/1083a71d-f12c-474c-992e-31393098ea20-kube-api-access-v6kg8\") pod \"1083a71d-f12c-474c-992e-31393098ea20\" (UID: \"1083a71d-f12c-474c-992e-31393098ea20\") "
Nov 01 03:42:17.641153 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:17.641006    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1083a71d-f12c-474c-992e-31393098ea20-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1083a71d-f12c-474c-992e-31393098ea20-openshift-service-ca\") pod \"1083a71d-f12c-474c-992e-31393098ea20\" (UID: \"1083a71d-f12c-474c-992e-31393098ea20\") "
Nov 01 03:42:17.641219 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:17.641184    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1083a71d-f12c-474c-992e-31393098ea20-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1083a71d-f12c-474c-992e-31393098ea20-metrics-certs\") pod \"1083a71d-f12c-474c-992e-31393098ea20\" (UID: \"1083a71d-f12c-474c-992e-31393098ea20\") "
Nov 01 03:42:17.641273 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:17.641246    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1083a71d-f12c-474c-992e-31393098ea20-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1083a71d-f12c-474c-992e-31393098ea20-maas-api-tls\") pod \"1083a71d-f12c-474c-992e-31393098ea20\" (UID: \"1083a71d-f12c-474c-992e-31393098ea20\") "
Nov 01 03:42:17.641375 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:17.641344    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1083a71d-f12c-474c-992e-31393098ea20-openshift-service-ca" pod "1083a71d-f12c-474c-992e-31393098ea20" (UID: "1083a71d-f12c-474c-992e-31393098ea20"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:42:17.641801 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:17.641783    4400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1083a71d-f12c-474c-992e-31393098ea20-openshift-service-ca\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:17.643678 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:17.643640    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1083a71d-f12c-474c-992e-31393098ea20-metrics-certs" pod "1083a71d-f12c-474c-992e-31393098ea20" (UID: "1083a71d-f12c-474c-992e-31393098ea20"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:42:17.643837 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:17.643812    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1083a71d-f12c-474c-992e-31393098ea20-kube-api-access-v6kg8" pod "1083a71d-f12c-474c-992e-31393098ea20" (UID: "1083a71d-f12c-474c-992e-31393098ea20"). InnerVolumeSpecName "kube-api-access-v6kg8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:42:17.643963 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:17.643938    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1083a71d-f12c-474c-992e-31393098ea20-maas-api-tls" pod "1083a71d-f12c-474c-992e-31393098ea20" (UID: "1083a71d-f12c-474c-992e-31393098ea20"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:42:17.742645 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:17.742615    4400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1083a71d-f12c-474c-992e-31393098ea20-metrics-certs\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:17.742645 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:17.742643    4400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1083a71d-f12c-474c-992e-31393098ea20-maas-api-tls\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:17.742840 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:17.742658    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6kg8\" (UniqueName: \"kubernetes.io/projected/1083a71d-f12c-474c-992e-31393098ea20-kube-api-access-v6kg8\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:18.057841 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:18.057758    4400 generic.go:356] "Generic (PLEG): container finished" podID="1083a71d-f12c-474c-992e-31393098ea20" containerID="64ed14ec931c30ce0b95173c5180ab1056f59d97ee15e8461dc1acaa17a6f111" exitCode=0
Nov 01 03:42:18.057841 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:18.057801    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2058fb-5995f847c9-mv55z"
Nov 01 03:42:18.057841 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:18.057833    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2058fb-5995f847c9-mv55z" event={"ID":"1083a71d-f12c-474c-992e-31393098ea20","Type":"ContainerDied","Data":"64ed14ec931c30ce0b95173c5180ab1056f59d97ee15e8461dc1acaa17a6f111"}
Nov 01 03:42:18.058076 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:18.057883    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2058fb-5995f847c9-mv55z" event={"ID":"1083a71d-f12c-474c-992e-31393098ea20","Type":"ContainerDied","Data":"67b8f70e4f2b6f4a3a486f48c3b43e66f9c344cee4d5cfc0f3d56f413572b977"}
Nov 01 03:42:18.058076 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:18.057904    4400 scope.go:122] "RemoveContainer" containerID="64ed14ec931c30ce0b95173c5180ab1056f59d97ee15e8461dc1acaa17a6f111"
Nov 01 03:42:18.069190 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:18.069173    4400 scope.go:122] "RemoveContainer" containerID="64ed14ec931c30ce0b95173c5180ab1056f59d97ee15e8461dc1acaa17a6f111"
Nov 01 03:42:18.069451 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:42:18.069430    4400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64ed14ec931c30ce0b95173c5180ab1056f59d97ee15e8461dc1acaa17a6f111\": container with ID starting with 64ed14ec931c30ce0b95173c5180ab1056f59d97ee15e8461dc1acaa17a6f111 not found: ID does not exist" containerID="64ed14ec931c30ce0b95173c5180ab1056f59d97ee15e8461dc1acaa17a6f111"
Nov 01 03:42:18.069493 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:18.069458    4400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64ed14ec931c30ce0b95173c5180ab1056f59d97ee15e8461dc1acaa17a6f111"} err="failed to get container status \"64ed14ec931c30ce0b95173c5180ab1056f59d97ee15e8461dc1acaa17a6f111\": rpc error: code = NotFound desc = could not find container \"64ed14ec931c30ce0b95173c5180ab1056f59d97ee15e8461dc1acaa17a6f111\": container with ID starting with 64ed14ec931c30ce0b95173c5180ab1056f59d97ee15e8461dc1acaa17a6f111 not found: ID does not exist"
Nov 01 03:42:18.230835 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:18.230805    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2058fb-5995f847c9-mv55z"]
Nov 01 03:42:18.242244 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:18.242217    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2058fb-5995f847c9-mv55z"]
Nov 01 03:42:18.538793 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:18.538761    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-438641-856ccc5949-tcnlc"]
Nov 01 03:42:18.539240 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:18.538992    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-438641-856ccc5949-tcnlc" podUID="6bdca032-ad0e-4346-8543-6eba340faf44" containerName="maas-api" containerID="cri-o://afdecec6d3ca60021733802b774b97804c6f2f66a005c4f1827435b1cd5b05ec" gracePeriod=30
Nov 01 03:42:18.869406 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:18.869381    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-438641-856ccc5949-tcnlc"
Nov 01 03:42:19.048863 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:19.048828    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bdca032-ad0e-4346-8543-6eba340faf44-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bdca032-ad0e-4346-8543-6eba340faf44-openshift-service-ca\") pod \"6bdca032-ad0e-4346-8543-6eba340faf44\" (UID: \"6bdca032-ad0e-4346-8543-6eba340faf44\") "
Nov 01 03:42:19.049034 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:19.049018    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bdca032-ad0e-4346-8543-6eba340faf44-kube-api-access-6tmdd\" (UniqueName: \"kubernetes.io/projected/6bdca032-ad0e-4346-8543-6eba340faf44-kube-api-access-6tmdd\") pod \"6bdca032-ad0e-4346-8543-6eba340faf44\" (UID: \"6bdca032-ad0e-4346-8543-6eba340faf44\") "
Nov 01 03:42:19.049144 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:19.049127    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6bdca032-ad0e-4346-8543-6eba340faf44-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bdca032-ad0e-4346-8543-6eba340faf44-metrics-certs\") pod \"6bdca032-ad0e-4346-8543-6eba340faf44\" (UID: \"6bdca032-ad0e-4346-8543-6eba340faf44\") "
Nov 01 03:42:19.049212 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:19.049178    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6bdca032-ad0e-4346-8543-6eba340faf44-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bdca032-ad0e-4346-8543-6eba340faf44-maas-api-tls\") pod \"6bdca032-ad0e-4346-8543-6eba340faf44\" (UID: \"6bdca032-ad0e-4346-8543-6eba340faf44\") "
Nov 01 03:42:19.049273 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:19.049241    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bdca032-ad0e-4346-8543-6eba340faf44-openshift-service-ca" pod "6bdca032-ad0e-4346-8543-6eba340faf44" (UID: "6bdca032-ad0e-4346-8543-6eba340faf44"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:42:19.049734 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:19.049710    4400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bdca032-ad0e-4346-8543-6eba340faf44-openshift-service-ca\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:19.051348 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:19.051322    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bdca032-ad0e-4346-8543-6eba340faf44-maas-api-tls" pod "6bdca032-ad0e-4346-8543-6eba340faf44" (UID: "6bdca032-ad0e-4346-8543-6eba340faf44"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:42:19.051435 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:19.051397    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bdca032-ad0e-4346-8543-6eba340faf44-kube-api-access-6tmdd" pod "6bdca032-ad0e-4346-8543-6eba340faf44" (UID: "6bdca032-ad0e-4346-8543-6eba340faf44"). InnerVolumeSpecName "kube-api-access-6tmdd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:42:19.051785 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:19.051764    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bdca032-ad0e-4346-8543-6eba340faf44-metrics-certs" pod "6bdca032-ad0e-4346-8543-6eba340faf44" (UID: "6bdca032-ad0e-4346-8543-6eba340faf44"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:42:19.067889 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:19.067860    4400 generic.go:356] "Generic (PLEG): container finished" podID="6bdca032-ad0e-4346-8543-6eba340faf44" containerID="afdecec6d3ca60021733802b774b97804c6f2f66a005c4f1827435b1cd5b05ec" exitCode=0
Nov 01 03:42:19.067983 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:19.067911    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-438641-856ccc5949-tcnlc"
Nov 01 03:42:19.067983 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:19.067938    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-438641-856ccc5949-tcnlc" event={"ID":"6bdca032-ad0e-4346-8543-6eba340faf44","Type":"ContainerDied","Data":"afdecec6d3ca60021733802b774b97804c6f2f66a005c4f1827435b1cd5b05ec"}
Nov 01 03:42:19.068064 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:19.067996    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-438641-856ccc5949-tcnlc" event={"ID":"6bdca032-ad0e-4346-8543-6eba340faf44","Type":"ContainerDied","Data":"d2bc6b5762622a412a39cfc797d82b79c69b13214be8b82034467bdab48208bc"}
Nov 01 03:42:19.068064 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:19.068018    4400 scope.go:122] "RemoveContainer" containerID="afdecec6d3ca60021733802b774b97804c6f2f66a005c4f1827435b1cd5b05ec"
Nov 01 03:42:19.081842 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:19.081812    4400 scope.go:122] "RemoveContainer" containerID="afdecec6d3ca60021733802b774b97804c6f2f66a005c4f1827435b1cd5b05ec"
Nov 01 03:42:19.082136 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:42:19.082113    4400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afdecec6d3ca60021733802b774b97804c6f2f66a005c4f1827435b1cd5b05ec\": container with ID starting with afdecec6d3ca60021733802b774b97804c6f2f66a005c4f1827435b1cd5b05ec not found: ID does not exist" containerID="afdecec6d3ca60021733802b774b97804c6f2f66a005c4f1827435b1cd5b05ec"
Nov 01 03:42:19.082205 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:19.082146    4400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afdecec6d3ca60021733802b774b97804c6f2f66a005c4f1827435b1cd5b05ec"} err="failed to get container status \"afdecec6d3ca60021733802b774b97804c6f2f66a005c4f1827435b1cd5b05ec\": rpc error: code = NotFound desc = could not find container \"afdecec6d3ca60021733802b774b97804c6f2f66a005c4f1827435b1cd5b05ec\": container with ID starting with afdecec6d3ca60021733802b774b97804c6f2f66a005c4f1827435b1cd5b05ec not found: ID does not exist"
Nov 01 03:42:19.157759 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:19.152135    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tmdd\" (UniqueName: \"kubernetes.io/projected/6bdca032-ad0e-4346-8543-6eba340faf44-kube-api-access-6tmdd\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:19.157759 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:19.152170    4400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bdca032-ad0e-4346-8543-6eba340faf44-metrics-certs\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:19.157759 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:19.152184    4400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bdca032-ad0e-4346-8543-6eba340faf44-maas-api-tls\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:19.244742 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:19.244707    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-438641-856ccc5949-tcnlc"]
Nov 01 03:42:19.250643 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:19.250614    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-438641-856ccc5949-tcnlc"]
Nov 01 03:42:19.616804 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:19.616771    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1083a71d-f12c-474c-992e-31393098ea20" path="/var/lib/kubelet/pods/1083a71d-f12c-474c-992e-31393098ea20/volumes"
Nov 01 03:42:19.617355 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:19.617337    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bdca032-ad0e-4346-8543-6eba340faf44" path="/var/lib/kubelet/pods/6bdca032-ad0e-4346-8543-6eba340faf44/volumes"
Nov 01 03:42:29.028741 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:29.028711    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-k2w66"]
Nov 01 03:42:29.029120 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:29.028911    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-k2w66" podUID="54027127-b211-45d7-a049-e2c6cf7ee964" containerName="istio-proxy" containerID="cri-o://3202fb001781b99e88ecc888ea3c59abb79506c67f2232937b12cd123da6392e" gracePeriod=30
Nov 01 03:42:31.265203 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:31.265156    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7081-2b8e82"]
Nov 01 03:42:31.266816 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:31.266783    4400 cpu_manager.go:414] "containerMap: removing container" podUID="1083a71d-f12c-474c-992e-31393098ea20" containerName="maas-api"
Nov 01 03:42:31.266996 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:31.266981    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1083a71d-f12c-474c-992e-31393098ea20" containerName="maas-api"
Nov 01 03:42:31.267107 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:31.267095    4400 cpu_manager.go:414] "containerMap: removing container" podUID="4988f434-dc04-41bd-8b3f-6e386bca1334" containerName="istio-proxy"
Nov 01 03:42:31.275342 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:31.275316    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4988f434-dc04-41bd-8b3f-6e386bca1334" containerName="istio-proxy"
Nov 01 03:42:31.275699 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:31.275665    4400 cpu_manager.go:414] "containerMap: removing container" podUID="c60732bb-7134-40b1-b5b1-48607d07bd68" containerName="maas-api"
Nov 01 03:42:31.275828 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:31.275814    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c60732bb-7134-40b1-b5b1-48607d07bd68" containerName="maas-api"
Nov 01 03:42:31.275931 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:31.275921    4400 cpu_manager.go:414] "containerMap: removing container" podUID="6bdca032-ad0e-4346-8543-6eba340faf44" containerName="maas-api"
Nov 01 03:42:31.276007 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:31.275996    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bdca032-ad0e-4346-8543-6eba340faf44" containerName="maas-api"
Nov 01 03:42:31.276095 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:31.276085    4400 cpu_manager.go:414] "containerMap: removing container" podUID="543cb5b7-afc2-4042-af65-69bf92772c80" containerName="istio-proxy"
Nov 01 03:42:31.276819 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:31.276788    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="543cb5b7-afc2-4042-af65-69bf92772c80" containerName="istio-proxy"
Nov 01 03:42:31.277309 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:31.277294    4400 cpu_manager.go:414] "containerMap: removing container" podUID="a03a2ca3-9463-4838-8c2f-a37293da9e6a" containerName="istio-proxy"
Nov 01 03:42:31.277408 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:31.277397    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a03a2ca3-9463-4838-8c2f-a37293da9e6a" containerName="istio-proxy"
Nov 01 03:42:31.277490 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:31.277481    4400 cpu_manager.go:414] "containerMap: removing container" podUID="3cb9c447-7466-4f2e-9449-d153b9b8ec2c" containerName="istio-proxy"
Nov 01 03:42:31.277557 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:31.277547    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cb9c447-7466-4f2e-9449-d153b9b8ec2c" containerName="istio-proxy"
Nov 01 03:42:31.278086 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:31.278072    4400 cpu_manager.go:414] "containerMap: removing container" podUID="ed1e2049-23c6-43e6-811c-fb0c5a9541b5" containerName="istio-proxy"
Nov 01 03:42:31.280268 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:31.279911    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed1e2049-23c6-43e6-811c-fb0c5a9541b5" containerName="istio-proxy"
Nov 01 03:42:31.280402 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:31.280390    4400 cpu_manager.go:414] "containerMap: removing container" podUID="884eee93-f740-4ccd-8ec2-12b26f39e775" containerName="maas-api"
Nov 01 03:42:31.280565 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:31.280551    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="884eee93-f740-4ccd-8ec2-12b26f39e775" containerName="maas-api"
Nov 01 03:42:31.280848 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:31.280833    4400 cpu_manager.go:414] "containerMap: removing container" podUID="863e73b9-dddf-41c3-9a3d-eca8d7e2b727" containerName="istio-proxy"
Nov 01 03:42:31.280962 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:31.280921    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="863e73b9-dddf-41c3-9a3d-eca8d7e2b727" containerName="istio-proxy"
Nov 01 03:42:31.281465 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:31.281438    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="4988f434-dc04-41bd-8b3f-6e386bca1334" containerName="istio-proxy"
Nov 01 03:42:31.281578 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:31.281471    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="c60732bb-7134-40b1-b5b1-48607d07bd68" containerName="maas-api"
Nov 01 03:42:31.281578 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:31.281484    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bdca032-ad0e-4346-8543-6eba340faf44" containerName="maas-api"
Nov 01 03:42:31.281578 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:31.281497    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="543cb5b7-afc2-4042-af65-69bf92772c80" containerName="istio-proxy"
Nov 01 03:42:31.281578 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:31.281512    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="a03a2ca3-9463-4838-8c2f-a37293da9e6a" containerName="istio-proxy"
Nov 01 03:42:31.281578 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:31.281523    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cb9c447-7466-4f2e-9449-d153b9b8ec2c" containerName="istio-proxy"
Nov 01 03:42:31.281578 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:31.281536    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed1e2049-23c6-43e6-811c-fb0c5a9541b5" containerName="istio-proxy"
Nov 01 03:42:31.281578 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:31.281545    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="884eee93-f740-4ccd-8ec2-12b26f39e775" containerName="maas-api"
Nov 01 03:42:31.281578 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:31.281555    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="863e73b9-dddf-41c3-9a3d-eca8d7e2b727" containerName="istio-proxy"
Nov 01 03:42:31.281578 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:31.281572    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="1083a71d-f12c-474c-992e-31393098ea20" containerName="maas-api"
Nov 01 03:42:31.303620 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:31.299921    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7081-2b8e82"
Nov 01 03:42:31.343578 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:31.343547    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7081-2b8e82"]
Nov 01 03:42:31.356821 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:31.356783    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvr4s\" (UniqueName: \"kubernetes.io/projected/79b90bfa-f43f-44b8-91dd-fe7918e64160-kube-api-access-zvr4s\") pod \"test-curl-7081-2b8e82\" (UID: \"79b90bfa-f43f-44b8-91dd-fe7918e64160\") " pod="opendatahub/test-curl-7081-2b8e82"
Nov 01 03:42:31.379046 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:31.379022    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-h2kzj\""
Nov 01 03:42:31.457848 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:31.457820    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvr4s\" (UniqueName: \"kubernetes.io/projected/79b90bfa-f43f-44b8-91dd-fe7918e64160-kube-api-access-zvr4s\") pod \"test-curl-7081-2b8e82\" (UID: \"79b90bfa-f43f-44b8-91dd-fe7918e64160\") " pod="opendatahub/test-curl-7081-2b8e82"
Nov 01 03:42:31.520464 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:31.520216    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvr4s\" (UniqueName: \"kubernetes.io/projected/79b90bfa-f43f-44b8-91dd-fe7918e64160-kube-api-access-zvr4s\") pod \"test-curl-7081-2b8e82\" (UID: \"79b90bfa-f43f-44b8-91dd-fe7918e64160\") " pod="opendatahub/test-curl-7081-2b8e82"
Nov 01 03:42:31.643098 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:31.643072    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7081-2b8e82"
Nov 01 03:42:31.748905 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:42:31.748869    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:42:32.024309 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:32.023867    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7081-2b8e82"]
Nov 01 03:42:32.024309 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:42:32.023947    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod79b90bfa_f43f_44b8_91dd_fe7918e64160.slice/crio-335cdf82f356cf59deccf232e22bc49b3aef49bfc96bd3a9666a892b8953c02a WatchSource:0}: Error finding container 335cdf82f356cf59deccf232e22bc49b3aef49bfc96bd3a9666a892b8953c02a: Status 404 returned error can't find the container with id 335cdf82f356cf59deccf232e22bc49b3aef49bfc96bd3a9666a892b8953c02a
Nov 01 03:42:32.212100 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:32.212068    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7081-2b8e82" event={"ID":"79b90bfa-f43f-44b8-91dd-fe7918e64160","Type":"ContainerStarted","Data":"91e8ec33a83e1b70e98f435c49a0586a426c4777e672535a99264e2da4be1a46"}
Nov 01 03:42:32.212272 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:32.212108    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7081-2b8e82" event={"ID":"79b90bfa-f43f-44b8-91dd-fe7918e64160","Type":"ContainerStarted","Data":"335cdf82f356cf59deccf232e22bc49b3aef49bfc96bd3a9666a892b8953c02a"}
Nov 01 03:42:33.710941 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:33.710889    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7081-2b8e82" podStartSLOduration=2.710872906 podStartE2EDuration="2.710872906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:42:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:42:32.32773069 +0000 UTC m=+2158.842376554" watchObservedRunningTime="2026-11-01 03:42:33.710872906 +0000 UTC m=+2160.225518768"
Nov 01 03:42:33.711409 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:33.711014    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7081-2b8e82"]
Nov 01 03:42:33.717156 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:33.717128    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7081-2b8e82"]
Nov 01 03:42:33.861978 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:33.861948    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7081-d641d5"]
Nov 01 03:42:33.862620 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:33.862605    4400 cpu_manager.go:414] "containerMap: removing container" podUID="79b90bfa-f43f-44b8-91dd-fe7918e64160" containerName="test-curl-7081-2b8e82"
Nov 01 03:42:33.862676 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:33.862621    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79b90bfa-f43f-44b8-91dd-fe7918e64160" containerName="test-curl-7081-2b8e82"
Nov 01 03:42:33.862777 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:33.862768    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="79b90bfa-f43f-44b8-91dd-fe7918e64160" containerName="test-curl-7081-2b8e82"
Nov 01 03:42:33.866304 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:33.866287    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7081-d641d5"
Nov 01 03:42:33.926907 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:33.926872    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7081-d641d5"]
Nov 01 03:42:34.032401 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:34.032328    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbsx7\" (UniqueName: \"kubernetes.io/projected/e2edb3c2-6fb4-49ea-b3e6-9f59761e1a10-kube-api-access-bbsx7\") pod \"test-curl-7081-d641d5\" (UID: \"e2edb3c2-6fb4-49ea-b3e6-9f59761e1a10\") " pod="opendatahub/test-curl-7081-d641d5"
Nov 01 03:42:34.134348 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:34.134316    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbsx7\" (UniqueName: \"kubernetes.io/projected/e2edb3c2-6fb4-49ea-b3e6-9f59761e1a10-kube-api-access-bbsx7\") pod \"test-curl-7081-d641d5\" (UID: \"e2edb3c2-6fb4-49ea-b3e6-9f59761e1a10\") " pod="opendatahub/test-curl-7081-d641d5"
Nov 01 03:42:34.243904 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:34.237085    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbsx7\" (UniqueName: \"kubernetes.io/projected/e2edb3c2-6fb4-49ea-b3e6-9f59761e1a10-kube-api-access-bbsx7\") pod \"test-curl-7081-d641d5\" (UID: \"e2edb3c2-6fb4-49ea-b3e6-9f59761e1a10\") " pod="opendatahub/test-curl-7081-d641d5"
Nov 01 03:42:34.258783 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:34.258199    4400 generic.go:356] "Generic (PLEG): container finished" podID="54027127-b211-45d7-a049-e2c6cf7ee964" containerID="3202fb001781b99e88ecc888ea3c59abb79506c67f2232937b12cd123da6392e" exitCode=0
Nov 01 03:42:34.258783 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:34.258473    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7081-2b8e82" podUID="79b90bfa-f43f-44b8-91dd-fe7918e64160" containerName="test-curl-7081-2b8e82" containerID="cri-o://91e8ec33a83e1b70e98f435c49a0586a426c4777e672535a99264e2da4be1a46" gracePeriod=2
Nov 01 03:42:34.258783 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:34.258525    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-k2w66" event={"ID":"54027127-b211-45d7-a049-e2c6cf7ee964","Type":"ContainerDied","Data":"3202fb001781b99e88ecc888ea3c59abb79506c67f2232937b12cd123da6392e"}
Nov 01 03:42:34.323705 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:42:34.317963    4400 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7081-2b8e82\" is forbidden: User \"system:node:ip-10-0-133-54.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-54.us-west-1.compute.internal' and this object" podUID="79b90bfa-f43f-44b8-91dd-fe7918e64160" pod="opendatahub/test-curl-7081-2b8e82"
Nov 01 03:42:34.361185 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:34.361155    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-k2w66"
Nov 01 03:42:34.413777 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:42:34.413735    4400 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7081-2b8e82\" is forbidden: User \"system:node:ip-10-0-133-54.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-54.us-west-1.compute.internal' and this object" podUID="79b90bfa-f43f-44b8-91dd-fe7918e64160" pod="opendatahub/test-curl-7081-2b8e82"
Nov 01 03:42:34.479600 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:34.479565    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7081-d641d5"
Nov 01 03:42:34.543076 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:34.542996    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54027127-b211-45d7-a049-e2c6cf7ee964-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54027127-b211-45d7-a049-e2c6cf7ee964-credential-socket\") pod \"54027127-b211-45d7-a049-e2c6cf7ee964\" (UID: \"54027127-b211-45d7-a049-e2c6cf7ee964\") "
Nov 01 03:42:34.543076 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:34.543063    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54027127-b211-45d7-a049-e2c6cf7ee964-kube-api-access-b4pbx\" (UniqueName: \"kubernetes.io/projected/54027127-b211-45d7-a049-e2c6cf7ee964-kube-api-access-b4pbx\") pod \"54027127-b211-45d7-a049-e2c6cf7ee964\" (UID: \"54027127-b211-45d7-a049-e2c6cf7ee964\") "
Nov 01 03:42:34.544152 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:34.543484    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54027127-b211-45d7-a049-e2c6cf7ee964-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54027127-b211-45d7-a049-e2c6cf7ee964-istio-data\") pod \"54027127-b211-45d7-a049-e2c6cf7ee964\" (UID: \"54027127-b211-45d7-a049-e2c6cf7ee964\") "
Nov 01 03:42:34.544152 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:34.543518    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54027127-b211-45d7-a049-e2c6cf7ee964-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54027127-b211-45d7-a049-e2c6cf7ee964-istio-envoy\") pod \"54027127-b211-45d7-a049-e2c6cf7ee964\" (UID: \"54027127-b211-45d7-a049-e2c6cf7ee964\") "
Nov 01 03:42:34.544152 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:34.543589    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54027127-b211-45d7-a049-e2c6cf7ee964-istio-token\" (UniqueName: \"kubernetes.io/projected/54027127-b211-45d7-a049-e2c6cf7ee964-istio-token\") pod \"54027127-b211-45d7-a049-e2c6cf7ee964\" (UID: \"54027127-b211-45d7-a049-e2c6cf7ee964\") "
Nov 01 03:42:34.544152 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:34.543631    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54027127-b211-45d7-a049-e2c6cf7ee964-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54027127-b211-45d7-a049-e2c6cf7ee964-workload-certs\") pod \"54027127-b211-45d7-a049-e2c6cf7ee964\" (UID: \"54027127-b211-45d7-a049-e2c6cf7ee964\") "
Nov 01 03:42:34.544152 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:34.543742    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54027127-b211-45d7-a049-e2c6cf7ee964-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54027127-b211-45d7-a049-e2c6cf7ee964-workload-socket\") pod \"54027127-b211-45d7-a049-e2c6cf7ee964\" (UID: \"54027127-b211-45d7-a049-e2c6cf7ee964\") "
Nov 01 03:42:34.544152 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:34.543874    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/54027127-b211-45d7-a049-e2c6cf7ee964-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54027127-b211-45d7-a049-e2c6cf7ee964-istio-podinfo\") pod \"54027127-b211-45d7-a049-e2c6cf7ee964\" (UID: \"54027127-b211-45d7-a049-e2c6cf7ee964\") "
Nov 01 03:42:34.544152 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:34.543900    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54027127-b211-45d7-a049-e2c6cf7ee964-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54027127-b211-45d7-a049-e2c6cf7ee964-istiod-ca-cert\") pod \"54027127-b211-45d7-a049-e2c6cf7ee964\" (UID: \"54027127-b211-45d7-a049-e2c6cf7ee964\") "
Nov 01 03:42:34.545573 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:34.545542    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54027127-b211-45d7-a049-e2c6cf7ee964-istiod-ca-cert" pod "54027127-b211-45d7-a049-e2c6cf7ee964" (UID: "54027127-b211-45d7-a049-e2c6cf7ee964"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:42:34.545943 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:34.545918    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54027127-b211-45d7-a049-e2c6cf7ee964-credential-socket" pod "54027127-b211-45d7-a049-e2c6cf7ee964" (UID: "54027127-b211-45d7-a049-e2c6cf7ee964"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:42:34.547121 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:34.547060    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54027127-b211-45d7-a049-e2c6cf7ee964-workload-socket" pod "54027127-b211-45d7-a049-e2c6cf7ee964" (UID: "54027127-b211-45d7-a049-e2c6cf7ee964"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:42:34.547239 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:34.547169    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54027127-b211-45d7-a049-e2c6cf7ee964-istio-data" pod "54027127-b211-45d7-a049-e2c6cf7ee964" (UID: "54027127-b211-45d7-a049-e2c6cf7ee964"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:42:34.547573 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:34.547545    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54027127-b211-45d7-a049-e2c6cf7ee964-workload-certs" pod "54027127-b211-45d7-a049-e2c6cf7ee964" (UID: "54027127-b211-45d7-a049-e2c6cf7ee964"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:42:34.548399 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:34.548372    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54027127-b211-45d7-a049-e2c6cf7ee964-kube-api-access-b4pbx" pod "54027127-b211-45d7-a049-e2c6cf7ee964" (UID: "54027127-b211-45d7-a049-e2c6cf7ee964"). InnerVolumeSpecName "kube-api-access-b4pbx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:42:34.549850 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:34.549825    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54027127-b211-45d7-a049-e2c6cf7ee964-istio-token" pod "54027127-b211-45d7-a049-e2c6cf7ee964" (UID: "54027127-b211-45d7-a049-e2c6cf7ee964"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:42:34.550265 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:34.550205    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/54027127-b211-45d7-a049-e2c6cf7ee964-istio-podinfo" pod "54027127-b211-45d7-a049-e2c6cf7ee964" (UID: "54027127-b211-45d7-a049-e2c6cf7ee964"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 03:42:34.551634 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:34.551607    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54027127-b211-45d7-a049-e2c6cf7ee964-istio-envoy" pod "54027127-b211-45d7-a049-e2c6cf7ee964" (UID: "54027127-b211-45d7-a049-e2c6cf7ee964"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:42:34.687728 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:34.679955    4400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54027127-b211-45d7-a049-e2c6cf7ee964-istio-podinfo\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:34.687728 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:34.679992    4400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54027127-b211-45d7-a049-e2c6cf7ee964-istiod-ca-cert\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:34.687728 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:34.680007    4400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54027127-b211-45d7-a049-e2c6cf7ee964-credential-socket\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:34.687728 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:34.680022    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4pbx\" (UniqueName: \"kubernetes.io/projected/54027127-b211-45d7-a049-e2c6cf7ee964-kube-api-access-b4pbx\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:34.687728 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:34.680038    4400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54027127-b211-45d7-a049-e2c6cf7ee964-istio-data\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:34.687728 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:34.680051    4400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54027127-b211-45d7-a049-e2c6cf7ee964-istio-envoy\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:34.687728 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:34.680064    4400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54027127-b211-45d7-a049-e2c6cf7ee964-istio-token\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:34.687728 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:34.680077    4400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54027127-b211-45d7-a049-e2c6cf7ee964-workload-certs\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:34.687728 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:34.680089    4400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54027127-b211-45d7-a049-e2c6cf7ee964-workload-socket\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:34.865059 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:34.865032    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7081-d641d5"]
Nov 01 03:42:34.865470 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:42:34.865130    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode2edb3c2_6fb4_49ea_b3e6_9f59761e1a10.slice/crio-dea0e86fa0229e06c723b58876df175cd5179b546a3175dca7a18aef3f1a9b8d WatchSource:0}: Error finding container dea0e86fa0229e06c723b58876df175cd5179b546a3175dca7a18aef3f1a9b8d: Status 404 returned error can't find the container with id dea0e86fa0229e06c723b58876df175cd5179b546a3175dca7a18aef3f1a9b8d
Nov 01 03:42:34.988563 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:34.988533    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 03:42:35.266653 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:35.266573    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-k2w66"
Nov 01 03:42:35.266653 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:35.266605    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-k2w66" event={"ID":"54027127-b211-45d7-a049-e2c6cf7ee964","Type":"ContainerDied","Data":"f2e1d82cbedc06db81d9174eedd595dccd0e91a8eb0736e04e5fdc8b66ef5a78"}
Nov 01 03:42:35.266897 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:35.266695    4400 scope.go:122] "RemoveContainer" containerID="3202fb001781b99e88ecc888ea3c59abb79506c67f2232937b12cd123da6392e"
Nov 01 03:42:35.268557 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:35.268531    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7081-d641d5" event={"ID":"e2edb3c2-6fb4-49ea-b3e6-9f59761e1a10","Type":"ContainerStarted","Data":"721a4660c236889d83d34f744b704e82d350e383b157ae20f76052f48bb464aa"}
Nov 01 03:42:35.268557 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:35.268562    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7081-d641d5" event={"ID":"e2edb3c2-6fb4-49ea-b3e6-9f59761e1a10","Type":"ContainerStarted","Data":"dea0e86fa0229e06c723b58876df175cd5179b546a3175dca7a18aef3f1a9b8d"}
Nov 01 03:42:35.372803 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:42:35.372771    4400 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7081-2b8e82\" is forbidden: User \"system:node:ip-10-0-133-54.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-54.us-west-1.compute.internal' and this object" podUID="79b90bfa-f43f-44b8-91dd-fe7918e64160" pod="opendatahub/test-curl-7081-2b8e82"
Nov 01 03:42:35.425150 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:42:35.425117    4400 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7081-2b8e82\" is forbidden: User \"system:node:ip-10-0-133-54.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-54.us-west-1.compute.internal' and this object" podUID="79b90bfa-f43f-44b8-91dd-fe7918e64160" pod="opendatahub/test-curl-7081-2b8e82"
Nov 01 03:42:35.551178 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:35.551079    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7081-d641d5" podStartSLOduration=2.551067058 podStartE2EDuration="2.551067058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:42:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:42:35.549216112 +0000 UTC m=+2162.063861975" watchObservedRunningTime="2026-11-01 03:42:35.551067058 +0000 UTC m=+2162.065712920"
Nov 01 03:42:35.674769 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:35.674741    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-k2w66"]
Nov 01 03:42:35.735709 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:35.734059    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-k2w66"]
Nov 01 03:42:36.523368 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:36.523314    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7081-2b8e82"
Nov 01 03:42:36.609716 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:36.606390    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79b90bfa-f43f-44b8-91dd-fe7918e64160-kube-api-access-zvr4s\" (UniqueName: \"kubernetes.io/projected/79b90bfa-f43f-44b8-91dd-fe7918e64160-kube-api-access-zvr4s\") pod \"79b90bfa-f43f-44b8-91dd-fe7918e64160\" (UID: \"79b90bfa-f43f-44b8-91dd-fe7918e64160\") "
Nov 01 03:42:36.654935 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:36.638942    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79b90bfa-f43f-44b8-91dd-fe7918e64160-kube-api-access-zvr4s" pod "79b90bfa-f43f-44b8-91dd-fe7918e64160" (UID: "79b90bfa-f43f-44b8-91dd-fe7918e64160"). InnerVolumeSpecName "kube-api-access-zvr4s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:42:36.708710 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:36.708650    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvr4s\" (UniqueName: \"kubernetes.io/projected/79b90bfa-f43f-44b8-91dd-fe7918e64160-kube-api-access-zvr4s\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:37.064144 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:37.064112    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7081-d641d5"]
Nov 01 03:42:37.068912 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:37.068883    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7081-d641d5"]
Nov 01 03:42:37.284108 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:37.284075    4400 generic.go:356] "Generic (PLEG): container finished" podID="79b90bfa-f43f-44b8-91dd-fe7918e64160" containerID="91e8ec33a83e1b70e98f435c49a0586a426c4777e672535a99264e2da4be1a46" exitCode=137
Nov 01 03:42:37.284285 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:37.284123    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7081-2b8e82"
Nov 01 03:42:37.284285 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:37.284185    4400 scope.go:122] "RemoveContainer" containerID="91e8ec33a83e1b70e98f435c49a0586a426c4777e672535a99264e2da4be1a46"
Nov 01 03:42:37.284418 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:37.284356    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7081-d641d5" podUID="e2edb3c2-6fb4-49ea-b3e6-9f59761e1a10" containerName="test-curl-7081-d641d5" containerID="cri-o://721a4660c236889d83d34f744b704e82d350e383b157ae20f76052f48bb464aa" gracePeriod=2
Nov 01 03:42:37.296239 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:37.296214    4400 scope.go:122] "RemoveContainer" containerID="91e8ec33a83e1b70e98f435c49a0586a426c4777e672535a99264e2da4be1a46"
Nov 01 03:42:37.296522 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:42:37.296492    4400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91e8ec33a83e1b70e98f435c49a0586a426c4777e672535a99264e2da4be1a46\": container with ID starting with 91e8ec33a83e1b70e98f435c49a0586a426c4777e672535a99264e2da4be1a46 not found: ID does not exist" containerID="91e8ec33a83e1b70e98f435c49a0586a426c4777e672535a99264e2da4be1a46"
Nov 01 03:42:37.296614 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:37.296529    4400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91e8ec33a83e1b70e98f435c49a0586a426c4777e672535a99264e2da4be1a46"} err="failed to get container status \"91e8ec33a83e1b70e98f435c49a0586a426c4777e672535a99264e2da4be1a46\": rpc error: code = NotFound desc = could not find container \"91e8ec33a83e1b70e98f435c49a0586a426c4777e672535a99264e2da4be1a46\": container with ID starting with 91e8ec33a83e1b70e98f435c49a0586a426c4777e672535a99264e2da4be1a46 not found: ID does not exist"
Nov 01 03:42:37.337082 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:42:37.337040    4400 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7081-d641d5\" is forbidden: User \"system:node:ip-10-0-133-54.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-54.us-west-1.compute.internal' and this object" podUID="e2edb3c2-6fb4-49ea-b3e6-9f59761e1a10" pod="opendatahub/test-curl-7081-d641d5"
Nov 01 03:42:37.389416 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:42:37.389387    4400 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7081-d641d5\" is forbidden: User \"system:node:ip-10-0-133-54.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-54.us-west-1.compute.internal' and this object" podUID="e2edb3c2-6fb4-49ea-b3e6-9f59761e1a10" pod="opendatahub/test-curl-7081-d641d5"
Nov 01 03:42:37.443795 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:42:37.443768    4400 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7081-2b8e82\" is forbidden: User \"system:node:ip-10-0-133-54.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-54.us-west-1.compute.internal' and this object" podUID="79b90bfa-f43f-44b8-91dd-fe7918e64160" pod="opendatahub/test-curl-7081-2b8e82"
Nov 01 03:42:37.615498 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:37.615451    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54027127-b211-45d7-a049-e2c6cf7ee964" path="/var/lib/kubelet/pods/54027127-b211-45d7-a049-e2c6cf7ee964/volumes"
Nov 01 03:42:37.615870 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:37.615854    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79b90bfa-f43f-44b8-91dd-fe7918e64160" path="/var/lib/kubelet/pods/79b90bfa-f43f-44b8-91dd-fe7918e64160/volumes"
Nov 01 03:42:39.579646 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:39.579622    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7081-d641d5"
Nov 01 03:42:39.639592 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:39.639563    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2edb3c2-6fb4-49ea-b3e6-9f59761e1a10-kube-api-access-bbsx7\" (UniqueName: \"kubernetes.io/projected/e2edb3c2-6fb4-49ea-b3e6-9f59761e1a10-kube-api-access-bbsx7\") pod \"e2edb3c2-6fb4-49ea-b3e6-9f59761e1a10\" (UID: \"e2edb3c2-6fb4-49ea-b3e6-9f59761e1a10\") "
Nov 01 03:42:39.641750 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:39.641723    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2edb3c2-6fb4-49ea-b3e6-9f59761e1a10-kube-api-access-bbsx7" pod "e2edb3c2-6fb4-49ea-b3e6-9f59761e1a10" (UID: "e2edb3c2-6fb4-49ea-b3e6-9f59761e1a10"). InnerVolumeSpecName "kube-api-access-bbsx7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:42:39.741273 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:39.741235    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbsx7\" (UniqueName: \"kubernetes.io/projected/e2edb3c2-6fb4-49ea-b3e6-9f59761e1a10-kube-api-access-bbsx7\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:40.304426 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:40.304390    4400 generic.go:356] "Generic (PLEG): container finished" podID="e2edb3c2-6fb4-49ea-b3e6-9f59761e1a10" containerID="721a4660c236889d83d34f744b704e82d350e383b157ae20f76052f48bb464aa" exitCode=137
Nov 01 03:42:40.304630 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:40.304441    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7081-d641d5"
Nov 01 03:42:40.304630 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:40.304487    4400 scope.go:122] "RemoveContainer" containerID="721a4660c236889d83d34f744b704e82d350e383b157ae20f76052f48bb464aa"
Nov 01 03:42:40.314900 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:40.314885    4400 scope.go:122] "RemoveContainer" containerID="721a4660c236889d83d34f744b704e82d350e383b157ae20f76052f48bb464aa"
Nov 01 03:42:40.315176 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:42:40.315160    4400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"721a4660c236889d83d34f744b704e82d350e383b157ae20f76052f48bb464aa\": container with ID starting with 721a4660c236889d83d34f744b704e82d350e383b157ae20f76052f48bb464aa not found: ID does not exist" containerID="721a4660c236889d83d34f744b704e82d350e383b157ae20f76052f48bb464aa"
Nov 01 03:42:40.315222 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:40.315184    4400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"721a4660c236889d83d34f744b704e82d350e383b157ae20f76052f48bb464aa"} err="failed to get container status \"721a4660c236889d83d34f744b704e82d350e383b157ae20f76052f48bb464aa\": rpc error: code = NotFound desc = could not find container \"721a4660c236889d83d34f744b704e82d350e383b157ae20f76052f48bb464aa\": container with ID starting with 721a4660c236889d83d34f744b704e82d350e383b157ae20f76052f48bb464aa not found: ID does not exist"
Nov 01 03:42:41.654174 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:41.654075    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2edb3c2-6fb4-49ea-b3e6-9f59761e1a10" path="/var/lib/kubelet/pods/e2edb3c2-6fb4-49ea-b3e6-9f59761e1a10/volumes"
Nov 01 03:42:48.146790 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:48.146750    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fdbdfefd-cb877c7cc-2xdgb"]
Nov 01 03:42:48.147895 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:48.147874    4400 cpu_manager.go:414] "containerMap: removing container" podUID="54027127-b211-45d7-a049-e2c6cf7ee964" containerName="istio-proxy"
Nov 01 03:42:48.147974 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:48.147899    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54027127-b211-45d7-a049-e2c6cf7ee964" containerName="istio-proxy"
Nov 01 03:42:48.147974 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:48.147926    4400 cpu_manager.go:414] "containerMap: removing container" podUID="e2edb3c2-6fb4-49ea-b3e6-9f59761e1a10" containerName="test-curl-7081-d641d5"
Nov 01 03:42:48.147974 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:48.147934    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2edb3c2-6fb4-49ea-b3e6-9f59761e1a10" containerName="test-curl-7081-d641d5"
Nov 01 03:42:48.148121 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:48.148106    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="54027127-b211-45d7-a049-e2c6cf7ee964" containerName="istio-proxy"
Nov 01 03:42:48.148171 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:48.148127    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2edb3c2-6fb4-49ea-b3e6-9f59761e1a10" containerName="test-curl-7081-d641d5"
Nov 01 03:42:48.153756 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:48.153736    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fdbdfefd-cb877c7cc-2xdgb"
Nov 01 03:42:48.209057 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:48.209025    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fdbdfefd-cb877c7cc-2xdgb"]
Nov 01 03:42:48.227897 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:48.227866    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f74835f6-1cf9-4580-bef8-c14f88b6978c-metrics-certs\") pod \"maas-api-e2e-mt-fdbdfefd-cb877c7cc-2xdgb\" (UID: \"f74835f6-1cf9-4580-bef8-c14f88b6978c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fdbdfefd-cb877c7cc-2xdgb"
Nov 01 03:42:48.228058 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:48.227926    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f74835f6-1cf9-4580-bef8-c14f88b6978c-openshift-service-ca\") pod \"maas-api-e2e-mt-fdbdfefd-cb877c7cc-2xdgb\" (UID: \"f74835f6-1cf9-4580-bef8-c14f88b6978c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fdbdfefd-cb877c7cc-2xdgb"
Nov 01 03:42:48.228112 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:48.228066    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bwv6\" (UniqueName: \"kubernetes.io/projected/f74835f6-1cf9-4580-bef8-c14f88b6978c-kube-api-access-5bwv6\") pod \"maas-api-e2e-mt-fdbdfefd-cb877c7cc-2xdgb\" (UID: \"f74835f6-1cf9-4580-bef8-c14f88b6978c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fdbdfefd-cb877c7cc-2xdgb"
Nov 01 03:42:48.228289 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:48.228271    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f74835f6-1cf9-4580-bef8-c14f88b6978c-maas-api-tls\") pod \"maas-api-e2e-mt-fdbdfefd-cb877c7cc-2xdgb\" (UID: \"f74835f6-1cf9-4580-bef8-c14f88b6978c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fdbdfefd-cb877c7cc-2xdgb"
Nov 01 03:42:48.255442 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:48.255407    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-fdbdfefd-serving-cert\""
Nov 01 03:42:48.329399 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:48.329364    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f74835f6-1cf9-4580-bef8-c14f88b6978c-openshift-service-ca\") pod \"maas-api-e2e-mt-fdbdfefd-cb877c7cc-2xdgb\" (UID: \"f74835f6-1cf9-4580-bef8-c14f88b6978c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fdbdfefd-cb877c7cc-2xdgb"
Nov 01 03:42:48.329570 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:48.329444    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bwv6\" (UniqueName: \"kubernetes.io/projected/f74835f6-1cf9-4580-bef8-c14f88b6978c-kube-api-access-5bwv6\") pod \"maas-api-e2e-mt-fdbdfefd-cb877c7cc-2xdgb\" (UID: \"f74835f6-1cf9-4580-bef8-c14f88b6978c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fdbdfefd-cb877c7cc-2xdgb"
Nov 01 03:42:48.329637 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:48.329617    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f74835f6-1cf9-4580-bef8-c14f88b6978c-maas-api-tls\") pod \"maas-api-e2e-mt-fdbdfefd-cb877c7cc-2xdgb\" (UID: \"f74835f6-1cf9-4580-bef8-c14f88b6978c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fdbdfefd-cb877c7cc-2xdgb"
Nov 01 03:42:48.329744 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:48.329732    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f74835f6-1cf9-4580-bef8-c14f88b6978c-metrics-certs\") pod \"maas-api-e2e-mt-fdbdfefd-cb877c7cc-2xdgb\" (UID: \"f74835f6-1cf9-4580-bef8-c14f88b6978c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fdbdfefd-cb877c7cc-2xdgb"
Nov 01 03:42:48.330446 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:42:48.329963    4400 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-fdbdfefd-serving-cert: secret "maas-api-e2e-mt-fdbdfefd-serving-cert" not found
Nov 01 03:42:48.330446 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:42:48.330038    4400 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f74835f6-1cf9-4580-bef8-c14f88b6978c-maas-api-tls podName:f74835f6-1cf9-4580-bef8-c14f88b6978c nodeName:}" failed. No retries permitted until 2026-11-01 03:42:48.830013837 +0000 UTC m=+2175.344659691 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f74835f6-1cf9-4580-bef8-c14f88b6978c-maas-api-tls") pod "maas-api-e2e-mt-fdbdfefd-cb877c7cc-2xdgb" (UID: "f74835f6-1cf9-4580-bef8-c14f88b6978c") : secret "maas-api-e2e-mt-fdbdfefd-serving-cert" not found
Nov 01 03:42:48.330446 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:48.330404    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f74835f6-1cf9-4580-bef8-c14f88b6978c-openshift-service-ca\") pod \"maas-api-e2e-mt-fdbdfefd-cb877c7cc-2xdgb\" (UID: \"f74835f6-1cf9-4580-bef8-c14f88b6978c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fdbdfefd-cb877c7cc-2xdgb"
Nov 01 03:42:48.332671 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:48.332650    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f74835f6-1cf9-4580-bef8-c14f88b6978c-metrics-certs\") pod \"maas-api-e2e-mt-fdbdfefd-cb877c7cc-2xdgb\" (UID: \"f74835f6-1cf9-4580-bef8-c14f88b6978c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fdbdfefd-cb877c7cc-2xdgb"
Nov 01 03:42:48.392513 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:48.392474    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bwv6\" (UniqueName: \"kubernetes.io/projected/f74835f6-1cf9-4580-bef8-c14f88b6978c-kube-api-access-5bwv6\") pod \"maas-api-e2e-mt-fdbdfefd-cb877c7cc-2xdgb\" (UID: \"f74835f6-1cf9-4580-bef8-c14f88b6978c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fdbdfefd-cb877c7cc-2xdgb"
Nov 01 03:42:48.838882 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:48.838846    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f74835f6-1cf9-4580-bef8-c14f88b6978c-maas-api-tls\") pod \"maas-api-e2e-mt-fdbdfefd-cb877c7cc-2xdgb\" (UID: \"f74835f6-1cf9-4580-bef8-c14f88b6978c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fdbdfefd-cb877c7cc-2xdgb"
Nov 01 03:42:48.841376 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:48.841348    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f74835f6-1cf9-4580-bef8-c14f88b6978c-maas-api-tls\") pod \"maas-api-e2e-mt-fdbdfefd-cb877c7cc-2xdgb\" (UID: \"f74835f6-1cf9-4580-bef8-c14f88b6978c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fdbdfefd-cb877c7cc-2xdgb"
Nov 01 03:42:49.068516 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:49.068478    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fdbdfefd-cb877c7cc-2xdgb"
Nov 01 03:42:49.496669 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:49.496630    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fdbdfefd-cb877c7cc-2xdgb"]
Nov 01 03:42:50.387873 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:50.387838    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fdbdfefd-cb877c7cc-2xdgb" event={"ID":"f74835f6-1cf9-4580-bef8-c14f88b6978c","Type":"ContainerStarted","Data":"2603e507fa0eff2971751a5aca77b9c5c155752eb15eb2fc8b46008e6c595b6e"}
Nov 01 03:42:51.396161 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:51.396121    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fdbdfefd-cb877c7cc-2xdgb" event={"ID":"f74835f6-1cf9-4580-bef8-c14f88b6978c","Type":"ContainerStarted","Data":"dca4f8aa1b7efd00d149a616c60d63be00390c45203b9d0f0d93d2fcf5b845db"}
Nov 01 03:42:51.396573 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:51.396248    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fdbdfefd-cb877c7cc-2xdgb"
Nov 01 03:42:57.409197 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:57.409164    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fdbdfefd-cb877c7cc-2xdgb"
Nov 01 03:42:57.533428 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:42:57.533375    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fdbdfefd-cb877c7cc-2xdgb" podStartSLOduration=8.733065591 podStartE2EDuration="9.533361254s" totalImagesPullingTime="800.295663ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:42:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:42:51.523549105 +0000 UTC m=+2178.038194978" watchObservedRunningTime="2026-11-01 03:42:57.533361254 +0000 UTC m=+2184.048007116"
Nov 01 03:43:01.730974 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:43:01.730937    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:43:07.674342 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:07.674305    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-e8c789/distinct-2-w4-e8c789-kserve-7977c64f4-wkxb2"]
Nov 01 03:43:07.725923 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:07.725893    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-e8c789/premium-w4-e8c789-kserve-9458f8874-dtpnm"]
Nov 01 03:43:20.509385 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:20.509307    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 03:43:22.695654 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:22.695598    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-e8c789/distinct-2-w4-e8c789-kserve-7977c64f4-wkxb2" podUID="bdb1bb9e-14f5-4be6-a028-c980119f4aa8" containerName="main" containerID="cri-o://14bb147afb82fe8ce409e1f8f4b10936e4771fed96d7a486fbb4200cab18dceb" gracePeriod=45
Nov 01 03:43:22.749997 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:22.749965    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-e8c789/premium-w4-e8c789-kserve-9458f8874-dtpnm" podUID="67bba91a-5ace-4d3f-89da-06bb9db17c78" containerName="main" containerID="cri-o://c217396d376bc52bb8ff8290331a2a1f5b616ad67868deec88e48b0863c34a71" gracePeriod=45
Nov 01 03:43:23.044622 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:23.044594    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e8c789/distinct-2-w4-e8c789-kserve-7977c64f4-wkxb2"
Nov 01 03:43:23.048044 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:23.048024    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e8c789/premium-w4-e8c789-kserve-9458f8874-dtpnm"
Nov 01 03:43:23.164376 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:23.164335    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdb1bb9e-14f5-4be6-a028-c980119f4aa8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdb1bb9e-14f5-4be6-a028-c980119f4aa8-tmp-dir\") pod \"bdb1bb9e-14f5-4be6-a028-c980119f4aa8\" (UID: \"bdb1bb9e-14f5-4be6-a028-c980119f4aa8\") "
Nov 01 03:43:23.164579 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:23.164472    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67bba91a-5ace-4d3f-89da-06bb9db17c78-home\" (UniqueName: \"kubernetes.io/empty-dir/67bba91a-5ace-4d3f-89da-06bb9db17c78-home\") pod \"67bba91a-5ace-4d3f-89da-06bb9db17c78\" (UID: \"67bba91a-5ace-4d3f-89da-06bb9db17c78\") "
Nov 01 03:43:23.164665 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:23.164579    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67bba91a-5ace-4d3f-89da-06bb9db17c78-tls-certs\" (UniqueName: \"kubernetes.io/secret/67bba91a-5ace-4d3f-89da-06bb9db17c78-tls-certs\") pod \"67bba91a-5ace-4d3f-89da-06bb9db17c78\" (UID: \"67bba91a-5ace-4d3f-89da-06bb9db17c78\") "
Nov 01 03:43:23.164665 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:23.164630    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67bba91a-5ace-4d3f-89da-06bb9db17c78-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67bba91a-5ace-4d3f-89da-06bb9db17c78-model-cache\") pod \"67bba91a-5ace-4d3f-89da-06bb9db17c78\" (UID: \"67bba91a-5ace-4d3f-89da-06bb9db17c78\") "
Nov 01 03:43:23.164823 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:23.164661    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdb1bb9e-14f5-4be6-a028-c980119f4aa8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdb1bb9e-14f5-4be6-a028-c980119f4aa8-model-cache\") pod \"bdb1bb9e-14f5-4be6-a028-c980119f4aa8\" (UID: \"bdb1bb9e-14f5-4be6-a028-c980119f4aa8\") "
Nov 01 03:43:23.164823 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:23.164780    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67bba91a-5ace-4d3f-89da-06bb9db17c78-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67bba91a-5ace-4d3f-89da-06bb9db17c78-tmp-dir\") pod \"67bba91a-5ace-4d3f-89da-06bb9db17c78\" (UID: \"67bba91a-5ace-4d3f-89da-06bb9db17c78\") "
Nov 01 03:43:23.164823 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:23.164813    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdb1bb9e-14f5-4be6-a028-c980119f4aa8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdb1bb9e-14f5-4be6-a028-c980119f4aa8-dshm\") pod \"bdb1bb9e-14f5-4be6-a028-c980119f4aa8\" (UID: \"bdb1bb9e-14f5-4be6-a028-c980119f4aa8\") "
Nov 01 03:43:23.164969 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:23.164852    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdb1bb9e-14f5-4be6-a028-c980119f4aa8-kube-api-access-rhwzn\" (UniqueName: \"kubernetes.io/projected/bdb1bb9e-14f5-4be6-a028-c980119f4aa8-kube-api-access-rhwzn\") pod \"bdb1bb9e-14f5-4be6-a028-c980119f4aa8\" (UID: \"bdb1bb9e-14f5-4be6-a028-c980119f4aa8\") "
Nov 01 03:43:23.164969 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:23.164925    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bdb1bb9e-14f5-4be6-a028-c980119f4aa8-tls-certs\" (UniqueName: \"kubernetes.io/secret/bdb1bb9e-14f5-4be6-a028-c980119f4aa8-tls-certs\") pod \"bdb1bb9e-14f5-4be6-a028-c980119f4aa8\" (UID: \"bdb1bb9e-14f5-4be6-a028-c980119f4aa8\") "
Nov 01 03:43:23.165070 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:23.164995    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67bba91a-5ace-4d3f-89da-06bb9db17c78-dshm\" (UniqueName: \"kubernetes.io/empty-dir/67bba91a-5ace-4d3f-89da-06bb9db17c78-dshm\") pod \"67bba91a-5ace-4d3f-89da-06bb9db17c78\" (UID: \"67bba91a-5ace-4d3f-89da-06bb9db17c78\") "
Nov 01 03:43:23.165070 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:23.165021    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdb1bb9e-14f5-4be6-a028-c980119f4aa8-home\" (UniqueName: \"kubernetes.io/empty-dir/bdb1bb9e-14f5-4be6-a028-c980119f4aa8-home\") pod \"bdb1bb9e-14f5-4be6-a028-c980119f4aa8\" (UID: \"bdb1bb9e-14f5-4be6-a028-c980119f4aa8\") "
Nov 01 03:43:23.165167 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:23.165067    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67bba91a-5ace-4d3f-89da-06bb9db17c78-kube-api-access-zngf4\" (UniqueName: \"kubernetes.io/projected/67bba91a-5ace-4d3f-89da-06bb9db17c78-kube-api-access-zngf4\") pod \"67bba91a-5ace-4d3f-89da-06bb9db17c78\" (UID: \"67bba91a-5ace-4d3f-89da-06bb9db17c78\") "
Nov 01 03:43:23.167081 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:23.166755    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdb1bb9e-14f5-4be6-a028-c980119f4aa8-tmp-dir" pod "bdb1bb9e-14f5-4be6-a028-c980119f4aa8" (UID: "bdb1bb9e-14f5-4be6-a028-c980119f4aa8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:43:23.167081 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:23.167017    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67bba91a-5ace-4d3f-89da-06bb9db17c78-tmp-dir" pod "67bba91a-5ace-4d3f-89da-06bb9db17c78" (UID: "67bba91a-5ace-4d3f-89da-06bb9db17c78"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:43:23.167276 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:23.167210    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67bba91a-5ace-4d3f-89da-06bb9db17c78-home" pod "67bba91a-5ace-4d3f-89da-06bb9db17c78" (UID: "67bba91a-5ace-4d3f-89da-06bb9db17c78"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:43:23.167276 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:23.167223    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67bba91a-5ace-4d3f-89da-06bb9db17c78-model-cache" pod "67bba91a-5ace-4d3f-89da-06bb9db17c78" (UID: "67bba91a-5ace-4d3f-89da-06bb9db17c78"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:43:23.167425 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:23.167400    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdb1bb9e-14f5-4be6-a028-c980119f4aa8-home" pod "bdb1bb9e-14f5-4be6-a028-c980119f4aa8" (UID: "bdb1bb9e-14f5-4be6-a028-c980119f4aa8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:43:23.167425 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:23.167412    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdb1bb9e-14f5-4be6-a028-c980119f4aa8-model-cache" pod "bdb1bb9e-14f5-4be6-a028-c980119f4aa8" (UID: "bdb1bb9e-14f5-4be6-a028-c980119f4aa8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:43:23.168904 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:23.168874    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67bba91a-5ace-4d3f-89da-06bb9db17c78-kube-api-access-zngf4" pod "67bba91a-5ace-4d3f-89da-06bb9db17c78" (UID: "67bba91a-5ace-4d3f-89da-06bb9db17c78"). InnerVolumeSpecName "kube-api-access-zngf4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:43:23.171010 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:23.170984    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdb1bb9e-14f5-4be6-a028-c980119f4aa8-tls-certs" pod "bdb1bb9e-14f5-4be6-a028-c980119f4aa8" (UID: "bdb1bb9e-14f5-4be6-a028-c980119f4aa8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:43:23.171010 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:23.170980    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67bba91a-5ace-4d3f-89da-06bb9db17c78-dshm" pod "67bba91a-5ace-4d3f-89da-06bb9db17c78" (UID: "67bba91a-5ace-4d3f-89da-06bb9db17c78"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:43:23.171132 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:23.171030    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdb1bb9e-14f5-4be6-a028-c980119f4aa8-kube-api-access-rhwzn" pod "bdb1bb9e-14f5-4be6-a028-c980119f4aa8" (UID: "bdb1bb9e-14f5-4be6-a028-c980119f4aa8"). InnerVolumeSpecName "kube-api-access-rhwzn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:43:23.171824 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:23.171796    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67bba91a-5ace-4d3f-89da-06bb9db17c78-tls-certs" pod "67bba91a-5ace-4d3f-89da-06bb9db17c78" (UID: "67bba91a-5ace-4d3f-89da-06bb9db17c78"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:43:23.172544 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:23.172517    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdb1bb9e-14f5-4be6-a028-c980119f4aa8-dshm" pod "bdb1bb9e-14f5-4be6-a028-c980119f4aa8" (UID: "bdb1bb9e-14f5-4be6-a028-c980119f4aa8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:43:23.266928 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:23.266849    4400 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdb1bb9e-14f5-4be6-a028-c980119f4aa8-tmp-dir\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:43:23.266928 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:23.266877    4400 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67bba91a-5ace-4d3f-89da-06bb9db17c78-home\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:43:23.266928 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:23.266888    4400 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67bba91a-5ace-4d3f-89da-06bb9db17c78-tls-certs\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:43:23.266928 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:23.266895    4400 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67bba91a-5ace-4d3f-89da-06bb9db17c78-model-cache\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:43:23.266928 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:23.266905    4400 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdb1bb9e-14f5-4be6-a028-c980119f4aa8-model-cache\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:43:23.266928 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:23.266914    4400 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67bba91a-5ace-4d3f-89da-06bb9db17c78-tmp-dir\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:43:23.266928 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:23.266922    4400 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdb1bb9e-14f5-4be6-a028-c980119f4aa8-dshm\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:43:23.266928 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:23.266930    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhwzn\" (UniqueName: \"kubernetes.io/projected/bdb1bb9e-14f5-4be6-a028-c980119f4aa8-kube-api-access-rhwzn\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:43:23.267291 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:23.266939    4400 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdb1bb9e-14f5-4be6-a028-c980119f4aa8-tls-certs\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:43:23.267291 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:23.266948    4400 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67bba91a-5ace-4d3f-89da-06bb9db17c78-dshm\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:43:23.267291 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:23.266956    4400 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdb1bb9e-14f5-4be6-a028-c980119f4aa8-home\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:43:23.267291 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:23.266964    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zngf4\" (UniqueName: \"kubernetes.io/projected/67bba91a-5ace-4d3f-89da-06bb9db17c78-kube-api-access-zngf4\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:43:23.626416 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:23.626381    4400 generic.go:356] "Generic (PLEG): container finished" podID="bdb1bb9e-14f5-4be6-a028-c980119f4aa8" containerID="14bb147afb82fe8ce409e1f8f4b10936e4771fed96d7a486fbb4200cab18dceb" exitCode=0
Nov 01 03:43:23.626606 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:23.626516    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e8c789/distinct-2-w4-e8c789-kserve-7977c64f4-wkxb2"
Nov 01 03:43:23.627904 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:23.627878    4400 generic.go:356] "Generic (PLEG): container finished" podID="67bba91a-5ace-4d3f-89da-06bb9db17c78" containerID="c217396d376bc52bb8ff8290331a2a1f5b616ad67868deec88e48b0863c34a71" exitCode=0
Nov 01 03:43:23.628054 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:23.627997    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e8c789/premium-w4-e8c789-kserve-9458f8874-dtpnm"
Nov 01 03:43:23.629424 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:23.629378    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e8c789/distinct-2-w4-e8c789-kserve-7977c64f4-wkxb2" event={"ID":"bdb1bb9e-14f5-4be6-a028-c980119f4aa8","Type":"ContainerDied","Data":"14bb147afb82fe8ce409e1f8f4b10936e4771fed96d7a486fbb4200cab18dceb"}
Nov 01 03:43:23.629545 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:23.629469    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e8c789/distinct-2-w4-e8c789-kserve-7977c64f4-wkxb2" event={"ID":"bdb1bb9e-14f5-4be6-a028-c980119f4aa8","Type":"ContainerDied","Data":"1da8d9451f703cd6903674237ef17bcd89db25bf6c63620a0771266436482224"}
Nov 01 03:43:23.629545 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:23.629486    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e8c789/premium-w4-e8c789-kserve-9458f8874-dtpnm" event={"ID":"67bba91a-5ace-4d3f-89da-06bb9db17c78","Type":"ContainerDied","Data":"c217396d376bc52bb8ff8290331a2a1f5b616ad67868deec88e48b0863c34a71"}
Nov 01 03:43:23.629545 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:23.629501    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e8c789/premium-w4-e8c789-kserve-9458f8874-dtpnm" event={"ID":"67bba91a-5ace-4d3f-89da-06bb9db17c78","Type":"ContainerDied","Data":"63e1c9437027d5b2d1d055975caaaac31de26a288c10798d49ab1051ee8e5e7e"}
Nov 01 03:43:23.630309 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:23.630279    4400 scope.go:122] "RemoveContainer" containerID="14bb147afb82fe8ce409e1f8f4b10936e4771fed96d7a486fbb4200cab18dceb"
Nov 01 03:43:23.649160 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:23.649141    4400 scope.go:122] "RemoveContainer" containerID="14bb147afb82fe8ce409e1f8f4b10936e4771fed96d7a486fbb4200cab18dceb"
Nov 01 03:43:23.649455 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:43:23.649437    4400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14bb147afb82fe8ce409e1f8f4b10936e4771fed96d7a486fbb4200cab18dceb\": container with ID starting with 14bb147afb82fe8ce409e1f8f4b10936e4771fed96d7a486fbb4200cab18dceb not found: ID does not exist" containerID="14bb147afb82fe8ce409e1f8f4b10936e4771fed96d7a486fbb4200cab18dceb"
Nov 01 03:43:23.649504 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:23.649463    4400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14bb147afb82fe8ce409e1f8f4b10936e4771fed96d7a486fbb4200cab18dceb"} err="failed to get container status \"14bb147afb82fe8ce409e1f8f4b10936e4771fed96d7a486fbb4200cab18dceb\": rpc error: code = NotFound desc = could not find container \"14bb147afb82fe8ce409e1f8f4b10936e4771fed96d7a486fbb4200cab18dceb\": container with ID starting with 14bb147afb82fe8ce409e1f8f4b10936e4771fed96d7a486fbb4200cab18dceb not found: ID does not exist"
Nov 01 03:43:23.649504 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:23.649478    4400 scope.go:122] "RemoveContainer" containerID="c217396d376bc52bb8ff8290331a2a1f5b616ad67868deec88e48b0863c34a71"
Nov 01 03:43:23.658155 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:23.658140    4400 scope.go:122] "RemoveContainer" containerID="c217396d376bc52bb8ff8290331a2a1f5b616ad67868deec88e48b0863c34a71"
Nov 01 03:43:23.658391 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:43:23.658373    4400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c217396d376bc52bb8ff8290331a2a1f5b616ad67868deec88e48b0863c34a71\": container with ID starting with c217396d376bc52bb8ff8290331a2a1f5b616ad67868deec88e48b0863c34a71 not found: ID does not exist" containerID="c217396d376bc52bb8ff8290331a2a1f5b616ad67868deec88e48b0863c34a71"
Nov 01 03:43:23.658458 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:23.658404    4400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c217396d376bc52bb8ff8290331a2a1f5b616ad67868deec88e48b0863c34a71"} err="failed to get container status \"c217396d376bc52bb8ff8290331a2a1f5b616ad67868deec88e48b0863c34a71\": rpc error: code = NotFound desc = could not find container \"c217396d376bc52bb8ff8290331a2a1f5b616ad67868deec88e48b0863c34a71\": container with ID starting with c217396d376bc52bb8ff8290331a2a1f5b616ad67868deec88e48b0863c34a71 not found: ID does not exist"
Nov 01 03:43:23.799512 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:23.799486    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-e8c789/premium-w4-e8c789-kserve-9458f8874-dtpnm"]
Nov 01 03:43:23.849292 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:23.849264    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-e8c789/premium-w4-e8c789-kserve-9458f8874-dtpnm"]
Nov 01 03:43:23.962008 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:23.961939    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-e8c789/distinct-2-w4-e8c789-kserve-7977c64f4-wkxb2"]
Nov 01 03:43:23.964915 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:23.964895    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-e8c789/distinct-2-w4-e8c789-kserve-7977c64f4-wkxb2"]
Nov 01 03:43:25.615613 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:25.615578    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67bba91a-5ace-4d3f-89da-06bb9db17c78" path="/var/lib/kubelet/pods/67bba91a-5ace-4d3f-89da-06bb9db17c78/volumes"
Nov 01 03:43:25.616062 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:25.616047    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdb1bb9e-14f5-4be6-a028-c980119f4aa8" path="/var/lib/kubelet/pods/bdb1bb9e-14f5-4be6-a028-c980119f4aa8/volumes"
Nov 01 03:43:31.729263 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:43:31.729227    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:43:44.511217 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:44.511187    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5fd6cf-78d6995bb5-ffqcs"]
Nov 01 03:43:44.511857 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:44.511844    4400 cpu_manager.go:414] "containerMap: removing container" podUID="67bba91a-5ace-4d3f-89da-06bb9db17c78" containerName="main"
Nov 01 03:43:44.511920 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:44.511858    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67bba91a-5ace-4d3f-89da-06bb9db17c78" containerName="main"
Nov 01 03:43:44.511920 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:44.511876    4400 cpu_manager.go:414] "containerMap: removing container" podUID="bdb1bb9e-14f5-4be6-a028-c980119f4aa8" containerName="main"
Nov 01 03:43:44.511920 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:44.511881    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdb1bb9e-14f5-4be6-a028-c980119f4aa8" containerName="main"
Nov 01 03:43:44.512024 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:44.512009    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="67bba91a-5ace-4d3f-89da-06bb9db17c78" containerName="main"
Nov 01 03:43:44.512024 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:44.512018    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdb1bb9e-14f5-4be6-a028-c980119f4aa8" containerName="main"
Nov 01 03:43:44.517389 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:44.517367    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5fd6cf-78d6995bb5-ffqcs"
Nov 01 03:43:44.575842 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:44.575811    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5fd6cf-78d6995bb5-ffqcs"]
Nov 01 03:43:44.621793 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:44.621765    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-5fd6cf-serving-cert\""
Nov 01 03:43:44.693030 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:44.692996    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73e54a1c-e8c0-4edc-991e-94cf8d331ff5-metrics-certs\") pod \"maas-api-e2e-api-a-5fd6cf-78d6995bb5-ffqcs\" (UID: \"73e54a1c-e8c0-4edc-991e-94cf8d331ff5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5fd6cf-78d6995bb5-ffqcs"
Nov 01 03:43:44.693175 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:44.693046    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73e54a1c-e8c0-4edc-991e-94cf8d331ff5-maas-api-tls\") pod \"maas-api-e2e-api-a-5fd6cf-78d6995bb5-ffqcs\" (UID: \"73e54a1c-e8c0-4edc-991e-94cf8d331ff5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5fd6cf-78d6995bb5-ffqcs"
Nov 01 03:43:44.693217 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:44.693182    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73e54a1c-e8c0-4edc-991e-94cf8d331ff5-openshift-service-ca\") pod \"maas-api-e2e-api-a-5fd6cf-78d6995bb5-ffqcs\" (UID: \"73e54a1c-e8c0-4edc-991e-94cf8d331ff5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5fd6cf-78d6995bb5-ffqcs"
Nov 01 03:43:44.693264 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:44.693225    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67hml\" (UniqueName: \"kubernetes.io/projected/73e54a1c-e8c0-4edc-991e-94cf8d331ff5-kube-api-access-67hml\") pod \"maas-api-e2e-api-a-5fd6cf-78d6995bb5-ffqcs\" (UID: \"73e54a1c-e8c0-4edc-991e-94cf8d331ff5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5fd6cf-78d6995bb5-ffqcs"
Nov 01 03:43:44.794670 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:44.794583    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73e54a1c-e8c0-4edc-991e-94cf8d331ff5-openshift-service-ca\") pod \"maas-api-e2e-api-a-5fd6cf-78d6995bb5-ffqcs\" (UID: \"73e54a1c-e8c0-4edc-991e-94cf8d331ff5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5fd6cf-78d6995bb5-ffqcs"
Nov 01 03:43:44.795160 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:44.795134    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67hml\" (UniqueName: \"kubernetes.io/projected/73e54a1c-e8c0-4edc-991e-94cf8d331ff5-kube-api-access-67hml\") pod \"maas-api-e2e-api-a-5fd6cf-78d6995bb5-ffqcs\" (UID: \"73e54a1c-e8c0-4edc-991e-94cf8d331ff5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5fd6cf-78d6995bb5-ffqcs"
Nov 01 03:43:44.795959 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:44.795921    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73e54a1c-e8c0-4edc-991e-94cf8d331ff5-metrics-certs\") pod \"maas-api-e2e-api-a-5fd6cf-78d6995bb5-ffqcs\" (UID: \"73e54a1c-e8c0-4edc-991e-94cf8d331ff5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5fd6cf-78d6995bb5-ffqcs"
Nov 01 03:43:44.797145 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:44.797115    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73e54a1c-e8c0-4edc-991e-94cf8d331ff5-maas-api-tls\") pod \"maas-api-e2e-api-a-5fd6cf-78d6995bb5-ffqcs\" (UID: \"73e54a1c-e8c0-4edc-991e-94cf8d331ff5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5fd6cf-78d6995bb5-ffqcs"
Nov 01 03:43:44.798046 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:44.795324    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73e54a1c-e8c0-4edc-991e-94cf8d331ff5-openshift-service-ca\") pod \"maas-api-e2e-api-a-5fd6cf-78d6995bb5-ffqcs\" (UID: \"73e54a1c-e8c0-4edc-991e-94cf8d331ff5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5fd6cf-78d6995bb5-ffqcs"
Nov 01 03:43:44.811812 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:44.811768    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73e54a1c-e8c0-4edc-991e-94cf8d331ff5-maas-api-tls\") pod \"maas-api-e2e-api-a-5fd6cf-78d6995bb5-ffqcs\" (UID: \"73e54a1c-e8c0-4edc-991e-94cf8d331ff5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5fd6cf-78d6995bb5-ffqcs"
Nov 01 03:43:44.812000 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:44.811821    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73e54a1c-e8c0-4edc-991e-94cf8d331ff5-metrics-certs\") pod \"maas-api-e2e-api-a-5fd6cf-78d6995bb5-ffqcs\" (UID: \"73e54a1c-e8c0-4edc-991e-94cf8d331ff5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5fd6cf-78d6995bb5-ffqcs"
Nov 01 03:43:44.858169 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:44.858145    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67hml\" (UniqueName: \"kubernetes.io/projected/73e54a1c-e8c0-4edc-991e-94cf8d331ff5-kube-api-access-67hml\") pod \"maas-api-e2e-api-a-5fd6cf-78d6995bb5-ffqcs\" (UID: \"73e54a1c-e8c0-4edc-991e-94cf8d331ff5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5fd6cf-78d6995bb5-ffqcs"
Nov 01 03:43:45.132937 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:45.132907    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5fd6cf-78d6995bb5-ffqcs"
Nov 01 03:43:45.566081 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:45.566056    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5fd6cf-78d6995bb5-ffqcs"]
Nov 01 03:43:45.566412 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:43:45.566152    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73e54a1c_e8c0_4edc_991e_94cf8d331ff5.slice/crio-f92f0f0d9d8eb9780619290abe7cd391a1d2050cb0e5dad055afa75dab3bc601 WatchSource:0}: Error finding container f92f0f0d9d8eb9780619290abe7cd391a1d2050cb0e5dad055afa75dab3bc601: Status 404 returned error can't find the container with id f92f0f0d9d8eb9780619290abe7cd391a1d2050cb0e5dad055afa75dab3bc601
Nov 01 03:43:45.824910 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:45.824698    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5fd6cf-78d6995bb5-ffqcs" event={"ID":"73e54a1c-e8c0-4edc-991e-94cf8d331ff5","Type":"ContainerStarted","Data":"f92f0f0d9d8eb9780619290abe7cd391a1d2050cb0e5dad055afa75dab3bc601"}
Nov 01 03:43:46.832709 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:46.832649    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5fd6cf-78d6995bb5-ffqcs" event={"ID":"73e54a1c-e8c0-4edc-991e-94cf8d331ff5","Type":"ContainerStarted","Data":"8a695ea3a1b403cbdcb348bf82cc5c47627203da7b9ff2f9009188668d61e686"}
Nov 01 03:43:46.833159 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:46.832751    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5fd6cf-78d6995bb5-ffqcs"
Nov 01 03:43:46.956430 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:46.956385    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5fd6cf-78d6995bb5-ffqcs" podStartSLOduration=2.073600129 podStartE2EDuration="2.95637233s" totalImagesPullingTime="882.772201ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:43:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:43:46.953604673 +0000 UTC m=+2233.468250535" watchObservedRunningTime="2026-11-01 03:43:46.95637233 +0000 UTC m=+2233.471018191"
Nov 01 03:43:52.843890 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:52.843865    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5fd6cf-78d6995bb5-ffqcs"
Nov 01 03:43:59.493241 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:59.493208    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d84de6-6f4ddfb696-m8smn"]
Nov 01 03:43:59.497784 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:59.497763    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d84de6-6f4ddfb696-m8smn"
Nov 01 03:43:59.553625 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:59.553606    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-d84de6-serving-cert\""
Nov 01 03:43:59.559521 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:59.559493    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d84de6-6f4ddfb696-m8smn"]
Nov 01 03:43:59.614235 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:59.614214    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0e53b77-f9a5-4c9b-9ccd-84d6d30c35e4-maas-api-tls\") pod \"maas-api-e2e-inf-b-d84de6-6f4ddfb696-m8smn\" (UID: \"c0e53b77-f9a5-4c9b-9ccd-84d6d30c35e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d84de6-6f4ddfb696-m8smn"
Nov 01 03:43:59.614353 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:59.614274    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0e53b77-f9a5-4c9b-9ccd-84d6d30c35e4-metrics-certs\") pod \"maas-api-e2e-inf-b-d84de6-6f4ddfb696-m8smn\" (UID: \"c0e53b77-f9a5-4c9b-9ccd-84d6d30c35e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d84de6-6f4ddfb696-m8smn"
Nov 01 03:43:59.614444 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:59.614425    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0e53b77-f9a5-4c9b-9ccd-84d6d30c35e4-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d84de6-6f4ddfb696-m8smn\" (UID: \"c0e53b77-f9a5-4c9b-9ccd-84d6d30c35e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d84de6-6f4ddfb696-m8smn"
Nov 01 03:43:59.614497 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:59.614462    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8vrn\" (UniqueName: \"kubernetes.io/projected/c0e53b77-f9a5-4c9b-9ccd-84d6d30c35e4-kube-api-access-q8vrn\") pod \"maas-api-e2e-inf-b-d84de6-6f4ddfb696-m8smn\" (UID: \"c0e53b77-f9a5-4c9b-9ccd-84d6d30c35e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d84de6-6f4ddfb696-m8smn"
Nov 01 03:43:59.715952 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:59.715922    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0e53b77-f9a5-4c9b-9ccd-84d6d30c35e4-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d84de6-6f4ddfb696-m8smn\" (UID: \"c0e53b77-f9a5-4c9b-9ccd-84d6d30c35e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d84de6-6f4ddfb696-m8smn"
Nov 01 03:43:59.716105 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:59.715971    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8vrn\" (UniqueName: \"kubernetes.io/projected/c0e53b77-f9a5-4c9b-9ccd-84d6d30c35e4-kube-api-access-q8vrn\") pod \"maas-api-e2e-inf-b-d84de6-6f4ddfb696-m8smn\" (UID: \"c0e53b77-f9a5-4c9b-9ccd-84d6d30c35e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d84de6-6f4ddfb696-m8smn"
Nov 01 03:43:59.716105 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:59.716054    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0e53b77-f9a5-4c9b-9ccd-84d6d30c35e4-maas-api-tls\") pod \"maas-api-e2e-inf-b-d84de6-6f4ddfb696-m8smn\" (UID: \"c0e53b77-f9a5-4c9b-9ccd-84d6d30c35e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d84de6-6f4ddfb696-m8smn"
Nov 01 03:43:59.716240 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:59.716186    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0e53b77-f9a5-4c9b-9ccd-84d6d30c35e4-metrics-certs\") pod \"maas-api-e2e-inf-b-d84de6-6f4ddfb696-m8smn\" (UID: \"c0e53b77-f9a5-4c9b-9ccd-84d6d30c35e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d84de6-6f4ddfb696-m8smn"
Nov 01 03:43:59.716240 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:43:59.716235    4400 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-d84de6-serving-cert: secret "maas-api-e2e-inf-b-d84de6-serving-cert" not found
Nov 01 03:43:59.716344 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:43:59.716289    4400 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0e53b77-f9a5-4c9b-9ccd-84d6d30c35e4-maas-api-tls podName:c0e53b77-f9a5-4c9b-9ccd-84d6d30c35e4 nodeName:}" failed. No retries permitted until 2026-11-01 03:44:00.216268572 +0000 UTC m=+2246.730914420 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c0e53b77-f9a5-4c9b-9ccd-84d6d30c35e4-maas-api-tls") pod "maas-api-e2e-inf-b-d84de6-6f4ddfb696-m8smn" (UID: "c0e53b77-f9a5-4c9b-9ccd-84d6d30c35e4") : secret "maas-api-e2e-inf-b-d84de6-serving-cert" not found
Nov 01 03:43:59.716622 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:59.716602    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0e53b77-f9a5-4c9b-9ccd-84d6d30c35e4-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d84de6-6f4ddfb696-m8smn\" (UID: \"c0e53b77-f9a5-4c9b-9ccd-84d6d30c35e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d84de6-6f4ddfb696-m8smn"
Nov 01 03:43:59.718668 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:59.718651    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0e53b77-f9a5-4c9b-9ccd-84d6d30c35e4-metrics-certs\") pod \"maas-api-e2e-inf-b-d84de6-6f4ddfb696-m8smn\" (UID: \"c0e53b77-f9a5-4c9b-9ccd-84d6d30c35e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d84de6-6f4ddfb696-m8smn"
Nov 01 03:43:59.812925 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:43:59.811442    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8vrn\" (UniqueName: \"kubernetes.io/projected/c0e53b77-f9a5-4c9b-9ccd-84d6d30c35e4-kube-api-access-q8vrn\") pod \"maas-api-e2e-inf-b-d84de6-6f4ddfb696-m8smn\" (UID: \"c0e53b77-f9a5-4c9b-9ccd-84d6d30c35e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d84de6-6f4ddfb696-m8smn"
Nov 01 03:44:00.235551 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:00.235518    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0e53b77-f9a5-4c9b-9ccd-84d6d30c35e4-maas-api-tls\") pod \"maas-api-e2e-inf-b-d84de6-6f4ddfb696-m8smn\" (UID: \"c0e53b77-f9a5-4c9b-9ccd-84d6d30c35e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d84de6-6f4ddfb696-m8smn"
Nov 01 03:44:00.238035 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:00.238014    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0e53b77-f9a5-4c9b-9ccd-84d6d30c35e4-maas-api-tls\") pod \"maas-api-e2e-inf-b-d84de6-6f4ddfb696-m8smn\" (UID: \"c0e53b77-f9a5-4c9b-9ccd-84d6d30c35e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d84de6-6f4ddfb696-m8smn"
Nov 01 03:44:00.411299 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:00.411272    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d84de6-6f4ddfb696-m8smn"
Nov 01 03:44:00.794049 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:00.794023    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d84de6-6f4ddfb696-m8smn"]
Nov 01 03:44:00.794469 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:44:00.794333    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0e53b77_f9a5_4c9b_9ccd_84d6d30c35e4.slice/crio-5de5d52f4d7383fc6a89bdebe2390bd354d721f6b4ba9bb3f3211cdfcbad762e WatchSource:0}: Error finding container 5de5d52f4d7383fc6a89bdebe2390bd354d721f6b4ba9bb3f3211cdfcbad762e: Status 404 returned error can't find the container with id 5de5d52f4d7383fc6a89bdebe2390bd354d721f6b4ba9bb3f3211cdfcbad762e
Nov 01 03:44:00.949240 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:00.949211    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d84de6-6f4ddfb696-m8smn" event={"ID":"c0e53b77-f9a5-4c9b-9ccd-84d6d30c35e4","Type":"ContainerStarted","Data":"5de5d52f4d7383fc6a89bdebe2390bd354d721f6b4ba9bb3f3211cdfcbad762e"}
Nov 01 03:44:01.731498 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:44:01.731459    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:44:01.957059 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:01.956978    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d84de6-6f4ddfb696-m8smn" event={"ID":"c0e53b77-f9a5-4c9b-9ccd-84d6d30c35e4","Type":"ContainerStarted","Data":"8a6902ae588681145eb3e271bc8227e201e7b491a14e3881ac5218fda0f616fe"}
Nov 01 03:44:01.957059 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:01.957056    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d84de6-6f4ddfb696-m8smn"
Nov 01 03:44:02.086308 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:02.086259    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d84de6-6f4ddfb696-m8smn" podStartSLOduration=2.246108293 podStartE2EDuration="3.086246176s" totalImagesPullingTime="840.137883ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:43:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:44:02.08286496 +0000 UTC m=+2248.597510837" watchObservedRunningTime="2026-11-01 03:44:02.086246176 +0000 UTC m=+2248.600892044"
Nov 01 03:44:03.865870 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:03.865834    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-pbw4s"]
Nov 01 03:44:03.988471 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:03.988424    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-pbw4s" podUID="82b7e135-bf31-448d-ae07-f8b859f15420" containerName="istio-proxy" containerID="cri-o://79be6af514c393018879b4f1dbd9f898188f176d545aa6bcccadbd8b3683c44f" gracePeriod=30
Nov 01 03:44:07.982434 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:07.982403    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d84de6-6f4ddfb696-m8smn"
Nov 01 03:44:08.919254 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:08.919217    4400 patch_prober.go:29] interesting pod/e2e-worker-w4-e8c789-openshift-default-774d997df-pbw4s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 03:44:08.919422 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:08.919284    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-pbw4s" podUID="82b7e135-bf31-448d-ae07-f8b859f15420" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 03:44:09.293799 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:09.293774    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-pbw4s"
Nov 01 03:44:09.338397 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:09.338360    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82b7e135-bf31-448d-ae07-f8b859f15420-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82b7e135-bf31-448d-ae07-f8b859f15420-workload-certs\") pod \"82b7e135-bf31-448d-ae07-f8b859f15420\" (UID: \"82b7e135-bf31-448d-ae07-f8b859f15420\") "
Nov 01 03:44:09.338397 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:09.338402    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82b7e135-bf31-448d-ae07-f8b859f15420-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82b7e135-bf31-448d-ae07-f8b859f15420-istio-envoy\") pod \"82b7e135-bf31-448d-ae07-f8b859f15420\" (UID: \"82b7e135-bf31-448d-ae07-f8b859f15420\") "
Nov 01 03:44:09.338633 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:09.338458    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82b7e135-bf31-448d-ae07-f8b859f15420-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82b7e135-bf31-448d-ae07-f8b859f15420-credential-socket\") pod \"82b7e135-bf31-448d-ae07-f8b859f15420\" (UID: \"82b7e135-bf31-448d-ae07-f8b859f15420\") "
Nov 01 03:44:09.338633 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:09.338491    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82b7e135-bf31-448d-ae07-f8b859f15420-kube-api-access-k2kgf\" (UniqueName: \"kubernetes.io/projected/82b7e135-bf31-448d-ae07-f8b859f15420-kube-api-access-k2kgf\") pod \"82b7e135-bf31-448d-ae07-f8b859f15420\" (UID: \"82b7e135-bf31-448d-ae07-f8b859f15420\") "
Nov 01 03:44:09.338633 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:09.338539    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/82b7e135-bf31-448d-ae07-f8b859f15420-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82b7e135-bf31-448d-ae07-f8b859f15420-istio-podinfo\") pod \"82b7e135-bf31-448d-ae07-f8b859f15420\" (UID: \"82b7e135-bf31-448d-ae07-f8b859f15420\") "
Nov 01 03:44:09.338633 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:09.338585    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82b7e135-bf31-448d-ae07-f8b859f15420-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82b7e135-bf31-448d-ae07-f8b859f15420-istio-data\") pod \"82b7e135-bf31-448d-ae07-f8b859f15420\" (UID: \"82b7e135-bf31-448d-ae07-f8b859f15420\") "
Nov 01 03:44:09.338633 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:09.338616    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82b7e135-bf31-448d-ae07-f8b859f15420-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82b7e135-bf31-448d-ae07-f8b859f15420-istiod-ca-cert\") pod \"82b7e135-bf31-448d-ae07-f8b859f15420\" (UID: \"82b7e135-bf31-448d-ae07-f8b859f15420\") "
Nov 01 03:44:09.338897 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:09.338646    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82b7e135-bf31-448d-ae07-f8b859f15420-workload-certs" pod "82b7e135-bf31-448d-ae07-f8b859f15420" (UID: "82b7e135-bf31-448d-ae07-f8b859f15420"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:44:09.338897 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:09.338789    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82b7e135-bf31-448d-ae07-f8b859f15420-istio-token\" (UniqueName: \"kubernetes.io/projected/82b7e135-bf31-448d-ae07-f8b859f15420-istio-token\") pod \"82b7e135-bf31-448d-ae07-f8b859f15420\" (UID: \"82b7e135-bf31-448d-ae07-f8b859f15420\") "
Nov 01 03:44:09.338897 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:09.338803    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82b7e135-bf31-448d-ae07-f8b859f15420-credential-socket" pod "82b7e135-bf31-448d-ae07-f8b859f15420" (UID: "82b7e135-bf31-448d-ae07-f8b859f15420"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:44:09.338897 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:09.338818    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82b7e135-bf31-448d-ae07-f8b859f15420-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82b7e135-bf31-448d-ae07-f8b859f15420-workload-socket\") pod \"82b7e135-bf31-448d-ae07-f8b859f15420\" (UID: \"82b7e135-bf31-448d-ae07-f8b859f15420\") "
Nov 01 03:44:09.339137 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:09.339112    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82b7e135-bf31-448d-ae07-f8b859f15420-istiod-ca-cert" pod "82b7e135-bf31-448d-ae07-f8b859f15420" (UID: "82b7e135-bf31-448d-ae07-f8b859f15420"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:44:09.339511 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:09.339489    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82b7e135-bf31-448d-ae07-f8b859f15420-workload-socket" pod "82b7e135-bf31-448d-ae07-f8b859f15420" (UID: "82b7e135-bf31-448d-ae07-f8b859f15420"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:44:09.339695 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:09.339661    4400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82b7e135-bf31-448d-ae07-f8b859f15420-istiod-ca-cert\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:44:09.339747 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:09.339717    4400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82b7e135-bf31-448d-ae07-f8b859f15420-workload-socket\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:44:09.339747 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:09.339734    4400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82b7e135-bf31-448d-ae07-f8b859f15420-workload-certs\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:44:09.339847 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:09.339747    4400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82b7e135-bf31-448d-ae07-f8b859f15420-credential-socket\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:44:09.339960 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:09.339940    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82b7e135-bf31-448d-ae07-f8b859f15420-istio-data" pod "82b7e135-bf31-448d-ae07-f8b859f15420" (UID: "82b7e135-bf31-448d-ae07-f8b859f15420"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:44:09.341015 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:09.340987    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82b7e135-bf31-448d-ae07-f8b859f15420-istio-envoy" pod "82b7e135-bf31-448d-ae07-f8b859f15420" (UID: "82b7e135-bf31-448d-ae07-f8b859f15420"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:44:09.341145 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:09.341045    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/82b7e135-bf31-448d-ae07-f8b859f15420-istio-podinfo" pod "82b7e135-bf31-448d-ae07-f8b859f15420" (UID: "82b7e135-bf31-448d-ae07-f8b859f15420"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 03:44:09.341344 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:09.341326    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82b7e135-bf31-448d-ae07-f8b859f15420-istio-token" pod "82b7e135-bf31-448d-ae07-f8b859f15420" (UID: "82b7e135-bf31-448d-ae07-f8b859f15420"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:44:09.341403 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:09.341354    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82b7e135-bf31-448d-ae07-f8b859f15420-kube-api-access-k2kgf" pod "82b7e135-bf31-448d-ae07-f8b859f15420" (UID: "82b7e135-bf31-448d-ae07-f8b859f15420"). InnerVolumeSpecName "kube-api-access-k2kgf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:44:09.441483 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:09.441450    4400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82b7e135-bf31-448d-ae07-f8b859f15420-istio-envoy\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:44:09.441483 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:09.441481    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2kgf\" (UniqueName: \"kubernetes.io/projected/82b7e135-bf31-448d-ae07-f8b859f15420-kube-api-access-k2kgf\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:44:09.441483 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:09.441491    4400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82b7e135-bf31-448d-ae07-f8b859f15420-istio-podinfo\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:44:09.441757 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:09.441500    4400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82b7e135-bf31-448d-ae07-f8b859f15420-istio-data\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:44:09.441757 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:09.441509    4400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82b7e135-bf31-448d-ae07-f8b859f15420-istio-token\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:44:10.028957 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:10.028868    4400 generic.go:356] "Generic (PLEG): container finished" podID="82b7e135-bf31-448d-ae07-f8b859f15420" containerID="79be6af514c393018879b4f1dbd9f898188f176d545aa6bcccadbd8b3683c44f" exitCode=0
Nov 01 03:44:10.028957 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:10.028921    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-pbw4s"
Nov 01 03:44:10.029144 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:10.028964    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-pbw4s" event={"ID":"82b7e135-bf31-448d-ae07-f8b859f15420","Type":"ContainerDied","Data":"79be6af514c393018879b4f1dbd9f898188f176d545aa6bcccadbd8b3683c44f"}
Nov 01 03:44:10.029144 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:10.029023    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-pbw4s" event={"ID":"82b7e135-bf31-448d-ae07-f8b859f15420","Type":"ContainerDied","Data":"f57894cca6b7d92496bf12533edf0b528f87a7b4caa94c0c395887a3c4ea390b"}
Nov 01 03:44:10.029144 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:10.029046    4400 scope.go:122] "RemoveContainer" containerID="79be6af514c393018879b4f1dbd9f898188f176d545aa6bcccadbd8b3683c44f"
Nov 01 03:44:10.040825 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:10.040807    4400 scope.go:122] "RemoveContainer" containerID="79be6af514c393018879b4f1dbd9f898188f176d545aa6bcccadbd8b3683c44f"
Nov 01 03:44:10.041138 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:44:10.041118    4400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79be6af514c393018879b4f1dbd9f898188f176d545aa6bcccadbd8b3683c44f\": container with ID starting with 79be6af514c393018879b4f1dbd9f898188f176d545aa6bcccadbd8b3683c44f not found: ID does not exist" containerID="79be6af514c393018879b4f1dbd9f898188f176d545aa6bcccadbd8b3683c44f"
Nov 01 03:44:10.041189 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:10.041147    4400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79be6af514c393018879b4f1dbd9f898188f176d545aa6bcccadbd8b3683c44f"} err="failed to get container status \"79be6af514c393018879b4f1dbd9f898188f176d545aa6bcccadbd8b3683c44f\": rpc error: code = NotFound desc = could not find container \"79be6af514c393018879b4f1dbd9f898188f176d545aa6bcccadbd8b3683c44f\": container with ID starting with 79be6af514c393018879b4f1dbd9f898188f176d545aa6bcccadbd8b3683c44f not found: ID does not exist"
Nov 01 03:44:10.251053 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:10.251010    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-pbw4s"]
Nov 01 03:44:10.306761 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:10.302112    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-pbw4s"]
Nov 01 03:44:11.615746 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:11.615714    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82b7e135-bf31-448d-ae07-f8b859f15420" path="/var/lib/kubelet/pods/82b7e135-bf31-448d-ae07-f8b859f15420/volumes"
Nov 01 03:44:11.939935 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:11.939863    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2906cb-68bd5bb9f9-d5gtm"]
Nov 01 03:44:11.940512 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:11.940492    4400 cpu_manager.go:414] "containerMap: removing container" podUID="82b7e135-bf31-448d-ae07-f8b859f15420" containerName="istio-proxy"
Nov 01 03:44:11.940582 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:11.940513    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82b7e135-bf31-448d-ae07-f8b859f15420" containerName="istio-proxy"
Nov 01 03:44:11.940666 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:11.940654    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="82b7e135-bf31-448d-ae07-f8b859f15420" containerName="istio-proxy"
Nov 01 03:44:11.945198 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:11.945181    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2906cb-68bd5bb9f9-d5gtm"
Nov 01 03:44:11.978342 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:11.978316    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2b4j\" (UniqueName: \"kubernetes.io/projected/7179f114-d7bf-4d4d-9a12-4a88e9041272-kube-api-access-d2b4j\") pod \"maas-api-e2e-api-a-2906cb-68bd5bb9f9-d5gtm\" (UID: \"7179f114-d7bf-4d4d-9a12-4a88e9041272\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2906cb-68bd5bb9f9-d5gtm"
Nov 01 03:44:11.978459 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:11.978387    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7179f114-d7bf-4d4d-9a12-4a88e9041272-metrics-certs\") pod \"maas-api-e2e-api-a-2906cb-68bd5bb9f9-d5gtm\" (UID: \"7179f114-d7bf-4d4d-9a12-4a88e9041272\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2906cb-68bd5bb9f9-d5gtm"
Nov 01 03:44:11.978573 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:11.978552    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7179f114-d7bf-4d4d-9a12-4a88e9041272-openshift-service-ca\") pod \"maas-api-e2e-api-a-2906cb-68bd5bb9f9-d5gtm\" (UID: \"7179f114-d7bf-4d4d-9a12-4a88e9041272\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2906cb-68bd5bb9f9-d5gtm"
Nov 01 03:44:11.978616 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:11.978585    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7179f114-d7bf-4d4d-9a12-4a88e9041272-maas-api-tls\") pod \"maas-api-e2e-api-a-2906cb-68bd5bb9f9-d5gtm\" (UID: \"7179f114-d7bf-4d4d-9a12-4a88e9041272\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2906cb-68bd5bb9f9-d5gtm"
Nov 01 03:44:12.006912 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:12.006887    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2906cb-68bd5bb9f9-d5gtm"]
Nov 01 03:44:12.051596 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:12.051570    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-2906cb-serving-cert\""
Nov 01 03:44:12.079408 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:12.079382    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2b4j\" (UniqueName: \"kubernetes.io/projected/7179f114-d7bf-4d4d-9a12-4a88e9041272-kube-api-access-d2b4j\") pod \"maas-api-e2e-api-a-2906cb-68bd5bb9f9-d5gtm\" (UID: \"7179f114-d7bf-4d4d-9a12-4a88e9041272\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2906cb-68bd5bb9f9-d5gtm"
Nov 01 03:44:12.079533 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:12.079448    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7179f114-d7bf-4d4d-9a12-4a88e9041272-metrics-certs\") pod \"maas-api-e2e-api-a-2906cb-68bd5bb9f9-d5gtm\" (UID: \"7179f114-d7bf-4d4d-9a12-4a88e9041272\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2906cb-68bd5bb9f9-d5gtm"
Nov 01 03:44:12.079583 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:12.079549    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7179f114-d7bf-4d4d-9a12-4a88e9041272-openshift-service-ca\") pod \"maas-api-e2e-api-a-2906cb-68bd5bb9f9-d5gtm\" (UID: \"7179f114-d7bf-4d4d-9a12-4a88e9041272\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2906cb-68bd5bb9f9-d5gtm"
Nov 01 03:44:12.079583 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:12.079567    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7179f114-d7bf-4d4d-9a12-4a88e9041272-maas-api-tls\") pod \"maas-api-e2e-api-a-2906cb-68bd5bb9f9-d5gtm\" (UID: \"7179f114-d7bf-4d4d-9a12-4a88e9041272\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2906cb-68bd5bb9f9-d5gtm"
Nov 01 03:44:12.079716 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:44:12.079695    4400 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-2906cb-serving-cert: secret "maas-api-e2e-api-a-2906cb-serving-cert" not found
Nov 01 03:44:12.079801 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:44:12.079764    4400 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7179f114-d7bf-4d4d-9a12-4a88e9041272-maas-api-tls podName:7179f114-d7bf-4d4d-9a12-4a88e9041272 nodeName:}" failed. No retries permitted until 2026-11-01 03:44:12.579746801 +0000 UTC m=+2259.094392643 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7179f114-d7bf-4d4d-9a12-4a88e9041272-maas-api-tls") pod "maas-api-e2e-api-a-2906cb-68bd5bb9f9-d5gtm" (UID: "7179f114-d7bf-4d4d-9a12-4a88e9041272") : secret "maas-api-e2e-api-a-2906cb-serving-cert" not found
Nov 01 03:44:12.080286 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:12.080264    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7179f114-d7bf-4d4d-9a12-4a88e9041272-openshift-service-ca\") pod \"maas-api-e2e-api-a-2906cb-68bd5bb9f9-d5gtm\" (UID: \"7179f114-d7bf-4d4d-9a12-4a88e9041272\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2906cb-68bd5bb9f9-d5gtm"
Nov 01 03:44:12.082020 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:12.081994    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7179f114-d7bf-4d4d-9a12-4a88e9041272-metrics-certs\") pod \"maas-api-e2e-api-a-2906cb-68bd5bb9f9-d5gtm\" (UID: \"7179f114-d7bf-4d4d-9a12-4a88e9041272\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2906cb-68bd5bb9f9-d5gtm"
Nov 01 03:44:12.142115 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:12.142085    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2b4j\" (UniqueName: \"kubernetes.io/projected/7179f114-d7bf-4d4d-9a12-4a88e9041272-kube-api-access-d2b4j\") pod \"maas-api-e2e-api-a-2906cb-68bd5bb9f9-d5gtm\" (UID: \"7179f114-d7bf-4d4d-9a12-4a88e9041272\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2906cb-68bd5bb9f9-d5gtm"
Nov 01 03:44:12.586542 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:12.586509    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7179f114-d7bf-4d4d-9a12-4a88e9041272-maas-api-tls\") pod \"maas-api-e2e-api-a-2906cb-68bd5bb9f9-d5gtm\" (UID: \"7179f114-d7bf-4d4d-9a12-4a88e9041272\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2906cb-68bd5bb9f9-d5gtm"
Nov 01 03:44:12.588964 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:12.588942    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7179f114-d7bf-4d4d-9a12-4a88e9041272-maas-api-tls\") pod \"maas-api-e2e-api-a-2906cb-68bd5bb9f9-d5gtm\" (UID: \"7179f114-d7bf-4d4d-9a12-4a88e9041272\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2906cb-68bd5bb9f9-d5gtm"
Nov 01 03:44:12.858452 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:12.858364    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2906cb-68bd5bb9f9-d5gtm"
Nov 01 03:44:13.288134 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:13.288043    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2906cb-68bd5bb9f9-d5gtm"]
Nov 01 03:44:14.061184 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:14.061150    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2906cb-68bd5bb9f9-d5gtm" event={"ID":"7179f114-d7bf-4d4d-9a12-4a88e9041272","Type":"ContainerStarted","Data":"a564771fa7da2fe4a34aa0b7b7c9aeddfa7e218192e4f0461173944de81e22ff"}
Nov 01 03:44:15.069157 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:15.069118    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2906cb-68bd5bb9f9-d5gtm" event={"ID":"7179f114-d7bf-4d4d-9a12-4a88e9041272","Type":"ContainerStarted","Data":"fd99d9d8d64260dcf723fed4bfc469e3b213edfe72535be200b276fcf39f83e5"}
Nov 01 03:44:15.069559 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:15.069249    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2906cb-68bd5bb9f9-d5gtm"
Nov 01 03:44:15.196641 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:15.196593    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2906cb-68bd5bb9f9-d5gtm" podStartSLOduration=3.335983942 podStartE2EDuration="4.196579473s" totalImagesPullingTime="860.595531ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:44:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:44:15.194029109 +0000 UTC m=+2261.708674972" watchObservedRunningTime="2026-11-01 03:44:15.196579473 +0000 UTC m=+2261.711225334"
Nov 01 03:44:19.950388 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:19.950354    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7072-6a4993"]
Nov 01 03:44:19.955258 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:19.955234    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7072-6a4993"
Nov 01 03:44:20.019180 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:20.019151    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7072-6a4993"]
Nov 01 03:44:20.042218 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:20.042189    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6jft\" (UniqueName: \"kubernetes.io/projected/34fb7732-c2d8-4785-ba96-d510f413015a-kube-api-access-s6jft\") pod \"test-curl-7072-6a4993\" (UID: \"34fb7732-c2d8-4785-ba96-d510f413015a\") " pod="opendatahub/test-curl-7072-6a4993"
Nov 01 03:44:20.065175 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:20.065151    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-h2kzj\""
Nov 01 03:44:20.143670 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:20.143638    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6jft\" (UniqueName: \"kubernetes.io/projected/34fb7732-c2d8-4785-ba96-d510f413015a-kube-api-access-s6jft\") pod \"test-curl-7072-6a4993\" (UID: \"34fb7732-c2d8-4785-ba96-d510f413015a\") " pod="opendatahub/test-curl-7072-6a4993"
Nov 01 03:44:20.206095 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:20.206027    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6jft\" (UniqueName: \"kubernetes.io/projected/34fb7732-c2d8-4785-ba96-d510f413015a-kube-api-access-s6jft\") pod \"test-curl-7072-6a4993\" (UID: \"34fb7732-c2d8-4785-ba96-d510f413015a\") " pod="opendatahub/test-curl-7072-6a4993"
Nov 01 03:44:20.268376 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:20.268352    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7072-6a4993"
Nov 01 03:44:20.644191 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:20.644166    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7072-6a4993"]
Nov 01 03:44:20.644191 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:44:20.644173    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod34fb7732_c2d8_4785_ba96_d510f413015a.slice/crio-4f00b0ea9be80368cc2fff8f75d7948944ad2f1453905e0398508061920a64dc WatchSource:0}: Error finding container 4f00b0ea9be80368cc2fff8f75d7948944ad2f1453905e0398508061920a64dc: Status 404 returned error can't find the container with id 4f00b0ea9be80368cc2fff8f75d7948944ad2f1453905e0398508061920a64dc
Nov 01 03:44:21.080829 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:21.080806    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2906cb-68bd5bb9f9-d5gtm"
Nov 01 03:44:21.108774 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:21.108738    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7072-6a4993" event={"ID":"34fb7732-c2d8-4785-ba96-d510f413015a","Type":"ContainerStarted","Data":"b88e92db766511c57e4b56a9a3a4548d789da84477d2ef54f8a01319e46ea293"}
Nov 01 03:44:21.108914 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:21.108778    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7072-6a4993" event={"ID":"34fb7732-c2d8-4785-ba96-d510f413015a","Type":"ContainerStarted","Data":"4f00b0ea9be80368cc2fff8f75d7948944ad2f1453905e0398508061920a64dc"}
Nov 01 03:44:21.323297 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:21.323237    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7072-6a4993" podStartSLOduration=2.323222056 podStartE2EDuration="2.323222056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:44:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:44:21.320137777 +0000 UTC m=+2267.834783639" watchObservedRunningTime="2026-11-01 03:44:21.323222056 +0000 UTC m=+2267.837867917"
Nov 01 03:44:22.841270 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:22.841241    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7072-6a4993"]
Nov 01 03:44:22.845178 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:22.845152    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7072-6a4993"]
Nov 01 03:44:23.121626 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:23.121537    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7072-6a4993" podUID="34fb7732-c2d8-4785-ba96-d510f413015a" containerName="test-curl-7072-6a4993" containerID="cri-o://b88e92db766511c57e4b56a9a3a4548d789da84477d2ef54f8a01319e46ea293" gracePeriod=2
Nov 01 03:44:23.174452 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:44:23.174417    4400 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7072-6a4993\" is forbidden: User \"system:node:ip-10-0-133-54.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-54.us-west-1.compute.internal' and this object" podUID="34fb7732-c2d8-4785-ba96-d510f413015a" pod="opendatahub/test-curl-7072-6a4993"
Nov 01 03:44:23.663262 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:44:23.663228    4400 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7072-6a4993\" is forbidden: User \"system:node:ip-10-0-133-54.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-54.us-west-1.compute.internal' and this object" podUID="34fb7732-c2d8-4785-ba96-d510f413015a" pod="opendatahub/test-curl-7072-6a4993"
Nov 01 03:44:25.381023 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:25.380958    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7072-6a4993"
Nov 01 03:44:25.514764 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:25.514718    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34fb7732-c2d8-4785-ba96-d510f413015a-kube-api-access-s6jft\" (UniqueName: \"kubernetes.io/projected/34fb7732-c2d8-4785-ba96-d510f413015a-kube-api-access-s6jft\") pod \"34fb7732-c2d8-4785-ba96-d510f413015a\" (UID: \"34fb7732-c2d8-4785-ba96-d510f413015a\") "
Nov 01 03:44:25.517098 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:25.517067    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34fb7732-c2d8-4785-ba96-d510f413015a-kube-api-access-s6jft" pod "34fb7732-c2d8-4785-ba96-d510f413015a" (UID: "34fb7732-c2d8-4785-ba96-d510f413015a"). InnerVolumeSpecName "kube-api-access-s6jft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:44:25.614950 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:25.614920    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34fb7732-c2d8-4785-ba96-d510f413015a" path="/var/lib/kubelet/pods/34fb7732-c2d8-4785-ba96-d510f413015a/volumes"
Nov 01 03:44:25.616009 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:25.615990    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6jft\" (UniqueName: \"kubernetes.io/projected/34fb7732-c2d8-4785-ba96-d510f413015a-kube-api-access-s6jft\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:44:26.145104 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:26.145058    4400 generic.go:356] "Generic (PLEG): container finished" podID="34fb7732-c2d8-4785-ba96-d510f413015a" containerID="b88e92db766511c57e4b56a9a3a4548d789da84477d2ef54f8a01319e46ea293" exitCode=137
Nov 01 03:44:26.145321 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:26.145113    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7072-6a4993"
Nov 01 03:44:26.145321 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:26.145142    4400 scope.go:122] "RemoveContainer" containerID="b88e92db766511c57e4b56a9a3a4548d789da84477d2ef54f8a01319e46ea293"
Nov 01 03:44:26.155458 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:26.155437    4400 scope.go:122] "RemoveContainer" containerID="b88e92db766511c57e4b56a9a3a4548d789da84477d2ef54f8a01319e46ea293"
Nov 01 03:44:26.155772 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:44:26.155749    4400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b88e92db766511c57e4b56a9a3a4548d789da84477d2ef54f8a01319e46ea293\": container with ID starting with b88e92db766511c57e4b56a9a3a4548d789da84477d2ef54f8a01319e46ea293 not found: ID does not exist" containerID="b88e92db766511c57e4b56a9a3a4548d789da84477d2ef54f8a01319e46ea293"
Nov 01 03:44:26.155869 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:26.155784    4400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b88e92db766511c57e4b56a9a3a4548d789da84477d2ef54f8a01319e46ea293"} err="failed to get container status \"b88e92db766511c57e4b56a9a3a4548d789da84477d2ef54f8a01319e46ea293\": rpc error: code = NotFound desc = could not find container \"b88e92db766511c57e4b56a9a3a4548d789da84477d2ef54f8a01319e46ea293\": container with ID starting with b88e92db766511c57e4b56a9a3a4548d789da84477d2ef54f8a01319e46ea293 not found: ID does not exist"
Nov 01 03:44:31.730509 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:44:31.730476    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:44:44.544155 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:44:44.544072    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 03:45:01.730640 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:45:01.730598    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:45:08.481469 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:08.481432    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-e8c789-574898d88f-mqljq"]
Nov 01 03:45:08.482052 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:08.481669    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e8c789-574898d88f-mqljq" podUID="9b31e254-c788-4169-9fc3-d286a54d50d7" containerName="maas-api" containerID="cri-o://fcc13b0315515141384a6057237f7c06f59f351c92e287002fd2060985638330" gracePeriod=30
Nov 01 03:45:08.795872 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:08.795848    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e8c789-574898d88f-mqljq"
Nov 01 03:45:08.973270 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:08.973238    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b31e254-c788-4169-9fc3-d286a54d50d7-kube-api-access-gk62r\" (UniqueName: \"kubernetes.io/projected/9b31e254-c788-4169-9fc3-d286a54d50d7-kube-api-access-gk62r\") pod \"9b31e254-c788-4169-9fc3-d286a54d50d7\" (UID: \"9b31e254-c788-4169-9fc3-d286a54d50d7\") "
Nov 01 03:45:08.973460 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:08.973305    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b31e254-c788-4169-9fc3-d286a54d50d7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b31e254-c788-4169-9fc3-d286a54d50d7-maas-api-tls\") pod \"9b31e254-c788-4169-9fc3-d286a54d50d7\" (UID: \"9b31e254-c788-4169-9fc3-d286a54d50d7\") "
Nov 01 03:45:08.973460 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:08.973329    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b31e254-c788-4169-9fc3-d286a54d50d7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b31e254-c788-4169-9fc3-d286a54d50d7-openshift-service-ca\") pod \"9b31e254-c788-4169-9fc3-d286a54d50d7\" (UID: \"9b31e254-c788-4169-9fc3-d286a54d50d7\") "
Nov 01 03:45:08.973460 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:08.973387    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b31e254-c788-4169-9fc3-d286a54d50d7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b31e254-c788-4169-9fc3-d286a54d50d7-metrics-certs\") pod \"9b31e254-c788-4169-9fc3-d286a54d50d7\" (UID: \"9b31e254-c788-4169-9fc3-d286a54d50d7\") "
Nov 01 03:45:08.973675 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:08.973652    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b31e254-c788-4169-9fc3-d286a54d50d7-openshift-service-ca" pod "9b31e254-c788-4169-9fc3-d286a54d50d7" (UID: "9b31e254-c788-4169-9fc3-d286a54d50d7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:45:08.973876 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:08.973857    4400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b31e254-c788-4169-9fc3-d286a54d50d7-openshift-service-ca\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:45:08.975623 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:08.975591    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b31e254-c788-4169-9fc3-d286a54d50d7-maas-api-tls" pod "9b31e254-c788-4169-9fc3-d286a54d50d7" (UID: "9b31e254-c788-4169-9fc3-d286a54d50d7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:45:08.975765 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:08.975617    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b31e254-c788-4169-9fc3-d286a54d50d7-kube-api-access-gk62r" pod "9b31e254-c788-4169-9fc3-d286a54d50d7" (UID: "9b31e254-c788-4169-9fc3-d286a54d50d7"). InnerVolumeSpecName "kube-api-access-gk62r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:45:08.975765 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:08.975651    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b31e254-c788-4169-9fc3-d286a54d50d7-metrics-certs" pod "9b31e254-c788-4169-9fc3-d286a54d50d7" (UID: "9b31e254-c788-4169-9fc3-d286a54d50d7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:45:09.090007 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:09.081820    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gk62r\" (UniqueName: \"kubernetes.io/projected/9b31e254-c788-4169-9fc3-d286a54d50d7-kube-api-access-gk62r\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:45:09.090007 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:09.081860    4400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b31e254-c788-4169-9fc3-d286a54d50d7-maas-api-tls\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:45:09.090007 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:09.081875    4400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b31e254-c788-4169-9fc3-d286a54d50d7-metrics-certs\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:45:09.487549 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:09.486616    4400 generic.go:356] "Generic (PLEG): container finished" podID="9b31e254-c788-4169-9fc3-d286a54d50d7" containerID="fcc13b0315515141384a6057237f7c06f59f351c92e287002fd2060985638330" exitCode=0
Nov 01 03:45:09.487549 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:09.486783    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e8c789-574898d88f-mqljq" event={"ID":"9b31e254-c788-4169-9fc3-d286a54d50d7","Type":"ContainerDied","Data":"fcc13b0315515141384a6057237f7c06f59f351c92e287002fd2060985638330"}
Nov 01 03:45:09.487549 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:09.486840    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e8c789-574898d88f-mqljq" event={"ID":"9b31e254-c788-4169-9fc3-d286a54d50d7","Type":"ContainerDied","Data":"bb9108d8a3f66d5d7a60322f9b7134a789079303be4ca1d39d0551675903288b"}
Nov 01 03:45:09.487549 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:09.486870    4400 scope.go:122] "RemoveContainer" containerID="fcc13b0315515141384a6057237f7c06f59f351c92e287002fd2060985638330"
Nov 01 03:45:09.487549 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:09.487033    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e8c789-574898d88f-mqljq"
Nov 01 03:45:09.515095 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:09.509656    4400 scope.go:122] "RemoveContainer" containerID="fcc13b0315515141384a6057237f7c06f59f351c92e287002fd2060985638330"
Nov 01 03:45:09.515095 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:45:09.512110    4400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcc13b0315515141384a6057237f7c06f59f351c92e287002fd2060985638330\": container with ID starting with fcc13b0315515141384a6057237f7c06f59f351c92e287002fd2060985638330 not found: ID does not exist" containerID="fcc13b0315515141384a6057237f7c06f59f351c92e287002fd2060985638330"
Nov 01 03:45:09.515095 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:09.512147    4400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcc13b0315515141384a6057237f7c06f59f351c92e287002fd2060985638330"} err="failed to get container status \"fcc13b0315515141384a6057237f7c06f59f351c92e287002fd2060985638330\": rpc error: code = NotFound desc = could not find container \"fcc13b0315515141384a6057237f7c06f59f351c92e287002fd2060985638330\": container with ID starting with fcc13b0315515141384a6057237f7c06f59f351c92e287002fd2060985638330 not found: ID does not exist"
Nov 01 03:45:09.659329 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:09.659297    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-e8c789-574898d88f-mqljq"]
Nov 01 03:45:09.663821 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:09.663798    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-e8c789-574898d88f-mqljq"]
Nov 01 03:45:11.615417 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:11.615385    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b31e254-c788-4169-9fc3-d286a54d50d7" path="/var/lib/kubelet/pods/9b31e254-c788-4169-9fc3-d286a54d50d7/volumes"
Nov 01 03:45:18.782627 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:18.782588    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-bpgj8"]
Nov 01 03:45:18.783047 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:18.782808    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-bpgj8" podUID="bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0" containerName="istio-proxy" containerID="cri-o://19377fb2c7b55ba8ad1a164d262d13e684c7ab57434d86ed7c4f7f2309ff9c23" gracePeriod=30
Nov 01 03:45:23.929566 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:23.929209    4400 patch_prober.go:29] interesting pod/e2e-worker-w4-e8c789-openshift-default-774d997df-bpgj8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Nov 01 03:45:23.929566 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:23.929288    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-bpgj8" podUID="bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Nov 01 03:45:24.103569 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:24.103539    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-bpgj8"
Nov 01 03:45:24.210800 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:24.210777    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0-istio-envoy\") pod \"bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0\" (UID: \"bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0\") "
Nov 01 03:45:24.210924 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:24.210810    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0-workload-socket\") pod \"bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0\" (UID: \"bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0\") "
Nov 01 03:45:24.210924 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:24.210873    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0-istio-token\" (UniqueName: \"kubernetes.io/projected/bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0-istio-token\") pod \"bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0\" (UID: \"bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0\") "
Nov 01 03:45:24.210924 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:24.210893    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0-istio-podinfo\") pod \"bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0\" (UID: \"bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0\") "
Nov 01 03:45:24.211088 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:24.210940    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0-istiod-ca-cert\") pod \"bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0\" (UID: \"bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0\") "
Nov 01 03:45:24.211088 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:24.210994    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0-credential-socket\") pod \"bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0\" (UID: \"bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0\") "
Nov 01 03:45:24.211088 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:24.211040    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0-workload-certs\") pod \"bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0\" (UID: \"bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0\") "
Nov 01 03:45:24.211235 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:24.211114    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0-workload-socket" pod "bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0" (UID: "bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:45:24.211235 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:24.211210    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0-istio-data\") pod \"bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0\" (UID: \"bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0\") "
Nov 01 03:45:24.211346 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:24.211333    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0-kube-api-access-v8s86\" (UniqueName: \"kubernetes.io/projected/bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0-kube-api-access-v8s86\") pod \"bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0\" (UID: \"bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0\") "
Nov 01 03:45:24.211410 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:24.211320    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0-credential-socket" pod "bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0" (UID: "bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:45:24.211410 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:24.211367    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0-workload-certs" pod "bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0" (UID: "bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:45:24.211506 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:24.211416    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0-istiod-ca-cert" pod "bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0" (UID: "bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:45:24.211768 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:24.211743    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0-istio-data" pod "bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0" (UID: "bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:45:24.211892 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:24.211877    4400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0-workload-socket\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:45:24.211959 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:24.211897    4400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0-istiod-ca-cert\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:45:24.211959 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:24.211913    4400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0-credential-socket\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:45:24.211959 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:24.211928    4400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0-workload-certs\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:45:24.211959 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:24.211943    4400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0-istio-data\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:45:24.213145 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:24.213121    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0-istio-token" pod "bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0" (UID: "bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:45:24.213251 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:24.213190    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0-istio-podinfo" pod "bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0" (UID: "bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 03:45:24.213499 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:24.213485    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0-istio-envoy" pod "bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0" (UID: "bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:45:24.213804 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:24.213790    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0-kube-api-access-v8s86" pod "bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0" (UID: "bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0"). InnerVolumeSpecName "kube-api-access-v8s86". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:45:24.313504 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:24.313477    4400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0-istio-token\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:45:24.313504 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:24.313501    4400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0-istio-podinfo\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:45:24.313671 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:24.313512    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8s86\" (UniqueName: \"kubernetes.io/projected/bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0-kube-api-access-v8s86\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:45:24.313671 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:24.313522    4400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0-istio-envoy\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:45:24.630640 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:24.630615    4400 generic.go:356] "Generic (PLEG): container finished" podID="bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0" containerID="19377fb2c7b55ba8ad1a164d262d13e684c7ab57434d86ed7c4f7f2309ff9c23" exitCode=0
Nov 01 03:45:24.630808 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:24.630652    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-bpgj8" event={"ID":"bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0","Type":"ContainerDied","Data":"19377fb2c7b55ba8ad1a164d262d13e684c7ab57434d86ed7c4f7f2309ff9c23"}
Nov 01 03:45:24.630808 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:24.630657    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-bpgj8"
Nov 01 03:45:24.630808 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:24.630707    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-bpgj8" event={"ID":"bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0","Type":"ContainerDied","Data":"bc60717e987690c1bb2aedd0576c7fce97ba2ff99bd686c7064c69c6f404cfa4"}
Nov 01 03:45:24.630808 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:24.630727    4400 scope.go:122] "RemoveContainer" containerID="19377fb2c7b55ba8ad1a164d262d13e684c7ab57434d86ed7c4f7f2309ff9c23"
Nov 01 03:45:24.641648 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:24.641629    4400 scope.go:122] "RemoveContainer" containerID="19377fb2c7b55ba8ad1a164d262d13e684c7ab57434d86ed7c4f7f2309ff9c23"
Nov 01 03:45:24.641927 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:45:24.641906    4400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19377fb2c7b55ba8ad1a164d262d13e684c7ab57434d86ed7c4f7f2309ff9c23\": container with ID starting with 19377fb2c7b55ba8ad1a164d262d13e684c7ab57434d86ed7c4f7f2309ff9c23 not found: ID does not exist" containerID="19377fb2c7b55ba8ad1a164d262d13e684c7ab57434d86ed7c4f7f2309ff9c23"
Nov 01 03:45:24.642020 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:24.641931    4400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19377fb2c7b55ba8ad1a164d262d13e684c7ab57434d86ed7c4f7f2309ff9c23"} err="failed to get container status \"19377fb2c7b55ba8ad1a164d262d13e684c7ab57434d86ed7c4f7f2309ff9c23\": rpc error: code = NotFound desc = could not find container \"19377fb2c7b55ba8ad1a164d262d13e684c7ab57434d86ed7c4f7f2309ff9c23\": container with ID starting with 19377fb2c7b55ba8ad1a164d262d13e684c7ab57434d86ed7c4f7f2309ff9c23 not found: ID does not exist"
Nov 01 03:45:24.800553 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:24.800527    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-bpgj8"]
Nov 01 03:45:24.851624 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:24.851591    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-bpgj8"]
Nov 01 03:45:25.615636 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:25.615600    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0" path="/var/lib/kubelet/pods/bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0/volumes"
Nov 01 03:45:29.176485 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:29.176453    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 03:45:30.962739 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:30.962705    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-417aa8-55c4f757fb-w744m"]
Nov 01 03:45:30.963374 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:30.963360    4400 cpu_manager.go:414] "containerMap: removing container" podUID="9b31e254-c788-4169-9fc3-d286a54d50d7" containerName="maas-api"
Nov 01 03:45:30.963429 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:30.963376    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b31e254-c788-4169-9fc3-d286a54d50d7" containerName="maas-api"
Nov 01 03:45:30.963429 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:30.963403    4400 cpu_manager.go:414] "containerMap: removing container" podUID="bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0" containerName="istio-proxy"
Nov 01 03:45:30.963429 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:30.963408    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0" containerName="istio-proxy"
Nov 01 03:45:30.963515 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:30.963432    4400 cpu_manager.go:414] "containerMap: removing container" podUID="34fb7732-c2d8-4785-ba96-d510f413015a" containerName="test-curl-7072-6a4993"
Nov 01 03:45:30.963515 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:30.963437    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34fb7732-c2d8-4785-ba96-d510f413015a" containerName="test-curl-7072-6a4993"
Nov 01 03:45:30.963587 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:30.963529    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="34fb7732-c2d8-4785-ba96-d510f413015a" containerName="test-curl-7072-6a4993"
Nov 01 03:45:30.963587 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:30.963540    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b31e254-c788-4169-9fc3-d286a54d50d7" containerName="maas-api"
Nov 01 03:45:30.963587 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:30.963550    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdfd43b2-b7ab-4d1a-a31e-5ebbd8ad50d0" containerName="istio-proxy"
Nov 01 03:45:30.966777 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:30.966762    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-417aa8-55c4f757fb-w744m"
Nov 01 03:45:31.020985 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:31.020963    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-417aa8-serving-cert\""
Nov 01 03:45:31.075862 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:31.075832    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-417aa8-55c4f757fb-w744m"]
Nov 01 03:45:31.111953 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:31.111928    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmsv2\" (UniqueName: \"kubernetes.io/projected/dc04277f-2785-4c65-a353-4afca8198519-kube-api-access-wmsv2\") pod \"maas-api-e2e-api-b-417aa8-55c4f757fb-w744m\" (UID: \"dc04277f-2785-4c65-a353-4afca8198519\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-417aa8-55c4f757fb-w744m"
Nov 01 03:45:31.112121 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:31.111984    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc04277f-2785-4c65-a353-4afca8198519-openshift-service-ca\") pod \"maas-api-e2e-api-b-417aa8-55c4f757fb-w744m\" (UID: \"dc04277f-2785-4c65-a353-4afca8198519\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-417aa8-55c4f757fb-w744m"
Nov 01 03:45:31.112217 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:31.112195    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc04277f-2785-4c65-a353-4afca8198519-maas-api-tls\") pod \"maas-api-e2e-api-b-417aa8-55c4f757fb-w744m\" (UID: \"dc04277f-2785-4c65-a353-4afca8198519\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-417aa8-55c4f757fb-w744m"
Nov 01 03:45:31.112280 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:31.112258    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc04277f-2785-4c65-a353-4afca8198519-metrics-certs\") pod \"maas-api-e2e-api-b-417aa8-55c4f757fb-w744m\" (UID: \"dc04277f-2785-4c65-a353-4afca8198519\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-417aa8-55c4f757fb-w744m"
Nov 01 03:45:31.214055 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:31.213954    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc04277f-2785-4c65-a353-4afca8198519-maas-api-tls\") pod \"maas-api-e2e-api-b-417aa8-55c4f757fb-w744m\" (UID: \"dc04277f-2785-4c65-a353-4afca8198519\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-417aa8-55c4f757fb-w744m"
Nov 01 03:45:31.214055 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:31.214031    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc04277f-2785-4c65-a353-4afca8198519-metrics-certs\") pod \"maas-api-e2e-api-b-417aa8-55c4f757fb-w744m\" (UID: \"dc04277f-2785-4c65-a353-4afca8198519\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-417aa8-55c4f757fb-w744m"
Nov 01 03:45:31.214295 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:45:31.214142    4400 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-417aa8-serving-cert: secret "maas-api-e2e-api-b-417aa8-serving-cert" not found
Nov 01 03:45:31.214295 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:45:31.214232    4400 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc04277f-2785-4c65-a353-4afca8198519-maas-api-tls podName:dc04277f-2785-4c65-a353-4afca8198519 nodeName:}" failed. No retries permitted until 2026-11-01 03:45:31.714198119 +0000 UTC m=+2338.228843973 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/dc04277f-2785-4c65-a353-4afca8198519-maas-api-tls") pod "maas-api-e2e-api-b-417aa8-55c4f757fb-w744m" (UID: "dc04277f-2785-4c65-a353-4afca8198519") : secret "maas-api-e2e-api-b-417aa8-serving-cert" not found
Nov 01 03:45:31.214422 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:31.214145    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmsv2\" (UniqueName: \"kubernetes.io/projected/dc04277f-2785-4c65-a353-4afca8198519-kube-api-access-wmsv2\") pod \"maas-api-e2e-api-b-417aa8-55c4f757fb-w744m\" (UID: \"dc04277f-2785-4c65-a353-4afca8198519\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-417aa8-55c4f757fb-w744m"
Nov 01 03:45:31.214477 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:31.214428    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc04277f-2785-4c65-a353-4afca8198519-openshift-service-ca\") pod \"maas-api-e2e-api-b-417aa8-55c4f757fb-w744m\" (UID: \"dc04277f-2785-4c65-a353-4afca8198519\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-417aa8-55c4f757fb-w744m"
Nov 01 03:45:31.215192 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:31.215169    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc04277f-2785-4c65-a353-4afca8198519-openshift-service-ca\") pod \"maas-api-e2e-api-b-417aa8-55c4f757fb-w744m\" (UID: \"dc04277f-2785-4c65-a353-4afca8198519\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-417aa8-55c4f757fb-w744m"
Nov 01 03:45:31.216984 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:31.216961    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc04277f-2785-4c65-a353-4afca8198519-metrics-certs\") pod \"maas-api-e2e-api-b-417aa8-55c4f757fb-w744m\" (UID: \"dc04277f-2785-4c65-a353-4afca8198519\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-417aa8-55c4f757fb-w744m"
Nov 01 03:45:31.290401 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:31.290369    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmsv2\" (UniqueName: \"kubernetes.io/projected/dc04277f-2785-4c65-a353-4afca8198519-kube-api-access-wmsv2\") pod \"maas-api-e2e-api-b-417aa8-55c4f757fb-w744m\" (UID: \"dc04277f-2785-4c65-a353-4afca8198519\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-417aa8-55c4f757fb-w744m"
Nov 01 03:45:31.722548 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:31.722509    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc04277f-2785-4c65-a353-4afca8198519-maas-api-tls\") pod \"maas-api-e2e-api-b-417aa8-55c4f757fb-w744m\" (UID: \"dc04277f-2785-4c65-a353-4afca8198519\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-417aa8-55c4f757fb-w744m"
Nov 01 03:45:31.724941 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:31.724921    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc04277f-2785-4c65-a353-4afca8198519-maas-api-tls\") pod \"maas-api-e2e-api-b-417aa8-55c4f757fb-w744m\" (UID: \"dc04277f-2785-4c65-a353-4afca8198519\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-417aa8-55c4f757fb-w744m"
Nov 01 03:45:31.730478 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:45:31.730447    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:45:31.880802 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:31.880781    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-417aa8-55c4f757fb-w744m"
Nov 01 03:45:32.261247 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:45:32.261218    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc04277f_2785_4c65_a353_4afca8198519.slice/crio-b2a33e2546d0a842a4821184a1de3d71381773baf491fe4ef0f1e2c9ea2d027f WatchSource:0}: Error finding container b2a33e2546d0a842a4821184a1de3d71381773baf491fe4ef0f1e2c9ea2d027f: Status 404 returned error can't find the container with id b2a33e2546d0a842a4821184a1de3d71381773baf491fe4ef0f1e2c9ea2d027f
Nov 01 03:45:32.261564 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:32.261454    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-417aa8-55c4f757fb-w744m"]
Nov 01 03:45:32.262695 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:32.262662    4400 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 03:45:32.679832 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:32.679795    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-417aa8-55c4f757fb-w744m" event={"ID":"dc04277f-2785-4c65-a353-4afca8198519","Type":"ContainerStarted","Data":"b2a33e2546d0a842a4821184a1de3d71381773baf491fe4ef0f1e2c9ea2d027f"}
Nov 01 03:45:33.687281 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:33.687241    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-417aa8-55c4f757fb-w744m" event={"ID":"dc04277f-2785-4c65-a353-4afca8198519","Type":"ContainerStarted","Data":"a94134333c8455c36ff5879d09e123eb04b1e373c40826bf6df20f56c2bad730"}
Nov 01 03:45:33.687667 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:33.687443    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-417aa8-55c4f757fb-w744m"
Nov 01 03:45:33.808660 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:33.808615    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-417aa8-55c4f757fb-w744m" podStartSLOduration=2.9327033289999997 podStartE2EDuration="3.808603421s" totalImagesPullingTime="875.900092ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:45:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:45:33.807936632 +0000 UTC m=+2340.322582493" watchObservedRunningTime="2026-11-01 03:45:33.808603421 +0000 UTC m=+2340.323249317"
Nov 01 03:45:35.587443 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:35.587411    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fdbdfefd-cb877c7cc-2xdgb"]
Nov 01 03:45:35.587890 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:35.587636    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fdbdfefd-cb877c7cc-2xdgb" podUID="f74835f6-1cf9-4580-bef8-c14f88b6978c" containerName="maas-api" containerID="cri-o://dca4f8aa1b7efd00d149a616c60d63be00390c45203b9d0f0d93d2fcf5b845db" gracePeriod=30
Nov 01 03:45:35.912502 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:35.912479    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fdbdfefd-cb877c7cc-2xdgb"
Nov 01 03:45:35.984570 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:35.984544    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f74835f6-1cf9-4580-bef8-c14f88b6978c-kube-api-access-5bwv6\" (UniqueName: \"kubernetes.io/projected/f74835f6-1cf9-4580-bef8-c14f88b6978c-kube-api-access-5bwv6\") pod \"f74835f6-1cf9-4580-bef8-c14f88b6978c\" (UID: \"f74835f6-1cf9-4580-bef8-c14f88b6978c\") "
Nov 01 03:45:35.984744 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:35.984601    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f74835f6-1cf9-4580-bef8-c14f88b6978c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f74835f6-1cf9-4580-bef8-c14f88b6978c-openshift-service-ca\") pod \"f74835f6-1cf9-4580-bef8-c14f88b6978c\" (UID: \"f74835f6-1cf9-4580-bef8-c14f88b6978c\") "
Nov 01 03:45:35.984744 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:35.984641    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f74835f6-1cf9-4580-bef8-c14f88b6978c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f74835f6-1cf9-4580-bef8-c14f88b6978c-maas-api-tls\") pod \"f74835f6-1cf9-4580-bef8-c14f88b6978c\" (UID: \"f74835f6-1cf9-4580-bef8-c14f88b6978c\") "
Nov 01 03:45:35.984744 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:35.984672    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f74835f6-1cf9-4580-bef8-c14f88b6978c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f74835f6-1cf9-4580-bef8-c14f88b6978c-metrics-certs\") pod \"f74835f6-1cf9-4580-bef8-c14f88b6978c\" (UID: \"f74835f6-1cf9-4580-bef8-c14f88b6978c\") "
Nov 01 03:45:35.985037 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:35.984994    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f74835f6-1cf9-4580-bef8-c14f88b6978c-openshift-service-ca" pod "f74835f6-1cf9-4580-bef8-c14f88b6978c" (UID: "f74835f6-1cf9-4580-bef8-c14f88b6978c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:45:35.985217 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:35.985200    4400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f74835f6-1cf9-4580-bef8-c14f88b6978c-openshift-service-ca\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:45:35.989705 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:35.987186    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f74835f6-1cf9-4580-bef8-c14f88b6978c-metrics-certs" pod "f74835f6-1cf9-4580-bef8-c14f88b6978c" (UID: "f74835f6-1cf9-4580-bef8-c14f88b6978c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:45:35.989705 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:35.987299    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f74835f6-1cf9-4580-bef8-c14f88b6978c-maas-api-tls" pod "f74835f6-1cf9-4580-bef8-c14f88b6978c" (UID: "f74835f6-1cf9-4580-bef8-c14f88b6978c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:45:35.989705 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:35.987372    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f74835f6-1cf9-4580-bef8-c14f88b6978c-kube-api-access-5bwv6" pod "f74835f6-1cf9-4580-bef8-c14f88b6978c" (UID: "f74835f6-1cf9-4580-bef8-c14f88b6978c"). InnerVolumeSpecName "kube-api-access-5bwv6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:45:36.093426 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:36.086241    4400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f74835f6-1cf9-4580-bef8-c14f88b6978c-maas-api-tls\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:45:36.093426 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:36.086267    4400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f74835f6-1cf9-4580-bef8-c14f88b6978c-metrics-certs\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:45:36.093426 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:36.086277    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bwv6\" (UniqueName: \"kubernetes.io/projected/f74835f6-1cf9-4580-bef8-c14f88b6978c-kube-api-access-5bwv6\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:45:36.711249 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:36.711214    4400 generic.go:356] "Generic (PLEG): container finished" podID="f74835f6-1cf9-4580-bef8-c14f88b6978c" containerID="dca4f8aa1b7efd00d149a616c60d63be00390c45203b9d0f0d93d2fcf5b845db" exitCode=0
Nov 01 03:45:36.711653 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:36.711341    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fdbdfefd-cb877c7cc-2xdgb" event={"ID":"f74835f6-1cf9-4580-bef8-c14f88b6978c","Type":"ContainerDied","Data":"dca4f8aa1b7efd00d149a616c60d63be00390c45203b9d0f0d93d2fcf5b845db"}
Nov 01 03:45:36.711653 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:36.711395    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fdbdfefd-cb877c7cc-2xdgb" event={"ID":"f74835f6-1cf9-4580-bef8-c14f88b6978c","Type":"ContainerDied","Data":"2603e507fa0eff2971751a5aca77b9c5c155752eb15eb2fc8b46008e6c595b6e"}
Nov 01 03:45:36.711653 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:36.711423    4400 scope.go:122] "RemoveContainer" containerID="dca4f8aa1b7efd00d149a616c60d63be00390c45203b9d0f0d93d2fcf5b845db"
Nov 01 03:45:36.711653 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:36.711442    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fdbdfefd-cb877c7cc-2xdgb"
Nov 01 03:45:36.722815 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:36.722795    4400 scope.go:122] "RemoveContainer" containerID="dca4f8aa1b7efd00d149a616c60d63be00390c45203b9d0f0d93d2fcf5b845db"
Nov 01 03:45:36.723110 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:45:36.723089    4400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dca4f8aa1b7efd00d149a616c60d63be00390c45203b9d0f0d93d2fcf5b845db\": container with ID starting with dca4f8aa1b7efd00d149a616c60d63be00390c45203b9d0f0d93d2fcf5b845db not found: ID does not exist" containerID="dca4f8aa1b7efd00d149a616c60d63be00390c45203b9d0f0d93d2fcf5b845db"
Nov 01 03:45:36.723169 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:36.723117    4400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dca4f8aa1b7efd00d149a616c60d63be00390c45203b9d0f0d93d2fcf5b845db"} err="failed to get container status \"dca4f8aa1b7efd00d149a616c60d63be00390c45203b9d0f0d93d2fcf5b845db\": rpc error: code = NotFound desc = could not find container \"dca4f8aa1b7efd00d149a616c60d63be00390c45203b9d0f0d93d2fcf5b845db\": container with ID starting with dca4f8aa1b7efd00d149a616c60d63be00390c45203b9d0f0d93d2fcf5b845db not found: ID does not exist"
Nov 01 03:45:36.882267 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:36.882241    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fdbdfefd-cb877c7cc-2xdgb"]
Nov 01 03:45:36.886990 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:36.886968    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fdbdfefd-cb877c7cc-2xdgb"]
Nov 01 03:45:37.084661 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:37.084633    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-22e11b/test-model-22e11b-kserve-f5fb75fb5-vkc4k"]
Nov 01 03:45:37.085337 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:37.085323    4400 cpu_manager.go:414] "containerMap: removing container" podUID="f74835f6-1cf9-4580-bef8-c14f88b6978c" containerName="maas-api"
Nov 01 03:45:37.085390 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:37.085338    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f74835f6-1cf9-4580-bef8-c14f88b6978c" containerName="maas-api"
Nov 01 03:45:37.085440 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:37.085432    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="f74835f6-1cf9-4580-bef8-c14f88b6978c" containerName="maas-api"
Nov 01 03:45:37.090059 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:37.090041    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-22e11b/test-model-22e11b-kserve-f5fb75fb5-vkc4k"
Nov 01 03:45:37.113352 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:37.113321    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7b0b2d1-ff0e-4bde-a832-430c7007749a-tls-certs\") pod \"test-model-22e11b-kserve-f5fb75fb5-vkc4k\" (UID: \"f7b0b2d1-ff0e-4bde-a832-430c7007749a\") " pod="ai-tenant-e2e-inf-a-22e11b/test-model-22e11b-kserve-f5fb75fb5-vkc4k"
Nov 01 03:45:37.113548 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:37.113531    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7b0b2d1-ff0e-4bde-a832-430c7007749a-home\") pod \"test-model-22e11b-kserve-f5fb75fb5-vkc4k\" (UID: \"f7b0b2d1-ff0e-4bde-a832-430c7007749a\") " pod="ai-tenant-e2e-inf-a-22e11b/test-model-22e11b-kserve-f5fb75fb5-vkc4k"
Nov 01 03:45:37.113664 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:37.113649    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq566\" (UniqueName: \"kubernetes.io/projected/f7b0b2d1-ff0e-4bde-a832-430c7007749a-kube-api-access-cq566\") pod \"test-model-22e11b-kserve-f5fb75fb5-vkc4k\" (UID: \"f7b0b2d1-ff0e-4bde-a832-430c7007749a\") " pod="ai-tenant-e2e-inf-a-22e11b/test-model-22e11b-kserve-f5fb75fb5-vkc4k"
Nov 01 03:45:37.113789 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:37.113774    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7b0b2d1-ff0e-4bde-a832-430c7007749a-model-cache\") pod \"test-model-22e11b-kserve-f5fb75fb5-vkc4k\" (UID: \"f7b0b2d1-ff0e-4bde-a832-430c7007749a\") " pod="ai-tenant-e2e-inf-a-22e11b/test-model-22e11b-kserve-f5fb75fb5-vkc4k"
Nov 01 03:45:37.114100 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:37.114082    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7b0b2d1-ff0e-4bde-a832-430c7007749a-dshm\") pod \"test-model-22e11b-kserve-f5fb75fb5-vkc4k\" (UID: \"f7b0b2d1-ff0e-4bde-a832-430c7007749a\") " pod="ai-tenant-e2e-inf-a-22e11b/test-model-22e11b-kserve-f5fb75fb5-vkc4k"
Nov 01 03:45:37.114215 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:37.114199    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7b0b2d1-ff0e-4bde-a832-430c7007749a-tmp-dir\") pod \"test-model-22e11b-kserve-f5fb75fb5-vkc4k\" (UID: \"f7b0b2d1-ff0e-4bde-a832-430c7007749a\") " pod="ai-tenant-e2e-inf-a-22e11b/test-model-22e11b-kserve-f5fb75fb5-vkc4k"
Nov 01 03:45:37.146024 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:37.145999    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-22e11b\"/\"test-model-22e11b-kserve-self-signed-certs\""
Nov 01 03:45:37.146169 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:37.146042    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-22e11b\"/\"default-dockercfg-stcqv\""
Nov 01 03:45:37.147039 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:37.147017    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-22e11b/test-model-22e11b-kserve-f5fb75fb5-vkc4k"]
Nov 01 03:45:37.216711 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:37.215455    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-22e11b\"/\"kube-root-ca.crt\""
Nov 01 03:45:37.216711 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:37.215594    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-22e11b\"/\"openshift-service-ca.crt\""
Nov 01 03:45:37.221021 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:37.218424    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7b0b2d1-ff0e-4bde-a832-430c7007749a-tls-certs\") pod \"test-model-22e11b-kserve-f5fb75fb5-vkc4k\" (UID: \"f7b0b2d1-ff0e-4bde-a832-430c7007749a\") " pod="ai-tenant-e2e-inf-a-22e11b/test-model-22e11b-kserve-f5fb75fb5-vkc4k"
Nov 01 03:45:37.221021 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:37.219167    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7b0b2d1-ff0e-4bde-a832-430c7007749a-home\") pod \"test-model-22e11b-kserve-f5fb75fb5-vkc4k\" (UID: \"f7b0b2d1-ff0e-4bde-a832-430c7007749a\") " pod="ai-tenant-e2e-inf-a-22e11b/test-model-22e11b-kserve-f5fb75fb5-vkc4k"
Nov 01 03:45:37.221021 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:37.219198    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq566\" (UniqueName: \"kubernetes.io/projected/f7b0b2d1-ff0e-4bde-a832-430c7007749a-kube-api-access-cq566\") pod \"test-model-22e11b-kserve-f5fb75fb5-vkc4k\" (UID: \"f7b0b2d1-ff0e-4bde-a832-430c7007749a\") " pod="ai-tenant-e2e-inf-a-22e11b/test-model-22e11b-kserve-f5fb75fb5-vkc4k"
Nov 01 03:45:37.221021 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:37.220014    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7b0b2d1-ff0e-4bde-a832-430c7007749a-home\") pod \"test-model-22e11b-kserve-f5fb75fb5-vkc4k\" (UID: \"f7b0b2d1-ff0e-4bde-a832-430c7007749a\") " pod="ai-tenant-e2e-inf-a-22e11b/test-model-22e11b-kserve-f5fb75fb5-vkc4k"
Nov 01 03:45:37.221021 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:37.220133    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7b0b2d1-ff0e-4bde-a832-430c7007749a-model-cache\") pod \"test-model-22e11b-kserve-f5fb75fb5-vkc4k\" (UID: \"f7b0b2d1-ff0e-4bde-a832-430c7007749a\") " pod="ai-tenant-e2e-inf-a-22e11b/test-model-22e11b-kserve-f5fb75fb5-vkc4k"
Nov 01 03:45:37.221021 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:37.220873    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7b0b2d1-ff0e-4bde-a832-430c7007749a-model-cache\") pod \"test-model-22e11b-kserve-f5fb75fb5-vkc4k\" (UID: \"f7b0b2d1-ff0e-4bde-a832-430c7007749a\") " pod="ai-tenant-e2e-inf-a-22e11b/test-model-22e11b-kserve-f5fb75fb5-vkc4k"
Nov 01 03:45:37.221021 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:37.220172    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7b0b2d1-ff0e-4bde-a832-430c7007749a-dshm\") pod \"test-model-22e11b-kserve-f5fb75fb5-vkc4k\" (UID: \"f7b0b2d1-ff0e-4bde-a832-430c7007749a\") " pod="ai-tenant-e2e-inf-a-22e11b/test-model-22e11b-kserve-f5fb75fb5-vkc4k"
Nov 01 03:45:37.221021 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:37.220952    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7b0b2d1-ff0e-4bde-a832-430c7007749a-tmp-dir\") pod \"test-model-22e11b-kserve-f5fb75fb5-vkc4k\" (UID: \"f7b0b2d1-ff0e-4bde-a832-430c7007749a\") " pod="ai-tenant-e2e-inf-a-22e11b/test-model-22e11b-kserve-f5fb75fb5-vkc4k"
Nov 01 03:45:37.221515 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:37.221415    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7b0b2d1-ff0e-4bde-a832-430c7007749a-tmp-dir\") pod \"test-model-22e11b-kserve-f5fb75fb5-vkc4k\" (UID: \"f7b0b2d1-ff0e-4bde-a832-430c7007749a\") " pod="ai-tenant-e2e-inf-a-22e11b/test-model-22e11b-kserve-f5fb75fb5-vkc4k"
Nov 01 03:45:37.278033 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:37.277998    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7b0b2d1-ff0e-4bde-a832-430c7007749a-dshm\") pod \"test-model-22e11b-kserve-f5fb75fb5-vkc4k\" (UID: \"f7b0b2d1-ff0e-4bde-a832-430c7007749a\") " pod="ai-tenant-e2e-inf-a-22e11b/test-model-22e11b-kserve-f5fb75fb5-vkc4k"
Nov 01 03:45:37.278298 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:37.278255    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7b0b2d1-ff0e-4bde-a832-430c7007749a-tls-certs\") pod \"test-model-22e11b-kserve-f5fb75fb5-vkc4k\" (UID: \"f7b0b2d1-ff0e-4bde-a832-430c7007749a\") " pod="ai-tenant-e2e-inf-a-22e11b/test-model-22e11b-kserve-f5fb75fb5-vkc4k"
Nov 01 03:45:37.285129 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:37.285100    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq566\" (UniqueName: \"kubernetes.io/projected/f7b0b2d1-ff0e-4bde-a832-430c7007749a-kube-api-access-cq566\") pod \"test-model-22e11b-kserve-f5fb75fb5-vkc4k\" (UID: \"f7b0b2d1-ff0e-4bde-a832-430c7007749a\") " pod="ai-tenant-e2e-inf-a-22e11b/test-model-22e11b-kserve-f5fb75fb5-vkc4k"
Nov 01 03:45:37.404535 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:37.404456    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-22e11b/test-model-22e11b-kserve-f5fb75fb5-vkc4k"
Nov 01 03:45:37.617049 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:37.617017    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f74835f6-1cf9-4580-bef8-c14f88b6978c" path="/var/lib/kubelet/pods/f74835f6-1cf9-4580-bef8-c14f88b6978c/volumes"
Nov 01 03:45:37.787107 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:37.786864    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-22e11b/test-model-22e11b-kserve-f5fb75fb5-vkc4k"]
Nov 01 03:45:37.788650 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:45:37.788614    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf7b0b2d1_ff0e_4bde_a832_430c7007749a.slice/crio-53db76dba6f7e6a68a58a18e876f03012fdc4581ef8674f8a415edd7e1663017 WatchSource:0}: Error finding container 53db76dba6f7e6a68a58a18e876f03012fdc4581ef8674f8a415edd7e1663017: Status 404 returned error can't find the container with id 53db76dba6f7e6a68a58a18e876f03012fdc4581ef8674f8a415edd7e1663017
Nov 01 03:45:38.727602 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:38.727566    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-22e11b/test-model-22e11b-kserve-f5fb75fb5-vkc4k" event={"ID":"f7b0b2d1-ff0e-4bde-a832-430c7007749a","Type":"ContainerStarted","Data":"d181aed4c212dfa838ff9ade2bdbab3fc57373f48213ac5681563c9e05bb67e4"}
Nov 01 03:45:38.727602 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:38.727602    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-22e11b/test-model-22e11b-kserve-f5fb75fb5-vkc4k" event={"ID":"f7b0b2d1-ff0e-4bde-a832-430c7007749a","Type":"ContainerStarted","Data":"53db76dba6f7e6a68a58a18e876f03012fdc4581ef8674f8a415edd7e1663017"}
Nov 01 03:45:38.847412 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:38.847363    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-22e11b/test-model-22e11b-kserve-f5fb75fb5-vkc4k" podStartSLOduration=1.84735189 podStartE2EDuration="1.84735189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:45:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:45:38.846884151 +0000 UTC m=+2345.361530015" watchObservedRunningTime="2026-11-01 03:45:38.84735189 +0000 UTC m=+2345.361997752"
Nov 01 03:45:39.404820 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:39.404795    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-22e11b/test-model-22e11b-kserve-f5fb75fb5-vkc4k"
Nov 01 03:45:39.698507 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:39.698445    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-417aa8-55c4f757fb-w744m"
Nov 01 03:45:42.135708 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:42.129820    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-da4f30-5d8cbd8644-kztdd"]
Nov 01 03:45:42.135708 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:42.130048    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da4f30-5d8cbd8644-kztdd" podUID="110908f1-9498-4f67-95b3-75b3e9a45571" containerName="maas-api" containerID="cri-o://d096d99affa4629c52253654eda4c14d54af895c25ef01d71a104696fea57890" gracePeriod=30
Nov 01 03:45:42.441543 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:42.441518    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da4f30-5d8cbd8644-kztdd"
Nov 01 03:45:42.640709 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:42.621150    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/110908f1-9498-4f67-95b3-75b3e9a45571-metrics-certs\" (UniqueName: \"kubernetes.io/secret/110908f1-9498-4f67-95b3-75b3e9a45571-metrics-certs\") pod \"110908f1-9498-4f67-95b3-75b3e9a45571\" (UID: \"110908f1-9498-4f67-95b3-75b3e9a45571\") "
Nov 01 03:45:42.640709 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:42.621285    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/110908f1-9498-4f67-95b3-75b3e9a45571-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/110908f1-9498-4f67-95b3-75b3e9a45571-openshift-service-ca\") pod \"110908f1-9498-4f67-95b3-75b3e9a45571\" (UID: \"110908f1-9498-4f67-95b3-75b3e9a45571\") "
Nov 01 03:45:42.640709 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:42.621386    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/110908f1-9498-4f67-95b3-75b3e9a45571-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/110908f1-9498-4f67-95b3-75b3e9a45571-maas-api-tls\") pod \"110908f1-9498-4f67-95b3-75b3e9a45571\" (UID: \"110908f1-9498-4f67-95b3-75b3e9a45571\") "
Nov 01 03:45:42.640709 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:42.621464    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/110908f1-9498-4f67-95b3-75b3e9a45571-kube-api-access-lv2kx\" (UniqueName: \"kubernetes.io/projected/110908f1-9498-4f67-95b3-75b3e9a45571-kube-api-access-lv2kx\") pod \"110908f1-9498-4f67-95b3-75b3e9a45571\" (UID: \"110908f1-9498-4f67-95b3-75b3e9a45571\") "
Nov 01 03:45:42.640709 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:42.625353    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/110908f1-9498-4f67-95b3-75b3e9a45571-openshift-service-ca" pod "110908f1-9498-4f67-95b3-75b3e9a45571" (UID: "110908f1-9498-4f67-95b3-75b3e9a45571"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:45:42.641107 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:42.640972    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/110908f1-9498-4f67-95b3-75b3e9a45571-metrics-certs" pod "110908f1-9498-4f67-95b3-75b3e9a45571" (UID: "110908f1-9498-4f67-95b3-75b3e9a45571"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:45:42.643711 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:42.643655    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/110908f1-9498-4f67-95b3-75b3e9a45571-maas-api-tls" pod "110908f1-9498-4f67-95b3-75b3e9a45571" (UID: "110908f1-9498-4f67-95b3-75b3e9a45571"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:45:42.646366 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:42.646302    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/110908f1-9498-4f67-95b3-75b3e9a45571-kube-api-access-lv2kx" pod "110908f1-9498-4f67-95b3-75b3e9a45571" (UID: "110908f1-9498-4f67-95b3-75b3e9a45571"). InnerVolumeSpecName "kube-api-access-lv2kx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:45:42.725898 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:42.725870    4400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/110908f1-9498-4f67-95b3-75b3e9a45571-metrics-certs\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:45:42.726329 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:42.726314    4400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/110908f1-9498-4f67-95b3-75b3e9a45571-openshift-service-ca\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:45:42.726413 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:42.726405    4400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/110908f1-9498-4f67-95b3-75b3e9a45571-maas-api-tls\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:45:42.726481 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:42.726470    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lv2kx\" (UniqueName: \"kubernetes.io/projected/110908f1-9498-4f67-95b3-75b3e9a45571-kube-api-access-lv2kx\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:45:42.755799 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:42.755768    4400 generic.go:356] "Generic (PLEG): container finished" podID="110908f1-9498-4f67-95b3-75b3e9a45571" containerID="d096d99affa4629c52253654eda4c14d54af895c25ef01d71a104696fea57890" exitCode=0
Nov 01 03:45:42.755947 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:42.755820    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da4f30-5d8cbd8644-kztdd"
Nov 01 03:45:42.755947 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:42.755845    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da4f30-5d8cbd8644-kztdd" event={"ID":"110908f1-9498-4f67-95b3-75b3e9a45571","Type":"ContainerDied","Data":"d096d99affa4629c52253654eda4c14d54af895c25ef01d71a104696fea57890"}
Nov 01 03:45:42.755947 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:42.755896    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da4f30-5d8cbd8644-kztdd" event={"ID":"110908f1-9498-4f67-95b3-75b3e9a45571","Type":"ContainerDied","Data":"c22184a0f4286a21cfdd1d04f772bf69047fdd78da2fb860ba29409c931ab365"}
Nov 01 03:45:42.755947 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:42.755918    4400 scope.go:122] "RemoveContainer" containerID="d096d99affa4629c52253654eda4c14d54af895c25ef01d71a104696fea57890"
Nov 01 03:45:42.766609 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:42.766593    4400 scope.go:122] "RemoveContainer" containerID="d096d99affa4629c52253654eda4c14d54af895c25ef01d71a104696fea57890"
Nov 01 03:45:42.766867 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:45:42.766849    4400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d096d99affa4629c52253654eda4c14d54af895c25ef01d71a104696fea57890\": container with ID starting with d096d99affa4629c52253654eda4c14d54af895c25ef01d71a104696fea57890 not found: ID does not exist" containerID="d096d99affa4629c52253654eda4c14d54af895c25ef01d71a104696fea57890"
Nov 01 03:45:42.766922 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:42.766873    4400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d096d99affa4629c52253654eda4c14d54af895c25ef01d71a104696fea57890"} err="failed to get container status \"d096d99affa4629c52253654eda4c14d54af895c25ef01d71a104696fea57890\": rpc error: code = NotFound desc = could not find container \"d096d99affa4629c52253654eda4c14d54af895c25ef01d71a104696fea57890\": container with ID starting with d096d99affa4629c52253654eda4c14d54af895c25ef01d71a104696fea57890 not found: ID does not exist"
Nov 01 03:45:43.008203 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:43.008144    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-da4f30-5d8cbd8644-kztdd"]
Nov 01 03:45:43.059319 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:43.059290    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-da4f30-5d8cbd8644-kztdd"]
Nov 01 03:45:43.615729 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:43.615699    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="110908f1-9498-4f67-95b3-75b3e9a45571" path="/var/lib/kubelet/pods/110908f1-9498-4f67-95b3-75b3e9a45571/volumes"
Nov 01 03:45:47.404863 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:47.404833    4400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-22e11b/test-model-22e11b-kserve-f5fb75fb5-vkc4k"
Nov 01 03:45:47.417068 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:47.417044    4400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-22e11b/test-model-22e11b-kserve-f5fb75fb5-vkc4k"
Nov 01 03:45:47.856233 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:47.856209    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-22e11b/test-model-22e11b-kserve-f5fb75fb5-vkc4k"
Nov 01 03:45:55.507502 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:55.507471    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-170ffa-6fdbdd6d4c-tvj7k"]
Nov 01 03:45:55.508148 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:55.508134    4400 cpu_manager.go:414] "containerMap: removing container" podUID="110908f1-9498-4f67-95b3-75b3e9a45571" containerName="maas-api"
Nov 01 03:45:55.508191 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:55.508150    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="110908f1-9498-4f67-95b3-75b3e9a45571" containerName="maas-api"
Nov 01 03:45:55.508285 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:55.508277    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="110908f1-9498-4f67-95b3-75b3e9a45571" containerName="maas-api"
Nov 01 03:45:55.513397 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:55.513375    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-170ffa-6fdbdd6d4c-tvj7k"
Nov 01 03:45:55.561375 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:55.561337    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xp9jm\" (UniqueName: \"kubernetes.io/projected/7159d3dd-647e-45b6-9f5b-1dbaecb70ad3-kube-api-access-xp9jm\") pod \"payload-pre-processing-e2e-api-b-170ffa-6fdbdd6d4c-tvj7k\" (UID: \"7159d3dd-647e-45b6-9f5b-1dbaecb70ad3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-170ffa-6fdbdd6d4c-tvj7k"
Nov 01 03:45:55.561567 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:55.561543    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7159d3dd-647e-45b6-9f5b-1dbaecb70ad3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-170ffa-6fdbdd6d4c-tvj7k\" (UID: \"7159d3dd-647e-45b6-9f5b-1dbaecb70ad3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-170ffa-6fdbdd6d4c-tvj7k"
Nov 01 03:45:55.573490 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:55.573454    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-170ffa-6fdbdd6d4c-tvj7k"]
Nov 01 03:45:55.625709 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:55.622083    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-170ffa-dockercfg-r64nw\""
Nov 01 03:45:55.625709 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:55.622187    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-170ffa\""
Nov 01 03:45:55.663645 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:55.663605    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xp9jm\" (UniqueName: \"kubernetes.io/projected/7159d3dd-647e-45b6-9f5b-1dbaecb70ad3-kube-api-access-xp9jm\") pod \"payload-pre-processing-e2e-api-b-170ffa-6fdbdd6d4c-tvj7k\" (UID: \"7159d3dd-647e-45b6-9f5b-1dbaecb70ad3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-170ffa-6fdbdd6d4c-tvj7k"
Nov 01 03:45:55.682585 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:55.682550    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7159d3dd-647e-45b6-9f5b-1dbaecb70ad3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-170ffa-6fdbdd6d4c-tvj7k\" (UID: \"7159d3dd-647e-45b6-9f5b-1dbaecb70ad3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-170ffa-6fdbdd6d4c-tvj7k"
Nov 01 03:45:55.696446 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:55.683922    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7159d3dd-647e-45b6-9f5b-1dbaecb70ad3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-170ffa-6fdbdd6d4c-tvj7k\" (UID: \"7159d3dd-647e-45b6-9f5b-1dbaecb70ad3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-170ffa-6fdbdd6d4c-tvj7k"
Nov 01 03:45:55.730018 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:55.729985    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xp9jm\" (UniqueName: \"kubernetes.io/projected/7159d3dd-647e-45b6-9f5b-1dbaecb70ad3-kube-api-access-xp9jm\") pod \"payload-pre-processing-e2e-api-b-170ffa-6fdbdd6d4c-tvj7k\" (UID: \"7159d3dd-647e-45b6-9f5b-1dbaecb70ad3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-170ffa-6fdbdd6d4c-tvj7k"
Nov 01 03:45:55.828043 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:55.828015    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-170ffa-6fdbdd6d4c-tvj7k"
Nov 01 03:45:56.213810 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:56.213778    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-170ffa-6fdbdd6d4c-tvj7k"]
Nov 01 03:45:56.214142 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:45:56.214107    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7159d3dd_647e_45b6_9f5b_1dbaecb70ad3.slice/crio-48eb69b848613b49ad9694619e0242e77451f6e33a5c1d8bbff024b1dff3a188 WatchSource:0}: Error finding container 48eb69b848613b49ad9694619e0242e77451f6e33a5c1d8bbff024b1dff3a188: Status 404 returned error can't find the container with id 48eb69b848613b49ad9694619e0242e77451f6e33a5c1d8bbff024b1dff3a188
Nov 01 03:45:56.943064 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:56.943023    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-170ffa-6fdbdd6d4c-tvj7k" event={"ID":"7159d3dd-647e-45b6-9f5b-1dbaecb70ad3","Type":"ContainerStarted","Data":"abd94975212d63844f27859b92ec15324b26c952a04836552deaf9d047ad8454"}
Nov 01 03:45:56.943448 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:56.943118    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-170ffa-6fdbdd6d4c-tvj7k" event={"ID":"7159d3dd-647e-45b6-9f5b-1dbaecb70ad3","Type":"ContainerStarted","Data":"48eb69b848613b49ad9694619e0242e77451f6e33a5c1d8bbff024b1dff3a188"}
Nov 01 03:45:56.943448 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:56.943144    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-170ffa-6fdbdd6d4c-tvj7k"
Nov 01 03:45:57.062227 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:57.062194    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-c8cb9a/distinct-2-w2-c8cb9a-kserve-5c5cfb454d-fh2cj"]
Nov 01 03:45:57.113772 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:45:57.113714    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-170ffa-6fdbdd6d4c-tvj7k" podStartSLOduration=2.11366903 podStartE2EDuration="2.11366903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:45:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:45:57.065769327 +0000 UTC m=+2363.580415213" watchObservedRunningTime="2026-11-01 03:45:57.11366903 +0000 UTC m=+2363.628314891"
Nov 01 03:46:01.730058 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:46:01.730024    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:46:07.952951 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:46:07.952924    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-170ffa-6fdbdd6d4c-tvj7k"
Nov 01 03:46:12.084448 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:46:12.084405    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-2-w2-c8cb9a-kserve-5c5cfb454d-fh2cj" podUID="2bcf3b85-3d4a-433f-94ad-0e954bf03e73" containerName="main" containerID="cri-o://50850fc9101f441bdefdec50b53aff9d82d329a0899d5d467e5f2e4eb4fe2e9f" gracePeriod=45
Nov 01 03:46:12.346934 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:46:12.346869    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-2-w2-c8cb9a-kserve-5c5cfb454d-fh2cj"
Nov 01 03:46:12.409433 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:46:12.409400    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bcf3b85-3d4a-433f-94ad-0e954bf03e73-tls-certs\" (UniqueName: \"kubernetes.io/secret/2bcf3b85-3d4a-433f-94ad-0e954bf03e73-tls-certs\") pod \"2bcf3b85-3d4a-433f-94ad-0e954bf03e73\" (UID: \"2bcf3b85-3d4a-433f-94ad-0e954bf03e73\") "
Nov 01 03:46:12.409622 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:46:12.409546    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bcf3b85-3d4a-433f-94ad-0e954bf03e73-home\" (UniqueName: \"kubernetes.io/empty-dir/2bcf3b85-3d4a-433f-94ad-0e954bf03e73-home\") pod \"2bcf3b85-3d4a-433f-94ad-0e954bf03e73\" (UID: \"2bcf3b85-3d4a-433f-94ad-0e954bf03e73\") "
Nov 01 03:46:12.409622 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:46:12.409606    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bcf3b85-3d4a-433f-94ad-0e954bf03e73-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bcf3b85-3d4a-433f-94ad-0e954bf03e73-tmp-dir\") pod \"2bcf3b85-3d4a-433f-94ad-0e954bf03e73\" (UID: \"2bcf3b85-3d4a-433f-94ad-0e954bf03e73\") "
Nov 01 03:46:12.409795 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:46:12.409651    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bcf3b85-3d4a-433f-94ad-0e954bf03e73-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bcf3b85-3d4a-433f-94ad-0e954bf03e73-model-cache\") pod \"2bcf3b85-3d4a-433f-94ad-0e954bf03e73\" (UID: \"2bcf3b85-3d4a-433f-94ad-0e954bf03e73\") "
Nov 01 03:46:12.409795 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:46:12.409731    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bcf3b85-3d4a-433f-94ad-0e954bf03e73-kube-api-access-zrvhv\" (UniqueName: \"kubernetes.io/projected/2bcf3b85-3d4a-433f-94ad-0e954bf03e73-kube-api-access-zrvhv\") pod \"2bcf3b85-3d4a-433f-94ad-0e954bf03e73\" (UID: \"2bcf3b85-3d4a-433f-94ad-0e954bf03e73\") "
Nov 01 03:46:12.409908 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:46:12.409857    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bcf3b85-3d4a-433f-94ad-0e954bf03e73-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bcf3b85-3d4a-433f-94ad-0e954bf03e73-dshm\") pod \"2bcf3b85-3d4a-433f-94ad-0e954bf03e73\" (UID: \"2bcf3b85-3d4a-433f-94ad-0e954bf03e73\") "
Nov 01 03:46:12.409908 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:46:12.409887    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bcf3b85-3d4a-433f-94ad-0e954bf03e73-home" pod "2bcf3b85-3d4a-433f-94ad-0e954bf03e73" (UID: "2bcf3b85-3d4a-433f-94ad-0e954bf03e73"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:46:12.410010 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:46:12.409921    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bcf3b85-3d4a-433f-94ad-0e954bf03e73-tmp-dir" pod "2bcf3b85-3d4a-433f-94ad-0e954bf03e73" (UID: "2bcf3b85-3d4a-433f-94ad-0e954bf03e73"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:46:12.410051 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:46:12.410017    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bcf3b85-3d4a-433f-94ad-0e954bf03e73-model-cache" pod "2bcf3b85-3d4a-433f-94ad-0e954bf03e73" (UID: "2bcf3b85-3d4a-433f-94ad-0e954bf03e73"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:46:12.410651 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:46:12.410626    4400 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2bcf3b85-3d4a-433f-94ad-0e954bf03e73-home\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:46:12.410808 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:46:12.410655    4400 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bcf3b85-3d4a-433f-94ad-0e954bf03e73-tmp-dir\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:46:12.410808 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:46:12.410671    4400 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bcf3b85-3d4a-433f-94ad-0e954bf03e73-model-cache\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:46:12.411938 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:46:12.411911    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bcf3b85-3d4a-433f-94ad-0e954bf03e73-tls-certs" pod "2bcf3b85-3d4a-433f-94ad-0e954bf03e73" (UID: "2bcf3b85-3d4a-433f-94ad-0e954bf03e73"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:46:12.412108 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:46:12.412084    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bcf3b85-3d4a-433f-94ad-0e954bf03e73-dshm" pod "2bcf3b85-3d4a-433f-94ad-0e954bf03e73" (UID: "2bcf3b85-3d4a-433f-94ad-0e954bf03e73"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:46:12.412199 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:46:12.412119    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bcf3b85-3d4a-433f-94ad-0e954bf03e73-kube-api-access-zrvhv" pod "2bcf3b85-3d4a-433f-94ad-0e954bf03e73" (UID: "2bcf3b85-3d4a-433f-94ad-0e954bf03e73"). InnerVolumeSpecName "kube-api-access-zrvhv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:46:12.512878 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:46:12.512829    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrvhv\" (UniqueName: \"kubernetes.io/projected/2bcf3b85-3d4a-433f-94ad-0e954bf03e73-kube-api-access-zrvhv\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:46:12.512878 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:46:12.512870    4400 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bcf3b85-3d4a-433f-94ad-0e954bf03e73-dshm\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:46:12.512878 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:46:12.512885    4400 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bcf3b85-3d4a-433f-94ad-0e954bf03e73-tls-certs\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:46:13.056059 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:46:13.056025    4400 generic.go:356] "Generic (PLEG): container finished" podID="2bcf3b85-3d4a-433f-94ad-0e954bf03e73" containerID="50850fc9101f441bdefdec50b53aff9d82d329a0899d5d467e5f2e4eb4fe2e9f" exitCode=0
Nov 01 03:46:13.056239 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:46:13.056124    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-2-w2-c8cb9a-kserve-5c5cfb454d-fh2cj"
Nov 01 03:46:13.056239 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:46:13.056163    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-2-w2-c8cb9a-kserve-5c5cfb454d-fh2cj" event={"ID":"2bcf3b85-3d4a-433f-94ad-0e954bf03e73","Type":"ContainerDied","Data":"50850fc9101f441bdefdec50b53aff9d82d329a0899d5d467e5f2e4eb4fe2e9f"}
Nov 01 03:46:13.056239 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:46:13.056207    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-2-w2-c8cb9a-kserve-5c5cfb454d-fh2cj" event={"ID":"2bcf3b85-3d4a-433f-94ad-0e954bf03e73","Type":"ContainerDied","Data":"2357d6f2a538784a4ef6fff4180d217eb9804ddd0a76b0117592aa2904de2ea8"}
Nov 01 03:46:13.056239 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:46:13.056224    4400 scope.go:122] "RemoveContainer" containerID="50850fc9101f441bdefdec50b53aff9d82d329a0899d5d467e5f2e4eb4fe2e9f"
Nov 01 03:46:13.067167 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:46:13.067147    4400 scope.go:122] "RemoveContainer" containerID="50850fc9101f441bdefdec50b53aff9d82d329a0899d5d467e5f2e4eb4fe2e9f"
Nov 01 03:46:13.067429 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:46:13.067404    4400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50850fc9101f441bdefdec50b53aff9d82d329a0899d5d467e5f2e4eb4fe2e9f\": container with ID starting with 50850fc9101f441bdefdec50b53aff9d82d329a0899d5d467e5f2e4eb4fe2e9f not found: ID does not exist" containerID="50850fc9101f441bdefdec50b53aff9d82d329a0899d5d467e5f2e4eb4fe2e9f"
Nov 01 03:46:13.067513 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:46:13.067430    4400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50850fc9101f441bdefdec50b53aff9d82d329a0899d5d467e5f2e4eb4fe2e9f"} err="failed to get container status \"50850fc9101f441bdefdec50b53aff9d82d329a0899d5d467e5f2e4eb4fe2e9f\": rpc error: code = NotFound desc = could not find container \"50850fc9101f441bdefdec50b53aff9d82d329a0899d5d467e5f2e4eb4fe2e9f\": container with ID starting with 50850fc9101f441bdefdec50b53aff9d82d329a0899d5d467e5f2e4eb4fe2e9f not found: ID does not exist"
Nov 01 03:46:13.225268 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:46:13.225235    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-c8cb9a/distinct-2-w2-c8cb9a-kserve-5c5cfb454d-fh2cj"]
Nov 01 03:46:13.276273 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:46:13.276234    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-c8cb9a/distinct-2-w2-c8cb9a-kserve-5c5cfb454d-fh2cj"]
Nov 01 03:46:13.337491 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:46:13.337455    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-2-w2-c8cb9a-kserve-5c5cfb454d-fh2cj" podUID="2bcf3b85-3d4a-433f-94ad-0e954bf03e73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:46:13.616668 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:46:13.616592    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bcf3b85-3d4a-433f-94ad-0e954bf03e73" path="/var/lib/kubelet/pods/2bcf3b85-3d4a-433f-94ad-0e954bf03e73/volumes"
Nov 01 03:46:31.730506 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:46:31.730470    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:46:33.259848 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:46:33.259814    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 03:46:39.300149 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:46:39.300114    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a279c2-75cd57db8-zwzs7"]
Nov 01 03:46:39.300912 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:46:39.300879    4400 cpu_manager.go:414] "containerMap: removing container" podUID="2bcf3b85-3d4a-433f-94ad-0e954bf03e73" containerName="main"
Nov 01 03:46:39.300912 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:46:39.300897    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bcf3b85-3d4a-433f-94ad-0e954bf03e73" containerName="main"
Nov 01 03:46:39.301082 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:46:39.301015    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bcf3b85-3d4a-433f-94ad-0e954bf03e73" containerName="main"
Nov 01 03:46:39.306042 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:46:39.306020    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a279c2-75cd57db8-zwzs7"
Nov 01 03:46:39.360007 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:46:39.359971    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-a279c2-serving-cert\""
Nov 01 03:46:39.406565 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:46:39.406525    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6592a8ed-219c-460c-b354-e0e8331cfa64-maas-api-tls\") pod \"maas-api-e2e-api-b-a279c2-75cd57db8-zwzs7\" (UID: \"6592a8ed-219c-460c-b354-e0e8331cfa64\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a279c2-75cd57db8-zwzs7"
Nov 01 03:46:39.406758 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:46:39.406662    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l6g2\" (UniqueName: \"kubernetes.io/projected/6592a8ed-219c-460c-b354-e0e8331cfa64-kube-api-access-7l6g2\") pod \"maas-api-e2e-api-b-a279c2-75cd57db8-zwzs7\" (UID: \"6592a8ed-219c-460c-b354-e0e8331cfa64\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a279c2-75cd57db8-zwzs7"
Nov 01 03:46:39.406931 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:46:39.406902    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6592a8ed-219c-460c-b354-e0e8331cfa64-metrics-certs\") pod \"maas-api-e2e-api-b-a279c2-75cd57db8-zwzs7\" (UID: \"6592a8ed-219c-460c-b354-e0e8331cfa64\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a279c2-75cd57db8-zwzs7"
Nov 01 03:46:39.407063 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:46:39.407045    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6592a8ed-219c-460c-b354-e0e8331cfa64-openshift-service-ca\") pod \"maas-api-e2e-api-b-a279c2-75cd57db8-zwzs7\" (UID: \"6592a8ed-219c-460c-b354-e0e8331cfa64\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a279c2-75cd57db8-zwzs7"
Nov 01 03:46:39.416950 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:46:39.416905    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a279c2-75cd57db8-zwzs7"]
Nov 01 03:46:39.508477 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:46:39.508439    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6592a8ed-219c-460c-b354-e0e8331cfa64-maas-api-tls\") pod \"maas-api-e2e-api-b-a279c2-75cd57db8-zwzs7\" (UID: \"6592a8ed-219c-460c-b354-e0e8331cfa64\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a279c2-75cd57db8-zwzs7"
Nov 01 03:46:39.508704 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:46:39.508524    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l6g2\" (UniqueName: \"kubernetes.io/projected/6592a8ed-219c-460c-b354-e0e8331cfa64-kube-api-access-7l6g2\") pod \"maas-api-e2e-api-b-a279c2-75cd57db8-zwzs7\" (UID: \"6592a8ed-219c-460c-b354-e0e8331cfa64\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a279c2-75cd57db8-zwzs7"
Nov 01 03:46:39.508704 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:46:39.508604    4400 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-a279c2-serving-cert: secret "maas-api-e2e-api-b-a279c2-serving-cert" not found
Nov 01 03:46:39.508704 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:46:39.508661    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6592a8ed-219c-460c-b354-e0e8331cfa64-metrics-certs\") pod \"maas-api-e2e-api-b-a279c2-75cd57db8-zwzs7\" (UID: \"6592a8ed-219c-460c-b354-e0e8331cfa64\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a279c2-75cd57db8-zwzs7"
Nov 01 03:46:39.508704 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:46:39.508695    4400 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6592a8ed-219c-460c-b354-e0e8331cfa64-maas-api-tls podName:6592a8ed-219c-460c-b354-e0e8331cfa64 nodeName:}" failed. No retries permitted until 2026-11-01 03:46:40.008658835 +0000 UTC m=+2406.523304689 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6592a8ed-219c-460c-b354-e0e8331cfa64-maas-api-tls") pod "maas-api-e2e-api-b-a279c2-75cd57db8-zwzs7" (UID: "6592a8ed-219c-460c-b354-e0e8331cfa64") : secret "maas-api-e2e-api-b-a279c2-serving-cert" not found
Nov 01 03:46:39.508947 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:46:39.508821    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6592a8ed-219c-460c-b354-e0e8331cfa64-openshift-service-ca\") pod \"maas-api-e2e-api-b-a279c2-75cd57db8-zwzs7\" (UID: \"6592a8ed-219c-460c-b354-e0e8331cfa64\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a279c2-75cd57db8-zwzs7"
Nov 01 03:46:39.509546 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:46:39.509527    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6592a8ed-219c-460c-b354-e0e8331cfa64-openshift-service-ca\") pod \"maas-api-e2e-api-b-a279c2-75cd57db8-zwzs7\" (UID: \"6592a8ed-219c-460c-b354-e0e8331cfa64\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a279c2-75cd57db8-zwzs7"
Nov 01 03:46:39.511229 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:46:39.511207    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6592a8ed-219c-460c-b354-e0e8331cfa64-metrics-certs\") pod \"maas-api-e2e-api-b-a279c2-75cd57db8-zwzs7\" (UID: \"6592a8ed-219c-460c-b354-e0e8331cfa64\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a279c2-75cd57db8-zwzs7"
Nov 01 03:46:39.586803 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:46:39.586768    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l6g2\" (UniqueName: \"kubernetes.io/projected/6592a8ed-219c-460c-b354-e0e8331cfa64-kube-api-access-7l6g2\") pod \"maas-api-e2e-api-b-a279c2-75cd57db8-zwzs7\" (UID: \"6592a8ed-219c-460c-b354-e0e8331cfa64\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a279c2-75cd57db8-zwzs7"
Nov 01 03:46:40.019218 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:46:40.019120    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6592a8ed-219c-460c-b354-e0e8331cfa64-maas-api-tls\") pod \"maas-api-e2e-api-b-a279c2-75cd57db8-zwzs7\" (UID: \"6592a8ed-219c-460c-b354-e0e8331cfa64\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a279c2-75cd57db8-zwzs7"
Nov 01 03:46:40.021784 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:46:40.021749    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6592a8ed-219c-460c-b354-e0e8331cfa64-maas-api-tls\") pod \"maas-api-e2e-api-b-a279c2-75cd57db8-zwzs7\" (UID: \"6592a8ed-219c-460c-b354-e0e8331cfa64\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a279c2-75cd57db8-zwzs7"
Nov 01 03:46:40.228991 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:46:40.228962    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a279c2-75cd57db8-zwzs7"
Nov 01 03:46:40.615515 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:46:40.615478    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6592a8ed_219c_460c_b354_e0e8331cfa64.slice/crio-ed78893dc4506ceb1e5f1460e0486a794021080191a04e9699b7a0c113910439 WatchSource:0}: Error finding container ed78893dc4506ceb1e5f1460e0486a794021080191a04e9699b7a0c113910439: Status 404 returned error can't find the container with id ed78893dc4506ceb1e5f1460e0486a794021080191a04e9699b7a0c113910439
Nov 01 03:46:40.615938 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:46:40.615714    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a279c2-75cd57db8-zwzs7"]
Nov 01 03:46:41.241256 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:46:41.241221    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a279c2-75cd57db8-zwzs7" event={"ID":"6592a8ed-219c-460c-b354-e0e8331cfa64","Type":"ContainerStarted","Data":"ed78893dc4506ceb1e5f1460e0486a794021080191a04e9699b7a0c113910439"}
Nov 01 03:46:42.251113 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:46:42.251079    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a279c2-75cd57db8-zwzs7" event={"ID":"6592a8ed-219c-460c-b354-e0e8331cfa64","Type":"ContainerStarted","Data":"1d67453df8ff48625555a62ac4167603500cd8842ddb62e8936cee456a4cb2b8"}
Nov 01 03:46:42.251538 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:46:42.251164    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a279c2-75cd57db8-zwzs7"
Nov 01 03:46:42.374818 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:46:42.374760    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a279c2-75cd57db8-zwzs7" podStartSLOduration=2.545898096 podStartE2EDuration="3.374740282s" totalImagesPullingTime="828.842186ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:46:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:46:42.371063696 +0000 UTC m=+2408.885709560" watchObservedRunningTime="2026-11-01 03:46:42.374740282 +0000 UTC m=+2408.889386150"
Nov 01 03:46:48.263187 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:46:48.263154    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a279c2-75cd57db8-zwzs7"
Nov 01 03:47:01.729016 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:47:01.728981    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:47:04.481158 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:04.481121    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-170ffa-56469b56d9-zwqkm"]
Nov 01 03:47:04.485492 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:04.485475    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-170ffa-56469b56d9-zwqkm"
Nov 01 03:47:04.540164 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:04.540131    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-170ffa-serving-cert\""
Nov 01 03:47:04.594579 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:04.594553    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-170ffa-56469b56d9-zwqkm"]
Nov 01 03:47:04.636673 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:04.636646    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de71d35c-8a34-4a1c-96d3-7ae4aa24fee2-maas-api-tls\") pod \"maas-api-e2e-api-b-170ffa-56469b56d9-zwqkm\" (UID: \"de71d35c-8a34-4a1c-96d3-7ae4aa24fee2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-170ffa-56469b56d9-zwqkm"
Nov 01 03:47:04.636813 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:04.636799    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwwzz\" (UniqueName: \"kubernetes.io/projected/de71d35c-8a34-4a1c-96d3-7ae4aa24fee2-kube-api-access-bwwzz\") pod \"maas-api-e2e-api-b-170ffa-56469b56d9-zwqkm\" (UID: \"de71d35c-8a34-4a1c-96d3-7ae4aa24fee2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-170ffa-56469b56d9-zwqkm"
Nov 01 03:47:04.636858 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:04.636824    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de71d35c-8a34-4a1c-96d3-7ae4aa24fee2-openshift-service-ca\") pod \"maas-api-e2e-api-b-170ffa-56469b56d9-zwqkm\" (UID: \"de71d35c-8a34-4a1c-96d3-7ae4aa24fee2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-170ffa-56469b56d9-zwqkm"
Nov 01 03:47:04.636906 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:04.636866    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de71d35c-8a34-4a1c-96d3-7ae4aa24fee2-metrics-certs\") pod \"maas-api-e2e-api-b-170ffa-56469b56d9-zwqkm\" (UID: \"de71d35c-8a34-4a1c-96d3-7ae4aa24fee2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-170ffa-56469b56d9-zwqkm"
Nov 01 03:47:04.738753 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:04.738638    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de71d35c-8a34-4a1c-96d3-7ae4aa24fee2-metrics-certs\") pod \"maas-api-e2e-api-b-170ffa-56469b56d9-zwqkm\" (UID: \"de71d35c-8a34-4a1c-96d3-7ae4aa24fee2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-170ffa-56469b56d9-zwqkm"
Nov 01 03:47:04.742704 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:04.739013    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de71d35c-8a34-4a1c-96d3-7ae4aa24fee2-maas-api-tls\") pod \"maas-api-e2e-api-b-170ffa-56469b56d9-zwqkm\" (UID: \"de71d35c-8a34-4a1c-96d3-7ae4aa24fee2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-170ffa-56469b56d9-zwqkm"
Nov 01 03:47:04.742704 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:04.739192    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwwzz\" (UniqueName: \"kubernetes.io/projected/de71d35c-8a34-4a1c-96d3-7ae4aa24fee2-kube-api-access-bwwzz\") pod \"maas-api-e2e-api-b-170ffa-56469b56d9-zwqkm\" (UID: \"de71d35c-8a34-4a1c-96d3-7ae4aa24fee2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-170ffa-56469b56d9-zwqkm"
Nov 01 03:47:04.742704 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:04.739391    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de71d35c-8a34-4a1c-96d3-7ae4aa24fee2-openshift-service-ca\") pod \"maas-api-e2e-api-b-170ffa-56469b56d9-zwqkm\" (UID: \"de71d35c-8a34-4a1c-96d3-7ae4aa24fee2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-170ffa-56469b56d9-zwqkm"
Nov 01 03:47:04.750153 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:04.750126    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de71d35c-8a34-4a1c-96d3-7ae4aa24fee2-openshift-service-ca\") pod \"maas-api-e2e-api-b-170ffa-56469b56d9-zwqkm\" (UID: \"de71d35c-8a34-4a1c-96d3-7ae4aa24fee2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-170ffa-56469b56d9-zwqkm"
Nov 01 03:47:04.750344 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:47:04.750232    4400 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-170ffa-serving-cert: secret "maas-api-e2e-api-b-170ffa-serving-cert" not found
Nov 01 03:47:04.750492 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:47:04.750474    4400 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de71d35c-8a34-4a1c-96d3-7ae4aa24fee2-maas-api-tls podName:de71d35c-8a34-4a1c-96d3-7ae4aa24fee2 nodeName:}" failed. No retries permitted until 2026-11-01 03:47:05.250452766 +0000 UTC m=+2431.765098611 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/de71d35c-8a34-4a1c-96d3-7ae4aa24fee2-maas-api-tls") pod "maas-api-e2e-api-b-170ffa-56469b56d9-zwqkm" (UID: "de71d35c-8a34-4a1c-96d3-7ae4aa24fee2") : secret "maas-api-e2e-api-b-170ffa-serving-cert" not found
Nov 01 03:47:04.757821 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:04.757794    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de71d35c-8a34-4a1c-96d3-7ae4aa24fee2-metrics-certs\") pod \"maas-api-e2e-api-b-170ffa-56469b56d9-zwqkm\" (UID: \"de71d35c-8a34-4a1c-96d3-7ae4aa24fee2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-170ffa-56469b56d9-zwqkm"
Nov 01 03:47:04.813083 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:04.813054    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwwzz\" (UniqueName: \"kubernetes.io/projected/de71d35c-8a34-4a1c-96d3-7ae4aa24fee2-kube-api-access-bwwzz\") pod \"maas-api-e2e-api-b-170ffa-56469b56d9-zwqkm\" (UID: \"de71d35c-8a34-4a1c-96d3-7ae4aa24fee2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-170ffa-56469b56d9-zwqkm"
Nov 01 03:47:05.346268 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:05.346237    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de71d35c-8a34-4a1c-96d3-7ae4aa24fee2-maas-api-tls\") pod \"maas-api-e2e-api-b-170ffa-56469b56d9-zwqkm\" (UID: \"de71d35c-8a34-4a1c-96d3-7ae4aa24fee2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-170ffa-56469b56d9-zwqkm"
Nov 01 03:47:05.348961 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:05.348936    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de71d35c-8a34-4a1c-96d3-7ae4aa24fee2-maas-api-tls\") pod \"maas-api-e2e-api-b-170ffa-56469b56d9-zwqkm\" (UID: \"de71d35c-8a34-4a1c-96d3-7ae4aa24fee2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-170ffa-56469b56d9-zwqkm"
Nov 01 03:47:05.400066 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:05.400041    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-170ffa-56469b56d9-zwqkm"
Nov 01 03:47:05.466556 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:05.466459    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jq6dn"]
Nov 01 03:47:05.466814 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:05.466664    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jq6dn" podUID="842a8e86-2464-4f2e-88ac-6be1b68b0d7e" containerName="istio-proxy" containerID="cri-o://fac64634c4365b03c1ab82cb24c5cbd49bf564073dac524d897649a3e513dc1b" gracePeriod=30
Nov 01 03:47:05.784372 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:05.784344    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-170ffa-56469b56d9-zwqkm"]
Nov 01 03:47:05.785227 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:47:05.785207    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde71d35c_8a34_4a1c_96d3_7ae4aa24fee2.slice/crio-d8b319dfc0fa8c528cc95ad0ec1bfe7c6101fdf7faba73695f4e4693e9548ca0 WatchSource:0}: Error finding container d8b319dfc0fa8c528cc95ad0ec1bfe7c6101fdf7faba73695f4e4693e9548ca0: Status 404 returned error can't find the container with id d8b319dfc0fa8c528cc95ad0ec1bfe7c6101fdf7faba73695f4e4693e9548ca0
Nov 01 03:47:06.414625 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:06.414596    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-170ffa-56469b56d9-zwqkm" event={"ID":"de71d35c-8a34-4a1c-96d3-7ae4aa24fee2","Type":"ContainerStarted","Data":"d8b319dfc0fa8c528cc95ad0ec1bfe7c6101fdf7faba73695f4e4693e9548ca0"}
Nov 01 03:47:07.423259 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:07.423221    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-170ffa-56469b56d9-zwqkm" event={"ID":"de71d35c-8a34-4a1c-96d3-7ae4aa24fee2","Type":"ContainerStarted","Data":"b1d53409d7e78dd5c4ed0e262ae13ecf4765d73e5c2927db321c25c9a48afdb0"}
Nov 01 03:47:07.423763 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:07.423309    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-170ffa-56469b56d9-zwqkm"
Nov 01 03:47:07.560835 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:07.560768    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-170ffa-56469b56d9-zwqkm" podStartSLOduration=2.686323434 podStartE2EDuration="3.560750376s" totalImagesPullingTime="874.426942ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:47:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:47:07.558431151 +0000 UTC m=+2434.073077013" watchObservedRunningTime="2026-11-01 03:47:07.560750376 +0000 UTC m=+2434.075396237"
Nov 01 03:47:08.286063 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:08.286031    4400 patch_prober.go:29] interesting pod/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jq6dn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 03:47:08.286226 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:08.286093    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jq6dn" podUID="842a8e86-2464-4f2e-88ac-6be1b68b0d7e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 03:47:11.163662 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:11.163639    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jq6dn"
Nov 01 03:47:11.334085 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:11.333244    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/842a8e86-2464-4f2e-88ac-6be1b68b0d7e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/842a8e86-2464-4f2e-88ac-6be1b68b0d7e-workload-socket\") pod \"842a8e86-2464-4f2e-88ac-6be1b68b0d7e\" (UID: \"842a8e86-2464-4f2e-88ac-6be1b68b0d7e\") "
Nov 01 03:47:11.334085 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:11.333302    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/842a8e86-2464-4f2e-88ac-6be1b68b0d7e-kube-api-access-vqlcx\" (UniqueName: \"kubernetes.io/projected/842a8e86-2464-4f2e-88ac-6be1b68b0d7e-kube-api-access-vqlcx\") pod \"842a8e86-2464-4f2e-88ac-6be1b68b0d7e\" (UID: \"842a8e86-2464-4f2e-88ac-6be1b68b0d7e\") "
Nov 01 03:47:11.334085 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:11.333355    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/842a8e86-2464-4f2e-88ac-6be1b68b0d7e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/842a8e86-2464-4f2e-88ac-6be1b68b0d7e-istio-envoy\") pod \"842a8e86-2464-4f2e-88ac-6be1b68b0d7e\" (UID: \"842a8e86-2464-4f2e-88ac-6be1b68b0d7e\") "
Nov 01 03:47:11.334085 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:11.333386    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/842a8e86-2464-4f2e-88ac-6be1b68b0d7e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/842a8e86-2464-4f2e-88ac-6be1b68b0d7e-workload-certs\") pod \"842a8e86-2464-4f2e-88ac-6be1b68b0d7e\" (UID: \"842a8e86-2464-4f2e-88ac-6be1b68b0d7e\") "
Nov 01 03:47:11.334085 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:11.333486    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/842a8e86-2464-4f2e-88ac-6be1b68b0d7e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/842a8e86-2464-4f2e-88ac-6be1b68b0d7e-credential-socket\") pod \"842a8e86-2464-4f2e-88ac-6be1b68b0d7e\" (UID: \"842a8e86-2464-4f2e-88ac-6be1b68b0d7e\") "
Nov 01 03:47:11.334085 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:11.333560    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/842a8e86-2464-4f2e-88ac-6be1b68b0d7e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/842a8e86-2464-4f2e-88ac-6be1b68b0d7e-istio-data\") pod \"842a8e86-2464-4f2e-88ac-6be1b68b0d7e\" (UID: \"842a8e86-2464-4f2e-88ac-6be1b68b0d7e\") "
Nov 01 03:47:11.334085 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:11.333623    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/842a8e86-2464-4f2e-88ac-6be1b68b0d7e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/842a8e86-2464-4f2e-88ac-6be1b68b0d7e-istio-podinfo\") pod \"842a8e86-2464-4f2e-88ac-6be1b68b0d7e\" (UID: \"842a8e86-2464-4f2e-88ac-6be1b68b0d7e\") "
Nov 01 03:47:11.335513 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:11.335334    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/842a8e86-2464-4f2e-88ac-6be1b68b0d7e-workload-certs" pod "842a8e86-2464-4f2e-88ac-6be1b68b0d7e" (UID: "842a8e86-2464-4f2e-88ac-6be1b68b0d7e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:47:11.335871 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:11.335705    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/842a8e86-2464-4f2e-88ac-6be1b68b0d7e-credential-socket" pod "842a8e86-2464-4f2e-88ac-6be1b68b0d7e" (UID: "842a8e86-2464-4f2e-88ac-6be1b68b0d7e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:47:11.336037 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:11.336003    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/842a8e86-2464-4f2e-88ac-6be1b68b0d7e-istio-data" pod "842a8e86-2464-4f2e-88ac-6be1b68b0d7e" (UID: "842a8e86-2464-4f2e-88ac-6be1b68b0d7e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:47:11.336654 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:11.336620    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/842a8e86-2464-4f2e-88ac-6be1b68b0d7e-workload-socket" pod "842a8e86-2464-4f2e-88ac-6be1b68b0d7e" (UID: "842a8e86-2464-4f2e-88ac-6be1b68b0d7e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:47:11.336919 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:11.336900    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/842a8e86-2464-4f2e-88ac-6be1b68b0d7e-istio-token\" (UniqueName: \"kubernetes.io/projected/842a8e86-2464-4f2e-88ac-6be1b68b0d7e-istio-token\") pod \"842a8e86-2464-4f2e-88ac-6be1b68b0d7e\" (UID: \"842a8e86-2464-4f2e-88ac-6be1b68b0d7e\") "
Nov 01 03:47:11.337401 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:11.337384    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/842a8e86-2464-4f2e-88ac-6be1b68b0d7e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/842a8e86-2464-4f2e-88ac-6be1b68b0d7e-istiod-ca-cert\") pod \"842a8e86-2464-4f2e-88ac-6be1b68b0d7e\" (UID: \"842a8e86-2464-4f2e-88ac-6be1b68b0d7e\") "
Nov 01 03:47:11.341760 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:11.341739    4400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/842a8e86-2464-4f2e-88ac-6be1b68b0d7e-istio-data\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:47:11.341907 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:11.341893    4400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/842a8e86-2464-4f2e-88ac-6be1b68b0d7e-workload-socket\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:47:11.341993 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:11.341982    4400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/842a8e86-2464-4f2e-88ac-6be1b68b0d7e-workload-certs\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:47:11.342078 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:11.342067    4400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/842a8e86-2464-4f2e-88ac-6be1b68b0d7e-credential-socket\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:47:11.342161 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:11.340915    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/842a8e86-2464-4f2e-88ac-6be1b68b0d7e-istiod-ca-cert" pod "842a8e86-2464-4f2e-88ac-6be1b68b0d7e" (UID: "842a8e86-2464-4f2e-88ac-6be1b68b0d7e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:47:11.352735 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:11.347293    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/842a8e86-2464-4f2e-88ac-6be1b68b0d7e-istio-podinfo" pod "842a8e86-2464-4f2e-88ac-6be1b68b0d7e" (UID: "842a8e86-2464-4f2e-88ac-6be1b68b0d7e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 03:47:11.352735 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:11.347610    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/842a8e86-2464-4f2e-88ac-6be1b68b0d7e-istio-envoy" pod "842a8e86-2464-4f2e-88ac-6be1b68b0d7e" (UID: "842a8e86-2464-4f2e-88ac-6be1b68b0d7e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:47:11.353184 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:11.353149    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/842a8e86-2464-4f2e-88ac-6be1b68b0d7e-kube-api-access-vqlcx" pod "842a8e86-2464-4f2e-88ac-6be1b68b0d7e" (UID: "842a8e86-2464-4f2e-88ac-6be1b68b0d7e"). InnerVolumeSpecName "kube-api-access-vqlcx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:47:11.356727 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:11.356124    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/842a8e86-2464-4f2e-88ac-6be1b68b0d7e-istio-token" pod "842a8e86-2464-4f2e-88ac-6be1b68b0d7e" (UID: "842a8e86-2464-4f2e-88ac-6be1b68b0d7e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:47:11.443069 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:11.443036    4400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/842a8e86-2464-4f2e-88ac-6be1b68b0d7e-istio-token\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:47:11.443069 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:11.443063    4400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/842a8e86-2464-4f2e-88ac-6be1b68b0d7e-istiod-ca-cert\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:47:11.443069 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:11.443073    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqlcx\" (UniqueName: \"kubernetes.io/projected/842a8e86-2464-4f2e-88ac-6be1b68b0d7e-kube-api-access-vqlcx\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:47:11.443305 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:11.443083    4400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/842a8e86-2464-4f2e-88ac-6be1b68b0d7e-istio-envoy\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:47:11.443305 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:11.443092    4400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/842a8e86-2464-4f2e-88ac-6be1b68b0d7e-istio-podinfo\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:47:11.494747 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:11.494713    4400 generic.go:356] "Generic (PLEG): container finished" podID="842a8e86-2464-4f2e-88ac-6be1b68b0d7e" containerID="fac64634c4365b03c1ab82cb24c5cbd49bf564073dac524d897649a3e513dc1b" exitCode=0
Nov 01 03:47:11.494975 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:11.494749    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jq6dn" event={"ID":"842a8e86-2464-4f2e-88ac-6be1b68b0d7e","Type":"ContainerDied","Data":"fac64634c4365b03c1ab82cb24c5cbd49bf564073dac524d897649a3e513dc1b"}
Nov 01 03:47:11.495129 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:11.495019    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jq6dn" event={"ID":"842a8e86-2464-4f2e-88ac-6be1b68b0d7e","Type":"ContainerDied","Data":"bd2ee24343b9a9a9307912b55691710d831e1e84cec6bc648c9ddcf7f7a175a0"}
Nov 01 03:47:11.495129 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:11.494803    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jq6dn"
Nov 01 03:47:11.495129 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:11.495056    4400 scope.go:122] "RemoveContainer" containerID="fac64634c4365b03c1ab82cb24c5cbd49bf564073dac524d897649a3e513dc1b"
Nov 01 03:47:11.509017 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:11.508932    4400 scope.go:122] "RemoveContainer" containerID="fac64634c4365b03c1ab82cb24c5cbd49bf564073dac524d897649a3e513dc1b"
Nov 01 03:47:11.509340 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:47:11.509313    4400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fac64634c4365b03c1ab82cb24c5cbd49bf564073dac524d897649a3e513dc1b\": container with ID starting with fac64634c4365b03c1ab82cb24c5cbd49bf564073dac524d897649a3e513dc1b not found: ID does not exist" containerID="fac64634c4365b03c1ab82cb24c5cbd49bf564073dac524d897649a3e513dc1b"
Nov 01 03:47:11.509424 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:11.509351    4400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fac64634c4365b03c1ab82cb24c5cbd49bf564073dac524d897649a3e513dc1b"} err="failed to get container status \"fac64634c4365b03c1ab82cb24c5cbd49bf564073dac524d897649a3e513dc1b\": rpc error: code = NotFound desc = could not find container \"fac64634c4365b03c1ab82cb24c5cbd49bf564073dac524d897649a3e513dc1b\": container with ID starting with fac64634c4365b03c1ab82cb24c5cbd49bf564073dac524d897649a3e513dc1b not found: ID does not exist"
Nov 01 03:47:11.746240 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:11.746153    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jq6dn"]
Nov 01 03:47:11.757417 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:11.757387    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jq6dn"]
Nov 01 03:47:13.472897 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:13.472867    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-170ffa-56469b56d9-zwqkm"
Nov 01 03:47:13.651547 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:13.651108    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="842a8e86-2464-4f2e-88ac-6be1b68b0d7e" path="/var/lib/kubelet/pods/842a8e86-2464-4f2e-88ac-6be1b68b0d7e/volumes"
Nov 01 03:47:13.700204 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:13.698115    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-g8ft5"]
Nov 01 03:47:13.700204 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:13.698358    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-g8ft5" podUID="f593b1bc-8cc9-4147-88d9-9f441b0d7159" containerName="istio-proxy" containerID="cri-o://f8322c8af545e7159a93ede36f7338261eb7539c557a2eb9ab7d200d487ca7bb" gracePeriod=30
Nov 01 03:47:15.907749 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:15.907715    4400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-g8ft5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 03:47:15.908141 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:15.907764    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-g8ft5" podUID="f593b1bc-8cc9-4147-88d9-9f441b0d7159" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 03:47:19.210013 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:19.209957    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-g8ft5"
Nov 01 03:47:19.237694 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:19.237655    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f593b1bc-8cc9-4147-88d9-9f441b0d7159-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f593b1bc-8cc9-4147-88d9-9f441b0d7159-istio-data\") pod \"f593b1bc-8cc9-4147-88d9-9f441b0d7159\" (UID: \"f593b1bc-8cc9-4147-88d9-9f441b0d7159\") "
Nov 01 03:47:19.237833 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:19.237713    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f593b1bc-8cc9-4147-88d9-9f441b0d7159-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f593b1bc-8cc9-4147-88d9-9f441b0d7159-istiod-ca-cert\") pod \"f593b1bc-8cc9-4147-88d9-9f441b0d7159\" (UID: \"f593b1bc-8cc9-4147-88d9-9f441b0d7159\") "
Nov 01 03:47:19.237833 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:19.237735    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f593b1bc-8cc9-4147-88d9-9f441b0d7159-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f593b1bc-8cc9-4147-88d9-9f441b0d7159-workload-socket\") pod \"f593b1bc-8cc9-4147-88d9-9f441b0d7159\" (UID: \"f593b1bc-8cc9-4147-88d9-9f441b0d7159\") "
Nov 01 03:47:19.237833 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:19.237759    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f593b1bc-8cc9-4147-88d9-9f441b0d7159-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f593b1bc-8cc9-4147-88d9-9f441b0d7159-istio-envoy\") pod \"f593b1bc-8cc9-4147-88d9-9f441b0d7159\" (UID: \"f593b1bc-8cc9-4147-88d9-9f441b0d7159\") "
Nov 01 03:47:19.237833 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:19.237786    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f593b1bc-8cc9-4147-88d9-9f441b0d7159-istio-token\" (UniqueName: \"kubernetes.io/projected/f593b1bc-8cc9-4147-88d9-9f441b0d7159-istio-token\") pod \"f593b1bc-8cc9-4147-88d9-9f441b0d7159\" (UID: \"f593b1bc-8cc9-4147-88d9-9f441b0d7159\") "
Nov 01 03:47:19.238037 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:19.237900    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f593b1bc-8cc9-4147-88d9-9f441b0d7159-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f593b1bc-8cc9-4147-88d9-9f441b0d7159-workload-certs\") pod \"f593b1bc-8cc9-4147-88d9-9f441b0d7159\" (UID: \"f593b1bc-8cc9-4147-88d9-9f441b0d7159\") "
Nov 01 03:47:19.238037 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:19.237917    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f593b1bc-8cc9-4147-88d9-9f441b0d7159-istio-data" pod "f593b1bc-8cc9-4147-88d9-9f441b0d7159" (UID: "f593b1bc-8cc9-4147-88d9-9f441b0d7159"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:47:19.238037 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:19.237930    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f593b1bc-8cc9-4147-88d9-9f441b0d7159-kube-api-access-9xf8v\" (UniqueName: \"kubernetes.io/projected/f593b1bc-8cc9-4147-88d9-9f441b0d7159-kube-api-access-9xf8v\") pod \"f593b1bc-8cc9-4147-88d9-9f441b0d7159\" (UID: \"f593b1bc-8cc9-4147-88d9-9f441b0d7159\") "
Nov 01 03:47:19.238037 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:19.238000    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f593b1bc-8cc9-4147-88d9-9f441b0d7159-workload-socket" pod "f593b1bc-8cc9-4147-88d9-9f441b0d7159" (UID: "f593b1bc-8cc9-4147-88d9-9f441b0d7159"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:47:19.238231 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:19.238082    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f593b1bc-8cc9-4147-88d9-9f441b0d7159-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f593b1bc-8cc9-4147-88d9-9f441b0d7159-istio-podinfo\") pod \"f593b1bc-8cc9-4147-88d9-9f441b0d7159\" (UID: \"f593b1bc-8cc9-4147-88d9-9f441b0d7159\") "
Nov 01 03:47:19.238231 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:19.238142    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f593b1bc-8cc9-4147-88d9-9f441b0d7159-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f593b1bc-8cc9-4147-88d9-9f441b0d7159-credential-socket\") pod \"f593b1bc-8cc9-4147-88d9-9f441b0d7159\" (UID: \"f593b1bc-8cc9-4147-88d9-9f441b0d7159\") "
Nov 01 03:47:19.238231 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:19.238181    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f593b1bc-8cc9-4147-88d9-9f441b0d7159-workload-certs" pod "f593b1bc-8cc9-4147-88d9-9f441b0d7159" (UID: "f593b1bc-8cc9-4147-88d9-9f441b0d7159"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:47:19.238361 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:19.238239    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f593b1bc-8cc9-4147-88d9-9f441b0d7159-istiod-ca-cert" pod "f593b1bc-8cc9-4147-88d9-9f441b0d7159" (UID: "f593b1bc-8cc9-4147-88d9-9f441b0d7159"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:47:19.238512 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:19.238487    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f593b1bc-8cc9-4147-88d9-9f441b0d7159-credential-socket" pod "f593b1bc-8cc9-4147-88d9-9f441b0d7159" (UID: "f593b1bc-8cc9-4147-88d9-9f441b0d7159"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:47:19.238815 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:19.238729    4400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f593b1bc-8cc9-4147-88d9-9f441b0d7159-credential-socket\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:47:19.238815 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:19.238750    4400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f593b1bc-8cc9-4147-88d9-9f441b0d7159-istio-data\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:47:19.238815 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:19.238763    4400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f593b1bc-8cc9-4147-88d9-9f441b0d7159-istiod-ca-cert\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:47:19.238815 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:19.238777    4400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f593b1bc-8cc9-4147-88d9-9f441b0d7159-workload-socket\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:47:19.238815 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:19.238792    4400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f593b1bc-8cc9-4147-88d9-9f441b0d7159-workload-certs\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:47:19.240179 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:19.240149    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f593b1bc-8cc9-4147-88d9-9f441b0d7159-kube-api-access-9xf8v" pod "f593b1bc-8cc9-4147-88d9-9f441b0d7159" (UID: "f593b1bc-8cc9-4147-88d9-9f441b0d7159"). InnerVolumeSpecName "kube-api-access-9xf8v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:47:19.240666 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:19.240647    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f593b1bc-8cc9-4147-88d9-9f441b0d7159-istio-envoy" pod "f593b1bc-8cc9-4147-88d9-9f441b0d7159" (UID: "f593b1bc-8cc9-4147-88d9-9f441b0d7159"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:47:19.240774 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:19.240701    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f593b1bc-8cc9-4147-88d9-9f441b0d7159-istio-podinfo" pod "f593b1bc-8cc9-4147-88d9-9f441b0d7159" (UID: "f593b1bc-8cc9-4147-88d9-9f441b0d7159"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 03:47:19.240774 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:19.240739    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f593b1bc-8cc9-4147-88d9-9f441b0d7159-istio-token" pod "f593b1bc-8cc9-4147-88d9-9f441b0d7159" (UID: "f593b1bc-8cc9-4147-88d9-9f441b0d7159"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:47:19.340038 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:19.340013    4400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f593b1bc-8cc9-4147-88d9-9f441b0d7159-istio-envoy\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:47:19.340038 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:19.340036    4400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f593b1bc-8cc9-4147-88d9-9f441b0d7159-istio-token\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:47:19.340162 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:19.340047    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xf8v\" (UniqueName: \"kubernetes.io/projected/f593b1bc-8cc9-4147-88d9-9f441b0d7159-kube-api-access-9xf8v\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:47:19.340162 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:19.340058    4400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f593b1bc-8cc9-4147-88d9-9f441b0d7159-istio-podinfo\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:47:19.637440 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:19.637409    4400 generic.go:356] "Generic (PLEG): container finished" podID="f593b1bc-8cc9-4147-88d9-9f441b0d7159" containerID="f8322c8af545e7159a93ede36f7338261eb7539c557a2eb9ab7d200d487ca7bb" exitCode=0
Nov 01 03:47:19.637582 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:19.637467    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-g8ft5"
Nov 01 03:47:19.637582 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:19.637502    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-g8ft5" event={"ID":"f593b1bc-8cc9-4147-88d9-9f441b0d7159","Type":"ContainerDied","Data":"f8322c8af545e7159a93ede36f7338261eb7539c557a2eb9ab7d200d487ca7bb"}
Nov 01 03:47:19.637582 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:19.637570    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-g8ft5" event={"ID":"f593b1bc-8cc9-4147-88d9-9f441b0d7159","Type":"ContainerDied","Data":"dda22dad46011c94f25a868920fdd2eae4542f3eb542e6dff259daea2d10f7ff"}
Nov 01 03:47:19.637745 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:19.637603    4400 scope.go:122] "RemoveContainer" containerID="f8322c8af545e7159a93ede36f7338261eb7539c557a2eb9ab7d200d487ca7bb"
Nov 01 03:47:19.647486 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:19.647467    4400 scope.go:122] "RemoveContainer" containerID="f8322c8af545e7159a93ede36f7338261eb7539c557a2eb9ab7d200d487ca7bb"
Nov 01 03:47:19.647796 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:47:19.647777    4400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8322c8af545e7159a93ede36f7338261eb7539c557a2eb9ab7d200d487ca7bb\": container with ID starting with f8322c8af545e7159a93ede36f7338261eb7539c557a2eb9ab7d200d487ca7bb not found: ID does not exist" containerID="f8322c8af545e7159a93ede36f7338261eb7539c557a2eb9ab7d200d487ca7bb"
Nov 01 03:47:19.647876 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:19.647806    4400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8322c8af545e7159a93ede36f7338261eb7539c557a2eb9ab7d200d487ca7bb"} err="failed to get container status \"f8322c8af545e7159a93ede36f7338261eb7539c557a2eb9ab7d200d487ca7bb\": rpc error: code = NotFound desc = could not find container \"f8322c8af545e7159a93ede36f7338261eb7539c557a2eb9ab7d200d487ca7bb\": container with ID starting with f8322c8af545e7159a93ede36f7338261eb7539c557a2eb9ab7d200d487ca7bb not found: ID does not exist"
Nov 01 03:47:19.808007 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:19.807972    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-g8ft5"]
Nov 01 03:47:19.859115 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:19.859084    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-g8ft5"]
Nov 01 03:47:21.623526 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:21.623485    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f593b1bc-8cc9-4147-88d9-9f441b0d7159" path="/var/lib/kubelet/pods/f593b1bc-8cc9-4147-88d9-9f441b0d7159/volumes"
Nov 01 03:47:31.730259 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:47:31.730218    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:47:43.908463 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:43.908369    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-rtwgm"]
Nov 01 03:47:43.909284 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:43.909262    4400 cpu_manager.go:414] "containerMap: removing container" podUID="f593b1bc-8cc9-4147-88d9-9f441b0d7159" containerName="istio-proxy"
Nov 01 03:47:43.909388 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:43.909287    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f593b1bc-8cc9-4147-88d9-9f441b0d7159" containerName="istio-proxy"
Nov 01 03:47:43.909388 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:43.909356    4400 cpu_manager.go:414] "containerMap: removing container" podUID="842a8e86-2464-4f2e-88ac-6be1b68b0d7e" containerName="istio-proxy"
Nov 01 03:47:43.909388 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:43.909365    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="842a8e86-2464-4f2e-88ac-6be1b68b0d7e" containerName="istio-proxy"
Nov 01 03:47:43.909537 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:43.909527    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="842a8e86-2464-4f2e-88ac-6be1b68b0d7e" containerName="istio-proxy"
Nov 01 03:47:43.909588 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:43.909545    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="f593b1bc-8cc9-4147-88d9-9f441b0d7159" containerName="istio-proxy"
Nov 01 03:47:43.914313 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:43.914292    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-rtwgm"
Nov 01 03:47:43.922536 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:43.922511    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9547cd7-2c26-4515-91fe-8aebb8f8b263-istio-token\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-rtwgm\" (UID: \"b9547cd7-2c26-4515-91fe-8aebb8f8b263\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-rtwgm"
Nov 01 03:47:43.922640 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:43.922548    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9547cd7-2c26-4515-91fe-8aebb8f8b263-workload-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-rtwgm\" (UID: \"b9547cd7-2c26-4515-91fe-8aebb8f8b263\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-rtwgm"
Nov 01 03:47:43.922751 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:43.922729    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9547cd7-2c26-4515-91fe-8aebb8f8b263-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-rtwgm\" (UID: \"b9547cd7-2c26-4515-91fe-8aebb8f8b263\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-rtwgm"
Nov 01 03:47:43.922798 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:43.922765    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gswm9\" (UniqueName: \"kubernetes.io/projected/b9547cd7-2c26-4515-91fe-8aebb8f8b263-kube-api-access-gswm9\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-rtwgm\" (UID: \"b9547cd7-2c26-4515-91fe-8aebb8f8b263\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-rtwgm"
Nov 01 03:47:43.922798 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:43.922783    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9547cd7-2c26-4515-91fe-8aebb8f8b263-workload-certs\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-rtwgm\" (UID: \"b9547cd7-2c26-4515-91fe-8aebb8f8b263\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-rtwgm"
Nov 01 03:47:43.922903 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:43.922889    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9547cd7-2c26-4515-91fe-8aebb8f8b263-istio-envoy\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-rtwgm\" (UID: \"b9547cd7-2c26-4515-91fe-8aebb8f8b263\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-rtwgm"
Nov 01 03:47:43.922953 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:43.922939    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9547cd7-2c26-4515-91fe-8aebb8f8b263-istio-data\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-rtwgm\" (UID: \"b9547cd7-2c26-4515-91fe-8aebb8f8b263\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-rtwgm"
Nov 01 03:47:43.923032 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:43.923018    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9547cd7-2c26-4515-91fe-8aebb8f8b263-credential-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-rtwgm\" (UID: \"b9547cd7-2c26-4515-91fe-8aebb8f8b263\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-rtwgm"
Nov 01 03:47:43.923067 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:43.923055    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9547cd7-2c26-4515-91fe-8aebb8f8b263-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-rtwgm\" (UID: \"b9547cd7-2c26-4515-91fe-8aebb8f8b263\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-rtwgm"
Nov 01 03:47:44.024089 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:44.024049    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9547cd7-2c26-4515-91fe-8aebb8f8b263-istio-envoy\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-rtwgm\" (UID: \"b9547cd7-2c26-4515-91fe-8aebb8f8b263\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-rtwgm"
Nov 01 03:47:44.024369 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:44.024345    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9547cd7-2c26-4515-91fe-8aebb8f8b263-istio-data\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-rtwgm\" (UID: \"b9547cd7-2c26-4515-91fe-8aebb8f8b263\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-rtwgm"
Nov 01 03:47:44.024551 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:44.024532    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9547cd7-2c26-4515-91fe-8aebb8f8b263-credential-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-rtwgm\" (UID: \"b9547cd7-2c26-4515-91fe-8aebb8f8b263\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-rtwgm"
Nov 01 03:47:44.024729 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:44.024674    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9547cd7-2c26-4515-91fe-8aebb8f8b263-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-rtwgm\" (UID: \"b9547cd7-2c26-4515-91fe-8aebb8f8b263\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-rtwgm"
Nov 01 03:47:44.024854 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:44.024759    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9547cd7-2c26-4515-91fe-8aebb8f8b263-istio-token\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-rtwgm\" (UID: \"b9547cd7-2c26-4515-91fe-8aebb8f8b263\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-rtwgm"
Nov 01 03:47:44.024854 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:44.024782    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9547cd7-2c26-4515-91fe-8aebb8f8b263-workload-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-rtwgm\" (UID: \"b9547cd7-2c26-4515-91fe-8aebb8f8b263\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-rtwgm"
Nov 01 03:47:44.026895 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:44.024896    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9547cd7-2c26-4515-91fe-8aebb8f8b263-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-rtwgm\" (UID: \"b9547cd7-2c26-4515-91fe-8aebb8f8b263\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-rtwgm"
Nov 01 03:47:44.026895 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:44.024928    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gswm9\" (UniqueName: \"kubernetes.io/projected/b9547cd7-2c26-4515-91fe-8aebb8f8b263-kube-api-access-gswm9\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-rtwgm\" (UID: \"b9547cd7-2c26-4515-91fe-8aebb8f8b263\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-rtwgm"
Nov 01 03:47:44.026895 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:44.024955    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9547cd7-2c26-4515-91fe-8aebb8f8b263-workload-certs\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-rtwgm\" (UID: \"b9547cd7-2c26-4515-91fe-8aebb8f8b263\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-rtwgm"
Nov 01 03:47:44.026895 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:44.025350    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9547cd7-2c26-4515-91fe-8aebb8f8b263-workload-certs\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-rtwgm\" (UID: \"b9547cd7-2c26-4515-91fe-8aebb8f8b263\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-rtwgm"
Nov 01 03:47:44.026895 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:44.024205    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-rtwgm"]
Nov 01 03:47:44.026895 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:44.025734    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9547cd7-2c26-4515-91fe-8aebb8f8b263-istio-data\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-rtwgm\" (UID: \"b9547cd7-2c26-4515-91fe-8aebb8f8b263\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-rtwgm"
Nov 01 03:47:44.026895 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:44.025989    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9547cd7-2c26-4515-91fe-8aebb8f8b263-credential-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-rtwgm\" (UID: \"b9547cd7-2c26-4515-91fe-8aebb8f8b263\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-rtwgm"
Nov 01 03:47:44.026895 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:44.026602    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9547cd7-2c26-4515-91fe-8aebb8f8b263-workload-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-rtwgm\" (UID: \"b9547cd7-2c26-4515-91fe-8aebb8f8b263\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-rtwgm"
Nov 01 03:47:44.027563 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:44.027538    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9547cd7-2c26-4515-91fe-8aebb8f8b263-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-rtwgm\" (UID: \"b9547cd7-2c26-4515-91fe-8aebb8f8b263\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-rtwgm"
Nov 01 03:47:44.028854 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:44.028830    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9547cd7-2c26-4515-91fe-8aebb8f8b263-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-rtwgm\" (UID: \"b9547cd7-2c26-4515-91fe-8aebb8f8b263\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-rtwgm"
Nov 01 03:47:44.029083 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:44.029063    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9547cd7-2c26-4515-91fe-8aebb8f8b263-istio-envoy\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-rtwgm\" (UID: \"b9547cd7-2c26-4515-91fe-8aebb8f8b263\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-rtwgm"
Nov 01 03:47:44.089297 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:44.089266    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9547cd7-2c26-4515-91fe-8aebb8f8b263-istio-token\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-rtwgm\" (UID: \"b9547cd7-2c26-4515-91fe-8aebb8f8b263\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-rtwgm"
Nov 01 03:47:44.089448 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:44.089306    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gswm9\" (UniqueName: \"kubernetes.io/projected/b9547cd7-2c26-4515-91fe-8aebb8f8b263-kube-api-access-gswm9\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-rtwgm\" (UID: \"b9547cd7-2c26-4515-91fe-8aebb8f8b263\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-rtwgm"
Nov 01 03:47:44.229087 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:44.229003    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-rtwgm"
Nov 01 03:47:44.661316 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:44.661285    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-rtwgm"]
Nov 01 03:47:44.661440 ip-10-0-133-54 kubenswrapper[4400]: W1101 03:47:44.661321    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9547cd7_2c26_4515_91fe_8aebb8f8b263.slice/crio-d8ac465b1abf6600a53cf440464fe84170d99fa57eae2f3faddd69219f574fd1 WatchSource:0}: Error finding container d8ac465b1abf6600a53cf440464fe84170d99fa57eae2f3faddd69219f574fd1: Status 404 returned error can't find the container with id d8ac465b1abf6600a53cf440464fe84170d99fa57eae2f3faddd69219f574fd1
Nov 01 03:47:44.663976 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:44.663942    4400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 03:47:44.664095 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:44.664044    4400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 03:47:44.664095 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:44.664089    4400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 03:47:44.796613 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:44.796567    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-rtwgm" event={"ID":"b9547cd7-2c26-4515-91fe-8aebb8f8b263","Type":"ContainerStarted","Data":"49c26872e8c9c526817ae6fd546816c3d0c45c09b1cccb6a91b693d6335b4c09"}
Nov 01 03:47:44.796791 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:44.796653    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-rtwgm" event={"ID":"b9547cd7-2c26-4515-91fe-8aebb8f8b263","Type":"ContainerStarted","Data":"d8ac465b1abf6600a53cf440464fe84170d99fa57eae2f3faddd69219f574fd1"}
Nov 01 03:47:44.922062 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:44.921963    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-rtwgm" podStartSLOduration=1.9219492919999999 podStartE2EDuration="1.921949292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:47:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:47:44.920151073 +0000 UTC m=+2471.434796937" watchObservedRunningTime="2026-11-01 03:47:44.921949292 +0000 UTC m=+2471.436595154"
Nov 01 03:47:45.229484 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:45.229387    4400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-rtwgm"
Nov 01 03:47:45.230772 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:45.230746    4400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-rtwgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body=
Nov 01 03:47:45.230900 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:45.230797    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-rtwgm" podUID="b9547cd7-2c26-4515-91fe-8aebb8f8b263" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused"
Nov 01 03:47:46.229600 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:46.229561    4400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-rtwgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body=
Nov 01 03:47:46.230040 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:46.229635    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-rtwgm" podUID="b9547cd7-2c26-4515-91fe-8aebb8f8b263" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused"
Nov 01 03:47:47.229644 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:47.229607    4400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-rtwgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body=
Nov 01 03:47:47.230102 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:47.229676    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-rtwgm" podUID="b9547cd7-2c26-4515-91fe-8aebb8f8b263" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused"
Nov 01 03:47:48.230211 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:48.230175    4400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-rtwgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body=
Nov 01 03:47:48.230643 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:48.230239    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-rtwgm" podUID="b9547cd7-2c26-4515-91fe-8aebb8f8b263" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused"
Nov 01 03:47:49.229651 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:49.229606    4400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-rtwgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body=
Nov 01 03:47:49.229894 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:49.229668    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-rtwgm" podUID="b9547cd7-2c26-4515-91fe-8aebb8f8b263" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused"
Nov 01 03:47:50.230109 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:50.230072    4400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-rtwgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body=
Nov 01 03:47:50.230595 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:50.230488    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-rtwgm" podUID="b9547cd7-2c26-4515-91fe-8aebb8f8b263" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused"
Nov 01 03:47:51.230077 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:51.230037    4400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-rtwgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body=
Nov 01 03:47:51.230269 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:51.230110    4400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-rtwgm" podUID="b9547cd7-2c26-4515-91fe-8aebb8f8b263" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused"
Nov 01 03:47:53.005361 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:53.005329    4400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-rtwgm"
Nov 01 03:47:53.006142 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:53.006115    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-rtwgm"
Nov 01 03:47:53.006869 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:47:53.006851    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-rtwgm"
Nov 01 03:48:01.730952 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:48:01.730908    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:48:02.875127 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:48:02.875085    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-7pxrv"]
Nov 01 03:48:02.875622 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:48:02.875357    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-7pxrv" podUID="925a7174-5bbd-4400-869a-c3b62c79199e" containerName="istio-proxy" containerID="cri-o://718caceaa9e3e2ecb3957206d063b432e74ff5f13c6dcc5aaa7bfc51d2bd4e2b" gracePeriod=30
Nov 01 03:48:04.247598 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:48:04.247568    4400 patch_prober.go:29] interesting pod/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-7pxrv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 03:48:04.247990 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:48:04.247620    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-7pxrv" podUID="925a7174-5bbd-4400-869a-c3b62c79199e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 03:48:08.019781 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:48:08.019742    4400 generic.go:356] "Generic (PLEG): container finished" podID="925a7174-5bbd-4400-869a-c3b62c79199e" containerID="718caceaa9e3e2ecb3957206d063b432e74ff5f13c6dcc5aaa7bfc51d2bd4e2b" exitCode=0
Nov 01 03:48:08.020254 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:48:08.019923    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-7pxrv" event={"ID":"925a7174-5bbd-4400-869a-c3b62c79199e","Type":"ContainerDied","Data":"718caceaa9e3e2ecb3957206d063b432e74ff5f13c6dcc5aaa7bfc51d2bd4e2b"}
Nov 01 03:48:08.219208 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:48:08.219174    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-7pxrv"
Nov 01 03:48:08.301348 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:48:08.301319    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/925a7174-5bbd-4400-869a-c3b62c79199e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/925a7174-5bbd-4400-869a-c3b62c79199e-credential-socket\") pod \"925a7174-5bbd-4400-869a-c3b62c79199e\" (UID: \"925a7174-5bbd-4400-869a-c3b62c79199e\") "
Nov 01 03:48:08.301520 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:48:08.301359    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/925a7174-5bbd-4400-869a-c3b62c79199e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/925a7174-5bbd-4400-869a-c3b62c79199e-workload-certs\") pod \"925a7174-5bbd-4400-869a-c3b62c79199e\" (UID: \"925a7174-5bbd-4400-869a-c3b62c79199e\") "
Nov 01 03:48:08.301520 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:48:08.301385    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/925a7174-5bbd-4400-869a-c3b62c79199e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/925a7174-5bbd-4400-869a-c3b62c79199e-workload-socket\") pod \"925a7174-5bbd-4400-869a-c3b62c79199e\" (UID: \"925a7174-5bbd-4400-869a-c3b62c79199e\") "
Nov 01 03:48:08.301520 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:48:08.301432    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/925a7174-5bbd-4400-869a-c3b62c79199e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/925a7174-5bbd-4400-869a-c3b62c79199e-istio-envoy\") pod \"925a7174-5bbd-4400-869a-c3b62c79199e\" (UID: \"925a7174-5bbd-4400-869a-c3b62c79199e\") "
Nov 01 03:48:08.301520 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:48:08.301459    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/925a7174-5bbd-4400-869a-c3b62c79199e-kube-api-access-f8bfx\" (UniqueName: \"kubernetes.io/projected/925a7174-5bbd-4400-869a-c3b62c79199e-kube-api-access-f8bfx\") pod \"925a7174-5bbd-4400-869a-c3b62c79199e\" (UID: \"925a7174-5bbd-4400-869a-c3b62c79199e\") "
Nov 01 03:48:08.301782 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:48:08.301610    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/925a7174-5bbd-4400-869a-c3b62c79199e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/925a7174-5bbd-4400-869a-c3b62c79199e-istiod-ca-cert\") pod \"925a7174-5bbd-4400-869a-c3b62c79199e\" (UID: \"925a7174-5bbd-4400-869a-c3b62c79199e\") "
Nov 01 03:48:08.301782 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:48:08.301639    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/925a7174-5bbd-4400-869a-c3b62c79199e-credential-socket" pod "925a7174-5bbd-4400-869a-c3b62c79199e" (UID: "925a7174-5bbd-4400-869a-c3b62c79199e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:48:08.301782 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:48:08.301671    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/925a7174-5bbd-4400-869a-c3b62c79199e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/925a7174-5bbd-4400-869a-c3b62c79199e-istio-podinfo\") pod \"925a7174-5bbd-4400-869a-c3b62c79199e\" (UID: \"925a7174-5bbd-4400-869a-c3b62c79199e\") "
Nov 01 03:48:08.301782 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:48:08.301700    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/925a7174-5bbd-4400-869a-c3b62c79199e-workload-certs" pod "925a7174-5bbd-4400-869a-c3b62c79199e" (UID: "925a7174-5bbd-4400-869a-c3b62c79199e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:48:08.301782 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:48:08.301744    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/925a7174-5bbd-4400-869a-c3b62c79199e-istio-token\" (UniqueName: \"kubernetes.io/projected/925a7174-5bbd-4400-869a-c3b62c79199e-istio-token\") pod \"925a7174-5bbd-4400-869a-c3b62c79199e\" (UID: \"925a7174-5bbd-4400-869a-c3b62c79199e\") "
Nov 01 03:48:08.301782 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:48:08.301778    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/925a7174-5bbd-4400-869a-c3b62c79199e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/925a7174-5bbd-4400-869a-c3b62c79199e-istio-data\") pod \"925a7174-5bbd-4400-869a-c3b62c79199e\" (UID: \"925a7174-5bbd-4400-869a-c3b62c79199e\") "
Nov 01 03:48:08.302110 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:48:08.302032    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/925a7174-5bbd-4400-869a-c3b62c79199e-istiod-ca-cert" pod "925a7174-5bbd-4400-869a-c3b62c79199e" (UID: "925a7174-5bbd-4400-869a-c3b62c79199e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:48:08.302110 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:48:08.302054    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/925a7174-5bbd-4400-869a-c3b62c79199e-workload-socket" pod "925a7174-5bbd-4400-869a-c3b62c79199e" (UID: "925a7174-5bbd-4400-869a-c3b62c79199e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:48:08.302220 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:48:08.302175    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/925a7174-5bbd-4400-869a-c3b62c79199e-istio-data" pod "925a7174-5bbd-4400-869a-c3b62c79199e" (UID: "925a7174-5bbd-4400-869a-c3b62c79199e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:48:08.302887 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:48:08.302862    4400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/925a7174-5bbd-4400-869a-c3b62c79199e-credential-socket\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:48:08.303083 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:48:08.302892    4400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/925a7174-5bbd-4400-869a-c3b62c79199e-workload-certs\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:48:08.303083 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:48:08.302908    4400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/925a7174-5bbd-4400-869a-c3b62c79199e-workload-socket\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:48:08.303083 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:48:08.302922    4400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/925a7174-5bbd-4400-869a-c3b62c79199e-istiod-ca-cert\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:48:08.303083 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:48:08.302936    4400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/925a7174-5bbd-4400-869a-c3b62c79199e-istio-data\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:48:08.304068 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:48:08.304041    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/925a7174-5bbd-4400-869a-c3b62c79199e-istio-envoy" pod "925a7174-5bbd-4400-869a-c3b62c79199e" (UID: "925a7174-5bbd-4400-869a-c3b62c79199e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:48:08.304351 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:48:08.304327    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/925a7174-5bbd-4400-869a-c3b62c79199e-istio-podinfo" pod "925a7174-5bbd-4400-869a-c3b62c79199e" (UID: "925a7174-5bbd-4400-869a-c3b62c79199e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 03:48:08.304661 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:48:08.304630    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/925a7174-5bbd-4400-869a-c3b62c79199e-istio-token" pod "925a7174-5bbd-4400-869a-c3b62c79199e" (UID: "925a7174-5bbd-4400-869a-c3b62c79199e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:48:08.304803 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:48:08.304728    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/925a7174-5bbd-4400-869a-c3b62c79199e-kube-api-access-f8bfx" pod "925a7174-5bbd-4400-869a-c3b62c79199e" (UID: "925a7174-5bbd-4400-869a-c3b62c79199e"). InnerVolumeSpecName "kube-api-access-f8bfx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:48:08.404345 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:48:08.404306    4400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/925a7174-5bbd-4400-869a-c3b62c79199e-istio-envoy\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:48:08.404506 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:48:08.404350    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8bfx\" (UniqueName: \"kubernetes.io/projected/925a7174-5bbd-4400-869a-c3b62c79199e-kube-api-access-f8bfx\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:48:08.404506 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:48:08.404367    4400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/925a7174-5bbd-4400-869a-c3b62c79199e-istio-podinfo\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:48:08.404506 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:48:08.404385    4400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/925a7174-5bbd-4400-869a-c3b62c79199e-istio-token\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:48:08.487215 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:48:08.487132    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 03:48:09.027441 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:48:09.027407    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-7pxrv"
Nov 01 03:48:09.027924 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:48:09.027442    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-7pxrv" event={"ID":"925a7174-5bbd-4400-869a-c3b62c79199e","Type":"ContainerDied","Data":"5459c3bef4a99f337d1599c19577643697d2d29055350be670590003bc26f2fa"}
Nov 01 03:48:09.027924 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:48:09.027517    4400 scope.go:122] "RemoveContainer" containerID="718caceaa9e3e2ecb3957206d063b432e74ff5f13c6dcc5aaa7bfc51d2bd4e2b"
Nov 01 03:48:09.201155 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:48:09.201125    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-7pxrv"]
Nov 01 03:48:09.209765 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:48:09.209739    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-7pxrv"]
Nov 01 03:48:09.631648 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:48:09.631615    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="925a7174-5bbd-4400-869a-c3b62c79199e" path="/var/lib/kubelet/pods/925a7174-5bbd-4400-869a-c3b62c79199e/volumes"
Nov 01 03:48:31.730021 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:48:31.729987    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:48:36.969918 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:48:36.969885    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-d7f535/distinct-w3-d7f535-kserve-7d69fb9558-88lq8"]
Nov 01 03:48:43.079458 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:48:43.079418    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-d7f535-67cfcb74d9-lzhln"]
Nov 01 03:48:43.195699 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:48:43.195654    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-d7f535-8659cb78cf-wxmbp"]
Nov 01 03:48:45.546364 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:48:45.546334    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 03:48:48.080192 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:48:48.080152    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-d7f535-67cfcb74d9-lzhln" podUID="ed1ec2ab-443d-42d9-9221-20cdf668121a" containerName="payload-processing" containerID="cri-o://0600998a757bb0c1749ee1486fd4e78f1d41db1b5bb4c74e86d3797d5f22786c" gracePeriod=25
Nov 01 03:48:48.196415 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:48:48.196372    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d7f535-8659cb78cf-wxmbp" podUID="a6b65b4c-f4d1-49a4-a4d3-7ef365d8aac9" containerName="payload-pre-processing" containerID="cri-o://d156422b113d9f662ae479f0a27ca60e20b80380fdce48654cd5bc7b2177028e" gracePeriod=25
Nov 01 03:48:48.788673 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:48:48.788637    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d7f535-8659cb78cf-wxmbp" podUID="a6b65b4c-f4d1-49a4-a4d3-7ef365d8aac9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 03:48:49.795760 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:48:49.795723    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-d7f535-67cfcb74d9-lzhln" podUID="ed1ec2ab-443d-42d9-9221-20cdf668121a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 03:48:51.991444 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:48:51.991410    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-d7f535/distinct-w3-d7f535-kserve-7d69fb9558-88lq8" podUID="65358d08-83e6-45c5-a17e-f1dbde164345" containerName="main" containerID="cri-o://5358aa2bcf8023a7cc57d249f7e1b5d1d2964a94922972b7c0dab7e001a04b16" gracePeriod=45
Nov 01 03:48:52.270990 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:48:52.270964    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d7f535/distinct-w3-d7f535-kserve-7d69fb9558-88lq8"
Nov 01 03:48:52.305763 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:48:52.305738    4400 generic.go:356] "Generic (PLEG): container finished" podID="65358d08-83e6-45c5-a17e-f1dbde164345" containerID="5358aa2bcf8023a7cc57d249f7e1b5d1d2964a94922972b7c0dab7e001a04b16" exitCode=0
Nov 01 03:48:52.305893 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:48:52.305789    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d7f535/distinct-w3-d7f535-kserve-7d69fb9558-88lq8"
Nov 01 03:48:52.305893 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:48:52.305823    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d7f535/distinct-w3-d7f535-kserve-7d69fb9558-88lq8" event={"ID":"65358d08-83e6-45c5-a17e-f1dbde164345","Type":"ContainerDied","Data":"5358aa2bcf8023a7cc57d249f7e1b5d1d2964a94922972b7c0dab7e001a04b16"}
Nov 01 03:48:52.305893 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:48:52.305865    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d7f535/distinct-w3-d7f535-kserve-7d69fb9558-88lq8" event={"ID":"65358d08-83e6-45c5-a17e-f1dbde164345","Type":"ContainerDied","Data":"5340cf56d9db1941d947404f2bfc1632048158212a8c75f82ee4340f185a1cb4"}
Nov 01 03:48:52.305893 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:48:52.305883    4400 scope.go:122] "RemoveContainer" containerID="5358aa2bcf8023a7cc57d249f7e1b5d1d2964a94922972b7c0dab7e001a04b16"
Nov 01 03:48:52.315300 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:48:52.315277    4400 scope.go:122] "RemoveContainer" containerID="5358aa2bcf8023a7cc57d249f7e1b5d1d2964a94922972b7c0dab7e001a04b16"
Nov 01 03:48:52.315555 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:48:52.315534    4400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5358aa2bcf8023a7cc57d249f7e1b5d1d2964a94922972b7c0dab7e001a04b16\": container with ID starting with 5358aa2bcf8023a7cc57d249f7e1b5d1d2964a94922972b7c0dab7e001a04b16 not found: ID does not exist" containerID="5358aa2bcf8023a7cc57d249f7e1b5d1d2964a94922972b7c0dab7e001a04b16"
Nov 01 03:48:52.315629 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:48:52.315561    4400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5358aa2bcf8023a7cc57d249f7e1b5d1d2964a94922972b7c0dab7e001a04b16"} err="failed to get container status \"5358aa2bcf8023a7cc57d249f7e1b5d1d2964a94922972b7c0dab7e001a04b16\": rpc error: code = NotFound desc = could not find container \"5358aa2bcf8023a7cc57d249f7e1b5d1d2964a94922972b7c0dab7e001a04b16\": container with ID starting with 5358aa2bcf8023a7cc57d249f7e1b5d1d2964a94922972b7c0dab7e001a04b16 not found: ID does not exist"
Nov 01 03:48:52.336891 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:48:52.336872    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65358d08-83e6-45c5-a17e-f1dbde164345-tls-certs\" (UniqueName: \"kubernetes.io/secret/65358d08-83e6-45c5-a17e-f1dbde164345-tls-certs\") pod \"65358d08-83e6-45c5-a17e-f1dbde164345\" (UID: \"65358d08-83e6-45c5-a17e-f1dbde164345\") "
Nov 01 03:48:52.337005 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:48:52.336932    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65358d08-83e6-45c5-a17e-f1dbde164345-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65358d08-83e6-45c5-a17e-f1dbde164345-tmp-dir\") pod \"65358d08-83e6-45c5-a17e-f1dbde164345\" (UID: \"65358d08-83e6-45c5-a17e-f1dbde164345\") "
Nov 01 03:48:52.337071 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:48:52.337024    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65358d08-83e6-45c5-a17e-f1dbde164345-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65358d08-83e6-45c5-a17e-f1dbde164345-model-cache\") pod \"65358d08-83e6-45c5-a17e-f1dbde164345\" (UID: \"65358d08-83e6-45c5-a17e-f1dbde164345\") "
Nov 01 03:48:52.337071 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:48:52.337059    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65358d08-83e6-45c5-a17e-f1dbde164345-kube-api-access-qlfxt\" (UniqueName: \"kubernetes.io/projected/65358d08-83e6-45c5-a17e-f1dbde164345-kube-api-access-qlfxt\") pod \"65358d08-83e6-45c5-a17e-f1dbde164345\" (UID: \"65358d08-83e6-45c5-a17e-f1dbde164345\") "
Nov 01 03:48:52.337186 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:48:52.337093    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65358d08-83e6-45c5-a17e-f1dbde164345-home\" (UniqueName: \"kubernetes.io/empty-dir/65358d08-83e6-45c5-a17e-f1dbde164345-home\") pod \"65358d08-83e6-45c5-a17e-f1dbde164345\" (UID: \"65358d08-83e6-45c5-a17e-f1dbde164345\") "
Nov 01 03:48:52.337186 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:48:52.337109    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65358d08-83e6-45c5-a17e-f1dbde164345-dshm\" (UniqueName: \"kubernetes.io/empty-dir/65358d08-83e6-45c5-a17e-f1dbde164345-dshm\") pod \"65358d08-83e6-45c5-a17e-f1dbde164345\" (UID: \"65358d08-83e6-45c5-a17e-f1dbde164345\") "
Nov 01 03:48:52.337186 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:48:52.337107    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65358d08-83e6-45c5-a17e-f1dbde164345-tmp-dir" pod "65358d08-83e6-45c5-a17e-f1dbde164345" (UID: "65358d08-83e6-45c5-a17e-f1dbde164345"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:48:52.337346 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:48:52.337293    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65358d08-83e6-45c5-a17e-f1dbde164345-model-cache" pod "65358d08-83e6-45c5-a17e-f1dbde164345" (UID: "65358d08-83e6-45c5-a17e-f1dbde164345"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:48:52.337346 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:48:52.337331    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65358d08-83e6-45c5-a17e-f1dbde164345-home" pod "65358d08-83e6-45c5-a17e-f1dbde164345" (UID: "65358d08-83e6-45c5-a17e-f1dbde164345"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:48:52.337636 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:48:52.337617    4400 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65358d08-83e6-45c5-a17e-f1dbde164345-home\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:48:52.337759 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:48:52.337641    4400 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65358d08-83e6-45c5-a17e-f1dbde164345-tmp-dir\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:48:52.337759 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:48:52.337654    4400 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65358d08-83e6-45c5-a17e-f1dbde164345-model-cache\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:48:52.339192 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:48:52.339170    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65358d08-83e6-45c5-a17e-f1dbde164345-dshm" pod "65358d08-83e6-45c5-a17e-f1dbde164345" (UID: "65358d08-83e6-45c5-a17e-f1dbde164345"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:48:52.339259 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:48:52.339220    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65358d08-83e6-45c5-a17e-f1dbde164345-kube-api-access-qlfxt" pod "65358d08-83e6-45c5-a17e-f1dbde164345" (UID: "65358d08-83e6-45c5-a17e-f1dbde164345"). InnerVolumeSpecName "kube-api-access-qlfxt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:48:52.339259 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:48:52.339233    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65358d08-83e6-45c5-a17e-f1dbde164345-tls-certs" pod "65358d08-83e6-45c5-a17e-f1dbde164345" (UID: "65358d08-83e6-45c5-a17e-f1dbde164345"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:48:52.439222 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:48:52.439190    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlfxt\" (UniqueName: \"kubernetes.io/projected/65358d08-83e6-45c5-a17e-f1dbde164345-kube-api-access-qlfxt\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:48:52.439222 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:48:52.439226    4400 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65358d08-83e6-45c5-a17e-f1dbde164345-dshm\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:48:52.439399 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:48:52.439240    4400 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65358d08-83e6-45c5-a17e-f1dbde164345-tls-certs\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:48:52.730027 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:48:52.729993    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-d7f535/distinct-w3-d7f535-kserve-7d69fb9558-88lq8"]
Nov 01 03:48:52.734177 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:48:52.734149    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-d7f535/distinct-w3-d7f535-kserve-7d69fb9558-88lq8"]
Nov 01 03:48:53.616487 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:48:53.616445    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65358d08-83e6-45c5-a17e-f1dbde164345" path="/var/lib/kubelet/pods/65358d08-83e6-45c5-a17e-f1dbde164345/volumes"
Nov 01 03:48:58.788737 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:48:58.788673    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d7f535-8659cb78cf-wxmbp" podUID="a6b65b4c-f4d1-49a4-a4d3-7ef365d8aac9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 03:48:59.795395 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:48:59.795364    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-d7f535-67cfcb74d9-lzhln" podUID="ed1ec2ab-443d-42d9-9221-20cdf668121a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 03:49:01.730072 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:49:01.730033    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:49:08.359615 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:49:08.359589    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-d7f535-67cfcb74d9-lzhln"
Nov 01 03:49:08.410470 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:49:08.410439    4400 generic.go:356] "Generic (PLEG): container finished" podID="a6b65b4c-f4d1-49a4-a4d3-7ef365d8aac9" containerID="d156422b113d9f662ae479f0a27ca60e20b80380fdce48654cd5bc7b2177028e" exitCode=0
Nov 01 03:49:08.410635 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:49:08.410513    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d7f535-8659cb78cf-wxmbp" event={"ID":"a6b65b4c-f4d1-49a4-a4d3-7ef365d8aac9","Type":"ContainerDied","Data":"d156422b113d9f662ae479f0a27ca60e20b80380fdce48654cd5bc7b2177028e"}
Nov 01 03:49:08.411932 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:49:08.411906    4400 generic.go:356] "Generic (PLEG): container finished" podID="ed1ec2ab-443d-42d9-9221-20cdf668121a" containerID="0600998a757bb0c1749ee1486fd4e78f1d41db1b5bb4c74e86d3797d5f22786c" exitCode=0
Nov 01 03:49:08.412074 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:49:08.411960    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-d7f535-67cfcb74d9-lzhln"
Nov 01 03:49:08.412074 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:49:08.411984    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-d7f535-67cfcb74d9-lzhln" event={"ID":"ed1ec2ab-443d-42d9-9221-20cdf668121a","Type":"ContainerDied","Data":"0600998a757bb0c1749ee1486fd4e78f1d41db1b5bb4c74e86d3797d5f22786c"}
Nov 01 03:49:08.412074 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:49:08.412046    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-d7f535-67cfcb74d9-lzhln" event={"ID":"ed1ec2ab-443d-42d9-9221-20cdf668121a","Type":"ContainerDied","Data":"bcd8c50498b0bfa5306832eafe11b506ab00d647fd6b6678b7381edab1956b44"}
Nov 01 03:49:08.412074 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:49:08.412074    4400 scope.go:122] "RemoveContainer" containerID="0600998a757bb0c1749ee1486fd4e78f1d41db1b5bb4c74e86d3797d5f22786c"
Nov 01 03:49:08.424932 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:49:08.424908    4400 scope.go:122] "RemoveContainer" containerID="0600998a757bb0c1749ee1486fd4e78f1d41db1b5bb4c74e86d3797d5f22786c"
Nov 01 03:49:08.425323 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:49:08.425258    4400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0600998a757bb0c1749ee1486fd4e78f1d41db1b5bb4c74e86d3797d5f22786c\": container with ID starting with 0600998a757bb0c1749ee1486fd4e78f1d41db1b5bb4c74e86d3797d5f22786c not found: ID does not exist" containerID="0600998a757bb0c1749ee1486fd4e78f1d41db1b5bb4c74e86d3797d5f22786c"
Nov 01 03:49:08.425323 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:49:08.425295    4400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0600998a757bb0c1749ee1486fd4e78f1d41db1b5bb4c74e86d3797d5f22786c"} err="failed to get container status \"0600998a757bb0c1749ee1486fd4e78f1d41db1b5bb4c74e86d3797d5f22786c\": rpc error: code = NotFound desc = could not find container \"0600998a757bb0c1749ee1486fd4e78f1d41db1b5bb4c74e86d3797d5f22786c\": container with ID starting with 0600998a757bb0c1749ee1486fd4e78f1d41db1b5bb4c74e86d3797d5f22786c not found: ID does not exist"
Nov 01 03:49:08.440529 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:49:08.440509    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed1ec2ab-443d-42d9-9221-20cdf668121a-kube-api-access-s2n9n\" (UniqueName: \"kubernetes.io/projected/ed1ec2ab-443d-42d9-9221-20cdf668121a-kube-api-access-s2n9n\") pod \"ed1ec2ab-443d-42d9-9221-20cdf668121a\" (UID: \"ed1ec2ab-443d-42d9-9221-20cdf668121a\") "
Nov 01 03:49:08.440649 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:49:08.440638    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed1ec2ab-443d-42d9-9221-20cdf668121a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed1ec2ab-443d-42d9-9221-20cdf668121a-plugins-config-volume\") pod \"ed1ec2ab-443d-42d9-9221-20cdf668121a\" (UID: \"ed1ec2ab-443d-42d9-9221-20cdf668121a\") "
Nov 01 03:49:08.441030 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:49:08.441007    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed1ec2ab-443d-42d9-9221-20cdf668121a-plugins-config-volume" pod "ed1ec2ab-443d-42d9-9221-20cdf668121a" (UID: "ed1ec2ab-443d-42d9-9221-20cdf668121a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:49:08.441131 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:49:08.441113    4400 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed1ec2ab-443d-42d9-9221-20cdf668121a-plugins-config-volume\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:49:08.442589 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:49:08.442570    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed1ec2ab-443d-42d9-9221-20cdf668121a-kube-api-access-s2n9n" pod "ed1ec2ab-443d-42d9-9221-20cdf668121a" (UID: "ed1ec2ab-443d-42d9-9221-20cdf668121a"). InnerVolumeSpecName "kube-api-access-s2n9n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:49:08.456373 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:49:08.456353    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d7f535-8659cb78cf-wxmbp"
Nov 01 03:49:08.542337 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:49:08.542308    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6b65b4c-f4d1-49a4-a4d3-7ef365d8aac9-kube-api-access-r7rvh\" (UniqueName: \"kubernetes.io/projected/a6b65b4c-f4d1-49a4-a4d3-7ef365d8aac9-kube-api-access-r7rvh\") pod \"a6b65b4c-f4d1-49a4-a4d3-7ef365d8aac9\" (UID: \"a6b65b4c-f4d1-49a4-a4d3-7ef365d8aac9\") "
Nov 01 03:49:08.542504 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:49:08.542439    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6b65b4c-f4d1-49a4-a4d3-7ef365d8aac9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6b65b4c-f4d1-49a4-a4d3-7ef365d8aac9-plugins-config-volume\") pod \"a6b65b4c-f4d1-49a4-a4d3-7ef365d8aac9\" (UID: \"a6b65b4c-f4d1-49a4-a4d3-7ef365d8aac9\") "
Nov 01 03:49:08.542902 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:49:08.542878    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6b65b4c-f4d1-49a4-a4d3-7ef365d8aac9-plugins-config-volume" pod "a6b65b4c-f4d1-49a4-a4d3-7ef365d8aac9" (UID: "a6b65b4c-f4d1-49a4-a4d3-7ef365d8aac9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:49:08.543053 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:49:08.543039    4400 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6b65b4c-f4d1-49a4-a4d3-7ef365d8aac9-plugins-config-volume\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:49:08.543089 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:49:08.543056    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2n9n\" (UniqueName: \"kubernetes.io/projected/ed1ec2ab-443d-42d9-9221-20cdf668121a-kube-api-access-s2n9n\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:49:08.544532 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:49:08.544509    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6b65b4c-f4d1-49a4-a4d3-7ef365d8aac9-kube-api-access-r7rvh" pod "a6b65b4c-f4d1-49a4-a4d3-7ef365d8aac9" (UID: "a6b65b4c-f4d1-49a4-a4d3-7ef365d8aac9"). InnerVolumeSpecName "kube-api-access-r7rvh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:49:08.644462 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:49:08.644378    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7rvh\" (UniqueName: \"kubernetes.io/projected/a6b65b4c-f4d1-49a4-a4d3-7ef365d8aac9-kube-api-access-r7rvh\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:49:08.839757 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:49:08.839718    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-d7f535-67cfcb74d9-lzhln"]
Nov 01 03:49:08.844363 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:49:08.844332    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-d7f535-67cfcb74d9-lzhln"]
Nov 01 03:49:09.419742 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:49:09.419630    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d7f535-8659cb78cf-wxmbp"
Nov 01 03:49:09.419742 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:49:09.419626    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d7f535-8659cb78cf-wxmbp" event={"ID":"a6b65b4c-f4d1-49a4-a4d3-7ef365d8aac9","Type":"ContainerDied","Data":"23dcab7dc5582e89866262bf403961e478accfc3ca6e967bd6646edf3e3609b3"}
Nov 01 03:49:09.420268 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:49:09.419796    4400 scope.go:122] "RemoveContainer" containerID="d156422b113d9f662ae479f0a27ca60e20b80380fdce48654cd5bc7b2177028e"
Nov 01 03:49:09.592598 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:49:09.592561    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-d7f535-8659cb78cf-wxmbp"]
Nov 01 03:49:09.615358 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:49:09.615323    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed1ec2ab-443d-42d9-9221-20cdf668121a" path="/var/lib/kubelet/pods/ed1ec2ab-443d-42d9-9221-20cdf668121a/volumes"
Nov 01 03:49:09.643534 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:49:09.643507    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-d7f535-8659cb78cf-wxmbp"]
Nov 01 03:49:11.615092 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:49:11.615060    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6b65b4c-f4d1-49a4-a4d3-7ef365d8aac9" path="/var/lib/kubelet/pods/a6b65b4c-f4d1-49a4-a4d3-7ef365d8aac9/volumes"
Nov 01 03:49:31.729820 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:49:31.729785    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:49:51.724166 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:49:51.724137    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 03:49:52.041276 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:49:52.041214    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-d7f535-6d5d847766-s754n"]
Nov 01 03:49:52.041423 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:49:52.041401    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d7f535-6d5d847766-s754n" podUID="aa3e1de2-5fb8-4692-b87f-5b0cb323fe03" containerName="maas-api" containerID="cri-o://a74b4e8bc2244072bd1de603bacd4e578d65c1332f6b0f1d90a540b0d50e3c54" gracePeriod=30
Nov 01 03:49:52.338564 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:49:52.338539    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d7f535-6d5d847766-s754n"
Nov 01 03:49:52.491167 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:49:52.491134    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa3e1de2-5fb8-4692-b87f-5b0cb323fe03-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa3e1de2-5fb8-4692-b87f-5b0cb323fe03-openshift-service-ca\") pod \"aa3e1de2-5fb8-4692-b87f-5b0cb323fe03\" (UID: \"aa3e1de2-5fb8-4692-b87f-5b0cb323fe03\") "
Nov 01 03:49:52.491339 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:49:52.491230    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa3e1de2-5fb8-4692-b87f-5b0cb323fe03-kube-api-access-7cm75\" (UniqueName: \"kubernetes.io/projected/aa3e1de2-5fb8-4692-b87f-5b0cb323fe03-kube-api-access-7cm75\") pod \"aa3e1de2-5fb8-4692-b87f-5b0cb323fe03\" (UID: \"aa3e1de2-5fb8-4692-b87f-5b0cb323fe03\") "
Nov 01 03:49:52.491339 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:49:52.491281    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa3e1de2-5fb8-4692-b87f-5b0cb323fe03-metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa3e1de2-5fb8-4692-b87f-5b0cb323fe03-metrics-certs\") pod \"aa3e1de2-5fb8-4692-b87f-5b0cb323fe03\" (UID: \"aa3e1de2-5fb8-4692-b87f-5b0cb323fe03\") "
Nov 01 03:49:52.491339 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:49:52.491319    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa3e1de2-5fb8-4692-b87f-5b0cb323fe03-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa3e1de2-5fb8-4692-b87f-5b0cb323fe03-maas-api-tls\") pod \"aa3e1de2-5fb8-4692-b87f-5b0cb323fe03\" (UID: \"aa3e1de2-5fb8-4692-b87f-5b0cb323fe03\") "
Nov 01 03:49:52.491509 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:49:52.491482    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa3e1de2-5fb8-4692-b87f-5b0cb323fe03-openshift-service-ca" pod "aa3e1de2-5fb8-4692-b87f-5b0cb323fe03" (UID: "aa3e1de2-5fb8-4692-b87f-5b0cb323fe03"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:49:52.491893 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:49:52.491871    4400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa3e1de2-5fb8-4692-b87f-5b0cb323fe03-openshift-service-ca\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:49:52.493750 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:49:52.493722    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa3e1de2-5fb8-4692-b87f-5b0cb323fe03-metrics-certs" pod "aa3e1de2-5fb8-4692-b87f-5b0cb323fe03" (UID: "aa3e1de2-5fb8-4692-b87f-5b0cb323fe03"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:49:52.493984 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:49:52.493968    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa3e1de2-5fb8-4692-b87f-5b0cb323fe03-maas-api-tls" pod "aa3e1de2-5fb8-4692-b87f-5b0cb323fe03" (UID: "aa3e1de2-5fb8-4692-b87f-5b0cb323fe03"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:49:52.494059 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:49:52.494023    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa3e1de2-5fb8-4692-b87f-5b0cb323fe03-kube-api-access-7cm75" pod "aa3e1de2-5fb8-4692-b87f-5b0cb323fe03" (UID: "aa3e1de2-5fb8-4692-b87f-5b0cb323fe03"). InnerVolumeSpecName "kube-api-access-7cm75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:49:52.593508 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:49:52.593485    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cm75\" (UniqueName: \"kubernetes.io/projected/aa3e1de2-5fb8-4692-b87f-5b0cb323fe03-kube-api-access-7cm75\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:49:52.593508 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:49:52.593507    4400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa3e1de2-5fb8-4692-b87f-5b0cb323fe03-metrics-certs\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:49:52.593655 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:49:52.593516    4400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa3e1de2-5fb8-4692-b87f-5b0cb323fe03-maas-api-tls\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:49:52.704618 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:49:52.704581    4400 generic.go:356] "Generic (PLEG): container finished" podID="aa3e1de2-5fb8-4692-b87f-5b0cb323fe03" containerID="a74b4e8bc2244072bd1de603bacd4e578d65c1332f6b0f1d90a540b0d50e3c54" exitCode=0
Nov 01 03:49:52.705077 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:49:52.704976    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d7f535-6d5d847766-s754n"
Nov 01 03:49:52.705241 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:49:52.704994    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d7f535-6d5d847766-s754n" event={"ID":"aa3e1de2-5fb8-4692-b87f-5b0cb323fe03","Type":"ContainerDied","Data":"a74b4e8bc2244072bd1de603bacd4e578d65c1332f6b0f1d90a540b0d50e3c54"}
Nov 01 03:49:52.705386 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:49:52.705369    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d7f535-6d5d847766-s754n" event={"ID":"aa3e1de2-5fb8-4692-b87f-5b0cb323fe03","Type":"ContainerDied","Data":"ef236b1156d231475c6b806f26559397c79308d74f48c1faab93e618d409eff3"}
Nov 01 03:49:52.705511 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:49:52.705431    4400 scope.go:122] "RemoveContainer" containerID="a74b4e8bc2244072bd1de603bacd4e578d65c1332f6b0f1d90a540b0d50e3c54"
Nov 01 03:49:52.730643 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:49:52.730622    4400 scope.go:122] "RemoveContainer" containerID="a74b4e8bc2244072bd1de603bacd4e578d65c1332f6b0f1d90a540b0d50e3c54"
Nov 01 03:49:52.731113 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:49:52.731077    4400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a74b4e8bc2244072bd1de603bacd4e578d65c1332f6b0f1d90a540b0d50e3c54\": container with ID starting with a74b4e8bc2244072bd1de603bacd4e578d65c1332f6b0f1d90a540b0d50e3c54 not found: ID does not exist" containerID="a74b4e8bc2244072bd1de603bacd4e578d65c1332f6b0f1d90a540b0d50e3c54"
Nov 01 03:49:52.731199 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:49:52.731120    4400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a74b4e8bc2244072bd1de603bacd4e578d65c1332f6b0f1d90a540b0d50e3c54"} err="failed to get container status \"a74b4e8bc2244072bd1de603bacd4e578d65c1332f6b0f1d90a540b0d50e3c54\": rpc error: code = NotFound desc = could not find container \"a74b4e8bc2244072bd1de603bacd4e578d65c1332f6b0f1d90a540b0d50e3c54\": container with ID starting with a74b4e8bc2244072bd1de603bacd4e578d65c1332f6b0f1d90a540b0d50e3c54 not found: ID does not exist"
Nov 01 03:49:52.876803 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:49:52.876726    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-d7f535-6d5d847766-s754n"]
Nov 01 03:49:52.879484 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:49:52.879464    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-d7f535-6d5d847766-s754n"]
Nov 01 03:49:53.615397 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:49:53.615365    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa3e1de2-5fb8-4692-b87f-5b0cb323fe03" path="/var/lib/kubelet/pods/aa3e1de2-5fb8-4692-b87f-5b0cb323fe03/volumes"
Nov 01 03:50:01.339093 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:01.339062    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-h57fm"]
Nov 01 03:50:01.339515 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:01.339283    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-h57fm" podUID="c6f13d0b-fb65-44b9-b627-ec07234e87bc" containerName="istio-proxy" containerID="cri-o://7b32988cce7d6ed88c8bd44b962326c2d0a18e16f1e464bf945e6bda6e83b638" gracePeriod=30
Nov 01 03:50:01.730001 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:50:01.729924    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:50:02.691877 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:02.691839    4400 patch_prober.go:29] interesting pod/e2e-worker-w3-d7f535-openshift-default-856bb56cf-h57fm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 03:50:02.692258 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:02.691904    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-h57fm" podUID="c6f13d0b-fb65-44b9-b627-ec07234e87bc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 03:50:03.297052 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:03.297022    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5fd6cf-78d6995bb5-ffqcs"]
Nov 01 03:50:03.297233 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:03.297207    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5fd6cf-78d6995bb5-ffqcs" podUID="73e54a1c-e8c0-4edc-991e-94cf8d331ff5" containerName="maas-api" containerID="cri-o://8a695ea3a1b403cbdcb348bf82cc5c47627203da7b9ff2f9009188668d61e686" gracePeriod=30
Nov 01 03:50:03.583941 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:03.583917    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5fd6cf-78d6995bb5-ffqcs"
Nov 01 03:50:03.619095 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:03.619068    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73e54a1c-e8c0-4edc-991e-94cf8d331ff5-kube-api-access-67hml\" (UniqueName: \"kubernetes.io/projected/73e54a1c-e8c0-4edc-991e-94cf8d331ff5-kube-api-access-67hml\") pod \"73e54a1c-e8c0-4edc-991e-94cf8d331ff5\" (UID: \"73e54a1c-e8c0-4edc-991e-94cf8d331ff5\") "
Nov 01 03:50:03.619250 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:03.619107    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73e54a1c-e8c0-4edc-991e-94cf8d331ff5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73e54a1c-e8c0-4edc-991e-94cf8d331ff5-openshift-service-ca\") pod \"73e54a1c-e8c0-4edc-991e-94cf8d331ff5\" (UID: \"73e54a1c-e8c0-4edc-991e-94cf8d331ff5\") "
Nov 01 03:50:03.619250 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:03.619138    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73e54a1c-e8c0-4edc-991e-94cf8d331ff5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73e54a1c-e8c0-4edc-991e-94cf8d331ff5-maas-api-tls\") pod \"73e54a1c-e8c0-4edc-991e-94cf8d331ff5\" (UID: \"73e54a1c-e8c0-4edc-991e-94cf8d331ff5\") "
Nov 01 03:50:03.619250 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:03.619204    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73e54a1c-e8c0-4edc-991e-94cf8d331ff5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/73e54a1c-e8c0-4edc-991e-94cf8d331ff5-metrics-certs\") pod \"73e54a1c-e8c0-4edc-991e-94cf8d331ff5\" (UID: \"73e54a1c-e8c0-4edc-991e-94cf8d331ff5\") "
Nov 01 03:50:03.619481 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:03.619452    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73e54a1c-e8c0-4edc-991e-94cf8d331ff5-openshift-service-ca" pod "73e54a1c-e8c0-4edc-991e-94cf8d331ff5" (UID: "73e54a1c-e8c0-4edc-991e-94cf8d331ff5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:50:03.619590 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:03.619576    4400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73e54a1c-e8c0-4edc-991e-94cf8d331ff5-openshift-service-ca\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:50:03.621389 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:03.621356    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73e54a1c-e8c0-4edc-991e-94cf8d331ff5-maas-api-tls" pod "73e54a1c-e8c0-4edc-991e-94cf8d331ff5" (UID: "73e54a1c-e8c0-4edc-991e-94cf8d331ff5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:50:03.621389 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:03.621376    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73e54a1c-e8c0-4edc-991e-94cf8d331ff5-metrics-certs" pod "73e54a1c-e8c0-4edc-991e-94cf8d331ff5" (UID: "73e54a1c-e8c0-4edc-991e-94cf8d331ff5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:50:03.621523 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:03.621381    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73e54a1c-e8c0-4edc-991e-94cf8d331ff5-kube-api-access-67hml" pod "73e54a1c-e8c0-4edc-991e-94cf8d331ff5" (UID: "73e54a1c-e8c0-4edc-991e-94cf8d331ff5"). InnerVolumeSpecName "kube-api-access-67hml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:50:03.724638 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:03.723809    4400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73e54a1c-e8c0-4edc-991e-94cf8d331ff5-metrics-certs\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:50:03.724638 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:03.723843    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67hml\" (UniqueName: \"kubernetes.io/projected/73e54a1c-e8c0-4edc-991e-94cf8d331ff5-kube-api-access-67hml\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:50:03.724638 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:03.723859    4400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73e54a1c-e8c0-4edc-991e-94cf8d331ff5-maas-api-tls\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:50:03.772330 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:03.772300    4400 generic.go:356] "Generic (PLEG): container finished" podID="73e54a1c-e8c0-4edc-991e-94cf8d331ff5" containerID="8a695ea3a1b403cbdcb348bf82cc5c47627203da7b9ff2f9009188668d61e686" exitCode=0
Nov 01 03:50:03.772495 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:03.772347    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5fd6cf-78d6995bb5-ffqcs"
Nov 01 03:50:03.772495 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:03.772371    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5fd6cf-78d6995bb5-ffqcs" event={"ID":"73e54a1c-e8c0-4edc-991e-94cf8d331ff5","Type":"ContainerDied","Data":"8a695ea3a1b403cbdcb348bf82cc5c47627203da7b9ff2f9009188668d61e686"}
Nov 01 03:50:03.772495 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:03.772409    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5fd6cf-78d6995bb5-ffqcs" event={"ID":"73e54a1c-e8c0-4edc-991e-94cf8d331ff5","Type":"ContainerDied","Data":"f92f0f0d9d8eb9780619290abe7cd391a1d2050cb0e5dad055afa75dab3bc601"}
Nov 01 03:50:03.772495 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:03.772429    4400 scope.go:122] "RemoveContainer" containerID="8a695ea3a1b403cbdcb348bf82cc5c47627203da7b9ff2f9009188668d61e686"
Nov 01 03:50:03.784135 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:03.783921    4400 scope.go:122] "RemoveContainer" containerID="8a695ea3a1b403cbdcb348bf82cc5c47627203da7b9ff2f9009188668d61e686"
Nov 01 03:50:03.785187 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:50:03.785116    4400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a695ea3a1b403cbdcb348bf82cc5c47627203da7b9ff2f9009188668d61e686\": container with ID starting with 8a695ea3a1b403cbdcb348bf82cc5c47627203da7b9ff2f9009188668d61e686 not found: ID does not exist" containerID="8a695ea3a1b403cbdcb348bf82cc5c47627203da7b9ff2f9009188668d61e686"
Nov 01 03:50:03.785187 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:03.785150    4400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a695ea3a1b403cbdcb348bf82cc5c47627203da7b9ff2f9009188668d61e686"} err="failed to get container status \"8a695ea3a1b403cbdcb348bf82cc5c47627203da7b9ff2f9009188668d61e686\": rpc error: code = NotFound desc = could not find container \"8a695ea3a1b403cbdcb348bf82cc5c47627203da7b9ff2f9009188668d61e686\": container with ID starting with 8a695ea3a1b403cbdcb348bf82cc5c47627203da7b9ff2f9009188668d61e686 not found: ID does not exist"
Nov 01 03:50:03.946188 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:03.946154    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5fd6cf-78d6995bb5-ffqcs"]
Nov 01 03:50:03.949584 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:03.949563    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5fd6cf-78d6995bb5-ffqcs"]
Nov 01 03:50:05.615135 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:05.615101    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73e54a1c-e8c0-4edc-991e-94cf8d331ff5" path="/var/lib/kubelet/pods/73e54a1c-e8c0-4edc-991e-94cf8d331ff5/volumes"
Nov 01 03:50:06.611790 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:06.611735    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-h57fm"
Nov 01 03:50:06.659043 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:06.659007    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6f13d0b-fb65-44b9-b627-ec07234e87bc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6f13d0b-fb65-44b9-b627-ec07234e87bc-credential-socket\") pod \"c6f13d0b-fb65-44b9-b627-ec07234e87bc\" (UID: \"c6f13d0b-fb65-44b9-b627-ec07234e87bc\") "
Nov 01 03:50:06.659541 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:06.659078    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6f13d0b-fb65-44b9-b627-ec07234e87bc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6f13d0b-fb65-44b9-b627-ec07234e87bc-workload-socket\") pod \"c6f13d0b-fb65-44b9-b627-ec07234e87bc\" (UID: \"c6f13d0b-fb65-44b9-b627-ec07234e87bc\") "
Nov 01 03:50:06.659541 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:06.659098    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6f13d0b-fb65-44b9-b627-ec07234e87bc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6f13d0b-fb65-44b9-b627-ec07234e87bc-workload-certs\") pod \"c6f13d0b-fb65-44b9-b627-ec07234e87bc\" (UID: \"c6f13d0b-fb65-44b9-b627-ec07234e87bc\") "
Nov 01 03:50:06.659541 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:06.659131    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6f13d0b-fb65-44b9-b627-ec07234e87bc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6f13d0b-fb65-44b9-b627-ec07234e87bc-istio-data\") pod \"c6f13d0b-fb65-44b9-b627-ec07234e87bc\" (UID: \"c6f13d0b-fb65-44b9-b627-ec07234e87bc\") "
Nov 01 03:50:06.659541 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:06.659158    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c6f13d0b-fb65-44b9-b627-ec07234e87bc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6f13d0b-fb65-44b9-b627-ec07234e87bc-istio-podinfo\") pod \"c6f13d0b-fb65-44b9-b627-ec07234e87bc\" (UID: \"c6f13d0b-fb65-44b9-b627-ec07234e87bc\") "
Nov 01 03:50:06.659541 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:06.659184    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6f13d0b-fb65-44b9-b627-ec07234e87bc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6f13d0b-fb65-44b9-b627-ec07234e87bc-istio-envoy\") pod \"c6f13d0b-fb65-44b9-b627-ec07234e87bc\" (UID: \"c6f13d0b-fb65-44b9-b627-ec07234e87bc\") "
Nov 01 03:50:06.659541 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:06.659251    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6f13d0b-fb65-44b9-b627-ec07234e87bc-istio-token\" (UniqueName: \"kubernetes.io/projected/c6f13d0b-fb65-44b9-b627-ec07234e87bc-istio-token\") pod \"c6f13d0b-fb65-44b9-b627-ec07234e87bc\" (UID: \"c6f13d0b-fb65-44b9-b627-ec07234e87bc\") "
Nov 01 03:50:06.659541 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:06.659281    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6f13d0b-fb65-44b9-b627-ec07234e87bc-credential-socket" pod "c6f13d0b-fb65-44b9-b627-ec07234e87bc" (UID: "c6f13d0b-fb65-44b9-b627-ec07234e87bc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:50:06.659541 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:06.659334    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6f13d0b-fb65-44b9-b627-ec07234e87bc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6f13d0b-fb65-44b9-b627-ec07234e87bc-istiod-ca-cert\") pod \"c6f13d0b-fb65-44b9-b627-ec07234e87bc\" (UID: \"c6f13d0b-fb65-44b9-b627-ec07234e87bc\") "
Nov 01 03:50:06.659541 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:06.659394    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6f13d0b-fb65-44b9-b627-ec07234e87bc-kube-api-access-qjxq5\" (UniqueName: \"kubernetes.io/projected/c6f13d0b-fb65-44b9-b627-ec07234e87bc-kube-api-access-qjxq5\") pod \"c6f13d0b-fb65-44b9-b627-ec07234e87bc\" (UID: \"c6f13d0b-fb65-44b9-b627-ec07234e87bc\") "
Nov 01 03:50:06.659541 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:06.659410    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6f13d0b-fb65-44b9-b627-ec07234e87bc-workload-socket" pod "c6f13d0b-fb65-44b9-b627-ec07234e87bc" (UID: "c6f13d0b-fb65-44b9-b627-ec07234e87bc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:50:06.659541 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:06.659451    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6f13d0b-fb65-44b9-b627-ec07234e87bc-workload-certs" pod "c6f13d0b-fb65-44b9-b627-ec07234e87bc" (UID: "c6f13d0b-fb65-44b9-b627-ec07234e87bc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:50:06.660389 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:06.659426    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6f13d0b-fb65-44b9-b627-ec07234e87bc-istio-data" pod "c6f13d0b-fb65-44b9-b627-ec07234e87bc" (UID: "c6f13d0b-fb65-44b9-b627-ec07234e87bc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:50:06.660389 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:06.659675    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6f13d0b-fb65-44b9-b627-ec07234e87bc-istiod-ca-cert" pod "c6f13d0b-fb65-44b9-b627-ec07234e87bc" (UID: "c6f13d0b-fb65-44b9-b627-ec07234e87bc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:50:06.660389 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:06.659999    4400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6f13d0b-fb65-44b9-b627-ec07234e87bc-istiod-ca-cert\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:50:06.660389 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:06.660021    4400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6f13d0b-fb65-44b9-b627-ec07234e87bc-credential-socket\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:50:06.660389 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:06.660036    4400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6f13d0b-fb65-44b9-b627-ec07234e87bc-workload-socket\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:50:06.660389 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:06.660049    4400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6f13d0b-fb65-44b9-b627-ec07234e87bc-workload-certs\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:50:06.660389 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:06.660061    4400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6f13d0b-fb65-44b9-b627-ec07234e87bc-istio-data\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:50:06.661553 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:06.661523    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c6f13d0b-fb65-44b9-b627-ec07234e87bc-istio-podinfo" pod "c6f13d0b-fb65-44b9-b627-ec07234e87bc" (UID: "c6f13d0b-fb65-44b9-b627-ec07234e87bc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 03:50:06.661725 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:06.661678    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6f13d0b-fb65-44b9-b627-ec07234e87bc-istio-token" pod "c6f13d0b-fb65-44b9-b627-ec07234e87bc" (UID: "c6f13d0b-fb65-44b9-b627-ec07234e87bc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:50:06.661786 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:06.661754    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6f13d0b-fb65-44b9-b627-ec07234e87bc-kube-api-access-qjxq5" pod "c6f13d0b-fb65-44b9-b627-ec07234e87bc" (UID: "c6f13d0b-fb65-44b9-b627-ec07234e87bc"). InnerVolumeSpecName "kube-api-access-qjxq5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:50:06.661892 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:06.661861    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6f13d0b-fb65-44b9-b627-ec07234e87bc-istio-envoy" pod "c6f13d0b-fb65-44b9-b627-ec07234e87bc" (UID: "c6f13d0b-fb65-44b9-b627-ec07234e87bc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:50:06.760918 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:06.760898    4400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6f13d0b-fb65-44b9-b627-ec07234e87bc-istio-podinfo\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:50:06.761048 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:06.760923    4400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6f13d0b-fb65-44b9-b627-ec07234e87bc-istio-envoy\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:50:06.761048 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:06.760937    4400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6f13d0b-fb65-44b9-b627-ec07234e87bc-istio-token\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:50:06.761048 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:06.760949    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjxq5\" (UniqueName: \"kubernetes.io/projected/c6f13d0b-fb65-44b9-b627-ec07234e87bc-kube-api-access-qjxq5\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:50:06.804144 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:06.804110    4400 generic.go:356] "Generic (PLEG): container finished" podID="c6f13d0b-fb65-44b9-b627-ec07234e87bc" containerID="7b32988cce7d6ed88c8bd44b962326c2d0a18e16f1e464bf945e6bda6e83b638" exitCode=0
Nov 01 03:50:06.804265 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:06.804158    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-h57fm" event={"ID":"c6f13d0b-fb65-44b9-b627-ec07234e87bc","Type":"ContainerDied","Data":"7b32988cce7d6ed88c8bd44b962326c2d0a18e16f1e464bf945e6bda6e83b638"}
Nov 01 03:50:06.804265 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:06.804172    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-h57fm"
Nov 01 03:50:06.804265 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:06.804229    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-h57fm" event={"ID":"c6f13d0b-fb65-44b9-b627-ec07234e87bc","Type":"ContainerDied","Data":"d3aea43c4b74817c08cc1f35bfa69ec5376b96995b501cb3b54035c1f8bdeae4"}
Nov 01 03:50:06.804265 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:06.804254    4400 scope.go:122] "RemoveContainer" containerID="7b32988cce7d6ed88c8bd44b962326c2d0a18e16f1e464bf945e6bda6e83b638"
Nov 01 03:50:06.814446 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:06.814431    4400 scope.go:122] "RemoveContainer" containerID="7b32988cce7d6ed88c8bd44b962326c2d0a18e16f1e464bf945e6bda6e83b638"
Nov 01 03:50:06.814798 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:50:06.814776    4400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b32988cce7d6ed88c8bd44b962326c2d0a18e16f1e464bf945e6bda6e83b638\": container with ID starting with 7b32988cce7d6ed88c8bd44b962326c2d0a18e16f1e464bf945e6bda6e83b638 not found: ID does not exist" containerID="7b32988cce7d6ed88c8bd44b962326c2d0a18e16f1e464bf945e6bda6e83b638"
Nov 01 03:50:06.814915 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:06.814808    4400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b32988cce7d6ed88c8bd44b962326c2d0a18e16f1e464bf945e6bda6e83b638"} err="failed to get container status \"7b32988cce7d6ed88c8bd44b962326c2d0a18e16f1e464bf945e6bda6e83b638\": rpc error: code = NotFound desc = could not find container \"7b32988cce7d6ed88c8bd44b962326c2d0a18e16f1e464bf945e6bda6e83b638\": container with ID starting with 7b32988cce7d6ed88c8bd44b962326c2d0a18e16f1e464bf945e6bda6e83b638 not found: ID does not exist"
Nov 01 03:50:06.978574 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:06.978512    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-h57fm"]
Nov 01 03:50:06.983478 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:06.983459    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-h57fm"]
Nov 01 03:50:07.614979 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:07.614951    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6f13d0b-fb65-44b9-b627-ec07234e87bc" path="/var/lib/kubelet/pods/c6f13d0b-fb65-44b9-b627-ec07234e87bc/volumes"
Nov 01 03:50:14.714306 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:14.714277    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2906cb-68bd5bb9f9-d5gtm"]
Nov 01 03:50:14.714741 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:14.714442    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2906cb-68bd5bb9f9-d5gtm" podUID="7179f114-d7bf-4d4d-9a12-4a88e9041272" containerName="maas-api" containerID="cri-o://fd99d9d8d64260dcf723fed4bfc469e3b213edfe72535be200b276fcf39f83e5" gracePeriod=30
Nov 01 03:50:14.853588 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:14.853558    4400 generic.go:356] "Generic (PLEG): container finished" podID="7179f114-d7bf-4d4d-9a12-4a88e9041272" containerID="fd99d9d8d64260dcf723fed4bfc469e3b213edfe72535be200b276fcf39f83e5" exitCode=0
Nov 01 03:50:14.853788 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:14.853627    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2906cb-68bd5bb9f9-d5gtm" event={"ID":"7179f114-d7bf-4d4d-9a12-4a88e9041272","Type":"ContainerDied","Data":"fd99d9d8d64260dcf723fed4bfc469e3b213edfe72535be200b276fcf39f83e5"}
Nov 01 03:50:15.013997 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:15.013971    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2906cb-68bd5bb9f9-d5gtm"
Nov 01 03:50:15.143302 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:15.143264    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7179f114-d7bf-4d4d-9a12-4a88e9041272-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7179f114-d7bf-4d4d-9a12-4a88e9041272-maas-api-tls\") pod \"7179f114-d7bf-4d4d-9a12-4a88e9041272\" (UID: \"7179f114-d7bf-4d4d-9a12-4a88e9041272\") "
Nov 01 03:50:15.143505 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:15.143405    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7179f114-d7bf-4d4d-9a12-4a88e9041272-kube-api-access-d2b4j\" (UniqueName: \"kubernetes.io/projected/7179f114-d7bf-4d4d-9a12-4a88e9041272-kube-api-access-d2b4j\") pod \"7179f114-d7bf-4d4d-9a12-4a88e9041272\" (UID: \"7179f114-d7bf-4d4d-9a12-4a88e9041272\") "
Nov 01 03:50:15.143505 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:15.143439    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7179f114-d7bf-4d4d-9a12-4a88e9041272-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7179f114-d7bf-4d4d-9a12-4a88e9041272-metrics-certs\") pod \"7179f114-d7bf-4d4d-9a12-4a88e9041272\" (UID: \"7179f114-d7bf-4d4d-9a12-4a88e9041272\") "
Nov 01 03:50:15.143640 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:15.143554    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7179f114-d7bf-4d4d-9a12-4a88e9041272-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7179f114-d7bf-4d4d-9a12-4a88e9041272-openshift-service-ca\") pod \"7179f114-d7bf-4d4d-9a12-4a88e9041272\" (UID: \"7179f114-d7bf-4d4d-9a12-4a88e9041272\") "
Nov 01 03:50:15.144442 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:15.144410    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7179f114-d7bf-4d4d-9a12-4a88e9041272-openshift-service-ca" pod "7179f114-d7bf-4d4d-9a12-4a88e9041272" (UID: "7179f114-d7bf-4d4d-9a12-4a88e9041272"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:50:15.146772 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:15.146671    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7179f114-d7bf-4d4d-9a12-4a88e9041272-metrics-certs" pod "7179f114-d7bf-4d4d-9a12-4a88e9041272" (UID: "7179f114-d7bf-4d4d-9a12-4a88e9041272"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:50:15.149139 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:15.149117    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7179f114-d7bf-4d4d-9a12-4a88e9041272-maas-api-tls" pod "7179f114-d7bf-4d4d-9a12-4a88e9041272" (UID: "7179f114-d7bf-4d4d-9a12-4a88e9041272"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:50:15.149753 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:15.149725    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7179f114-d7bf-4d4d-9a12-4a88e9041272-kube-api-access-d2b4j" pod "7179f114-d7bf-4d4d-9a12-4a88e9041272" (UID: "7179f114-d7bf-4d4d-9a12-4a88e9041272"). InnerVolumeSpecName "kube-api-access-d2b4j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:50:15.245644 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:15.245278    4400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7179f114-d7bf-4d4d-9a12-4a88e9041272-openshift-service-ca\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:50:15.245644 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:15.245308    4400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7179f114-d7bf-4d4d-9a12-4a88e9041272-maas-api-tls\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:50:15.245644 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:15.245322    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2b4j\" (UniqueName: \"kubernetes.io/projected/7179f114-d7bf-4d4d-9a12-4a88e9041272-kube-api-access-d2b4j\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:50:15.245644 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:15.245336    4400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7179f114-d7bf-4d4d-9a12-4a88e9041272-metrics-certs\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:50:15.861154 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:15.861123    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2906cb-68bd5bb9f9-d5gtm"
Nov 01 03:50:15.861563 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:15.861156    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2906cb-68bd5bb9f9-d5gtm" event={"ID":"7179f114-d7bf-4d4d-9a12-4a88e9041272","Type":"ContainerDied","Data":"a564771fa7da2fe4a34aa0b7b7c9aeddfa7e218192e4f0461173944de81e22ff"}
Nov 01 03:50:15.861563 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:15.861210    4400 scope.go:122] "RemoveContainer" containerID="fd99d9d8d64260dcf723fed4bfc469e3b213edfe72535be200b276fcf39f83e5"
Nov 01 03:50:16.044726 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:16.044646    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2906cb-68bd5bb9f9-d5gtm"]
Nov 01 03:50:16.097486 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:16.097439    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2906cb-68bd5bb9f9-d5gtm"]
Nov 01 03:50:17.614323 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:17.614286    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7179f114-d7bf-4d4d-9a12-4a88e9041272" path="/var/lib/kubelet/pods/7179f114-d7bf-4d4d-9a12-4a88e9041272/volumes"
Nov 01 03:50:31.728608 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:50:31.728573    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:50:36.354676 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:36.354644    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-170ffa-6fdbdd6d4c-tvj7k"]
Nov 01 03:50:39.883972 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:39.883942    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 03:50:41.355252 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:41.355215    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-170ffa-6fdbdd6d4c-tvj7k" podUID="7159d3dd-647e-45b6-9f5b-1dbaecb70ad3" containerName="payload-pre-processing" containerID="cri-o://abd94975212d63844f27859b92ec15324b26c952a04836552deaf9d047ad8454" gracePeriod=25
Nov 01 03:50:41.629979 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:41.629908    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-170ffa-6fdbdd6d4c-tvj7k"
Nov 01 03:50:41.756816 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:41.756786    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7159d3dd-647e-45b6-9f5b-1dbaecb70ad3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7159d3dd-647e-45b6-9f5b-1dbaecb70ad3-plugins-config-volume\") pod \"7159d3dd-647e-45b6-9f5b-1dbaecb70ad3\" (UID: \"7159d3dd-647e-45b6-9f5b-1dbaecb70ad3\") "
Nov 01 03:50:41.756974 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:41.756923    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7159d3dd-647e-45b6-9f5b-1dbaecb70ad3-kube-api-access-xp9jm\" (UniqueName: \"kubernetes.io/projected/7159d3dd-647e-45b6-9f5b-1dbaecb70ad3-kube-api-access-xp9jm\") pod \"7159d3dd-647e-45b6-9f5b-1dbaecb70ad3\" (UID: \"7159d3dd-647e-45b6-9f5b-1dbaecb70ad3\") "
Nov 01 03:50:41.757200 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:41.757178    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7159d3dd-647e-45b6-9f5b-1dbaecb70ad3-plugins-config-volume" pod "7159d3dd-647e-45b6-9f5b-1dbaecb70ad3" (UID: "7159d3dd-647e-45b6-9f5b-1dbaecb70ad3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:50:41.757896 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:41.757467    4400 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7159d3dd-647e-45b6-9f5b-1dbaecb70ad3-plugins-config-volume\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:50:41.759261 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:41.759231    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7159d3dd-647e-45b6-9f5b-1dbaecb70ad3-kube-api-access-xp9jm" pod "7159d3dd-647e-45b6-9f5b-1dbaecb70ad3" (UID: "7159d3dd-647e-45b6-9f5b-1dbaecb70ad3"). InnerVolumeSpecName "kube-api-access-xp9jm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:50:41.859093 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:41.859065    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xp9jm\" (UniqueName: \"kubernetes.io/projected/7159d3dd-647e-45b6-9f5b-1dbaecb70ad3-kube-api-access-xp9jm\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:50:42.036323 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:42.036261    4400 generic.go:356] "Generic (PLEG): container finished" podID="7159d3dd-647e-45b6-9f5b-1dbaecb70ad3" containerID="abd94975212d63844f27859b92ec15324b26c952a04836552deaf9d047ad8454" exitCode=0
Nov 01 03:50:42.036323 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:42.036313    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-170ffa-6fdbdd6d4c-tvj7k"
Nov 01 03:50:42.036470 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:42.036333    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-170ffa-6fdbdd6d4c-tvj7k" event={"ID":"7159d3dd-647e-45b6-9f5b-1dbaecb70ad3","Type":"ContainerDied","Data":"abd94975212d63844f27859b92ec15324b26c952a04836552deaf9d047ad8454"}
Nov 01 03:50:42.036470 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:42.036380    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-170ffa-6fdbdd6d4c-tvj7k" event={"ID":"7159d3dd-647e-45b6-9f5b-1dbaecb70ad3","Type":"ContainerDied","Data":"48eb69b848613b49ad9694619e0242e77451f6e33a5c1d8bbff024b1dff3a188"}
Nov 01 03:50:42.036470 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:42.036403    4400 scope.go:122] "RemoveContainer" containerID="abd94975212d63844f27859b92ec15324b26c952a04836552deaf9d047ad8454"
Nov 01 03:50:42.046331 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:42.046313    4400 scope.go:122] "RemoveContainer" containerID="abd94975212d63844f27859b92ec15324b26c952a04836552deaf9d047ad8454"
Nov 01 03:50:42.046594 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:50:42.046577    4400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abd94975212d63844f27859b92ec15324b26c952a04836552deaf9d047ad8454\": container with ID starting with abd94975212d63844f27859b92ec15324b26c952a04836552deaf9d047ad8454 not found: ID does not exist" containerID="abd94975212d63844f27859b92ec15324b26c952a04836552deaf9d047ad8454"
Nov 01 03:50:42.046650 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:42.046601    4400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abd94975212d63844f27859b92ec15324b26c952a04836552deaf9d047ad8454"} err="failed to get container status \"abd94975212d63844f27859b92ec15324b26c952a04836552deaf9d047ad8454\": rpc error: code = NotFound desc = could not find container \"abd94975212d63844f27859b92ec15324b26c952a04836552deaf9d047ad8454\": container with ID starting with abd94975212d63844f27859b92ec15324b26c952a04836552deaf9d047ad8454 not found: ID does not exist"
Nov 01 03:50:42.207230 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:42.207196    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-170ffa-6fdbdd6d4c-tvj7k"]
Nov 01 03:50:42.211391 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:42.211366    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-170ffa-6fdbdd6d4c-tvj7k"]
Nov 01 03:50:43.616032 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:50:43.616002    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7159d3dd-647e-45b6-9f5b-1dbaecb70ad3" path="/var/lib/kubelet/pods/7159d3dd-647e-45b6-9f5b-1dbaecb70ad3/volumes"
Nov 01 03:51:01.728821 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:51:01.728786    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:51:05.097053 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:05.097023    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-417aa8-55c4f757fb-w744m"]
Nov 01 03:51:05.097518 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:05.097226    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-417aa8-55c4f757fb-w744m" podUID="dc04277f-2785-4c65-a353-4afca8198519" containerName="maas-api" containerID="cri-o://a94134333c8455c36ff5879d09e123eb04b1e373c40826bf6df20f56c2bad730" gracePeriod=30
Nov 01 03:51:05.386314 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:05.386290    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-417aa8-55c4f757fb-w744m"
Nov 01 03:51:05.395656 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:05.395630    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc04277f-2785-4c65-a353-4afca8198519-kube-api-access-wmsv2\" (UniqueName: \"kubernetes.io/projected/dc04277f-2785-4c65-a353-4afca8198519-kube-api-access-wmsv2\") pod \"dc04277f-2785-4c65-a353-4afca8198519\" (UID: \"dc04277f-2785-4c65-a353-4afca8198519\") "
Nov 01 03:51:05.395822 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:05.395804    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc04277f-2785-4c65-a353-4afca8198519-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc04277f-2785-4c65-a353-4afca8198519-maas-api-tls\") pod \"dc04277f-2785-4c65-a353-4afca8198519\" (UID: \"dc04277f-2785-4c65-a353-4afca8198519\") "
Nov 01 03:51:05.395885 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:05.395855    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc04277f-2785-4c65-a353-4afca8198519-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc04277f-2785-4c65-a353-4afca8198519-metrics-certs\") pod \"dc04277f-2785-4c65-a353-4afca8198519\" (UID: \"dc04277f-2785-4c65-a353-4afca8198519\") "
Nov 01 03:51:05.395940 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:05.395898    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc04277f-2785-4c65-a353-4afca8198519-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc04277f-2785-4c65-a353-4afca8198519-openshift-service-ca\") pod \"dc04277f-2785-4c65-a353-4afca8198519\" (UID: \"dc04277f-2785-4c65-a353-4afca8198519\") "
Nov 01 03:51:05.396299 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:05.396234    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc04277f-2785-4c65-a353-4afca8198519-openshift-service-ca" pod "dc04277f-2785-4c65-a353-4afca8198519" (UID: "dc04277f-2785-4c65-a353-4afca8198519"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:51:05.396506 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:05.396491    4400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc04277f-2785-4c65-a353-4afca8198519-openshift-service-ca\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:51:05.397841 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:05.397814    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc04277f-2785-4c65-a353-4afca8198519-kube-api-access-wmsv2" pod "dc04277f-2785-4c65-a353-4afca8198519" (UID: "dc04277f-2785-4c65-a353-4afca8198519"). InnerVolumeSpecName "kube-api-access-wmsv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:51:05.398047 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:05.398021    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc04277f-2785-4c65-a353-4afca8198519-metrics-certs" pod "dc04277f-2785-4c65-a353-4afca8198519" (UID: "dc04277f-2785-4c65-a353-4afca8198519"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:51:05.398047 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:05.398038    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc04277f-2785-4c65-a353-4afca8198519-maas-api-tls" pod "dc04277f-2785-4c65-a353-4afca8198519" (UID: "dc04277f-2785-4c65-a353-4afca8198519"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:51:05.500873 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:05.500829    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmsv2\" (UniqueName: \"kubernetes.io/projected/dc04277f-2785-4c65-a353-4afca8198519-kube-api-access-wmsv2\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:51:05.500873 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:05.500863    4400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc04277f-2785-4c65-a353-4afca8198519-maas-api-tls\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:51:05.500873 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:05.500879    4400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc04277f-2785-4c65-a353-4afca8198519-metrics-certs\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:51:06.182311 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:06.182275    4400 generic.go:356] "Generic (PLEG): container finished" podID="dc04277f-2785-4c65-a353-4afca8198519" containerID="a94134333c8455c36ff5879d09e123eb04b1e373c40826bf6df20f56c2bad730" exitCode=0
Nov 01 03:51:06.182734 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:06.182319    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-417aa8-55c4f757fb-w744m"
Nov 01 03:51:06.182734 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:06.182364    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-417aa8-55c4f757fb-w744m" event={"ID":"dc04277f-2785-4c65-a353-4afca8198519","Type":"ContainerDied","Data":"a94134333c8455c36ff5879d09e123eb04b1e373c40826bf6df20f56c2bad730"}
Nov 01 03:51:06.182734 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:06.182412    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-417aa8-55c4f757fb-w744m" event={"ID":"dc04277f-2785-4c65-a353-4afca8198519","Type":"ContainerDied","Data":"b2a33e2546d0a842a4821184a1de3d71381773baf491fe4ef0f1e2c9ea2d027f"}
Nov 01 03:51:06.182734 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:06.182432    4400 scope.go:122] "RemoveContainer" containerID="a94134333c8455c36ff5879d09e123eb04b1e373c40826bf6df20f56c2bad730"
Nov 01 03:51:06.191994 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:06.191979    4400 scope.go:122] "RemoveContainer" containerID="a94134333c8455c36ff5879d09e123eb04b1e373c40826bf6df20f56c2bad730"
Nov 01 03:51:06.192243 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:51:06.192225    4400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a94134333c8455c36ff5879d09e123eb04b1e373c40826bf6df20f56c2bad730\": container with ID starting with a94134333c8455c36ff5879d09e123eb04b1e373c40826bf6df20f56c2bad730 not found: ID does not exist" containerID="a94134333c8455c36ff5879d09e123eb04b1e373c40826bf6df20f56c2bad730"
Nov 01 03:51:06.192306 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:06.192255    4400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a94134333c8455c36ff5879d09e123eb04b1e373c40826bf6df20f56c2bad730"} err="failed to get container status \"a94134333c8455c36ff5879d09e123eb04b1e373c40826bf6df20f56c2bad730\": rpc error: code = NotFound desc = could not find container \"a94134333c8455c36ff5879d09e123eb04b1e373c40826bf6df20f56c2bad730\": container with ID starting with a94134333c8455c36ff5879d09e123eb04b1e373c40826bf6df20f56c2bad730 not found: ID does not exist"
Nov 01 03:51:06.351790 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:06.351769    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-417aa8-55c4f757fb-w744m"]
Nov 01 03:51:06.361321 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:06.361291    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-417aa8-55c4f757fb-w744m"]
Nov 01 03:51:07.618635 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:07.618601    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc04277f-2785-4c65-a353-4afca8198519" path="/var/lib/kubelet/pods/dc04277f-2785-4c65-a353-4afca8198519/volumes"
Nov 01 03:51:23.093952 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:23.093917    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-22e11b/test-model-22e11b-kserve-f5fb75fb5-vkc4k"]
Nov 01 03:51:24.542815 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:24.542780    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 03:51:27.117519 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:27.117485    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-b12199-76bc58d68f-mdsld"]
Nov 01 03:51:27.227330 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:27.227303    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-b12199-6d8d9bb99-sjn42"]
Nov 01 03:51:31.728546 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:51:31.728513    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:51:32.118126 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:32.118076    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-b12199-76bc58d68f-mdsld" podUID="58d4db89-1d9c-46f7-a944-43b865313c8c" containerName="payload-processing" containerID="cri-o://9e5453964fd1a097ecc7e6ef842c74c17a90885bfdedbc48fa52eeb21e1c2911" gracePeriod=25
Nov 01 03:51:32.235271 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:32.235229    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b12199-6d8d9bb99-sjn42" podUID="1b60b3a2-9941-49b1-a528-ee002c10a700" containerName="payload-pre-processing" containerID="cri-o://9e4abb443db248ac27ccf6bcdd3337d3950db54c58f5a1a44e72c76daaa451dd" gracePeriod=25
Nov 01 03:51:32.390100 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:32.390072    4400 generic.go:356] "Generic (PLEG): container finished" podID="1b60b3a2-9941-49b1-a528-ee002c10a700" containerID="9e4abb443db248ac27ccf6bcdd3337d3950db54c58f5a1a44e72c76daaa451dd" exitCode=0
Nov 01 03:51:32.390246 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:32.390152    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b12199-6d8d9bb99-sjn42" event={"ID":"1b60b3a2-9941-49b1-a528-ee002c10a700","Type":"ContainerDied","Data":"9e4abb443db248ac27ccf6bcdd3337d3950db54c58f5a1a44e72c76daaa451dd"}
Nov 01 03:51:32.391859 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:32.391827    4400 generic.go:356] "Generic (PLEG): container finished" podID="58d4db89-1d9c-46f7-a944-43b865313c8c" containerID="9e5453964fd1a097ecc7e6ef842c74c17a90885bfdedbc48fa52eeb21e1c2911" exitCode=0
Nov 01 03:51:32.391974 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:32.391900    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-b12199-76bc58d68f-mdsld" event={"ID":"58d4db89-1d9c-46f7-a944-43b865313c8c","Type":"ContainerDied","Data":"9e5453964fd1a097ecc7e6ef842c74c17a90885bfdedbc48fa52eeb21e1c2911"}
Nov 01 03:51:32.419000 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:32.418965    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-b12199-76bc58d68f-mdsld"
Nov 01 03:51:32.529198 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:32.529173    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b12199-6d8d9bb99-sjn42"
Nov 01 03:51:32.576618 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:32.576593    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58d4db89-1d9c-46f7-a944-43b865313c8c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58d4db89-1d9c-46f7-a944-43b865313c8c-plugins-config-volume\") pod \"58d4db89-1d9c-46f7-a944-43b865313c8c\" (UID: \"58d4db89-1d9c-46f7-a944-43b865313c8c\") "
Nov 01 03:51:32.576856 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:32.576840    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58d4db89-1d9c-46f7-a944-43b865313c8c-kube-api-access-srhxs\" (UniqueName: \"kubernetes.io/projected/58d4db89-1d9c-46f7-a944-43b865313c8c-kube-api-access-srhxs\") pod \"58d4db89-1d9c-46f7-a944-43b865313c8c\" (UID: \"58d4db89-1d9c-46f7-a944-43b865313c8c\") "
Nov 01 03:51:32.577013 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:32.576987    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58d4db89-1d9c-46f7-a944-43b865313c8c-plugins-config-volume" pod "58d4db89-1d9c-46f7-a944-43b865313c8c" (UID: "58d4db89-1d9c-46f7-a944-43b865313c8c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:51:32.577233 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:32.577217    4400 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58d4db89-1d9c-46f7-a944-43b865313c8c-plugins-config-volume\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:51:32.578898 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:32.578874    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58d4db89-1d9c-46f7-a944-43b865313c8c-kube-api-access-srhxs" pod "58d4db89-1d9c-46f7-a944-43b865313c8c" (UID: "58d4db89-1d9c-46f7-a944-43b865313c8c"). InnerVolumeSpecName "kube-api-access-srhxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:51:32.677718 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:32.677610    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b60b3a2-9941-49b1-a528-ee002c10a700-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b60b3a2-9941-49b1-a528-ee002c10a700-plugins-config-volume\") pod \"1b60b3a2-9941-49b1-a528-ee002c10a700\" (UID: \"1b60b3a2-9941-49b1-a528-ee002c10a700\") "
Nov 01 03:51:32.677898 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:32.677756    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b60b3a2-9941-49b1-a528-ee002c10a700-kube-api-access-jdvrl\" (UniqueName: \"kubernetes.io/projected/1b60b3a2-9941-49b1-a528-ee002c10a700-kube-api-access-jdvrl\") pod \"1b60b3a2-9941-49b1-a528-ee002c10a700\" (UID: \"1b60b3a2-9941-49b1-a528-ee002c10a700\") "
Nov 01 03:51:32.678038 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:32.678007    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b60b3a2-9941-49b1-a528-ee002c10a700-plugins-config-volume" pod "1b60b3a2-9941-49b1-a528-ee002c10a700" (UID: "1b60b3a2-9941-49b1-a528-ee002c10a700"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:51:32.680029 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:32.678575    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srhxs\" (UniqueName: \"kubernetes.io/projected/58d4db89-1d9c-46f7-a944-43b865313c8c-kube-api-access-srhxs\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:51:32.680137 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:32.680041    4400 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b60b3a2-9941-49b1-a528-ee002c10a700-plugins-config-volume\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:51:32.680872 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:32.680837    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b60b3a2-9941-49b1-a528-ee002c10a700-kube-api-access-jdvrl" pod "1b60b3a2-9941-49b1-a528-ee002c10a700" (UID: "1b60b3a2-9941-49b1-a528-ee002c10a700"). InnerVolumeSpecName "kube-api-access-jdvrl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:51:32.780818 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:32.780789    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdvrl\" (UniqueName: \"kubernetes.io/projected/1b60b3a2-9941-49b1-a528-ee002c10a700-kube-api-access-jdvrl\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:51:33.404670 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:33.404586    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-b12199-76bc58d68f-mdsld"
Nov 01 03:51:33.404670 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:33.404614    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-b12199-76bc58d68f-mdsld" event={"ID":"58d4db89-1d9c-46f7-a944-43b865313c8c","Type":"ContainerDied","Data":"bc83a7f7288c96f2c07068dca6675539b8b7015d7f38c3fa7728107622f132e3"}
Nov 01 03:51:33.406824 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:33.404720    4400 scope.go:122] "RemoveContainer" containerID="9e5453964fd1a097ecc7e6ef842c74c17a90885bfdedbc48fa52eeb21e1c2911"
Nov 01 03:51:33.406824 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:33.406061    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b12199-6d8d9bb99-sjn42"
Nov 01 03:51:33.406824 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:33.406065    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b12199-6d8d9bb99-sjn42" event={"ID":"1b60b3a2-9941-49b1-a528-ee002c10a700","Type":"ContainerDied","Data":"2a59c4f64b5eb234f97c8288d48802f25277809962bcfa0008c1abeb3c2e9511"}
Nov 01 03:51:33.416217 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:33.416163    4400 scope.go:122] "RemoveContainer" containerID="9e4abb443db248ac27ccf6bcdd3337d3950db54c58f5a1a44e72c76daaa451dd"
Nov 01 03:51:33.573674 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:33.573645    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-b12199-76bc58d68f-mdsld"]
Nov 01 03:51:33.624540 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:33.624516    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-b12199-76bc58d68f-mdsld"]
Nov 01 03:51:33.693617 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:33.693561    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-b12199-6d8d9bb99-sjn42"]
Nov 01 03:51:33.701230 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:33.701207    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-b12199-6d8d9bb99-sjn42"]
Nov 01 03:51:35.614579 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:35.614546    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b60b3a2-9941-49b1-a528-ee002c10a700" path="/var/lib/kubelet/pods/1b60b3a2-9941-49b1-a528-ee002c10a700/volumes"
Nov 01 03:51:35.615047 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:35.615031    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58d4db89-1d9c-46f7-a944-43b865313c8c" path="/var/lib/kubelet/pods/58d4db89-1d9c-46f7-a944-43b865313c8c/volumes"
Nov 01 03:51:38.114584 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:38.114532    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-22e11b/test-model-22e11b-kserve-f5fb75fb5-vkc4k" podUID="f7b0b2d1-ff0e-4bde-a832-430c7007749a" containerName="main" containerID="cri-o://d181aed4c212dfa838ff9ade2bdbab3fc57373f48213ac5681563c9e05bb67e4" gracePeriod=45
Nov 01 03:51:38.382064 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:38.382012    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-22e11b/test-model-22e11b-kserve-f5fb75fb5-vkc4k"
Nov 01 03:51:38.437430 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:38.437402    4400 generic.go:356] "Generic (PLEG): container finished" podID="f7b0b2d1-ff0e-4bde-a832-430c7007749a" containerID="d181aed4c212dfa838ff9ade2bdbab3fc57373f48213ac5681563c9e05bb67e4" exitCode=0
Nov 01 03:51:38.437543 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:38.437442    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-22e11b/test-model-22e11b-kserve-f5fb75fb5-vkc4k"
Nov 01 03:51:38.437543 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:38.437444    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-22e11b/test-model-22e11b-kserve-f5fb75fb5-vkc4k" event={"ID":"f7b0b2d1-ff0e-4bde-a832-430c7007749a","Type":"ContainerDied","Data":"d181aed4c212dfa838ff9ade2bdbab3fc57373f48213ac5681563c9e05bb67e4"}
Nov 01 03:51:38.437543 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:38.437492    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-22e11b/test-model-22e11b-kserve-f5fb75fb5-vkc4k" event={"ID":"f7b0b2d1-ff0e-4bde-a832-430c7007749a","Type":"ContainerDied","Data":"53db76dba6f7e6a68a58a18e876f03012fdc4581ef8674f8a415edd7e1663017"}
Nov 01 03:51:38.437543 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:38.437511    4400 scope.go:122] "RemoveContainer" containerID="d181aed4c212dfa838ff9ade2bdbab3fc57373f48213ac5681563c9e05bb67e4"
Nov 01 03:51:38.446139 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:38.446123    4400 scope.go:122] "RemoveContainer" containerID="d181aed4c212dfa838ff9ade2bdbab3fc57373f48213ac5681563c9e05bb67e4"
Nov 01 03:51:38.446413 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:51:38.446387    4400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d181aed4c212dfa838ff9ade2bdbab3fc57373f48213ac5681563c9e05bb67e4\": container with ID starting with d181aed4c212dfa838ff9ade2bdbab3fc57373f48213ac5681563c9e05bb67e4 not found: ID does not exist" containerID="d181aed4c212dfa838ff9ade2bdbab3fc57373f48213ac5681563c9e05bb67e4"
Nov 01 03:51:38.446498 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:38.446424    4400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d181aed4c212dfa838ff9ade2bdbab3fc57373f48213ac5681563c9e05bb67e4"} err="failed to get container status \"d181aed4c212dfa838ff9ade2bdbab3fc57373f48213ac5681563c9e05bb67e4\": rpc error: code = NotFound desc = could not find container \"d181aed4c212dfa838ff9ade2bdbab3fc57373f48213ac5681563c9e05bb67e4\": container with ID starting with d181aed4c212dfa838ff9ade2bdbab3fc57373f48213ac5681563c9e05bb67e4 not found: ID does not exist"
Nov 01 03:51:38.543994 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:38.543971    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7b0b2d1-ff0e-4bde-a832-430c7007749a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7b0b2d1-ff0e-4bde-a832-430c7007749a-model-cache\") pod \"f7b0b2d1-ff0e-4bde-a832-430c7007749a\" (UID: \"f7b0b2d1-ff0e-4bde-a832-430c7007749a\") "
Nov 01 03:51:38.544098 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:38.544011    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7b0b2d1-ff0e-4bde-a832-430c7007749a-home\" (UniqueName: \"kubernetes.io/empty-dir/f7b0b2d1-ff0e-4bde-a832-430c7007749a-home\") pod \"f7b0b2d1-ff0e-4bde-a832-430c7007749a\" (UID: \"f7b0b2d1-ff0e-4bde-a832-430c7007749a\") "
Nov 01 03:51:38.544098 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:38.544057    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7b0b2d1-ff0e-4bde-a832-430c7007749a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7b0b2d1-ff0e-4bde-a832-430c7007749a-tmp-dir\") pod \"f7b0b2d1-ff0e-4bde-a832-430c7007749a\" (UID: \"f7b0b2d1-ff0e-4bde-a832-430c7007749a\") "
Nov 01 03:51:38.544098 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:38.544093    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7b0b2d1-ff0e-4bde-a832-430c7007749a-tls-certs\" (UniqueName: \"kubernetes.io/secret/f7b0b2d1-ff0e-4bde-a832-430c7007749a-tls-certs\") pod \"f7b0b2d1-ff0e-4bde-a832-430c7007749a\" (UID: \"f7b0b2d1-ff0e-4bde-a832-430c7007749a\") "
Nov 01 03:51:38.544200 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:38.544132    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7b0b2d1-ff0e-4bde-a832-430c7007749a-kube-api-access-cq566\" (UniqueName: \"kubernetes.io/projected/f7b0b2d1-ff0e-4bde-a832-430c7007749a-kube-api-access-cq566\") pod \"f7b0b2d1-ff0e-4bde-a832-430c7007749a\" (UID: \"f7b0b2d1-ff0e-4bde-a832-430c7007749a\") "
Nov 01 03:51:38.544248 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:38.544199    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7b0b2d1-ff0e-4bde-a832-430c7007749a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7b0b2d1-ff0e-4bde-a832-430c7007749a-dshm\") pod \"f7b0b2d1-ff0e-4bde-a832-430c7007749a\" (UID: \"f7b0b2d1-ff0e-4bde-a832-430c7007749a\") "
Nov 01 03:51:38.544284 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:38.544235    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7b0b2d1-ff0e-4bde-a832-430c7007749a-model-cache" pod "f7b0b2d1-ff0e-4bde-a832-430c7007749a" (UID: "f7b0b2d1-ff0e-4bde-a832-430c7007749a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:51:38.544284 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:38.544278    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7b0b2d1-ff0e-4bde-a832-430c7007749a-home" pod "f7b0b2d1-ff0e-4bde-a832-430c7007749a" (UID: "f7b0b2d1-ff0e-4bde-a832-430c7007749a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:51:38.544383 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:38.544362    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7b0b2d1-ff0e-4bde-a832-430c7007749a-tmp-dir" pod "f7b0b2d1-ff0e-4bde-a832-430c7007749a" (UID: "f7b0b2d1-ff0e-4bde-a832-430c7007749a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:51:38.544657 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:38.544637    4400 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7b0b2d1-ff0e-4bde-a832-430c7007749a-tmp-dir\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:51:38.544934 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:38.544658    4400 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7b0b2d1-ff0e-4bde-a832-430c7007749a-model-cache\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:51:38.544934 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:38.544671    4400 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7b0b2d1-ff0e-4bde-a832-430c7007749a-home\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:51:38.546361 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:38.546337    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7b0b2d1-ff0e-4bde-a832-430c7007749a-tls-certs" pod "f7b0b2d1-ff0e-4bde-a832-430c7007749a" (UID: "f7b0b2d1-ff0e-4bde-a832-430c7007749a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:51:38.546457 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:38.546362    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7b0b2d1-ff0e-4bde-a832-430c7007749a-dshm" pod "f7b0b2d1-ff0e-4bde-a832-430c7007749a" (UID: "f7b0b2d1-ff0e-4bde-a832-430c7007749a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:51:38.546457 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:38.546428    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7b0b2d1-ff0e-4bde-a832-430c7007749a-kube-api-access-cq566" pod "f7b0b2d1-ff0e-4bde-a832-430c7007749a" (UID: "f7b0b2d1-ff0e-4bde-a832-430c7007749a"). InnerVolumeSpecName "kube-api-access-cq566". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:51:38.645343 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:38.645283    4400 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7b0b2d1-ff0e-4bde-a832-430c7007749a-tls-certs\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:51:38.645343 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:38.645310    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cq566\" (UniqueName: \"kubernetes.io/projected/f7b0b2d1-ff0e-4bde-a832-430c7007749a-kube-api-access-cq566\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:51:38.645343 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:38.645324    4400 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7b0b2d1-ff0e-4bde-a832-430c7007749a-dshm\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:51:38.861225 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:38.861203    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-22e11b/test-model-22e11b-kserve-f5fb75fb5-vkc4k"]
Nov 01 03:51:38.865979 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:38.865958    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-22e11b/test-model-22e11b-kserve-f5fb75fb5-vkc4k"]
Nov 01 03:51:39.614586 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:39.614553    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7b0b2d1-ff0e-4bde-a832-430c7007749a" path="/var/lib/kubelet/pods/f7b0b2d1-ff0e-4bde-a832-430c7007749a/volumes"
Nov 01 03:51:50.560154 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:50.560122    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a279c2-75cd57db8-zwzs7"]
Nov 01 03:51:50.560607 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:50.560303    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a279c2-75cd57db8-zwzs7" podUID="6592a8ed-219c-460c-b354-e0e8331cfa64" containerName="maas-api" containerID="cri-o://1d67453df8ff48625555a62ac4167603500cd8842ddb62e8936cee456a4cb2b8" gracePeriod=30
Nov 01 03:51:50.842931 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:50.842905    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a279c2-75cd57db8-zwzs7"
Nov 01 03:51:50.879446 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:50.879420    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6592a8ed-219c-460c-b354-e0e8331cfa64-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6592a8ed-219c-460c-b354-e0e8331cfa64-maas-api-tls\") pod \"6592a8ed-219c-460c-b354-e0e8331cfa64\" (UID: \"6592a8ed-219c-460c-b354-e0e8331cfa64\") "
Nov 01 03:51:50.879564 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:50.879529    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6592a8ed-219c-460c-b354-e0e8331cfa64-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6592a8ed-219c-460c-b354-e0e8331cfa64-openshift-service-ca\") pod \"6592a8ed-219c-460c-b354-e0e8331cfa64\" (UID: \"6592a8ed-219c-460c-b354-e0e8331cfa64\") "
Nov 01 03:51:50.879623 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:50.879569    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6592a8ed-219c-460c-b354-e0e8331cfa64-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6592a8ed-219c-460c-b354-e0e8331cfa64-metrics-certs\") pod \"6592a8ed-219c-460c-b354-e0e8331cfa64\" (UID: \"6592a8ed-219c-460c-b354-e0e8331cfa64\") "
Nov 01 03:51:50.879726 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:50.879713    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6592a8ed-219c-460c-b354-e0e8331cfa64-kube-api-access-7l6g2\" (UniqueName: \"kubernetes.io/projected/6592a8ed-219c-460c-b354-e0e8331cfa64-kube-api-access-7l6g2\") pod \"6592a8ed-219c-460c-b354-e0e8331cfa64\" (UID: \"6592a8ed-219c-460c-b354-e0e8331cfa64\") "
Nov 01 03:51:50.879945 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:50.879920    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6592a8ed-219c-460c-b354-e0e8331cfa64-openshift-service-ca" pod "6592a8ed-219c-460c-b354-e0e8331cfa64" (UID: "6592a8ed-219c-460c-b354-e0e8331cfa64"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:51:50.880149 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:50.880134    4400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6592a8ed-219c-460c-b354-e0e8331cfa64-openshift-service-ca\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:51:50.881675 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:50.881657    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6592a8ed-219c-460c-b354-e0e8331cfa64-maas-api-tls" pod "6592a8ed-219c-460c-b354-e0e8331cfa64" (UID: "6592a8ed-219c-460c-b354-e0e8331cfa64"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:51:50.881843 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:50.881824    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6592a8ed-219c-460c-b354-e0e8331cfa64-kube-api-access-7l6g2" pod "6592a8ed-219c-460c-b354-e0e8331cfa64" (UID: "6592a8ed-219c-460c-b354-e0e8331cfa64"). InnerVolumeSpecName "kube-api-access-7l6g2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:51:50.881887 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:50.881824    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6592a8ed-219c-460c-b354-e0e8331cfa64-metrics-certs" pod "6592a8ed-219c-460c-b354-e0e8331cfa64" (UID: "6592a8ed-219c-460c-b354-e0e8331cfa64"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:51:50.981958 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:50.981875    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7l6g2\" (UniqueName: \"kubernetes.io/projected/6592a8ed-219c-460c-b354-e0e8331cfa64-kube-api-access-7l6g2\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:51:50.981958 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:50.981914    4400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6592a8ed-219c-460c-b354-e0e8331cfa64-maas-api-tls\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:51:50.981958 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:50.981929    4400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6592a8ed-219c-460c-b354-e0e8331cfa64-metrics-certs\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:51:51.548262 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:51.548223    4400 generic.go:356] "Generic (PLEG): container finished" podID="6592a8ed-219c-460c-b354-e0e8331cfa64" containerID="1d67453df8ff48625555a62ac4167603500cd8842ddb62e8936cee456a4cb2b8" exitCode=0
Nov 01 03:51:51.548479 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:51.548328    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a279c2-75cd57db8-zwzs7"
Nov 01 03:51:51.548479 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:51.548325    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a279c2-75cd57db8-zwzs7" event={"ID":"6592a8ed-219c-460c-b354-e0e8331cfa64","Type":"ContainerDied","Data":"1d67453df8ff48625555a62ac4167603500cd8842ddb62e8936cee456a4cb2b8"}
Nov 01 03:51:51.548599 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:51.548500    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a279c2-75cd57db8-zwzs7" event={"ID":"6592a8ed-219c-460c-b354-e0e8331cfa64","Type":"ContainerDied","Data":"ed78893dc4506ceb1e5f1460e0486a794021080191a04e9699b7a0c113910439"}
Nov 01 03:51:51.548599 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:51.548535    4400 scope.go:122] "RemoveContainer" containerID="1d67453df8ff48625555a62ac4167603500cd8842ddb62e8936cee456a4cb2b8"
Nov 01 03:51:51.574758 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:51.574625    4400 scope.go:122] "RemoveContainer" containerID="1d67453df8ff48625555a62ac4167603500cd8842ddb62e8936cee456a4cb2b8"
Nov 01 03:51:51.575107 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:51:51.575036    4400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d67453df8ff48625555a62ac4167603500cd8842ddb62e8936cee456a4cb2b8\": container with ID starting with 1d67453df8ff48625555a62ac4167603500cd8842ddb62e8936cee456a4cb2b8 not found: ID does not exist" containerID="1d67453df8ff48625555a62ac4167603500cd8842ddb62e8936cee456a4cb2b8"
Nov 01 03:51:51.575107 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:51.575076    4400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d67453df8ff48625555a62ac4167603500cd8842ddb62e8936cee456a4cb2b8"} err="failed to get container status \"1d67453df8ff48625555a62ac4167603500cd8842ddb62e8936cee456a4cb2b8\": rpc error: code = NotFound desc = could not find container \"1d67453df8ff48625555a62ac4167603500cd8842ddb62e8936cee456a4cb2b8\": container with ID starting with 1d67453df8ff48625555a62ac4167603500cd8842ddb62e8936cee456a4cb2b8 not found: ID does not exist"
Nov 01 03:51:51.719232 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:51.719205    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a279c2-75cd57db8-zwzs7"]
Nov 01 03:51:51.770026 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:51.770002    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a279c2-75cd57db8-zwzs7"]
Nov 01 03:51:53.614698 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:53.614652    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6592a8ed-219c-460c-b354-e0e8331cfa64" path="/var/lib/kubelet/pods/6592a8ed-219c-460c-b354-e0e8331cfa64/volumes"
Nov 01 03:51:56.958584 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:56.958546    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-170ffa-56469b56d9-zwqkm"]
Nov 01 03:51:56.959195 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:56.958803    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-170ffa-56469b56d9-zwqkm" podUID="de71d35c-8a34-4a1c-96d3-7ae4aa24fee2" containerName="maas-api" containerID="cri-o://b1d53409d7e78dd5c4ed0e262ae13ecf4765d73e5c2927db321c25c9a48afdb0" gracePeriod=30
Nov 01 03:51:57.242599 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:57.242568    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-170ffa-56469b56d9-zwqkm"
Nov 01 03:51:57.394970 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:57.394920    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de71d35c-8a34-4a1c-96d3-7ae4aa24fee2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de71d35c-8a34-4a1c-96d3-7ae4aa24fee2-maas-api-tls\") pod \"de71d35c-8a34-4a1c-96d3-7ae4aa24fee2\" (UID: \"de71d35c-8a34-4a1c-96d3-7ae4aa24fee2\") "
Nov 01 03:51:57.395189 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:57.395165    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de71d35c-8a34-4a1c-96d3-7ae4aa24fee2-kube-api-access-bwwzz\" (UniqueName: \"kubernetes.io/projected/de71d35c-8a34-4a1c-96d3-7ae4aa24fee2-kube-api-access-bwwzz\") pod \"de71d35c-8a34-4a1c-96d3-7ae4aa24fee2\" (UID: \"de71d35c-8a34-4a1c-96d3-7ae4aa24fee2\") "
Nov 01 03:51:57.395289 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:57.395274    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de71d35c-8a34-4a1c-96d3-7ae4aa24fee2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/de71d35c-8a34-4a1c-96d3-7ae4aa24fee2-metrics-certs\") pod \"de71d35c-8a34-4a1c-96d3-7ae4aa24fee2\" (UID: \"de71d35c-8a34-4a1c-96d3-7ae4aa24fee2\") "
Nov 01 03:51:57.395355 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:57.395315    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de71d35c-8a34-4a1c-96d3-7ae4aa24fee2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de71d35c-8a34-4a1c-96d3-7ae4aa24fee2-openshift-service-ca\") pod \"de71d35c-8a34-4a1c-96d3-7ae4aa24fee2\" (UID: \"de71d35c-8a34-4a1c-96d3-7ae4aa24fee2\") "
Nov 01 03:51:57.398407 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:57.397900    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de71d35c-8a34-4a1c-96d3-7ae4aa24fee2-openshift-service-ca" pod "de71d35c-8a34-4a1c-96d3-7ae4aa24fee2" (UID: "de71d35c-8a34-4a1c-96d3-7ae4aa24fee2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:51:57.406742 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:57.406676    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de71d35c-8a34-4a1c-96d3-7ae4aa24fee2-kube-api-access-bwwzz" pod "de71d35c-8a34-4a1c-96d3-7ae4aa24fee2" (UID: "de71d35c-8a34-4a1c-96d3-7ae4aa24fee2"). InnerVolumeSpecName "kube-api-access-bwwzz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:51:57.407179 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:57.407148    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de71d35c-8a34-4a1c-96d3-7ae4aa24fee2-maas-api-tls" pod "de71d35c-8a34-4a1c-96d3-7ae4aa24fee2" (UID: "de71d35c-8a34-4a1c-96d3-7ae4aa24fee2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:51:57.407858 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:57.407527    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de71d35c-8a34-4a1c-96d3-7ae4aa24fee2-metrics-certs" pod "de71d35c-8a34-4a1c-96d3-7ae4aa24fee2" (UID: "de71d35c-8a34-4a1c-96d3-7ae4aa24fee2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:51:57.511186 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:57.511012    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwwzz\" (UniqueName: \"kubernetes.io/projected/de71d35c-8a34-4a1c-96d3-7ae4aa24fee2-kube-api-access-bwwzz\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:51:57.511186 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:57.511052    4400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de71d35c-8a34-4a1c-96d3-7ae4aa24fee2-metrics-certs\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:51:57.511186 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:57.511067    4400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de71d35c-8a34-4a1c-96d3-7ae4aa24fee2-openshift-service-ca\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:51:57.511186 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:57.511081    4400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de71d35c-8a34-4a1c-96d3-7ae4aa24fee2-maas-api-tls\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:51:57.585781 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:57.585743    4400 generic.go:356] "Generic (PLEG): container finished" podID="de71d35c-8a34-4a1c-96d3-7ae4aa24fee2" containerID="b1d53409d7e78dd5c4ed0e262ae13ecf4765d73e5c2927db321c25c9a48afdb0" exitCode=0
Nov 01 03:51:57.585946 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:57.585795    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-170ffa-56469b56d9-zwqkm"
Nov 01 03:51:57.585946 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:57.585822    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-170ffa-56469b56d9-zwqkm" event={"ID":"de71d35c-8a34-4a1c-96d3-7ae4aa24fee2","Type":"ContainerDied","Data":"b1d53409d7e78dd5c4ed0e262ae13ecf4765d73e5c2927db321c25c9a48afdb0"}
Nov 01 03:51:57.585946 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:57.585876    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-170ffa-56469b56d9-zwqkm" event={"ID":"de71d35c-8a34-4a1c-96d3-7ae4aa24fee2","Type":"ContainerDied","Data":"d8b319dfc0fa8c528cc95ad0ec1bfe7c6101fdf7faba73695f4e4693e9548ca0"}
Nov 01 03:51:57.585946 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:57.585897    4400 scope.go:122] "RemoveContainer" containerID="b1d53409d7e78dd5c4ed0e262ae13ecf4765d73e5c2927db321c25c9a48afdb0"
Nov 01 03:51:57.596498 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:57.596480    4400 scope.go:122] "RemoveContainer" containerID="b1d53409d7e78dd5c4ed0e262ae13ecf4765d73e5c2927db321c25c9a48afdb0"
Nov 01 03:51:57.596791 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:51:57.596773    4400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1d53409d7e78dd5c4ed0e262ae13ecf4765d73e5c2927db321c25c9a48afdb0\": container with ID starting with b1d53409d7e78dd5c4ed0e262ae13ecf4765d73e5c2927db321c25c9a48afdb0 not found: ID does not exist" containerID="b1d53409d7e78dd5c4ed0e262ae13ecf4765d73e5c2927db321c25c9a48afdb0"
Nov 01 03:51:57.596856 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:57.596802    4400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1d53409d7e78dd5c4ed0e262ae13ecf4765d73e5c2927db321c25c9a48afdb0"} err="failed to get container status \"b1d53409d7e78dd5c4ed0e262ae13ecf4765d73e5c2927db321c25c9a48afdb0\": rpc error: code = NotFound desc = could not find container \"b1d53409d7e78dd5c4ed0e262ae13ecf4765d73e5c2927db321c25c9a48afdb0\": container with ID starting with b1d53409d7e78dd5c4ed0e262ae13ecf4765d73e5c2927db321c25c9a48afdb0 not found: ID does not exist"
Nov 01 03:51:57.756033 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:57.755634    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-170ffa-56469b56d9-zwqkm"]
Nov 01 03:51:57.758484 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:57.758467    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-170ffa-56469b56d9-zwqkm"]
Nov 01 03:51:59.614340 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:51:59.614307    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de71d35c-8a34-4a1c-96d3-7ae4aa24fee2" path="/var/lib/kubelet/pods/de71d35c-8a34-4a1c-96d3-7ae4aa24fee2/volumes"
Nov 01 03:52:01.729415 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:52:01.729383    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:52:06.570342 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:06.570310    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 03:52:12.942394 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:12.942358    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-ded2d7-54f99bddc-gjh62"]
Nov 01 03:52:13.050668 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:13.050632    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-ded2d7-748487bb8c-9w7mw"]
Nov 01 03:52:17.943996 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:17.943886    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-ded2d7-54f99bddc-gjh62" podUID="466179e3-7600-464e-9f7c-ec099bc8086d" containerName="payload-processing" containerID="cri-o://c2662c6c96fa2bb6dc157a754b41999ca03b687ca8158aee5a3832e0bce615e4" gracePeriod=25
Nov 01 03:52:18.051645 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:18.051593    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ded2d7-748487bb8c-9w7mw" podUID="7c1732ba-87a1-43cf-92d6-5edf0869787f" containerName="payload-pre-processing" containerID="cri-o://ca7234f808bb82dd2442bdf2c2bfba1bc875cde71f5216a7f6847d34d3c5fb82" gracePeriod=25
Nov 01 03:52:23.754287 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:23.754254    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-ded2d7-54f99bddc-gjh62" podUID="466179e3-7600-464e-9f7c-ec099bc8086d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 03:52:23.754717 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:23.754288    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ded2d7-748487bb8c-9w7mw" podUID="7c1732ba-87a1-43cf-92d6-5edf0869787f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 03:52:24.115076 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:24.115039    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-22e11b-75d4c95497-n67kh"]
Nov 01 03:52:24.115289 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:24.115266    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-22e11b-75d4c95497-n67kh" podUID="a617524f-b187-4692-aea7-89193c7bcad0" containerName="maas-api" containerID="cri-o://3839dc1d5020709f743a71b0fbfaae307a80d41d65c222176b02c52c64d428c7" gracePeriod=30
Nov 01 03:52:24.387534 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:24.387510    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-22e11b-75d4c95497-n67kh"
Nov 01 03:52:24.524720 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:24.506622    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a617524f-b187-4692-aea7-89193c7bcad0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a617524f-b187-4692-aea7-89193c7bcad0-openshift-service-ca\") pod \"a617524f-b187-4692-aea7-89193c7bcad0\" (UID: \"a617524f-b187-4692-aea7-89193c7bcad0\") "
Nov 01 03:52:24.524720 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:24.506769    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a617524f-b187-4692-aea7-89193c7bcad0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a617524f-b187-4692-aea7-89193c7bcad0-maas-api-tls\") pod \"a617524f-b187-4692-aea7-89193c7bcad0\" (UID: \"a617524f-b187-4692-aea7-89193c7bcad0\") "
Nov 01 03:52:24.524720 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:24.507004    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a617524f-b187-4692-aea7-89193c7bcad0-kube-api-access-pxj4d\" (UniqueName: \"kubernetes.io/projected/a617524f-b187-4692-aea7-89193c7bcad0-kube-api-access-pxj4d\") pod \"a617524f-b187-4692-aea7-89193c7bcad0\" (UID: \"a617524f-b187-4692-aea7-89193c7bcad0\") "
Nov 01 03:52:24.524720 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:24.507098    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a617524f-b187-4692-aea7-89193c7bcad0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a617524f-b187-4692-aea7-89193c7bcad0-metrics-certs\") pod \"a617524f-b187-4692-aea7-89193c7bcad0\" (UID: \"a617524f-b187-4692-aea7-89193c7bcad0\") "
Nov 01 03:52:24.524720 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:24.522016    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a617524f-b187-4692-aea7-89193c7bcad0-openshift-service-ca" pod "a617524f-b187-4692-aea7-89193c7bcad0" (UID: "a617524f-b187-4692-aea7-89193c7bcad0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:52:24.534714 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:24.533067    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a617524f-b187-4692-aea7-89193c7bcad0-maas-api-tls" pod "a617524f-b187-4692-aea7-89193c7bcad0" (UID: "a617524f-b187-4692-aea7-89193c7bcad0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:52:24.534714 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:24.533303    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a617524f-b187-4692-aea7-89193c7bcad0-kube-api-access-pxj4d" pod "a617524f-b187-4692-aea7-89193c7bcad0" (UID: "a617524f-b187-4692-aea7-89193c7bcad0"). InnerVolumeSpecName "kube-api-access-pxj4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:52:24.534714 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:24.533509    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a617524f-b187-4692-aea7-89193c7bcad0-metrics-certs" pod "a617524f-b187-4692-aea7-89193c7bcad0" (UID: "a617524f-b187-4692-aea7-89193c7bcad0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:52:24.617607 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:24.609501    4400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a617524f-b187-4692-aea7-89193c7bcad0-maas-api-tls\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:52:24.617607 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:24.609551    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxj4d\" (UniqueName: \"kubernetes.io/projected/a617524f-b187-4692-aea7-89193c7bcad0-kube-api-access-pxj4d\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:52:24.617607 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:24.609584    4400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a617524f-b187-4692-aea7-89193c7bcad0-metrics-certs\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:52:24.617607 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:24.609605    4400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a617524f-b187-4692-aea7-89193c7bcad0-openshift-service-ca\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:52:24.762954 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:24.762875    4400 generic.go:356] "Generic (PLEG): container finished" podID="a617524f-b187-4692-aea7-89193c7bcad0" containerID="3839dc1d5020709f743a71b0fbfaae307a80d41d65c222176b02c52c64d428c7" exitCode=0
Nov 01 03:52:24.762954 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:24.762914    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-22e11b-75d4c95497-n67kh" event={"ID":"a617524f-b187-4692-aea7-89193c7bcad0","Type":"ContainerDied","Data":"3839dc1d5020709f743a71b0fbfaae307a80d41d65c222176b02c52c64d428c7"}
Nov 01 03:52:24.762954 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:24.762932    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-22e11b-75d4c95497-n67kh"
Nov 01 03:52:24.763460 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:24.762995    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-22e11b-75d4c95497-n67kh" event={"ID":"a617524f-b187-4692-aea7-89193c7bcad0","Type":"ContainerDied","Data":"b781d0a044c5bc91ce40916c7a7ff76b902708b9625f39364069451368202c53"}
Nov 01 03:52:24.763460 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:24.763018    4400 scope.go:122] "RemoveContainer" containerID="3839dc1d5020709f743a71b0fbfaae307a80d41d65c222176b02c52c64d428c7"
Nov 01 03:52:24.772924 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:24.772910    4400 scope.go:122] "RemoveContainer" containerID="3839dc1d5020709f743a71b0fbfaae307a80d41d65c222176b02c52c64d428c7"
Nov 01 03:52:24.773173 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:52:24.773156    4400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3839dc1d5020709f743a71b0fbfaae307a80d41d65c222176b02c52c64d428c7\": container with ID starting with 3839dc1d5020709f743a71b0fbfaae307a80d41d65c222176b02c52c64d428c7 not found: ID does not exist" containerID="3839dc1d5020709f743a71b0fbfaae307a80d41d65c222176b02c52c64d428c7"
Nov 01 03:52:24.773213 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:24.773182    4400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3839dc1d5020709f743a71b0fbfaae307a80d41d65c222176b02c52c64d428c7"} err="failed to get container status \"3839dc1d5020709f743a71b0fbfaae307a80d41d65c222176b02c52c64d428c7\": rpc error: code = NotFound desc = could not find container \"3839dc1d5020709f743a71b0fbfaae307a80d41d65c222176b02c52c64d428c7\": container with ID starting with 3839dc1d5020709f743a71b0fbfaae307a80d41d65c222176b02c52c64d428c7 not found: ID does not exist"
Nov 01 03:52:24.931893 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:24.931870    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-22e11b-75d4c95497-n67kh"]
Nov 01 03:52:24.936386 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:24.936366    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-22e11b-75d4c95497-n67kh"]
Nov 01 03:52:25.615365 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:25.615337    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a617524f-b187-4692-aea7-89193c7bcad0" path="/var/lib/kubelet/pods/a617524f-b187-4692-aea7-89193c7bcad0/volumes"
Nov 01 03:52:31.728208 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:52:31.728173    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:52:33.753848 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:33.753818    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ded2d7-748487bb8c-9w7mw" podUID="7c1732ba-87a1-43cf-92d6-5edf0869787f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 03:52:33.754258 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:33.753816    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-ded2d7-54f99bddc-gjh62" podUID="466179e3-7600-464e-9f7c-ec099bc8086d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 03:52:33.811252 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:33.811226    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 03:52:38.207010 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:38.206947    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-ded2d7-54f99bddc-gjh62"
Nov 01 03:52:38.310243 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:38.310219    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ded2d7-748487bb8c-9w7mw"
Nov 01 03:52:38.354103 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:38.354079    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/466179e3-7600-464e-9f7c-ec099bc8086d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/466179e3-7600-464e-9f7c-ec099bc8086d-plugins-config-volume\") pod \"466179e3-7600-464e-9f7c-ec099bc8086d\" (UID: \"466179e3-7600-464e-9f7c-ec099bc8086d\") "
Nov 01 03:52:38.354228 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:38.354161    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/466179e3-7600-464e-9f7c-ec099bc8086d-kube-api-access-4498n\" (UniqueName: \"kubernetes.io/projected/466179e3-7600-464e-9f7c-ec099bc8086d-kube-api-access-4498n\") pod \"466179e3-7600-464e-9f7c-ec099bc8086d\" (UID: \"466179e3-7600-464e-9f7c-ec099bc8086d\") "
Nov 01 03:52:38.354514 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:38.354488    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/466179e3-7600-464e-9f7c-ec099bc8086d-plugins-config-volume" pod "466179e3-7600-464e-9f7c-ec099bc8086d" (UID: "466179e3-7600-464e-9f7c-ec099bc8086d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:52:38.356391 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:38.356361    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/466179e3-7600-464e-9f7c-ec099bc8086d-kube-api-access-4498n" pod "466179e3-7600-464e-9f7c-ec099bc8086d" (UID: "466179e3-7600-464e-9f7c-ec099bc8086d"). InnerVolumeSpecName "kube-api-access-4498n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:52:38.455408 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:38.455386    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c1732ba-87a1-43cf-92d6-5edf0869787f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c1732ba-87a1-43cf-92d6-5edf0869787f-plugins-config-volume\") pod \"7c1732ba-87a1-43cf-92d6-5edf0869787f\" (UID: \"7c1732ba-87a1-43cf-92d6-5edf0869787f\") "
Nov 01 03:52:38.455541 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:38.455495    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c1732ba-87a1-43cf-92d6-5edf0869787f-kube-api-access-fg2dp\" (UniqueName: \"kubernetes.io/projected/7c1732ba-87a1-43cf-92d6-5edf0869787f-kube-api-access-fg2dp\") pod \"7c1732ba-87a1-43cf-92d6-5edf0869787f\" (UID: \"7c1732ba-87a1-43cf-92d6-5edf0869787f\") "
Nov 01 03:52:38.455733 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:38.455715    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c1732ba-87a1-43cf-92d6-5edf0869787f-plugins-config-volume" pod "7c1732ba-87a1-43cf-92d6-5edf0869787f" (UID: "7c1732ba-87a1-43cf-92d6-5edf0869787f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:52:38.455959 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:38.455943    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4498n\" (UniqueName: \"kubernetes.io/projected/466179e3-7600-464e-9f7c-ec099bc8086d-kube-api-access-4498n\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:52:38.456033 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:38.455964    4400 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c1732ba-87a1-43cf-92d6-5edf0869787f-plugins-config-volume\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:52:38.456033 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:38.455977    4400 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/466179e3-7600-464e-9f7c-ec099bc8086d-plugins-config-volume\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:52:38.457572 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:38.457530    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c1732ba-87a1-43cf-92d6-5edf0869787f-kube-api-access-fg2dp" pod "7c1732ba-87a1-43cf-92d6-5edf0869787f" (UID: "7c1732ba-87a1-43cf-92d6-5edf0869787f"). InnerVolumeSpecName "kube-api-access-fg2dp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:52:38.556876 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:38.556855    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fg2dp\" (UniqueName: \"kubernetes.io/projected/7c1732ba-87a1-43cf-92d6-5edf0869787f-kube-api-access-fg2dp\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:52:38.842770 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:38.842738    4400 generic.go:356] "Generic (PLEG): container finished" podID="466179e3-7600-464e-9f7c-ec099bc8086d" containerID="c2662c6c96fa2bb6dc157a754b41999ca03b687ca8158aee5a3832e0bce615e4" exitCode=0
Nov 01 03:52:38.842891 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:38.842780    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-ded2d7-54f99bddc-gjh62"
Nov 01 03:52:38.842891 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:38.842826    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-ded2d7-54f99bddc-gjh62" event={"ID":"466179e3-7600-464e-9f7c-ec099bc8086d","Type":"ContainerDied","Data":"c2662c6c96fa2bb6dc157a754b41999ca03b687ca8158aee5a3832e0bce615e4"}
Nov 01 03:52:38.842891 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:38.842875    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-ded2d7-54f99bddc-gjh62" event={"ID":"466179e3-7600-464e-9f7c-ec099bc8086d","Type":"ContainerDied","Data":"75173a332a054796b1db6fb4afc582e064237a2eec7fb67c1e39b4863387a924"}
Nov 01 03:52:38.843010 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:38.842901    4400 scope.go:122] "RemoveContainer" containerID="c2662c6c96fa2bb6dc157a754b41999ca03b687ca8158aee5a3832e0bce615e4"
Nov 01 03:52:38.844430 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:38.844410    4400 generic.go:356] "Generic (PLEG): container finished" podID="7c1732ba-87a1-43cf-92d6-5edf0869787f" containerID="ca7234f808bb82dd2442bdf2c2bfba1bc875cde71f5216a7f6847d34d3c5fb82" exitCode=0
Nov 01 03:52:38.844570 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:38.844441    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ded2d7-748487bb8c-9w7mw" event={"ID":"7c1732ba-87a1-43cf-92d6-5edf0869787f","Type":"ContainerDied","Data":"ca7234f808bb82dd2442bdf2c2bfba1bc875cde71f5216a7f6847d34d3c5fb82"}
Nov 01 03:52:38.844570 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:38.844462    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ded2d7-748487bb8c-9w7mw"
Nov 01 03:52:38.844570 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:38.844475    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ded2d7-748487bb8c-9w7mw" event={"ID":"7c1732ba-87a1-43cf-92d6-5edf0869787f","Type":"ContainerDied","Data":"affac3f5fc0e9e934c6b9e17cba4cb6ff8097f3b47b7a2fba872d56dc119799c"}
Nov 01 03:52:38.852450 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:38.852433    4400 scope.go:122] "RemoveContainer" containerID="c2662c6c96fa2bb6dc157a754b41999ca03b687ca8158aee5a3832e0bce615e4"
Nov 01 03:52:38.852771 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:52:38.852749    4400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2662c6c96fa2bb6dc157a754b41999ca03b687ca8158aee5a3832e0bce615e4\": container with ID starting with c2662c6c96fa2bb6dc157a754b41999ca03b687ca8158aee5a3832e0bce615e4 not found: ID does not exist" containerID="c2662c6c96fa2bb6dc157a754b41999ca03b687ca8158aee5a3832e0bce615e4"
Nov 01 03:52:38.852845 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:38.852782    4400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2662c6c96fa2bb6dc157a754b41999ca03b687ca8158aee5a3832e0bce615e4"} err="failed to get container status \"c2662c6c96fa2bb6dc157a754b41999ca03b687ca8158aee5a3832e0bce615e4\": rpc error: code = NotFound desc = could not find container \"c2662c6c96fa2bb6dc157a754b41999ca03b687ca8158aee5a3832e0bce615e4\": container with ID starting with c2662c6c96fa2bb6dc157a754b41999ca03b687ca8158aee5a3832e0bce615e4 not found: ID does not exist"
Nov 01 03:52:38.852845 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:38.852797    4400 scope.go:122] "RemoveContainer" containerID="ca7234f808bb82dd2442bdf2c2bfba1bc875cde71f5216a7f6847d34d3c5fb82"
Nov 01 03:52:38.860806 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:38.860788    4400 scope.go:122] "RemoveContainer" containerID="ca7234f808bb82dd2442bdf2c2bfba1bc875cde71f5216a7f6847d34d3c5fb82"
Nov 01 03:52:38.861055 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:52:38.861039    4400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca7234f808bb82dd2442bdf2c2bfba1bc875cde71f5216a7f6847d34d3c5fb82\": container with ID starting with ca7234f808bb82dd2442bdf2c2bfba1bc875cde71f5216a7f6847d34d3c5fb82 not found: ID does not exist" containerID="ca7234f808bb82dd2442bdf2c2bfba1bc875cde71f5216a7f6847d34d3c5fb82"
Nov 01 03:52:38.861098 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:38.861061    4400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca7234f808bb82dd2442bdf2c2bfba1bc875cde71f5216a7f6847d34d3c5fb82"} err="failed to get container status \"ca7234f808bb82dd2442bdf2c2bfba1bc875cde71f5216a7f6847d34d3c5fb82\": rpc error: code = NotFound desc = could not find container \"ca7234f808bb82dd2442bdf2c2bfba1bc875cde71f5216a7f6847d34d3c5fb82\": container with ID starting with ca7234f808bb82dd2442bdf2c2bfba1bc875cde71f5216a7f6847d34d3c5fb82 not found: ID does not exist"
Nov 01 03:52:39.013281 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:39.013256    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-ded2d7-748487bb8c-9w7mw"]
Nov 01 03:52:39.017307 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:39.017286    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-ded2d7-748487bb8c-9w7mw"]
Nov 01 03:52:39.173145 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:39.173093    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-ded2d7-54f99bddc-gjh62"]
Nov 01 03:52:39.179072 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:39.179052    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-ded2d7-54f99bddc-gjh62"]
Nov 01 03:52:39.613968 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:39.613941    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="466179e3-7600-464e-9f7c-ec099bc8086d" path="/var/lib/kubelet/pods/466179e3-7600-464e-9f7c-ec099bc8086d/volumes"
Nov 01 03:52:39.614459 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:39.614442    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c1732ba-87a1-43cf-92d6-5edf0869787f" path="/var/lib/kubelet/pods/7c1732ba-87a1-43cf-92d6-5edf0869787f/volumes"
Nov 01 03:52:45.456648 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:45.456615    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-b12199-9c46654f4-j5wwb"]
Nov 01 03:52:45.457058 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:45.456804    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b12199-9c46654f4-j5wwb" podUID="a0737af1-9631-4f69-ba0d-18e6cd2765d1" containerName="maas-api" containerID="cri-o://70edca7784a169295f94713bbab5f5dc0becbb0ba9a39106f15feb2ea987bc74" gracePeriod=30
Nov 01 03:52:45.725429 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:45.725406    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b12199-9c46654f4-j5wwb"
Nov 01 03:52:45.829577 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:45.829547    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0737af1-9631-4f69-ba0d-18e6cd2765d1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0737af1-9631-4f69-ba0d-18e6cd2765d1-maas-api-tls\") pod \"a0737af1-9631-4f69-ba0d-18e6cd2765d1\" (UID: \"a0737af1-9631-4f69-ba0d-18e6cd2765d1\") "
Nov 01 03:52:45.829808 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:45.829789    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0737af1-9631-4f69-ba0d-18e6cd2765d1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0737af1-9631-4f69-ba0d-18e6cd2765d1-metrics-certs\") pod \"a0737af1-9631-4f69-ba0d-18e6cd2765d1\" (UID: \"a0737af1-9631-4f69-ba0d-18e6cd2765d1\") "
Nov 01 03:52:45.829899 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:45.829862    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0737af1-9631-4f69-ba0d-18e6cd2765d1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0737af1-9631-4f69-ba0d-18e6cd2765d1-openshift-service-ca\") pod \"a0737af1-9631-4f69-ba0d-18e6cd2765d1\" (UID: \"a0737af1-9631-4f69-ba0d-18e6cd2765d1\") "
Nov 01 03:52:45.829956 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:45.829894    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0737af1-9631-4f69-ba0d-18e6cd2765d1-kube-api-access-cxnfw\" (UniqueName: \"kubernetes.io/projected/a0737af1-9631-4f69-ba0d-18e6cd2765d1-kube-api-access-cxnfw\") pod \"a0737af1-9631-4f69-ba0d-18e6cd2765d1\" (UID: \"a0737af1-9631-4f69-ba0d-18e6cd2765d1\") "
Nov 01 03:52:45.830371 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:45.830339    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0737af1-9631-4f69-ba0d-18e6cd2765d1-openshift-service-ca" pod "a0737af1-9631-4f69-ba0d-18e6cd2765d1" (UID: "a0737af1-9631-4f69-ba0d-18e6cd2765d1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:52:45.832815 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:45.832785    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0737af1-9631-4f69-ba0d-18e6cd2765d1-maas-api-tls" pod "a0737af1-9631-4f69-ba0d-18e6cd2765d1" (UID: "a0737af1-9631-4f69-ba0d-18e6cd2765d1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:52:45.833224 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:45.833182    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0737af1-9631-4f69-ba0d-18e6cd2765d1-metrics-certs" pod "a0737af1-9631-4f69-ba0d-18e6cd2765d1" (UID: "a0737af1-9631-4f69-ba0d-18e6cd2765d1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:52:45.833333 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:45.833238    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0737af1-9631-4f69-ba0d-18e6cd2765d1-kube-api-access-cxnfw" pod "a0737af1-9631-4f69-ba0d-18e6cd2765d1" (UID: "a0737af1-9631-4f69-ba0d-18e6cd2765d1"). InnerVolumeSpecName "kube-api-access-cxnfw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:52:45.886441 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:45.886409    4400 generic.go:356] "Generic (PLEG): container finished" podID="a0737af1-9631-4f69-ba0d-18e6cd2765d1" containerID="70edca7784a169295f94713bbab5f5dc0becbb0ba9a39106f15feb2ea987bc74" exitCode=0
Nov 01 03:52:45.886558 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:45.886449    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b12199-9c46654f4-j5wwb"
Nov 01 03:52:45.886558 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:45.886485    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b12199-9c46654f4-j5wwb" event={"ID":"a0737af1-9631-4f69-ba0d-18e6cd2765d1","Type":"ContainerDied","Data":"70edca7784a169295f94713bbab5f5dc0becbb0ba9a39106f15feb2ea987bc74"}
Nov 01 03:52:45.886558 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:45.886541    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b12199-9c46654f4-j5wwb" event={"ID":"a0737af1-9631-4f69-ba0d-18e6cd2765d1","Type":"ContainerDied","Data":"bc42d49ad26e64b2895f3fe62119a0ff1b8873f6dedf36fb682acc12051cafce"}
Nov 01 03:52:45.886722 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:45.886561    4400 scope.go:122] "RemoveContainer" containerID="70edca7784a169295f94713bbab5f5dc0becbb0ba9a39106f15feb2ea987bc74"
Nov 01 03:52:45.896360 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:45.896342    4400 scope.go:122] "RemoveContainer" containerID="70edca7784a169295f94713bbab5f5dc0becbb0ba9a39106f15feb2ea987bc74"
Nov 01 03:52:45.896636 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:52:45.896612    4400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70edca7784a169295f94713bbab5f5dc0becbb0ba9a39106f15feb2ea987bc74\": container with ID starting with 70edca7784a169295f94713bbab5f5dc0becbb0ba9a39106f15feb2ea987bc74 not found: ID does not exist" containerID="70edca7784a169295f94713bbab5f5dc0becbb0ba9a39106f15feb2ea987bc74"
Nov 01 03:52:45.896702 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:45.896645    4400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70edca7784a169295f94713bbab5f5dc0becbb0ba9a39106f15feb2ea987bc74"} err="failed to get container status \"70edca7784a169295f94713bbab5f5dc0becbb0ba9a39106f15feb2ea987bc74\": rpc error: code = NotFound desc = could not find container \"70edca7784a169295f94713bbab5f5dc0becbb0ba9a39106f15feb2ea987bc74\": container with ID starting with 70edca7784a169295f94713bbab5f5dc0becbb0ba9a39106f15feb2ea987bc74 not found: ID does not exist"
Nov 01 03:52:45.931471 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:45.931446    4400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0737af1-9631-4f69-ba0d-18e6cd2765d1-metrics-certs\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:52:45.931471 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:45.931472    4400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0737af1-9631-4f69-ba0d-18e6cd2765d1-openshift-service-ca\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:52:45.931603 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:45.931485    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxnfw\" (UniqueName: \"kubernetes.io/projected/a0737af1-9631-4f69-ba0d-18e6cd2765d1-kube-api-access-cxnfw\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:52:45.931603 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:45.931498    4400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0737af1-9631-4f69-ba0d-18e6cd2765d1-maas-api-tls\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:52:46.073757 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:46.066813    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-b12199-9c46654f4-j5wwb"]
Nov 01 03:52:46.073757 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:46.071276    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-b12199-9c46654f4-j5wwb"]
Nov 01 03:52:46.828222 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:46.828185    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-7cd000-795dd66c4d-l77dz"]
Nov 01 03:52:46.828649 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:46.828397    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-7cd000-795dd66c4d-l77dz" podUID="d2550fb6-41b8-4237-b186-237684022a2a" containerName="maas-api" containerID="cri-o://e69c737a5c1aa8d30d17e262e0bc9134c8d8d72c805cdc0ff43b1fbe82a9d7cf" gracePeriod=30
Nov 01 03:52:47.161315 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:47.161290    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-7cd000-795dd66c4d-l77dz"
Nov 01 03:52:47.321082 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:47.321047    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2550fb6-41b8-4237-b186-237684022a2a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2550fb6-41b8-4237-b186-237684022a2a-maas-api-tls\") pod \"d2550fb6-41b8-4237-b186-237684022a2a\" (UID: \"d2550fb6-41b8-4237-b186-237684022a2a\") "
Nov 01 03:52:47.321255 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:47.321126    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2550fb6-41b8-4237-b186-237684022a2a-kube-api-access-9gm7k\" (UniqueName: \"kubernetes.io/projected/d2550fb6-41b8-4237-b186-237684022a2a-kube-api-access-9gm7k\") pod \"d2550fb6-41b8-4237-b186-237684022a2a\" (UID: \"d2550fb6-41b8-4237-b186-237684022a2a\") "
Nov 01 03:52:47.321333 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:47.321296    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2550fb6-41b8-4237-b186-237684022a2a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2550fb6-41b8-4237-b186-237684022a2a-openshift-service-ca\") pod \"d2550fb6-41b8-4237-b186-237684022a2a\" (UID: \"d2550fb6-41b8-4237-b186-237684022a2a\") "
Nov 01 03:52:47.321333 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:47.321325    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2550fb6-41b8-4237-b186-237684022a2a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2550fb6-41b8-4237-b186-237684022a2a-metrics-certs\") pod \"d2550fb6-41b8-4237-b186-237684022a2a\" (UID: \"d2550fb6-41b8-4237-b186-237684022a2a\") "
Nov 01 03:52:47.322218 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:47.322186    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2550fb6-41b8-4237-b186-237684022a2a-openshift-service-ca" pod "d2550fb6-41b8-4237-b186-237684022a2a" (UID: "d2550fb6-41b8-4237-b186-237684022a2a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:52:47.323535 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:47.323511    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2550fb6-41b8-4237-b186-237684022a2a-maas-api-tls" pod "d2550fb6-41b8-4237-b186-237684022a2a" (UID: "d2550fb6-41b8-4237-b186-237684022a2a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:52:47.323641 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:47.323584    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2550fb6-41b8-4237-b186-237684022a2a-kube-api-access-9gm7k" pod "d2550fb6-41b8-4237-b186-237684022a2a" (UID: "d2550fb6-41b8-4237-b186-237684022a2a"). InnerVolumeSpecName "kube-api-access-9gm7k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:52:47.324137 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:47.324118    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2550fb6-41b8-4237-b186-237684022a2a-metrics-certs" pod "d2550fb6-41b8-4237-b186-237684022a2a" (UID: "d2550fb6-41b8-4237-b186-237684022a2a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:52:47.428757 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:47.427760    4400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2550fb6-41b8-4237-b186-237684022a2a-openshift-service-ca\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:52:47.428757 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:47.427810    4400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2550fb6-41b8-4237-b186-237684022a2a-metrics-certs\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:52:47.428757 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:47.427823    4400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2550fb6-41b8-4237-b186-237684022a2a-maas-api-tls\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:52:47.428757 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:47.427835    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gm7k\" (UniqueName: \"kubernetes.io/projected/d2550fb6-41b8-4237-b186-237684022a2a-kube-api-access-9gm7k\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:52:47.618307 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:47.618273    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0737af1-9631-4f69-ba0d-18e6cd2765d1" path="/var/lib/kubelet/pods/a0737af1-9631-4f69-ba0d-18e6cd2765d1/volumes"
Nov 01 03:52:47.903775 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:47.903743    4400 generic.go:356] "Generic (PLEG): container finished" podID="d2550fb6-41b8-4237-b186-237684022a2a" containerID="e69c737a5c1aa8d30d17e262e0bc9134c8d8d72c805cdc0ff43b1fbe82a9d7cf" exitCode=0
Nov 01 03:52:47.904250 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:47.903788    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-7cd000-795dd66c4d-l77dz"
Nov 01 03:52:47.904250 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:47.903827    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-7cd000-795dd66c4d-l77dz" event={"ID":"d2550fb6-41b8-4237-b186-237684022a2a","Type":"ContainerDied","Data":"e69c737a5c1aa8d30d17e262e0bc9134c8d8d72c805cdc0ff43b1fbe82a9d7cf"}
Nov 01 03:52:47.904250 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:47.903889    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-7cd000-795dd66c4d-l77dz" event={"ID":"d2550fb6-41b8-4237-b186-237684022a2a","Type":"ContainerDied","Data":"92313d051f53f4b188a03d4fa2fb3a0e36c711cdb646bff46087d4a76417531e"}
Nov 01 03:52:47.904250 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:47.903902    4400 scope.go:122] "RemoveContainer" containerID="e69c737a5c1aa8d30d17e262e0bc9134c8d8d72c805cdc0ff43b1fbe82a9d7cf"
Nov 01 03:52:47.913307 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:47.913290    4400 scope.go:122] "RemoveContainer" containerID="e69c737a5c1aa8d30d17e262e0bc9134c8d8d72c805cdc0ff43b1fbe82a9d7cf"
Nov 01 03:52:47.913594 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:52:47.913550    4400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e69c737a5c1aa8d30d17e262e0bc9134c8d8d72c805cdc0ff43b1fbe82a9d7cf\": container with ID starting with e69c737a5c1aa8d30d17e262e0bc9134c8d8d72c805cdc0ff43b1fbe82a9d7cf not found: ID does not exist" containerID="e69c737a5c1aa8d30d17e262e0bc9134c8d8d72c805cdc0ff43b1fbe82a9d7cf"
Nov 01 03:52:47.913751 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:47.913590    4400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e69c737a5c1aa8d30d17e262e0bc9134c8d8d72c805cdc0ff43b1fbe82a9d7cf"} err="failed to get container status \"e69c737a5c1aa8d30d17e262e0bc9134c8d8d72c805cdc0ff43b1fbe82a9d7cf\": rpc error: code = NotFound desc = could not find container \"e69c737a5c1aa8d30d17e262e0bc9134c8d8d72c805cdc0ff43b1fbe82a9d7cf\": container with ID starting with e69c737a5c1aa8d30d17e262e0bc9134c8d8d72c805cdc0ff43b1fbe82a9d7cf not found: ID does not exist"
Nov 01 03:52:48.077123 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:48.077097    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-7cd000-795dd66c4d-l77dz"]
Nov 01 03:52:48.128570 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:48.128539    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-7cd000-795dd66c4d-l77dz"]
Nov 01 03:52:49.614095 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:49.614061    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2550fb6-41b8-4237-b186-237684022a2a" path="/var/lib/kubelet/pods/d2550fb6-41b8-4237-b186-237684022a2a/volumes"
Nov 01 03:52:52.201054 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:52.201021    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 03:52:53.408421 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:53.408387    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d84de6-6f4ddfb696-m8smn"]
Nov 01 03:52:53.408907 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:53.408648    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d84de6-6f4ddfb696-m8smn" podUID="c0e53b77-f9a5-4c9b-9ccd-84d6d30c35e4" containerName="maas-api" containerID="cri-o://8a6902ae588681145eb3e271bc8227e201e7b491a14e3881ac5218fda0f616fe" gracePeriod=30
Nov 01 03:52:53.672430 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:53.672407    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d84de6-6f4ddfb696-m8smn"
Nov 01 03:52:53.699763 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:53.699741    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0e53b77-f9a5-4c9b-9ccd-84d6d30c35e4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0e53b77-f9a5-4c9b-9ccd-84d6d30c35e4-maas-api-tls\") pod \"c0e53b77-f9a5-4c9b-9ccd-84d6d30c35e4\" (UID: \"c0e53b77-f9a5-4c9b-9ccd-84d6d30c35e4\") "
Nov 01 03:52:53.699868 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:53.699816    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0e53b77-f9a5-4c9b-9ccd-84d6d30c35e4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0e53b77-f9a5-4c9b-9ccd-84d6d30c35e4-openshift-service-ca\") pod \"c0e53b77-f9a5-4c9b-9ccd-84d6d30c35e4\" (UID: \"c0e53b77-f9a5-4c9b-9ccd-84d6d30c35e4\") "
Nov 01 03:52:53.699905 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:53.699876    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0e53b77-f9a5-4c9b-9ccd-84d6d30c35e4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0e53b77-f9a5-4c9b-9ccd-84d6d30c35e4-metrics-certs\") pod \"c0e53b77-f9a5-4c9b-9ccd-84d6d30c35e4\" (UID: \"c0e53b77-f9a5-4c9b-9ccd-84d6d30c35e4\") "
Nov 01 03:52:53.699905 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:53.699903    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0e53b77-f9a5-4c9b-9ccd-84d6d30c35e4-kube-api-access-q8vrn\" (UniqueName: \"kubernetes.io/projected/c0e53b77-f9a5-4c9b-9ccd-84d6d30c35e4-kube-api-access-q8vrn\") pod \"c0e53b77-f9a5-4c9b-9ccd-84d6d30c35e4\" (UID: \"c0e53b77-f9a5-4c9b-9ccd-84d6d30c35e4\") "
Nov 01 03:52:53.700136 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:53.700109    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0e53b77-f9a5-4c9b-9ccd-84d6d30c35e4-openshift-service-ca" pod "c0e53b77-f9a5-4c9b-9ccd-84d6d30c35e4" (UID: "c0e53b77-f9a5-4c9b-9ccd-84d6d30c35e4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:52:53.700281 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:53.700261    4400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0e53b77-f9a5-4c9b-9ccd-84d6d30c35e4-openshift-service-ca\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:52:53.702056 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:53.702024    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0e53b77-f9a5-4c9b-9ccd-84d6d30c35e4-metrics-certs" pod "c0e53b77-f9a5-4c9b-9ccd-84d6d30c35e4" (UID: "c0e53b77-f9a5-4c9b-9ccd-84d6d30c35e4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:52:53.702169 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:53.702096    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0e53b77-f9a5-4c9b-9ccd-84d6d30c35e4-maas-api-tls" pod "c0e53b77-f9a5-4c9b-9ccd-84d6d30c35e4" (UID: "c0e53b77-f9a5-4c9b-9ccd-84d6d30c35e4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:52:53.702169 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:53.702117    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0e53b77-f9a5-4c9b-9ccd-84d6d30c35e4-kube-api-access-q8vrn" pod "c0e53b77-f9a5-4c9b-9ccd-84d6d30c35e4" (UID: "c0e53b77-f9a5-4c9b-9ccd-84d6d30c35e4"). InnerVolumeSpecName "kube-api-access-q8vrn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:52:53.801269 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:53.801190    4400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0e53b77-f9a5-4c9b-9ccd-84d6d30c35e4-metrics-certs\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:52:53.801269 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:53.801228    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8vrn\" (UniqueName: \"kubernetes.io/projected/c0e53b77-f9a5-4c9b-9ccd-84d6d30c35e4-kube-api-access-q8vrn\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:52:53.801269 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:53.801242    4400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0e53b77-f9a5-4c9b-9ccd-84d6d30c35e4-maas-api-tls\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:52:53.944151 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:53.944061    4400 generic.go:356] "Generic (PLEG): container finished" podID="c0e53b77-f9a5-4c9b-9ccd-84d6d30c35e4" containerID="8a6902ae588681145eb3e271bc8227e201e7b491a14e3881ac5218fda0f616fe" exitCode=0
Nov 01 03:52:53.944151 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:53.944104    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d84de6-6f4ddfb696-m8smn"
Nov 01 03:52:53.944151 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:53.944118    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d84de6-6f4ddfb696-m8smn" event={"ID":"c0e53b77-f9a5-4c9b-9ccd-84d6d30c35e4","Type":"ContainerDied","Data":"8a6902ae588681145eb3e271bc8227e201e7b491a14e3881ac5218fda0f616fe"}
Nov 01 03:52:53.944394 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:53.944173    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d84de6-6f4ddfb696-m8smn" event={"ID":"c0e53b77-f9a5-4c9b-9ccd-84d6d30c35e4","Type":"ContainerDied","Data":"5de5d52f4d7383fc6a89bdebe2390bd354d721f6b4ba9bb3f3211cdfcbad762e"}
Nov 01 03:52:53.944394 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:53.944202    4400 scope.go:122] "RemoveContainer" containerID="8a6902ae588681145eb3e271bc8227e201e7b491a14e3881ac5218fda0f616fe"
Nov 01 03:52:53.954731 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:53.954713    4400 scope.go:122] "RemoveContainer" containerID="8a6902ae588681145eb3e271bc8227e201e7b491a14e3881ac5218fda0f616fe"
Nov 01 03:52:53.954992 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:52:53.954974    4400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a6902ae588681145eb3e271bc8227e201e7b491a14e3881ac5218fda0f616fe\": container with ID starting with 8a6902ae588681145eb3e271bc8227e201e7b491a14e3881ac5218fda0f616fe not found: ID does not exist" containerID="8a6902ae588681145eb3e271bc8227e201e7b491a14e3881ac5218fda0f616fe"
Nov 01 03:52:53.955053 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:53.955000    4400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a6902ae588681145eb3e271bc8227e201e7b491a14e3881ac5218fda0f616fe"} err="failed to get container status \"8a6902ae588681145eb3e271bc8227e201e7b491a14e3881ac5218fda0f616fe\": rpc error: code = NotFound desc = could not find container \"8a6902ae588681145eb3e271bc8227e201e7b491a14e3881ac5218fda0f616fe\": container with ID starting with 8a6902ae588681145eb3e271bc8227e201e7b491a14e3881ac5218fda0f616fe not found: ID does not exist"
Nov 01 03:52:54.112146 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:54.112120    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d84de6-6f4ddfb696-m8smn"]
Nov 01 03:52:54.115769 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:54.115750    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d84de6-6f4ddfb696-m8smn"]
Nov 01 03:52:55.614856 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:52:55.614827    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0e53b77-f9a5-4c9b-9ccd-84d6d30c35e4" path="/var/lib/kubelet/pods/c0e53b77-f9a5-4c9b-9ccd-84d6d30c35e4/volumes"
Nov 01 03:53:01.728175 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:53:01.728139    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:53:08.285546 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:08.285511    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-c22dd8/distinct-w5-c22dd8-kserve-5c65c9bcd7-nnf7d"]
Nov 01 03:53:09.994908 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:09.994877    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-211380-7d964dd845-2blzj"]
Nov 01 03:53:10.113729 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:10.113701    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-211380-598d8b59f-gs5lv"]
Nov 01 03:53:11.621381 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:11.621354    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-211380-5b58f48f46-7r8nn"]
Nov 01 03:53:11.621885 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:11.621507    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-211380-5b58f48f46-7r8nn" podUID="21b19cc7-731a-45fb-b864-da76b9b41be0" containerName="maas-api" containerID="cri-o://8d161bda216feb0fc151394f7b0bdb6eca47125ba1651f0101f06762bfc19049" gracePeriod=30
Nov 01 03:53:11.678663 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:11.678635    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 03:53:11.906600 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:11.906229    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-211380-5b58f48f46-7r8nn"
Nov 01 03:53:11.997819 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:11.997781    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21b19cc7-731a-45fb-b864-da76b9b41be0-kube-api-access-pkztn\" (UniqueName: \"kubernetes.io/projected/21b19cc7-731a-45fb-b864-da76b9b41be0-kube-api-access-pkztn\") pod \"21b19cc7-731a-45fb-b864-da76b9b41be0\" (UID: \"21b19cc7-731a-45fb-b864-da76b9b41be0\") "
Nov 01 03:53:11.998007 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:11.997912    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21b19cc7-731a-45fb-b864-da76b9b41be0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21b19cc7-731a-45fb-b864-da76b9b41be0-openshift-service-ca\") pod \"21b19cc7-731a-45fb-b864-da76b9b41be0\" (UID: \"21b19cc7-731a-45fb-b864-da76b9b41be0\") "
Nov 01 03:53:11.998007 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:11.997950    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21b19cc7-731a-45fb-b864-da76b9b41be0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/21b19cc7-731a-45fb-b864-da76b9b41be0-metrics-certs\") pod \"21b19cc7-731a-45fb-b864-da76b9b41be0\" (UID: \"21b19cc7-731a-45fb-b864-da76b9b41be0\") "
Nov 01 03:53:11.998007 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:11.997994    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21b19cc7-731a-45fb-b864-da76b9b41be0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21b19cc7-731a-45fb-b864-da76b9b41be0-maas-api-tls\") pod \"21b19cc7-731a-45fb-b864-da76b9b41be0\" (UID: \"21b19cc7-731a-45fb-b864-da76b9b41be0\") "
Nov 01 03:53:11.998326 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:11.998281    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21b19cc7-731a-45fb-b864-da76b9b41be0-openshift-service-ca" pod "21b19cc7-731a-45fb-b864-da76b9b41be0" (UID: "21b19cc7-731a-45fb-b864-da76b9b41be0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:53:11.998505 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:11.998479    4400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21b19cc7-731a-45fb-b864-da76b9b41be0-openshift-service-ca\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:12.000532 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:12.000476    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21b19cc7-731a-45fb-b864-da76b9b41be0-maas-api-tls" pod "21b19cc7-731a-45fb-b864-da76b9b41be0" (UID: "21b19cc7-731a-45fb-b864-da76b9b41be0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:53:12.000649 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:12.000628    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21b19cc7-731a-45fb-b864-da76b9b41be0-kube-api-access-pkztn" pod "21b19cc7-731a-45fb-b864-da76b9b41be0" (UID: "21b19cc7-731a-45fb-b864-da76b9b41be0"). InnerVolumeSpecName "kube-api-access-pkztn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:53:12.000772 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:12.000724    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21b19cc7-731a-45fb-b864-da76b9b41be0-metrics-certs" pod "21b19cc7-731a-45fb-b864-da76b9b41be0" (UID: "21b19cc7-731a-45fb-b864-da76b9b41be0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:53:12.062131 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:12.062095    4400 generic.go:356] "Generic (PLEG): container finished" podID="21b19cc7-731a-45fb-b864-da76b9b41be0" containerID="8d161bda216feb0fc151394f7b0bdb6eca47125ba1651f0101f06762bfc19049" exitCode=0
Nov 01 03:53:12.062306 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:12.062267    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-211380-5b58f48f46-7r8nn" event={"ID":"21b19cc7-731a-45fb-b864-da76b9b41be0","Type":"ContainerDied","Data":"8d161bda216feb0fc151394f7b0bdb6eca47125ba1651f0101f06762bfc19049"}
Nov 01 03:53:12.062371 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:12.062317    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-211380-5b58f48f46-7r8nn" event={"ID":"21b19cc7-731a-45fb-b864-da76b9b41be0","Type":"ContainerDied","Data":"8d6d4e6b0fd6c347263f9ce4308d48eac8a1750266489ae7f0b4b8249b721822"}
Nov 01 03:53:12.062371 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:12.062346    4400 scope.go:122] "RemoveContainer" containerID="8d161bda216feb0fc151394f7b0bdb6eca47125ba1651f0101f06762bfc19049"
Nov 01 03:53:12.062515 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:12.062501    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-211380-5b58f48f46-7r8nn"
Nov 01 03:53:12.078885 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:12.078863    4400 scope.go:122] "RemoveContainer" containerID="8d161bda216feb0fc151394f7b0bdb6eca47125ba1651f0101f06762bfc19049"
Nov 01 03:53:12.079163 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:53:12.079142    4400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d161bda216feb0fc151394f7b0bdb6eca47125ba1651f0101f06762bfc19049\": container with ID starting with 8d161bda216feb0fc151394f7b0bdb6eca47125ba1651f0101f06762bfc19049 not found: ID does not exist" containerID="8d161bda216feb0fc151394f7b0bdb6eca47125ba1651f0101f06762bfc19049"
Nov 01 03:53:12.079206 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:12.079173    4400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d161bda216feb0fc151394f7b0bdb6eca47125ba1651f0101f06762bfc19049"} err="failed to get container status \"8d161bda216feb0fc151394f7b0bdb6eca47125ba1651f0101f06762bfc19049\": rpc error: code = NotFound desc = could not find container \"8d161bda216feb0fc151394f7b0bdb6eca47125ba1651f0101f06762bfc19049\": container with ID starting with 8d161bda216feb0fc151394f7b0bdb6eca47125ba1651f0101f06762bfc19049 not found: ID does not exist"
Nov 01 03:53:12.099900 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:12.099879    4400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21b19cc7-731a-45fb-b864-da76b9b41be0-metrics-certs\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:12.100020 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:12.099903    4400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21b19cc7-731a-45fb-b864-da76b9b41be0-maas-api-tls\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:12.100020 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:12.099917    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkztn\" (UniqueName: \"kubernetes.io/projected/21b19cc7-731a-45fb-b864-da76b9b41be0-kube-api-access-pkztn\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:12.233253 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:12.233165    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-211380-5b58f48f46-7r8nn"]
Nov 01 03:53:12.237002 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:12.236976    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-211380-5b58f48f46-7r8nn"]
Nov 01 03:53:13.615555 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:13.615525    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21b19cc7-731a-45fb-b864-da76b9b41be0" path="/var/lib/kubelet/pods/21b19cc7-731a-45fb-b864-da76b9b41be0/volumes"
Nov 01 03:53:14.995448 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:14.995414    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-211380-7d964dd845-2blzj" podUID="0a5c6392-1b3d-4823-a5fa-f91005df6759" containerName="payload-processing" containerID="cri-o://745e0f7dbdb631f410a4d8da5d2a8b2505a246b25322f6411cf68b85030848bf" gracePeriod=25
Nov 01 03:53:15.114834 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:15.114806    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-211380-598d8b59f-gs5lv" podUID="bc0b9a54-e2e5-44e2-bb9f-62949ff3db59" containerName="payload-pre-processing" containerID="cri-o://5596f24ad7026543a059004bcd039a00dc996e096fe2bf55de0c7f51b6242e5f" gracePeriod=25
Nov 01 03:53:15.251574 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:15.251521    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-211380-7d964dd845-2blzj"
Nov 01 03:53:15.308951 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:15.308920    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-c22dd8-6986b5b4cf-r9pws"]
Nov 01 03:53:15.331701 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:15.331656    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a5c6392-1b3d-4823-a5fa-f91005df6759-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a5c6392-1b3d-4823-a5fa-f91005df6759-plugins-config-volume\") pod \"0a5c6392-1b3d-4823-a5fa-f91005df6759\" (UID: \"0a5c6392-1b3d-4823-a5fa-f91005df6759\") "
Nov 01 03:53:15.331840 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:15.331744    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a5c6392-1b3d-4823-a5fa-f91005df6759-kube-api-access-xppfh\" (UniqueName: \"kubernetes.io/projected/0a5c6392-1b3d-4823-a5fa-f91005df6759-kube-api-access-xppfh\") pod \"0a5c6392-1b3d-4823-a5fa-f91005df6759\" (UID: \"0a5c6392-1b3d-4823-a5fa-f91005df6759\") "
Nov 01 03:53:15.331994 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:15.331972    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a5c6392-1b3d-4823-a5fa-f91005df6759-plugins-config-volume" pod "0a5c6392-1b3d-4823-a5fa-f91005df6759" (UID: "0a5c6392-1b3d-4823-a5fa-f91005df6759"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:53:15.332240 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:15.332222    4400 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a5c6392-1b3d-4823-a5fa-f91005df6759-plugins-config-volume\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:15.334826 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:15.334799    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a5c6392-1b3d-4823-a5fa-f91005df6759-kube-api-access-xppfh" pod "0a5c6392-1b3d-4823-a5fa-f91005df6759" (UID: "0a5c6392-1b3d-4823-a5fa-f91005df6759"). InnerVolumeSpecName "kube-api-access-xppfh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:53:15.366833 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:15.366811    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-211380-598d8b59f-gs5lv"
Nov 01 03:53:15.432833 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:15.432806    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc0b9a54-e2e5-44e2-bb9f-62949ff3db59-kube-api-access-fgsx7\" (UniqueName: \"kubernetes.io/projected/bc0b9a54-e2e5-44e2-bb9f-62949ff3db59-kube-api-access-fgsx7\") pod \"bc0b9a54-e2e5-44e2-bb9f-62949ff3db59\" (UID: \"bc0b9a54-e2e5-44e2-bb9f-62949ff3db59\") "
Nov 01 03:53:15.432998 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:15.432975    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc0b9a54-e2e5-44e2-bb9f-62949ff3db59-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc0b9a54-e2e5-44e2-bb9f-62949ff3db59-plugins-config-volume\") pod \"bc0b9a54-e2e5-44e2-bb9f-62949ff3db59\" (UID: \"bc0b9a54-e2e5-44e2-bb9f-62949ff3db59\") "
Nov 01 03:53:15.433283 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:15.433266    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xppfh\" (UniqueName: \"kubernetes.io/projected/0a5c6392-1b3d-4823-a5fa-f91005df6759-kube-api-access-xppfh\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:15.433350 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:15.433275    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc0b9a54-e2e5-44e2-bb9f-62949ff3db59-plugins-config-volume" pod "bc0b9a54-e2e5-44e2-bb9f-62949ff3db59" (UID: "bc0b9a54-e2e5-44e2-bb9f-62949ff3db59"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:53:15.435328 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:15.435307    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc0b9a54-e2e5-44e2-bb9f-62949ff3db59-kube-api-access-fgsx7" pod "bc0b9a54-e2e5-44e2-bb9f-62949ff3db59" (UID: "bc0b9a54-e2e5-44e2-bb9f-62949ff3db59"). InnerVolumeSpecName "kube-api-access-fgsx7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:53:15.534864 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:15.534801    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgsx7\" (UniqueName: \"kubernetes.io/projected/bc0b9a54-e2e5-44e2-bb9f-62949ff3db59-kube-api-access-fgsx7\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:15.534864 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:15.534825    4400 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc0b9a54-e2e5-44e2-bb9f-62949ff3db59-plugins-config-volume\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:16.094019 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:16.093982    4400 generic.go:356] "Generic (PLEG): container finished" podID="0a5c6392-1b3d-4823-a5fa-f91005df6759" containerID="745e0f7dbdb631f410a4d8da5d2a8b2505a246b25322f6411cf68b85030848bf" exitCode=0
Nov 01 03:53:16.094387 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:16.094031    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-211380-7d964dd845-2blzj"
Nov 01 03:53:16.094387 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:16.094061    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-211380-7d964dd845-2blzj" event={"ID":"0a5c6392-1b3d-4823-a5fa-f91005df6759","Type":"ContainerDied","Data":"745e0f7dbdb631f410a4d8da5d2a8b2505a246b25322f6411cf68b85030848bf"}
Nov 01 03:53:16.094387 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:16.094106    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-211380-7d964dd845-2blzj" event={"ID":"0a5c6392-1b3d-4823-a5fa-f91005df6759","Type":"ContainerDied","Data":"0c910f73bf0234d1487d8c7b65bcb996f458848583e5adc2be2938583cc0f4dc"}
Nov 01 03:53:16.094387 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:16.094118    4400 scope.go:122] "RemoveContainer" containerID="745e0f7dbdb631f410a4d8da5d2a8b2505a246b25322f6411cf68b85030848bf"
Nov 01 03:53:16.095534 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:16.095512    4400 generic.go:356] "Generic (PLEG): container finished" podID="bc0b9a54-e2e5-44e2-bb9f-62949ff3db59" containerID="5596f24ad7026543a059004bcd039a00dc996e096fe2bf55de0c7f51b6242e5f" exitCode=0
Nov 01 03:53:16.095621 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:16.095568    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-211380-598d8b59f-gs5lv"
Nov 01 03:53:16.095736 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:16.095710    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-211380-598d8b59f-gs5lv" event={"ID":"bc0b9a54-e2e5-44e2-bb9f-62949ff3db59","Type":"ContainerDied","Data":"5596f24ad7026543a059004bcd039a00dc996e096fe2bf55de0c7f51b6242e5f"}
Nov 01 03:53:16.095891 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:16.095757    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-211380-598d8b59f-gs5lv" event={"ID":"bc0b9a54-e2e5-44e2-bb9f-62949ff3db59","Type":"ContainerDied","Data":"b5e464fb8bb89ceed7eaf1c1fc7310b219d6c8f514e987af2498e4c8130e9427"}
Nov 01 03:53:16.103463 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:16.103449    4400 scope.go:122] "RemoveContainer" containerID="745e0f7dbdb631f410a4d8da5d2a8b2505a246b25322f6411cf68b85030848bf"
Nov 01 03:53:16.103713 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:53:16.103693    4400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"745e0f7dbdb631f410a4d8da5d2a8b2505a246b25322f6411cf68b85030848bf\": container with ID starting with 745e0f7dbdb631f410a4d8da5d2a8b2505a246b25322f6411cf68b85030848bf not found: ID does not exist" containerID="745e0f7dbdb631f410a4d8da5d2a8b2505a246b25322f6411cf68b85030848bf"
Nov 01 03:53:16.103781 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:16.103724    4400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"745e0f7dbdb631f410a4d8da5d2a8b2505a246b25322f6411cf68b85030848bf"} err="failed to get container status \"745e0f7dbdb631f410a4d8da5d2a8b2505a246b25322f6411cf68b85030848bf\": rpc error: code = NotFound desc = could not find container \"745e0f7dbdb631f410a4d8da5d2a8b2505a246b25322f6411cf68b85030848bf\": container with ID starting with 745e0f7dbdb631f410a4d8da5d2a8b2505a246b25322f6411cf68b85030848bf not found: ID does not exist"
Nov 01 03:53:16.103781 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:16.103738    4400 scope.go:122] "RemoveContainer" containerID="5596f24ad7026543a059004bcd039a00dc996e096fe2bf55de0c7f51b6242e5f"
Nov 01 03:53:16.111834 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:16.111818    4400 scope.go:122] "RemoveContainer" containerID="5596f24ad7026543a059004bcd039a00dc996e096fe2bf55de0c7f51b6242e5f"
Nov 01 03:53:16.112059 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:53:16.112041    4400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5596f24ad7026543a059004bcd039a00dc996e096fe2bf55de0c7f51b6242e5f\": container with ID starting with 5596f24ad7026543a059004bcd039a00dc996e096fe2bf55de0c7f51b6242e5f not found: ID does not exist" containerID="5596f24ad7026543a059004bcd039a00dc996e096fe2bf55de0c7f51b6242e5f"
Nov 01 03:53:16.112105 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:16.112066    4400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5596f24ad7026543a059004bcd039a00dc996e096fe2bf55de0c7f51b6242e5f"} err="failed to get container status \"5596f24ad7026543a059004bcd039a00dc996e096fe2bf55de0c7f51b6242e5f\": rpc error: code = NotFound desc = could not find container \"5596f24ad7026543a059004bcd039a00dc996e096fe2bf55de0c7f51b6242e5f\": container with ID starting with 5596f24ad7026543a059004bcd039a00dc996e096fe2bf55de0c7f51b6242e5f not found: ID does not exist"
Nov 01 03:53:16.262817 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:16.262768    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-211380-598d8b59f-gs5lv"]
Nov 01 03:53:16.267848 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:16.267828    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-211380-598d8b59f-gs5lv"]
Nov 01 03:53:16.381180 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:16.381153    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-211380-7d964dd845-2blzj"]
Nov 01 03:53:16.384612 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:16.384593    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-211380-7d964dd845-2blzj"]
Nov 01 03:53:17.613966 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:17.613935    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a5c6392-1b3d-4823-a5fa-f91005df6759" path="/var/lib/kubelet/pods/0a5c6392-1b3d-4823-a5fa-f91005df6759/volumes"
Nov 01 03:53:17.614324 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:17.614313    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc0b9a54-e2e5-44e2-bb9f-62949ff3db59" path="/var/lib/kubelet/pods/bc0b9a54-e2e5-44e2-bb9f-62949ff3db59/volumes"
Nov 01 03:53:19.381194 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:19.381166    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-tl6jm"]
Nov 01 03:53:19.381625 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:19.381372    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-tl6jm" podUID="51a974d4-ac5e-4dc5-b5b4-2fac0f648c06" containerName="istio-proxy" containerID="cri-o://f2a4f73c0b087f7e1b0de473be68dab046bdf6875ceaf7484965722206db7cbe" gracePeriod=30
Nov 01 03:53:20.309638 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:20.309598    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-c22dd8-6986b5b4cf-r9pws" podUID="1c1fcbce-b87d-45df-877f-89394755bd40" containerName="payload-processing" containerID="cri-o://5dd44e49425b514ac9cfbaf1ab6c857adebdbe1429747db6b3d497edb31f0f0e" gracePeriod=25
Nov 01 03:53:23.305982 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:23.305946    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-w5-c22dd8-kserve-5c65c9bcd7-nnf7d" podUID="9cd440c6-7191-4fae-a806-f8728975b3cf" containerName="main" containerID="cri-o://ad9717c2c4f658e6682eb6e046f42a4a1a10e2fc0876e2db5a78c8aacffc45e0" gracePeriod=45
Nov 01 03:53:23.356867 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:23.356839    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-w5-c22dd8-kserve-5c65c9bcd7-nnf7d" podUID="9cd440c6-7191-4fae-a806-f8728975b3cf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": dial tcp 10.132.0.66:8000: connect: connection refused"
Nov 01 03:53:23.548755 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:23.548733    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-w5-c22dd8-kserve-5c65c9bcd7-nnf7d"
Nov 01 03:53:23.640989 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:23.640964    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cd440c6-7191-4fae-a806-f8728975b3cf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cd440c6-7191-4fae-a806-f8728975b3cf-dshm\") pod \"9cd440c6-7191-4fae-a806-f8728975b3cf\" (UID: \"9cd440c6-7191-4fae-a806-f8728975b3cf\") "
Nov 01 03:53:23.641125 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:23.640995    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9cd440c6-7191-4fae-a806-f8728975b3cf-tls-certs\" (UniqueName: \"kubernetes.io/secret/9cd440c6-7191-4fae-a806-f8728975b3cf-tls-certs\") pod \"9cd440c6-7191-4fae-a806-f8728975b3cf\" (UID: \"9cd440c6-7191-4fae-a806-f8728975b3cf\") "
Nov 01 03:53:23.641125 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:23.641042    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cd440c6-7191-4fae-a806-f8728975b3cf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cd440c6-7191-4fae-a806-f8728975b3cf-model-cache\") pod \"9cd440c6-7191-4fae-a806-f8728975b3cf\" (UID: \"9cd440c6-7191-4fae-a806-f8728975b3cf\") "
Nov 01 03:53:23.641125 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:23.641065    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cd440c6-7191-4fae-a806-f8728975b3cf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cd440c6-7191-4fae-a806-f8728975b3cf-tmp-dir\") pod \"9cd440c6-7191-4fae-a806-f8728975b3cf\" (UID: \"9cd440c6-7191-4fae-a806-f8728975b3cf\") "
Nov 01 03:53:23.641296 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:23.641145    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cd440c6-7191-4fae-a806-f8728975b3cf-home\" (UniqueName: \"kubernetes.io/empty-dir/9cd440c6-7191-4fae-a806-f8728975b3cf-home\") pod \"9cd440c6-7191-4fae-a806-f8728975b3cf\" (UID: \"9cd440c6-7191-4fae-a806-f8728975b3cf\") "
Nov 01 03:53:23.641296 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:23.641168    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cd440c6-7191-4fae-a806-f8728975b3cf-kube-api-access-hsr5l\" (UniqueName: \"kubernetes.io/projected/9cd440c6-7191-4fae-a806-f8728975b3cf-kube-api-access-hsr5l\") pod \"9cd440c6-7191-4fae-a806-f8728975b3cf\" (UID: \"9cd440c6-7191-4fae-a806-f8728975b3cf\") "
Nov 01 03:53:23.641399 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:23.641350    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cd440c6-7191-4fae-a806-f8728975b3cf-model-cache" pod "9cd440c6-7191-4fae-a806-f8728975b3cf" (UID: "9cd440c6-7191-4fae-a806-f8728975b3cf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:53:23.641471 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:23.641452    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cd440c6-7191-4fae-a806-f8728975b3cf-tmp-dir" pod "9cd440c6-7191-4fae-a806-f8728975b3cf" (UID: "9cd440c6-7191-4fae-a806-f8728975b3cf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:53:23.641617 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:23.641597    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cd440c6-7191-4fae-a806-f8728975b3cf-home" pod "9cd440c6-7191-4fae-a806-f8728975b3cf" (UID: "9cd440c6-7191-4fae-a806-f8728975b3cf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:53:23.641747 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:23.641729    4400 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cd440c6-7191-4fae-a806-f8728975b3cf-tmp-dir\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:23.641823 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:23.641753    4400 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9cd440c6-7191-4fae-a806-f8728975b3cf-home\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:23.641823 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:23.641764    4400 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cd440c6-7191-4fae-a806-f8728975b3cf-model-cache\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:23.643457 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:23.643437    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cd440c6-7191-4fae-a806-f8728975b3cf-kube-api-access-hsr5l" pod "9cd440c6-7191-4fae-a806-f8728975b3cf" (UID: "9cd440c6-7191-4fae-a806-f8728975b3cf"). InnerVolumeSpecName "kube-api-access-hsr5l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:53:23.643665 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:23.643645    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cd440c6-7191-4fae-a806-f8728975b3cf-dshm" pod "9cd440c6-7191-4fae-a806-f8728975b3cf" (UID: "9cd440c6-7191-4fae-a806-f8728975b3cf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:53:23.643665 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:23.643653    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cd440c6-7191-4fae-a806-f8728975b3cf-tls-certs" pod "9cd440c6-7191-4fae-a806-f8728975b3cf" (UID: "9cd440c6-7191-4fae-a806-f8728975b3cf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:53:23.743538 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:23.743495    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsr5l\" (UniqueName: \"kubernetes.io/projected/9cd440c6-7191-4fae-a806-f8728975b3cf-kube-api-access-hsr5l\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:23.743538 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:23.743538    4400 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cd440c6-7191-4fae-a806-f8728975b3cf-dshm\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:23.743811 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:23.743554    4400 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cd440c6-7191-4fae-a806-f8728975b3cf-tls-certs\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:23.885662 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:23.885627    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-c22dd8-6986b5b4cf-r9pws" podUID="1c1fcbce-b87d-45df-877f-89394755bd40" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 03:53:24.165263 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:24.165231    4400 generic.go:356] "Generic (PLEG): container finished" podID="9cd440c6-7191-4fae-a806-f8728975b3cf" containerID="ad9717c2c4f658e6682eb6e046f42a4a1a10e2fc0876e2db5a78c8aacffc45e0" exitCode=0
Nov 01 03:53:24.165431 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:24.165290    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-w5-c22dd8-kserve-5c65c9bcd7-nnf7d"
Nov 01 03:53:24.165431 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:24.165317    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-w5-c22dd8-kserve-5c65c9bcd7-nnf7d" event={"ID":"9cd440c6-7191-4fae-a806-f8728975b3cf","Type":"ContainerDied","Data":"ad9717c2c4f658e6682eb6e046f42a4a1a10e2fc0876e2db5a78c8aacffc45e0"}
Nov 01 03:53:24.165431 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:24.165359    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-w5-c22dd8-kserve-5c65c9bcd7-nnf7d" event={"ID":"9cd440c6-7191-4fae-a806-f8728975b3cf","Type":"ContainerDied","Data":"b741237d2ccf3e77a3acd97fea72e0f7fc4b4a0f1066994e69d380fc543e6dd5"}
Nov 01 03:53:24.165431 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:24.165376    4400 scope.go:122] "RemoveContainer" containerID="ad9717c2c4f658e6682eb6e046f42a4a1a10e2fc0876e2db5a78c8aacffc45e0"
Nov 01 03:53:24.175240 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:24.175224    4400 scope.go:122] "RemoveContainer" containerID="ad9717c2c4f658e6682eb6e046f42a4a1a10e2fc0876e2db5a78c8aacffc45e0"
Nov 01 03:53:24.175497 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:53:24.175479    4400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad9717c2c4f658e6682eb6e046f42a4a1a10e2fc0876e2db5a78c8aacffc45e0\": container with ID starting with ad9717c2c4f658e6682eb6e046f42a4a1a10e2fc0876e2db5a78c8aacffc45e0 not found: ID does not exist" containerID="ad9717c2c4f658e6682eb6e046f42a4a1a10e2fc0876e2db5a78c8aacffc45e0"
Nov 01 03:53:24.175542 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:24.175507    4400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad9717c2c4f658e6682eb6e046f42a4a1a10e2fc0876e2db5a78c8aacffc45e0"} err="failed to get container status \"ad9717c2c4f658e6682eb6e046f42a4a1a10e2fc0876e2db5a78c8aacffc45e0\": rpc error: code = NotFound desc = could not find container \"ad9717c2c4f658e6682eb6e046f42a4a1a10e2fc0876e2db5a78c8aacffc45e0\": container with ID starting with ad9717c2c4f658e6682eb6e046f42a4a1a10e2fc0876e2db5a78c8aacffc45e0 not found: ID does not exist"
Nov 01 03:53:24.334031 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:24.334006    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-c22dd8/distinct-w5-c22dd8-kserve-5c65c9bcd7-nnf7d"]
Nov 01 03:53:24.385173 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:24.385148    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-c22dd8/distinct-w5-c22dd8-kserve-5c65c9bcd7-nnf7d"]
Nov 01 03:53:24.634641 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:24.634620    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-tl6jm"
Nov 01 03:53:24.754309 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:24.754237    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51a974d4-ac5e-4dc5-b5b4-2fac0f648c06-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51a974d4-ac5e-4dc5-b5b4-2fac0f648c06-credential-socket\") pod \"51a974d4-ac5e-4dc5-b5b4-2fac0f648c06\" (UID: \"51a974d4-ac5e-4dc5-b5b4-2fac0f648c06\") "
Nov 01 03:53:24.754309 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:24.754277    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51a974d4-ac5e-4dc5-b5b4-2fac0f648c06-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51a974d4-ac5e-4dc5-b5b4-2fac0f648c06-workload-certs\") pod \"51a974d4-ac5e-4dc5-b5b4-2fac0f648c06\" (UID: \"51a974d4-ac5e-4dc5-b5b4-2fac0f648c06\") "
Nov 01 03:53:24.754309 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:24.754296    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51a974d4-ac5e-4dc5-b5b4-2fac0f648c06-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51a974d4-ac5e-4dc5-b5b4-2fac0f648c06-istiod-ca-cert\") pod \"51a974d4-ac5e-4dc5-b5b4-2fac0f648c06\" (UID: \"51a974d4-ac5e-4dc5-b5b4-2fac0f648c06\") "
Nov 01 03:53:24.754569 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:24.754318    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51a974d4-ac5e-4dc5-b5b4-2fac0f648c06-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51a974d4-ac5e-4dc5-b5b4-2fac0f648c06-workload-socket\") pod \"51a974d4-ac5e-4dc5-b5b4-2fac0f648c06\" (UID: \"51a974d4-ac5e-4dc5-b5b4-2fac0f648c06\") "
Nov 01 03:53:24.754569 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:24.754353    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/51a974d4-ac5e-4dc5-b5b4-2fac0f648c06-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51a974d4-ac5e-4dc5-b5b4-2fac0f648c06-istio-podinfo\") pod \"51a974d4-ac5e-4dc5-b5b4-2fac0f648c06\" (UID: \"51a974d4-ac5e-4dc5-b5b4-2fac0f648c06\") "
Nov 01 03:53:24.754569 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:24.754447    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51a974d4-ac5e-4dc5-b5b4-2fac0f648c06-kube-api-access-hvt22\" (UniqueName: \"kubernetes.io/projected/51a974d4-ac5e-4dc5-b5b4-2fac0f648c06-kube-api-access-hvt22\") pod \"51a974d4-ac5e-4dc5-b5b4-2fac0f648c06\" (UID: \"51a974d4-ac5e-4dc5-b5b4-2fac0f648c06\") "
Nov 01 03:53:24.754569 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:24.754548    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51a974d4-ac5e-4dc5-b5b4-2fac0f648c06-workload-certs" pod "51a974d4-ac5e-4dc5-b5b4-2fac0f648c06" (UID: "51a974d4-ac5e-4dc5-b5b4-2fac0f648c06"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:53:24.754569 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:24.754567    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51a974d4-ac5e-4dc5-b5b4-2fac0f648c06-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51a974d4-ac5e-4dc5-b5b4-2fac0f648c06-istio-envoy\") pod \"51a974d4-ac5e-4dc5-b5b4-2fac0f648c06\" (UID: \"51a974d4-ac5e-4dc5-b5b4-2fac0f648c06\") "
Nov 01 03:53:24.755571 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:24.754713    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51a974d4-ac5e-4dc5-b5b4-2fac0f648c06-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51a974d4-ac5e-4dc5-b5b4-2fac0f648c06-istio-data\") pod \"51a974d4-ac5e-4dc5-b5b4-2fac0f648c06\" (UID: \"51a974d4-ac5e-4dc5-b5b4-2fac0f648c06\") "
Nov 01 03:53:24.755571 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:24.754747    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51a974d4-ac5e-4dc5-b5b4-2fac0f648c06-istio-token\" (UniqueName: \"kubernetes.io/projected/51a974d4-ac5e-4dc5-b5b4-2fac0f648c06-istio-token\") pod \"51a974d4-ac5e-4dc5-b5b4-2fac0f648c06\" (UID: \"51a974d4-ac5e-4dc5-b5b4-2fac0f648c06\") "
Nov 01 03:53:24.755571 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:24.754564    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51a974d4-ac5e-4dc5-b5b4-2fac0f648c06-credential-socket" pod "51a974d4-ac5e-4dc5-b5b4-2fac0f648c06" (UID: "51a974d4-ac5e-4dc5-b5b4-2fac0f648c06"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:53:24.755571 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:24.754841    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51a974d4-ac5e-4dc5-b5b4-2fac0f648c06-workload-socket" pod "51a974d4-ac5e-4dc5-b5b4-2fac0f648c06" (UID: "51a974d4-ac5e-4dc5-b5b4-2fac0f648c06"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:53:24.755571 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:24.754907    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51a974d4-ac5e-4dc5-b5b4-2fac0f648c06-istiod-ca-cert" pod "51a974d4-ac5e-4dc5-b5b4-2fac0f648c06" (UID: "51a974d4-ac5e-4dc5-b5b4-2fac0f648c06"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:53:24.755571 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:24.754978    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51a974d4-ac5e-4dc5-b5b4-2fac0f648c06-istio-data" pod "51a974d4-ac5e-4dc5-b5b4-2fac0f648c06" (UID: "51a974d4-ac5e-4dc5-b5b4-2fac0f648c06"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:53:24.755571 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:24.755265    4400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51a974d4-ac5e-4dc5-b5b4-2fac0f648c06-istio-data\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:24.755571 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:24.755288    4400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51a974d4-ac5e-4dc5-b5b4-2fac0f648c06-credential-socket\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:24.755571 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:24.755301    4400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51a974d4-ac5e-4dc5-b5b4-2fac0f648c06-workload-certs\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:24.755571 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:24.755313    4400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51a974d4-ac5e-4dc5-b5b4-2fac0f648c06-istiod-ca-cert\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:24.755571 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:24.755325    4400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51a974d4-ac5e-4dc5-b5b4-2fac0f648c06-workload-socket\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:24.757749 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:24.757726    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51a974d4-ac5e-4dc5-b5b4-2fac0f648c06-istio-envoy" pod "51a974d4-ac5e-4dc5-b5b4-2fac0f648c06" (UID: "51a974d4-ac5e-4dc5-b5b4-2fac0f648c06"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:53:24.757905 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:24.757884    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/51a974d4-ac5e-4dc5-b5b4-2fac0f648c06-istio-podinfo" pod "51a974d4-ac5e-4dc5-b5b4-2fac0f648c06" (UID: "51a974d4-ac5e-4dc5-b5b4-2fac0f648c06"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 03:53:24.757962 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:24.757911    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51a974d4-ac5e-4dc5-b5b4-2fac0f648c06-kube-api-access-hvt22" pod "51a974d4-ac5e-4dc5-b5b4-2fac0f648c06" (UID: "51a974d4-ac5e-4dc5-b5b4-2fac0f648c06"). InnerVolumeSpecName "kube-api-access-hvt22". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:53:24.758191 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:24.758170    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51a974d4-ac5e-4dc5-b5b4-2fac0f648c06-istio-token" pod "51a974d4-ac5e-4dc5-b5b4-2fac0f648c06" (UID: "51a974d4-ac5e-4dc5-b5b4-2fac0f648c06"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:53:24.856120 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:24.856096    4400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51a974d4-ac5e-4dc5-b5b4-2fac0f648c06-istio-podinfo\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:24.856120 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:24.856118    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvt22\" (UniqueName: \"kubernetes.io/projected/51a974d4-ac5e-4dc5-b5b4-2fac0f648c06-kube-api-access-hvt22\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:24.856248 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:24.856128    4400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51a974d4-ac5e-4dc5-b5b4-2fac0f648c06-istio-envoy\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:24.856248 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:24.856135    4400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51a974d4-ac5e-4dc5-b5b4-2fac0f648c06-istio-token\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:25.172436 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:25.172405    4400 generic.go:356] "Generic (PLEG): container finished" podID="51a974d4-ac5e-4dc5-b5b4-2fac0f648c06" containerID="f2a4f73c0b087f7e1b0de473be68dab046bdf6875ceaf7484965722206db7cbe" exitCode=0
Nov 01 03:53:25.172662 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:25.172462    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-tl6jm"
Nov 01 03:53:25.172662 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:25.172504    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-tl6jm" event={"ID":"51a974d4-ac5e-4dc5-b5b4-2fac0f648c06","Type":"ContainerDied","Data":"f2a4f73c0b087f7e1b0de473be68dab046bdf6875ceaf7484965722206db7cbe"}
Nov 01 03:53:25.172662 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:25.172590    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-tl6jm" event={"ID":"51a974d4-ac5e-4dc5-b5b4-2fac0f648c06","Type":"ContainerDied","Data":"4f1a83ca6f15fd85c6392811686b8785943d75b997cbd8c73b7c8a7cea34a408"}
Nov 01 03:53:25.172662 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:25.172617    4400 scope.go:122] "RemoveContainer" containerID="f2a4f73c0b087f7e1b0de473be68dab046bdf6875ceaf7484965722206db7cbe"
Nov 01 03:53:25.182456 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:25.182431    4400 scope.go:122] "RemoveContainer" containerID="f2a4f73c0b087f7e1b0de473be68dab046bdf6875ceaf7484965722206db7cbe"
Nov 01 03:53:25.182721 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:53:25.182677    4400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2a4f73c0b087f7e1b0de473be68dab046bdf6875ceaf7484965722206db7cbe\": container with ID starting with f2a4f73c0b087f7e1b0de473be68dab046bdf6875ceaf7484965722206db7cbe not found: ID does not exist" containerID="f2a4f73c0b087f7e1b0de473be68dab046bdf6875ceaf7484965722206db7cbe"
Nov 01 03:53:25.182783 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:25.182731    4400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2a4f73c0b087f7e1b0de473be68dab046bdf6875ceaf7484965722206db7cbe"} err="failed to get container status \"f2a4f73c0b087f7e1b0de473be68dab046bdf6875ceaf7484965722206db7cbe\": rpc error: code = NotFound desc = could not find container \"f2a4f73c0b087f7e1b0de473be68dab046bdf6875ceaf7484965722206db7cbe\": container with ID starting with f2a4f73c0b087f7e1b0de473be68dab046bdf6875ceaf7484965722206db7cbe not found: ID does not exist"
Nov 01 03:53:25.341403 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:25.341379    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-tl6jm"]
Nov 01 03:53:25.345615 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:25.345596    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-tl6jm"]
Nov 01 03:53:25.614692 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:25.614663    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51a974d4-ac5e-4dc5-b5b4-2fac0f648c06" path="/var/lib/kubelet/pods/51a974d4-ac5e-4dc5-b5b4-2fac0f648c06/volumes"
Nov 01 03:53:25.615072 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:25.615059    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cd440c6-7191-4fae-a806-f8728975b3cf" path="/var/lib/kubelet/pods/9cd440c6-7191-4fae-a806-f8728975b3cf/volumes"
Nov 01 03:53:30.096378 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:30.096350    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 03:53:31.728189 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:53:31.728156    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:53:33.885748 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:33.885717    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-c22dd8-6986b5b4cf-r9pws" podUID="1c1fcbce-b87d-45df-877f-89394755bd40" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 03:53:34.564451 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:34.564412    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-c22dd8-5ccf5b65f-tqnf8"]
Nov 01 03:53:34.564760 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:34.564677    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c22dd8-5ccf5b65f-tqnf8" podUID="ac5c9c0b-3403-4f6a-b5f6-500a3df457dd" containerName="maas-api" containerID="cri-o://0ee4bdb00f9ad0168239112aaee64579a19f7b481cf34349a18896a1cfeaa826" gracePeriod=30
Nov 01 03:53:34.830552 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:34.830529    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c22dd8-5ccf5b65f-tqnf8"
Nov 01 03:53:34.947473 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:34.947450    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac5c9c0b-3403-4f6a-b5f6-500a3df457dd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac5c9c0b-3403-4f6a-b5f6-500a3df457dd-openshift-service-ca\") pod \"ac5c9c0b-3403-4f6a-b5f6-500a3df457dd\" (UID: \"ac5c9c0b-3403-4f6a-b5f6-500a3df457dd\") "
Nov 01 03:53:34.947902 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:34.947524    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac5c9c0b-3403-4f6a-b5f6-500a3df457dd-kube-api-access-zcvjs\" (UniqueName: \"kubernetes.io/projected/ac5c9c0b-3403-4f6a-b5f6-500a3df457dd-kube-api-access-zcvjs\") pod \"ac5c9c0b-3403-4f6a-b5f6-500a3df457dd\" (UID: \"ac5c9c0b-3403-4f6a-b5f6-500a3df457dd\") "
Nov 01 03:53:34.947902 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:34.947543    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac5c9c0b-3403-4f6a-b5f6-500a3df457dd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac5c9c0b-3403-4f6a-b5f6-500a3df457dd-maas-api-tls\") pod \"ac5c9c0b-3403-4f6a-b5f6-500a3df457dd\" (UID: \"ac5c9c0b-3403-4f6a-b5f6-500a3df457dd\") "
Nov 01 03:53:34.947902 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:34.947567    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac5c9c0b-3403-4f6a-b5f6-500a3df457dd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac5c9c0b-3403-4f6a-b5f6-500a3df457dd-metrics-certs\") pod \"ac5c9c0b-3403-4f6a-b5f6-500a3df457dd\" (UID: \"ac5c9c0b-3403-4f6a-b5f6-500a3df457dd\") "
Nov 01 03:53:34.947902 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:34.947812    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac5c9c0b-3403-4f6a-b5f6-500a3df457dd-openshift-service-ca" pod "ac5c9c0b-3403-4f6a-b5f6-500a3df457dd" (UID: "ac5c9c0b-3403-4f6a-b5f6-500a3df457dd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:53:34.949787 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:34.949765    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac5c9c0b-3403-4f6a-b5f6-500a3df457dd-kube-api-access-zcvjs" pod "ac5c9c0b-3403-4f6a-b5f6-500a3df457dd" (UID: "ac5c9c0b-3403-4f6a-b5f6-500a3df457dd"). InnerVolumeSpecName "kube-api-access-zcvjs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:53:34.949876 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:34.949805    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac5c9c0b-3403-4f6a-b5f6-500a3df457dd-maas-api-tls" pod "ac5c9c0b-3403-4f6a-b5f6-500a3df457dd" (UID: "ac5c9c0b-3403-4f6a-b5f6-500a3df457dd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:53:34.949923 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:34.949909    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac5c9c0b-3403-4f6a-b5f6-500a3df457dd-metrics-certs" pod "ac5c9c0b-3403-4f6a-b5f6-500a3df457dd" (UID: "ac5c9c0b-3403-4f6a-b5f6-500a3df457dd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:53:35.049207 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:35.049175    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcvjs\" (UniqueName: \"kubernetes.io/projected/ac5c9c0b-3403-4f6a-b5f6-500a3df457dd-kube-api-access-zcvjs\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:35.049207 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:35.049202    4400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac5c9c0b-3403-4f6a-b5f6-500a3df457dd-maas-api-tls\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:35.049207 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:35.049211    4400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac5c9c0b-3403-4f6a-b5f6-500a3df457dd-metrics-certs\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:35.049435 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:35.049222    4400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac5c9c0b-3403-4f6a-b5f6-500a3df457dd-openshift-service-ca\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:35.231258 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:35.231177    4400 generic.go:356] "Generic (PLEG): container finished" podID="ac5c9c0b-3403-4f6a-b5f6-500a3df457dd" containerID="0ee4bdb00f9ad0168239112aaee64579a19f7b481cf34349a18896a1cfeaa826" exitCode=0
Nov 01 03:53:35.231258 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:35.231226    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c22dd8-5ccf5b65f-tqnf8"
Nov 01 03:53:35.231484 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:35.231257    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c22dd8-5ccf5b65f-tqnf8" event={"ID":"ac5c9c0b-3403-4f6a-b5f6-500a3df457dd","Type":"ContainerDied","Data":"0ee4bdb00f9ad0168239112aaee64579a19f7b481cf34349a18896a1cfeaa826"}
Nov 01 03:53:35.231484 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:35.231311    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c22dd8-5ccf5b65f-tqnf8" event={"ID":"ac5c9c0b-3403-4f6a-b5f6-500a3df457dd","Type":"ContainerDied","Data":"38129c4ebbe1d3f84d9d8197dd19f8213278224ba31be0ee6801350ba49473d3"}
Nov 01 03:53:35.231484 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:35.231340    4400 scope.go:122] "RemoveContainer" containerID="0ee4bdb00f9ad0168239112aaee64579a19f7b481cf34349a18896a1cfeaa826"
Nov 01 03:53:35.241085 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:35.241066    4400 scope.go:122] "RemoveContainer" containerID="0ee4bdb00f9ad0168239112aaee64579a19f7b481cf34349a18896a1cfeaa826"
Nov 01 03:53:35.241322 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:53:35.241305    4400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ee4bdb00f9ad0168239112aaee64579a19f7b481cf34349a18896a1cfeaa826\": container with ID starting with 0ee4bdb00f9ad0168239112aaee64579a19f7b481cf34349a18896a1cfeaa826 not found: ID does not exist" containerID="0ee4bdb00f9ad0168239112aaee64579a19f7b481cf34349a18896a1cfeaa826"
Nov 01 03:53:35.241380 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:35.241332    4400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ee4bdb00f9ad0168239112aaee64579a19f7b481cf34349a18896a1cfeaa826"} err="failed to get container status \"0ee4bdb00f9ad0168239112aaee64579a19f7b481cf34349a18896a1cfeaa826\": rpc error: code = NotFound desc = could not find container \"0ee4bdb00f9ad0168239112aaee64579a19f7b481cf34349a18896a1cfeaa826\": container with ID starting with 0ee4bdb00f9ad0168239112aaee64579a19f7b481cf34349a18896a1cfeaa826 not found: ID does not exist"
Nov 01 03:53:35.402308 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:35.402285    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-c22dd8-5ccf5b65f-tqnf8"]
Nov 01 03:53:35.409048 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:35.409029    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-c22dd8-5ccf5b65f-tqnf8"]
Nov 01 03:53:35.614573 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:35.614543    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac5c9c0b-3403-4f6a-b5f6-500a3df457dd" path="/var/lib/kubelet/pods/ac5c9c0b-3403-4f6a-b5f6-500a3df457dd/volumes"
Nov 01 03:53:38.213805 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:38.213770    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-19faa5/distinct-2-w6-19faa5-kserve-58db847577-hrzlw"]
Nov 01 03:53:40.556286 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:40.556257    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-c22dd8-6986b5b4cf-r9pws"
Nov 01 03:53:40.701767 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:40.701741    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c1fcbce-b87d-45df-877f-89394755bd40-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c1fcbce-b87d-45df-877f-89394755bd40-plugins-config-volume\") pod \"1c1fcbce-b87d-45df-877f-89394755bd40\" (UID: \"1c1fcbce-b87d-45df-877f-89394755bd40\") "
Nov 01 03:53:40.701906 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:40.701801    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c1fcbce-b87d-45df-877f-89394755bd40-kube-api-access-rbr4n\" (UniqueName: \"kubernetes.io/projected/1c1fcbce-b87d-45df-877f-89394755bd40-kube-api-access-rbr4n\") pod \"1c1fcbce-b87d-45df-877f-89394755bd40\" (UID: \"1c1fcbce-b87d-45df-877f-89394755bd40\") "
Nov 01 03:53:40.702185 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:40.702144    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c1fcbce-b87d-45df-877f-89394755bd40-plugins-config-volume" pod "1c1fcbce-b87d-45df-877f-89394755bd40" (UID: "1c1fcbce-b87d-45df-877f-89394755bd40"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:53:40.703980 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:40.703957    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c1fcbce-b87d-45df-877f-89394755bd40-kube-api-access-rbr4n" pod "1c1fcbce-b87d-45df-877f-89394755bd40" (UID: "1c1fcbce-b87d-45df-877f-89394755bd40"). InnerVolumeSpecName "kube-api-access-rbr4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:53:40.803289 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:40.803262    4400 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c1fcbce-b87d-45df-877f-89394755bd40-plugins-config-volume\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:40.803411 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:40.803298    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbr4n\" (UniqueName: \"kubernetes.io/projected/1c1fcbce-b87d-45df-877f-89394755bd40-kube-api-access-rbr4n\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:41.262782 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:41.262758    4400 generic.go:356] "Generic (PLEG): container finished" podID="1c1fcbce-b87d-45df-877f-89394755bd40" containerID="5dd44e49425b514ac9cfbaf1ab6c857adebdbe1429747db6b3d497edb31f0f0e" exitCode=0
Nov 01 03:53:41.262904 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:41.262799    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-c22dd8-6986b5b4cf-r9pws"
Nov 01 03:53:41.262904 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:41.262839    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-c22dd8-6986b5b4cf-r9pws" event={"ID":"1c1fcbce-b87d-45df-877f-89394755bd40","Type":"ContainerDied","Data":"5dd44e49425b514ac9cfbaf1ab6c857adebdbe1429747db6b3d497edb31f0f0e"}
Nov 01 03:53:41.262904 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:41.262882    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-c22dd8-6986b5b4cf-r9pws" event={"ID":"1c1fcbce-b87d-45df-877f-89394755bd40","Type":"ContainerDied","Data":"fb7fa7169957f52d6da0ff53dac0aa77f00e3962a977210bca90515b07f300be"}
Nov 01 03:53:41.262904 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:41.262901    4400 scope.go:122] "RemoveContainer" containerID="5dd44e49425b514ac9cfbaf1ab6c857adebdbe1429747db6b3d497edb31f0f0e"
Nov 01 03:53:41.272344 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:41.272330    4400 scope.go:122] "RemoveContainer" containerID="5dd44e49425b514ac9cfbaf1ab6c857adebdbe1429747db6b3d497edb31f0f0e"
Nov 01 03:53:41.272617 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:53:41.272602    4400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5dd44e49425b514ac9cfbaf1ab6c857adebdbe1429747db6b3d497edb31f0f0e\": container with ID starting with 5dd44e49425b514ac9cfbaf1ab6c857adebdbe1429747db6b3d497edb31f0f0e not found: ID does not exist" containerID="5dd44e49425b514ac9cfbaf1ab6c857adebdbe1429747db6b3d497edb31f0f0e"
Nov 01 03:53:41.272659 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:41.272624    4400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dd44e49425b514ac9cfbaf1ab6c857adebdbe1429747db6b3d497edb31f0f0e"} err="failed to get container status \"5dd44e49425b514ac9cfbaf1ab6c857adebdbe1429747db6b3d497edb31f0f0e\": rpc error: code = NotFound desc = could not find container \"5dd44e49425b514ac9cfbaf1ab6c857adebdbe1429747db6b3d497edb31f0f0e\": container with ID starting with 5dd44e49425b514ac9cfbaf1ab6c857adebdbe1429747db6b3d497edb31f0f0e not found: ID does not exist"
Nov 01 03:53:41.451702 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:41.451658    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-c22dd8-6986b5b4cf-r9pws"]
Nov 01 03:53:41.456779 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:41.456755    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-c22dd8-6986b5b4cf-r9pws"]
Nov 01 03:53:41.612935 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:41.612904    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c1fcbce-b87d-45df-877f-89394755bd40" path="/var/lib/kubelet/pods/1c1fcbce-b87d-45df-877f-89394755bd40/volumes"
Nov 01 03:53:44.590266 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:44.590186    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-19faa5-85bf5575cc-9lhg9"]
Nov 01 03:53:44.698022 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:44.697944    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-19faa5-67dd968c47-scbcb"]
Nov 01 03:53:49.591043 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:49.591007    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-19faa5-85bf5575cc-9lhg9" podUID="92f2c828-dd25-4783-ab42-3840e6417e29" containerName="payload-processing" containerID="cri-o://748594fad63267be4df1395de90ff1f53acd8f169ac92b6a9f5dbaf08cd01410" gracePeriod=25
Nov 01 03:53:49.699347 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:49.699319    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-19faa5-67dd968c47-scbcb" podUID="a73d166a-a23f-4e1f-a653-4abf676d9acb" containerName="payload-pre-processing" containerID="cri-o://22bfd96ee23fec39b6a6de2dcdf4a74680c53d9865718c4ddd4a76a298e00a87" gracePeriod=25
Nov 01 03:53:53.234633 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:53.234582    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-19faa5/distinct-2-w6-19faa5-kserve-58db847577-hrzlw" podUID="062959ee-ec7c-4385-98d5-cf4a240852e5" containerName="main" containerID="cri-o://3391d65238eaaf6631628730676c33368ca0469d55c523ed38a27a089a79b54a" gracePeriod=45
Nov 01 03:53:53.480664 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:53.480636    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-19faa5/distinct-2-w6-19faa5-kserve-58db847577-hrzlw"
Nov 01 03:53:53.536979 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:53.536920    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/062959ee-ec7c-4385-98d5-cf4a240852e5-home\" (UniqueName: \"kubernetes.io/empty-dir/062959ee-ec7c-4385-98d5-cf4a240852e5-home\") pod \"062959ee-ec7c-4385-98d5-cf4a240852e5\" (UID: \"062959ee-ec7c-4385-98d5-cf4a240852e5\") "
Nov 01 03:53:53.537118 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:53.537054    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/062959ee-ec7c-4385-98d5-cf4a240852e5-tls-certs\" (UniqueName: \"kubernetes.io/secret/062959ee-ec7c-4385-98d5-cf4a240852e5-tls-certs\") pod \"062959ee-ec7c-4385-98d5-cf4a240852e5\" (UID: \"062959ee-ec7c-4385-98d5-cf4a240852e5\") "
Nov 01 03:53:53.537184 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:53.537123    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/062959ee-ec7c-4385-98d5-cf4a240852e5-home" pod "062959ee-ec7c-4385-98d5-cf4a240852e5" (UID: "062959ee-ec7c-4385-98d5-cf4a240852e5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:53:53.537184 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:53.537123    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/062959ee-ec7c-4385-98d5-cf4a240852e5-kube-api-access-mmptv\" (UniqueName: \"kubernetes.io/projected/062959ee-ec7c-4385-98d5-cf4a240852e5-kube-api-access-mmptv\") pod \"062959ee-ec7c-4385-98d5-cf4a240852e5\" (UID: \"062959ee-ec7c-4385-98d5-cf4a240852e5\") "
Nov 01 03:53:53.537295 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:53.537186    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/062959ee-ec7c-4385-98d5-cf4a240852e5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/062959ee-ec7c-4385-98d5-cf4a240852e5-model-cache\") pod \"062959ee-ec7c-4385-98d5-cf4a240852e5\" (UID: \"062959ee-ec7c-4385-98d5-cf4a240852e5\") "
Nov 01 03:53:53.537295 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:53.537238    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/062959ee-ec7c-4385-98d5-cf4a240852e5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/062959ee-ec7c-4385-98d5-cf4a240852e5-tmp-dir\") pod \"062959ee-ec7c-4385-98d5-cf4a240852e5\" (UID: \"062959ee-ec7c-4385-98d5-cf4a240852e5\") "
Nov 01 03:53:53.537295 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:53.537257    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/062959ee-ec7c-4385-98d5-cf4a240852e5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/062959ee-ec7c-4385-98d5-cf4a240852e5-dshm\") pod \"062959ee-ec7c-4385-98d5-cf4a240852e5\" (UID: \"062959ee-ec7c-4385-98d5-cf4a240852e5\") "
Nov 01 03:53:53.537481 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:53.537453    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/062959ee-ec7c-4385-98d5-cf4a240852e5-model-cache" pod "062959ee-ec7c-4385-98d5-cf4a240852e5" (UID: "062959ee-ec7c-4385-98d5-cf4a240852e5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:53:53.537521 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:53.537465    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/062959ee-ec7c-4385-98d5-cf4a240852e5-tmp-dir" pod "062959ee-ec7c-4385-98d5-cf4a240852e5" (UID: "062959ee-ec7c-4385-98d5-cf4a240852e5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:53:53.537942 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:53.537921    4400 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/062959ee-ec7c-4385-98d5-cf4a240852e5-tmp-dir\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:53.538036 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:53.537946    4400 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/062959ee-ec7c-4385-98d5-cf4a240852e5-home\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:53.538036 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:53.537959    4400 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/062959ee-ec7c-4385-98d5-cf4a240852e5-model-cache\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:53.539500 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:53.539474    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/062959ee-ec7c-4385-98d5-cf4a240852e5-tls-certs" pod "062959ee-ec7c-4385-98d5-cf4a240852e5" (UID: "062959ee-ec7c-4385-98d5-cf4a240852e5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:53:53.539500 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:53.539493    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/062959ee-ec7c-4385-98d5-cf4a240852e5-dshm" pod "062959ee-ec7c-4385-98d5-cf4a240852e5" (UID: "062959ee-ec7c-4385-98d5-cf4a240852e5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:53:53.539615 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:53.539519    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/062959ee-ec7c-4385-98d5-cf4a240852e5-kube-api-access-mmptv" pod "062959ee-ec7c-4385-98d5-cf4a240852e5" (UID: "062959ee-ec7c-4385-98d5-cf4a240852e5"). InnerVolumeSpecName "kube-api-access-mmptv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:53:53.638939 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:53.638911    4400 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/062959ee-ec7c-4385-98d5-cf4a240852e5-tls-certs\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:53.638939 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:53.638937    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmptv\" (UniqueName: \"kubernetes.io/projected/062959ee-ec7c-4385-98d5-cf4a240852e5-kube-api-access-mmptv\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:53.639054 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:53.638948    4400 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/062959ee-ec7c-4385-98d5-cf4a240852e5-dshm\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:54.330004 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:54.329972    4400 generic.go:356] "Generic (PLEG): container finished" podID="062959ee-ec7c-4385-98d5-cf4a240852e5" containerID="3391d65238eaaf6631628730676c33368ca0469d55c523ed38a27a089a79b54a" exitCode=0
Nov 01 03:53:54.330422 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:54.330016    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-19faa5/distinct-2-w6-19faa5-kserve-58db847577-hrzlw"
Nov 01 03:53:54.330422 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:54.330036    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-19faa5/distinct-2-w6-19faa5-kserve-58db847577-hrzlw" event={"ID":"062959ee-ec7c-4385-98d5-cf4a240852e5","Type":"ContainerDied","Data":"3391d65238eaaf6631628730676c33368ca0469d55c523ed38a27a089a79b54a"}
Nov 01 03:53:54.330422 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:54.330065    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-19faa5/distinct-2-w6-19faa5-kserve-58db847577-hrzlw" event={"ID":"062959ee-ec7c-4385-98d5-cf4a240852e5","Type":"ContainerDied","Data":"a9d95d204ab7acadfeecbaed868026e6e58093cf117d7e27f8054416a1c470de"}
Nov 01 03:53:54.330422 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:54.330080    4400 scope.go:122] "RemoveContainer" containerID="3391d65238eaaf6631628730676c33368ca0469d55c523ed38a27a089a79b54a"
Nov 01 03:53:54.339061 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:54.339043    4400 scope.go:122] "RemoveContainer" containerID="3391d65238eaaf6631628730676c33368ca0469d55c523ed38a27a089a79b54a"
Nov 01 03:53:54.339303 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:53:54.339283    4400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3391d65238eaaf6631628730676c33368ca0469d55c523ed38a27a089a79b54a\": container with ID starting with 3391d65238eaaf6631628730676c33368ca0469d55c523ed38a27a089a79b54a not found: ID does not exist" containerID="3391d65238eaaf6631628730676c33368ca0469d55c523ed38a27a089a79b54a"
Nov 01 03:53:54.339358 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:54.339311    4400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3391d65238eaaf6631628730676c33368ca0469d55c523ed38a27a089a79b54a"} err="failed to get container status \"3391d65238eaaf6631628730676c33368ca0469d55c523ed38a27a089a79b54a\": rpc error: code = NotFound desc = could not find container \"3391d65238eaaf6631628730676c33368ca0469d55c523ed38a27a089a79b54a\": container with ID starting with 3391d65238eaaf6631628730676c33368ca0469d55c523ed38a27a089a79b54a not found: ID does not exist"
Nov 01 03:53:54.500443 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:54.500422    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-19faa5/distinct-2-w6-19faa5-kserve-58db847577-hrzlw"]
Nov 01 03:53:54.506819 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:54.506800    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-19faa5/distinct-2-w6-19faa5-kserve-58db847577-hrzlw"]
Nov 01 03:53:55.619662 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:55.619626    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="062959ee-ec7c-4385-98d5-cf4a240852e5" path="/var/lib/kubelet/pods/062959ee-ec7c-4385-98d5-cf4a240852e5/volumes"
Nov 01 03:53:56.773023 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:56.772988    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-19faa5-67dd968c47-scbcb" podUID="a73d166a-a23f-4e1f-a653-4abf676d9acb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 03:53:56.773023 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:53:56.773011    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-19faa5-85bf5575cc-9lhg9" podUID="92f2c828-dd25-4783-ab42-3840e6417e29" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 03:54:01.288903 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:01.288875    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 03:54:01.727728 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:54:01.727670    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:54:06.451827 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:06.451798    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-19faa5-b9cb6f5d-7tr6v"]
Nov 01 03:54:06.452285 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:06.451970    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-19faa5-b9cb6f5d-7tr6v" podUID="0e73051a-861a-4bb4-b3cf-b06d447768aa" containerName="maas-api" containerID="cri-o://b6422dc5f05f33558d1ddb0ed04e5726feda67fa66fd8bdd3444270d577b41b0" gracePeriod=30
Nov 01 03:54:06.724928 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:06.724902    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-19faa5-b9cb6f5d-7tr6v"
Nov 01 03:54:06.772895 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:06.772859    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-19faa5-85bf5575cc-9lhg9" podUID="92f2c828-dd25-4783-ab42-3840e6417e29" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 03:54:06.773029 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:06.772859    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-19faa5-67dd968c47-scbcb" podUID="a73d166a-a23f-4e1f-a653-4abf676d9acb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 03:54:06.862990 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:06.862959    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e73051a-861a-4bb4-b3cf-b06d447768aa-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e73051a-861a-4bb4-b3cf-b06d447768aa-maas-api-tls\") pod \"0e73051a-861a-4bb4-b3cf-b06d447768aa\" (UID: \"0e73051a-861a-4bb4-b3cf-b06d447768aa\") "
Nov 01 03:54:06.863152 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:06.863054    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e73051a-861a-4bb4-b3cf-b06d447768aa-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e73051a-861a-4bb4-b3cf-b06d447768aa-openshift-service-ca\") pod \"0e73051a-861a-4bb4-b3cf-b06d447768aa\" (UID: \"0e73051a-861a-4bb4-b3cf-b06d447768aa\") "
Nov 01 03:54:06.863152 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:06.863097    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e73051a-861a-4bb4-b3cf-b06d447768aa-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e73051a-861a-4bb4-b3cf-b06d447768aa-metrics-certs\") pod \"0e73051a-861a-4bb4-b3cf-b06d447768aa\" (UID: \"0e73051a-861a-4bb4-b3cf-b06d447768aa\") "
Nov 01 03:54:06.863265 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:06.863162    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e73051a-861a-4bb4-b3cf-b06d447768aa-kube-api-access-fn4w9\" (UniqueName: \"kubernetes.io/projected/0e73051a-861a-4bb4-b3cf-b06d447768aa-kube-api-access-fn4w9\") pod \"0e73051a-861a-4bb4-b3cf-b06d447768aa\" (UID: \"0e73051a-861a-4bb4-b3cf-b06d447768aa\") "
Nov 01 03:54:06.863394 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:06.863363    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e73051a-861a-4bb4-b3cf-b06d447768aa-openshift-service-ca" pod "0e73051a-861a-4bb4-b3cf-b06d447768aa" (UID: "0e73051a-861a-4bb4-b3cf-b06d447768aa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:54:06.863578 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:06.863556    4400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e73051a-861a-4bb4-b3cf-b06d447768aa-openshift-service-ca\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:54:06.865325 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:06.865298    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e73051a-861a-4bb4-b3cf-b06d447768aa-kube-api-access-fn4w9" pod "0e73051a-861a-4bb4-b3cf-b06d447768aa" (UID: "0e73051a-861a-4bb4-b3cf-b06d447768aa"). InnerVolumeSpecName "kube-api-access-fn4w9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:54:06.865325 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:06.865320    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e73051a-861a-4bb4-b3cf-b06d447768aa-metrics-certs" pod "0e73051a-861a-4bb4-b3cf-b06d447768aa" (UID: "0e73051a-861a-4bb4-b3cf-b06d447768aa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:54:06.865461 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:06.865340    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e73051a-861a-4bb4-b3cf-b06d447768aa-maas-api-tls" pod "0e73051a-861a-4bb4-b3cf-b06d447768aa" (UID: "0e73051a-861a-4bb4-b3cf-b06d447768aa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:54:06.979454 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:06.979317    4400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e73051a-861a-4bb4-b3cf-b06d447768aa-maas-api-tls\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:54:06.979454 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:06.979357    4400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e73051a-861a-4bb4-b3cf-b06d447768aa-metrics-certs\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:54:06.979454 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:06.979372    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fn4w9\" (UniqueName: \"kubernetes.io/projected/0e73051a-861a-4bb4-b3cf-b06d447768aa-kube-api-access-fn4w9\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:54:07.395760 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:07.395719    4400 generic.go:356] "Generic (PLEG): container finished" podID="0e73051a-861a-4bb4-b3cf-b06d447768aa" containerID="b6422dc5f05f33558d1ddb0ed04e5726feda67fa66fd8bdd3444270d577b41b0" exitCode=0
Nov 01 03:54:07.395933 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:07.395812    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-19faa5-b9cb6f5d-7tr6v"
Nov 01 03:54:07.395933 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:07.395842    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-19faa5-b9cb6f5d-7tr6v" event={"ID":"0e73051a-861a-4bb4-b3cf-b06d447768aa","Type":"ContainerDied","Data":"b6422dc5f05f33558d1ddb0ed04e5726feda67fa66fd8bdd3444270d577b41b0"}
Nov 01 03:54:07.395933 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:07.395893    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-19faa5-b9cb6f5d-7tr6v" event={"ID":"0e73051a-861a-4bb4-b3cf-b06d447768aa","Type":"ContainerDied","Data":"4078ef7d628944ee04a0e7c2b1ee93a99cabaab1468d26aca36920917c01803f"}
Nov 01 03:54:07.395933 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:07.395908    4400 scope.go:122] "RemoveContainer" containerID="b6422dc5f05f33558d1ddb0ed04e5726feda67fa66fd8bdd3444270d577b41b0"
Nov 01 03:54:07.405752 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:07.405738    4400 scope.go:122] "RemoveContainer" containerID="b6422dc5f05f33558d1ddb0ed04e5726feda67fa66fd8bdd3444270d577b41b0"
Nov 01 03:54:07.406016 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:54:07.405996    4400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6422dc5f05f33558d1ddb0ed04e5726feda67fa66fd8bdd3444270d577b41b0\": container with ID starting with b6422dc5f05f33558d1ddb0ed04e5726feda67fa66fd8bdd3444270d577b41b0 not found: ID does not exist" containerID="b6422dc5f05f33558d1ddb0ed04e5726feda67fa66fd8bdd3444270d577b41b0"
Nov 01 03:54:07.406069 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:07.406023    4400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6422dc5f05f33558d1ddb0ed04e5726feda67fa66fd8bdd3444270d577b41b0"} err="failed to get container status \"b6422dc5f05f33558d1ddb0ed04e5726feda67fa66fd8bdd3444270d577b41b0\": rpc error: code = NotFound desc = could not find container \"b6422dc5f05f33558d1ddb0ed04e5726feda67fa66fd8bdd3444270d577b41b0\": container with ID starting with b6422dc5f05f33558d1ddb0ed04e5726feda67fa66fd8bdd3444270d577b41b0 not found: ID does not exist"
Nov 01 03:54:07.564576 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:07.564553    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-19faa5-b9cb6f5d-7tr6v"]
Nov 01 03:54:07.615563 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:07.615539    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-19faa5-b9cb6f5d-7tr6v"]
Nov 01 03:54:09.614460 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:09.614431    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e73051a-861a-4bb4-b3cf-b06d447768aa" path="/var/lib/kubelet/pods/0e73051a-861a-4bb4-b3cf-b06d447768aa/volumes"
Nov 01 03:54:09.838043 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:09.838018    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-19faa5-85bf5575cc-9lhg9"
Nov 01 03:54:09.929892 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:09.929869    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-19faa5-67dd968c47-scbcb"
Nov 01 03:54:10.009849 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:10.009826    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a73d166a-a23f-4e1f-a653-4abf676d9acb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a73d166a-a23f-4e1f-a653-4abf676d9acb-plugins-config-volume\") pod \"a73d166a-a23f-4e1f-a653-4abf676d9acb\" (UID: \"a73d166a-a23f-4e1f-a653-4abf676d9acb\") "
Nov 01 03:54:10.009989 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:10.009873    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92f2c828-dd25-4783-ab42-3840e6417e29-kube-api-access-tcjs2\" (UniqueName: \"kubernetes.io/projected/92f2c828-dd25-4783-ab42-3840e6417e29-kube-api-access-tcjs2\") pod \"92f2c828-dd25-4783-ab42-3840e6417e29\" (UID: \"92f2c828-dd25-4783-ab42-3840e6417e29\") "
Nov 01 03:54:10.009989 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:10.009907    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a73d166a-a23f-4e1f-a653-4abf676d9acb-kube-api-access-qgzx5\" (UniqueName: \"kubernetes.io/projected/a73d166a-a23f-4e1f-a653-4abf676d9acb-kube-api-access-qgzx5\") pod \"a73d166a-a23f-4e1f-a653-4abf676d9acb\" (UID: \"a73d166a-a23f-4e1f-a653-4abf676d9acb\") "
Nov 01 03:54:10.009989 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:10.009944    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92f2c828-dd25-4783-ab42-3840e6417e29-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92f2c828-dd25-4783-ab42-3840e6417e29-plugins-config-volume\") pod \"92f2c828-dd25-4783-ab42-3840e6417e29\" (UID: \"92f2c828-dd25-4783-ab42-3840e6417e29\") "
Nov 01 03:54:10.010175 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:10.010153    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a73d166a-a23f-4e1f-a653-4abf676d9acb-plugins-config-volume" pod "a73d166a-a23f-4e1f-a653-4abf676d9acb" (UID: "a73d166a-a23f-4e1f-a653-4abf676d9acb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:54:10.010347 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:10.010330    4400 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a73d166a-a23f-4e1f-a653-4abf676d9acb-plugins-config-volume\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:54:10.010414 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:10.010356    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92f2c828-dd25-4783-ab42-3840e6417e29-plugins-config-volume" pod "92f2c828-dd25-4783-ab42-3840e6417e29" (UID: "92f2c828-dd25-4783-ab42-3840e6417e29"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:54:10.011944 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:10.011924    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a73d166a-a23f-4e1f-a653-4abf676d9acb-kube-api-access-qgzx5" pod "a73d166a-a23f-4e1f-a653-4abf676d9acb" (UID: "a73d166a-a23f-4e1f-a653-4abf676d9acb"). InnerVolumeSpecName "kube-api-access-qgzx5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:54:10.012288 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:10.012271    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92f2c828-dd25-4783-ab42-3840e6417e29-kube-api-access-tcjs2" pod "92f2c828-dd25-4783-ab42-3840e6417e29" (UID: "92f2c828-dd25-4783-ab42-3840e6417e29"). InnerVolumeSpecName "kube-api-access-tcjs2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:54:10.111053 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:10.111024    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcjs2\" (UniqueName: \"kubernetes.io/projected/92f2c828-dd25-4783-ab42-3840e6417e29-kube-api-access-tcjs2\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:54:10.111053 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:10.111049    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgzx5\" (UniqueName: \"kubernetes.io/projected/a73d166a-a23f-4e1f-a653-4abf676d9acb-kube-api-access-qgzx5\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:54:10.111192 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:10.111062    4400 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92f2c828-dd25-4783-ab42-3840e6417e29-plugins-config-volume\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:54:10.422842 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:10.422814    4400 generic.go:356] "Generic (PLEG): container finished" podID="92f2c828-dd25-4783-ab42-3840e6417e29" containerID="748594fad63267be4df1395de90ff1f53acd8f169ac92b6a9f5dbaf08cd01410" exitCode=0
Nov 01 03:54:10.422983 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:10.422860    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-19faa5-85bf5575cc-9lhg9"
Nov 01 03:54:10.422983 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:10.422903    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-19faa5-85bf5575cc-9lhg9" event={"ID":"92f2c828-dd25-4783-ab42-3840e6417e29","Type":"ContainerDied","Data":"748594fad63267be4df1395de90ff1f53acd8f169ac92b6a9f5dbaf08cd01410"}
Nov 01 03:54:10.422983 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:10.422953    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-19faa5-85bf5575cc-9lhg9" event={"ID":"92f2c828-dd25-4783-ab42-3840e6417e29","Type":"ContainerDied","Data":"d5c20d86c070c9197f2d05d0cd737c967dcbb27aa8f54c07fde6cbe0e4f1e4a1"}
Nov 01 03:54:10.422983 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:10.422983    4400 scope.go:122] "RemoveContainer" containerID="748594fad63267be4df1395de90ff1f53acd8f169ac92b6a9f5dbaf08cd01410"
Nov 01 03:54:10.424182 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:10.424160    4400 generic.go:356] "Generic (PLEG): container finished" podID="a73d166a-a23f-4e1f-a653-4abf676d9acb" containerID="22bfd96ee23fec39b6a6de2dcdf4a74680c53d9865718c4ddd4a76a298e00a87" exitCode=0
Nov 01 03:54:10.424261 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:10.424206    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-19faa5-67dd968c47-scbcb"
Nov 01 03:54:10.424261 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:10.424247    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-19faa5-67dd968c47-scbcb" event={"ID":"a73d166a-a23f-4e1f-a653-4abf676d9acb","Type":"ContainerDied","Data":"22bfd96ee23fec39b6a6de2dcdf4a74680c53d9865718c4ddd4a76a298e00a87"}
Nov 01 03:54:10.424373 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:10.424285    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-19faa5-67dd968c47-scbcb" event={"ID":"a73d166a-a23f-4e1f-a653-4abf676d9acb","Type":"ContainerDied","Data":"3fb90508fb83989514f53006685fc4c0fae5f4b496ebe454a580f99947f5964b"}
Nov 01 03:54:10.431807 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:10.431791    4400 scope.go:122] "RemoveContainer" containerID="748594fad63267be4df1395de90ff1f53acd8f169ac92b6a9f5dbaf08cd01410"
Nov 01 03:54:10.432050 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:54:10.432029    4400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"748594fad63267be4df1395de90ff1f53acd8f169ac92b6a9f5dbaf08cd01410\": container with ID starting with 748594fad63267be4df1395de90ff1f53acd8f169ac92b6a9f5dbaf08cd01410 not found: ID does not exist" containerID="748594fad63267be4df1395de90ff1f53acd8f169ac92b6a9f5dbaf08cd01410"
Nov 01 03:54:10.432129 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:10.432061    4400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"748594fad63267be4df1395de90ff1f53acd8f169ac92b6a9f5dbaf08cd01410"} err="failed to get container status \"748594fad63267be4df1395de90ff1f53acd8f169ac92b6a9f5dbaf08cd01410\": rpc error: code = NotFound desc = could not find container \"748594fad63267be4df1395de90ff1f53acd8f169ac92b6a9f5dbaf08cd01410\": container with ID starting with 748594fad63267be4df1395de90ff1f53acd8f169ac92b6a9f5dbaf08cd01410 not found: ID does not exist"
Nov 01 03:54:10.432129 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:10.432083    4400 scope.go:122] "RemoveContainer" containerID="22bfd96ee23fec39b6a6de2dcdf4a74680c53d9865718c4ddd4a76a298e00a87"
Nov 01 03:54:10.439919 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:10.439899    4400 scope.go:122] "RemoveContainer" containerID="22bfd96ee23fec39b6a6de2dcdf4a74680c53d9865718c4ddd4a76a298e00a87"
Nov 01 03:54:10.440160 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:54:10.440140    4400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22bfd96ee23fec39b6a6de2dcdf4a74680c53d9865718c4ddd4a76a298e00a87\": container with ID starting with 22bfd96ee23fec39b6a6de2dcdf4a74680c53d9865718c4ddd4a76a298e00a87 not found: ID does not exist" containerID="22bfd96ee23fec39b6a6de2dcdf4a74680c53d9865718c4ddd4a76a298e00a87"
Nov 01 03:54:10.440221 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:10.440169    4400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22bfd96ee23fec39b6a6de2dcdf4a74680c53d9865718c4ddd4a76a298e00a87"} err="failed to get container status \"22bfd96ee23fec39b6a6de2dcdf4a74680c53d9865718c4ddd4a76a298e00a87\": rpc error: code = NotFound desc = could not find container \"22bfd96ee23fec39b6a6de2dcdf4a74680c53d9865718c4ddd4a76a298e00a87\": container with ID starting with 22bfd96ee23fec39b6a6de2dcdf4a74680c53d9865718c4ddd4a76a298e00a87 not found: ID does not exist"
Nov 01 03:54:10.591386 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:10.591360    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-19faa5-85bf5575cc-9lhg9"]
Nov 01 03:54:10.642427 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:10.642407    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-19faa5-85bf5575cc-9lhg9"]
Nov 01 03:54:10.711310 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:10.711259    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-19faa5-67dd968c47-scbcb"]
Nov 01 03:54:10.722511 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:10.722489    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-19faa5-67dd968c47-scbcb"]
Nov 01 03:54:11.613032 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:11.613005    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92f2c828-dd25-4783-ab42-3840e6417e29" path="/var/lib/kubelet/pods/92f2c828-dd25-4783-ab42-3840e6417e29/volumes"
Nov 01 03:54:11.613360 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:11.613347    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a73d166a-a23f-4e1f-a653-4abf676d9acb" path="/var/lib/kubelet/pods/a73d166a-a23f-4e1f-a653-4abf676d9acb/volumes"
Nov 01 03:54:17.121671 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:17.121624    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wqmgx"]
Nov 01 03:54:17.122212 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:17.121882    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wqmgx" podUID="66ca3614-7687-403d-a5c0-359421d8584b" containerName="istio-proxy" containerID="cri-o://c775e544a2c5155452178a9c1619ea5d76521fb7efdaaab69565bcd7645bd028" gracePeriod=30
Nov 01 03:54:19.082153 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:19.082121    4400 patch_prober.go:29] interesting pod/e2e-worker-w6-19faa5-openshift-default-59c44967-wqmgx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 03:54:19.082506 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:19.082183    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wqmgx" podUID="66ca3614-7687-403d-a5c0-359421d8584b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 03:54:22.374847 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:22.374824    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wqmgx"
Nov 01 03:54:22.501325 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:22.501259    4400 generic.go:356] "Generic (PLEG): container finished" podID="66ca3614-7687-403d-a5c0-359421d8584b" containerID="c775e544a2c5155452178a9c1619ea5d76521fb7efdaaab69565bcd7645bd028" exitCode=0
Nov 01 03:54:22.501435 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:22.501359    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wqmgx"
Nov 01 03:54:22.501435 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:22.501396    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wqmgx" event={"ID":"66ca3614-7687-403d-a5c0-359421d8584b","Type":"ContainerDied","Data":"c775e544a2c5155452178a9c1619ea5d76521fb7efdaaab69565bcd7645bd028"}
Nov 01 03:54:22.501523 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:22.501444    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wqmgx" event={"ID":"66ca3614-7687-403d-a5c0-359421d8584b","Type":"ContainerDied","Data":"5088e8235268c93be85c23d6ae9e80a1fb19ebd931518cb965c3d53999ed1add"}
Nov 01 03:54:22.501523 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:22.501465    4400 scope.go:122] "RemoveContainer" containerID="c775e544a2c5155452178a9c1619ea5d76521fb7efdaaab69565bcd7645bd028"
Nov 01 03:54:22.510489 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:22.510473    4400 scope.go:122] "RemoveContainer" containerID="c775e544a2c5155452178a9c1619ea5d76521fb7efdaaab69565bcd7645bd028"
Nov 01 03:54:22.510759 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:54:22.510740    4400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c775e544a2c5155452178a9c1619ea5d76521fb7efdaaab69565bcd7645bd028\": container with ID starting with c775e544a2c5155452178a9c1619ea5d76521fb7efdaaab69565bcd7645bd028 not found: ID does not exist" containerID="c775e544a2c5155452178a9c1619ea5d76521fb7efdaaab69565bcd7645bd028"
Nov 01 03:54:22.510821 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:22.510767    4400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c775e544a2c5155452178a9c1619ea5d76521fb7efdaaab69565bcd7645bd028"} err="failed to get container status \"c775e544a2c5155452178a9c1619ea5d76521fb7efdaaab69565bcd7645bd028\": rpc error: code = NotFound desc = could not find container \"c775e544a2c5155452178a9c1619ea5d76521fb7efdaaab69565bcd7645bd028\": container with ID starting with c775e544a2c5155452178a9c1619ea5d76521fb7efdaaab69565bcd7645bd028 not found: ID does not exist"
Nov 01 03:54:22.522075 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:22.522054    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66ca3614-7687-403d-a5c0-359421d8584b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66ca3614-7687-403d-a5c0-359421d8584b-workload-certs\") pod \"66ca3614-7687-403d-a5c0-359421d8584b\" (UID: \"66ca3614-7687-403d-a5c0-359421d8584b\") "
Nov 01 03:54:22.522180 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:22.522132    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66ca3614-7687-403d-a5c0-359421d8584b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66ca3614-7687-403d-a5c0-359421d8584b-istio-data\") pod \"66ca3614-7687-403d-a5c0-359421d8584b\" (UID: \"66ca3614-7687-403d-a5c0-359421d8584b\") "
Nov 01 03:54:22.522180 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:22.522164    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66ca3614-7687-403d-a5c0-359421d8584b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66ca3614-7687-403d-a5c0-359421d8584b-istio-envoy\") pod \"66ca3614-7687-403d-a5c0-359421d8584b\" (UID: \"66ca3614-7687-403d-a5c0-359421d8584b\") "
Nov 01 03:54:22.522289 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:22.522215    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66ca3614-7687-403d-a5c0-359421d8584b-istio-token\" (UniqueName: \"kubernetes.io/projected/66ca3614-7687-403d-a5c0-359421d8584b-istio-token\") pod \"66ca3614-7687-403d-a5c0-359421d8584b\" (UID: \"66ca3614-7687-403d-a5c0-359421d8584b\") "
Nov 01 03:54:22.522289 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:22.522243    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66ca3614-7687-403d-a5c0-359421d8584b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66ca3614-7687-403d-a5c0-359421d8584b-workload-socket\") pod \"66ca3614-7687-403d-a5c0-359421d8584b\" (UID: \"66ca3614-7687-403d-a5c0-359421d8584b\") "
Nov 01 03:54:22.522289 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:22.522270    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66ca3614-7687-403d-a5c0-359421d8584b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66ca3614-7687-403d-a5c0-359421d8584b-credential-socket\") pod \"66ca3614-7687-403d-a5c0-359421d8584b\" (UID: \"66ca3614-7687-403d-a5c0-359421d8584b\") "
Nov 01 03:54:22.522434 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:22.522306    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66ca3614-7687-403d-a5c0-359421d8584b-workload-certs" pod "66ca3614-7687-403d-a5c0-359421d8584b" (UID: "66ca3614-7687-403d-a5c0-359421d8584b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:54:22.522434 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:22.522319    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/66ca3614-7687-403d-a5c0-359421d8584b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66ca3614-7687-403d-a5c0-359421d8584b-istio-podinfo\") pod \"66ca3614-7687-403d-a5c0-359421d8584b\" (UID: \"66ca3614-7687-403d-a5c0-359421d8584b\") "
Nov 01 03:54:22.522434 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:22.522337    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66ca3614-7687-403d-a5c0-359421d8584b-istio-data" pod "66ca3614-7687-403d-a5c0-359421d8584b" (UID: "66ca3614-7687-403d-a5c0-359421d8584b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:54:22.522434 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:22.522370    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66ca3614-7687-403d-a5c0-359421d8584b-kube-api-access-bkqv6\" (UniqueName: \"kubernetes.io/projected/66ca3614-7687-403d-a5c0-359421d8584b-kube-api-access-bkqv6\") pod \"66ca3614-7687-403d-a5c0-359421d8584b\" (UID: \"66ca3614-7687-403d-a5c0-359421d8584b\") "
Nov 01 03:54:22.522610 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:22.522439    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66ca3614-7687-403d-a5c0-359421d8584b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66ca3614-7687-403d-a5c0-359421d8584b-istiod-ca-cert\") pod \"66ca3614-7687-403d-a5c0-359421d8584b\" (UID: \"66ca3614-7687-403d-a5c0-359421d8584b\") "
Nov 01 03:54:22.522610 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:22.522540    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66ca3614-7687-403d-a5c0-359421d8584b-credential-socket" pod "66ca3614-7687-403d-a5c0-359421d8584b" (UID: "66ca3614-7687-403d-a5c0-359421d8584b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:54:22.522610 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:22.522553    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66ca3614-7687-403d-a5c0-359421d8584b-workload-socket" pod "66ca3614-7687-403d-a5c0-359421d8584b" (UID: "66ca3614-7687-403d-a5c0-359421d8584b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:54:22.522845 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:22.522824    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66ca3614-7687-403d-a5c0-359421d8584b-istiod-ca-cert" pod "66ca3614-7687-403d-a5c0-359421d8584b" (UID: "66ca3614-7687-403d-a5c0-359421d8584b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:54:22.522962 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:22.522865    4400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66ca3614-7687-403d-a5c0-359421d8584b-workload-certs\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:54:22.522962 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:22.522885    4400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66ca3614-7687-403d-a5c0-359421d8584b-istio-data\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:54:22.522962 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:22.522900    4400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66ca3614-7687-403d-a5c0-359421d8584b-workload-socket\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:54:22.522962 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:22.522914    4400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66ca3614-7687-403d-a5c0-359421d8584b-credential-socket\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:54:22.524496 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:22.524460    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66ca3614-7687-403d-a5c0-359421d8584b-istio-token" pod "66ca3614-7687-403d-a5c0-359421d8584b" (UID: "66ca3614-7687-403d-a5c0-359421d8584b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:54:22.524592 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:22.524573    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66ca3614-7687-403d-a5c0-359421d8584b-istio-envoy" pod "66ca3614-7687-403d-a5c0-359421d8584b" (UID: "66ca3614-7687-403d-a5c0-359421d8584b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:54:22.524884 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:22.524866    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/66ca3614-7687-403d-a5c0-359421d8584b-istio-podinfo" pod "66ca3614-7687-403d-a5c0-359421d8584b" (UID: "66ca3614-7687-403d-a5c0-359421d8584b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 03:54:22.524964 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:22.524944    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66ca3614-7687-403d-a5c0-359421d8584b-kube-api-access-bkqv6" pod "66ca3614-7687-403d-a5c0-359421d8584b" (UID: "66ca3614-7687-403d-a5c0-359421d8584b"). InnerVolumeSpecName "kube-api-access-bkqv6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:54:22.624153 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:22.624121    4400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66ca3614-7687-403d-a5c0-359421d8584b-istio-envoy\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:54:22.624153 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:22.624147    4400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66ca3614-7687-403d-a5c0-359421d8584b-istio-token\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:54:22.624313 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:22.624160    4400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66ca3614-7687-403d-a5c0-359421d8584b-istio-podinfo\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:54:22.624313 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:22.624172    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkqv6\" (UniqueName: \"kubernetes.io/projected/66ca3614-7687-403d-a5c0-359421d8584b-kube-api-access-bkqv6\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:54:22.624313 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:22.624185    4400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66ca3614-7687-403d-a5c0-359421d8584b-istiod-ca-cert\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:54:22.926013 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:22.925984    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wqmgx"]
Nov 01 03:54:22.931728 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:22.931702    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wqmgx"]
Nov 01 03:54:23.613374 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:23.613344    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66ca3614-7687-403d-a5c0-359421d8584b" path="/var/lib/kubelet/pods/66ca3614-7687-403d-a5c0-359421d8584b/volumes"
Nov 01 03:54:24.377012 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:24.376984    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 03:54:31.726671 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:54:31.726633    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:54:52.738906 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:54:52.738876    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 03:55:01.728991 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:55:01.728958    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:55:31.727644 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:55:31.727602    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:55:48.466554 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:55:48.466519    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 03:56:01.727329 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:56:01.727294    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:56:09.577186 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:56:09.577154    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 03:56:20.810131 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:56:20.810085    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 03:56:31.732267 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:56:31.732219    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:56:40.709262 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:56:40.709224    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 03:56:57.474134 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:56:57.474054    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 03:57:01.730398 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:57:01.730359    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:57:28.511629 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:57:28.511539    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 03:57:30.990902 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:57:30.990857    4400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-8xrcf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 03:57:30.991366 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:57:30.990956    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-8xrcf" podUID="fa4d3ec1-8b26-4c06-8290-87c055cdca0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:57:31.731759 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:57:31.731706    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:57:39.281983 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:57:39.281944    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 03:57:52.020511 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:57:52.020480    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 03:58:01.731541 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:58:01.731506    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:58:17.199959 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:58:17.199876    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 03:58:27.012377 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:58:27.012338    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 03:58:31.730482 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:58:31.730441    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:58:45.299044 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:58:45.299011    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 03:58:58.067471 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:58:58.067433    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 03:59:01.730547 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:59:01.730510    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:59:17.800932 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:59:17.800893    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 03:59:31.729934 ip-10-0-133-54 kubenswrapper[4400]: E1101 03:59:31.729898    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 03:59:40.005601 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:59:40.005563    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 03:59:41.617710 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:59:41.614814    4400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-zkjpt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 03:59:41.617710 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:59:41.614914    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zkjpt" podUID="7204663c-ef51-407b-8b56-65e9e54a2d51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:59:53.102087 ip-10-0-133-54 kubenswrapper[4400]: I1101 03:59:53.101873    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 04:00:01.730912 ip-10-0-133-54 kubenswrapper[4400]: E1101 04:00:01.730873    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 04:00:03.874562 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:00:03.874526    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 04:00:26.903930 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:00:26.903893    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 04:00:31.730393 ip-10-0-133-54 kubenswrapper[4400]: E1101 04:00:31.730353    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 04:00:37.084709 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:00:37.084639    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 04:00:50.708558 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:00:50.708523    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 04:01:01.731929 ip-10-0-133-54 kubenswrapper[4400]: E1101 04:01:01.731887    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 04:01:12.606052 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:01:12.606018    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 04:01:23.313436 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:01:23.313352    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 04:01:31.730298 ip-10-0-133-54 kubenswrapper[4400]: E1101 04:01:31.730263    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 04:01:41.902403 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:01:41.902371    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 04:01:52.407204 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:01:52.407165    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 04:02:01.741827 ip-10-0-133-54 kubenswrapper[4400]: E1101 04:02:01.741784    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 04:02:10.539815 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.539777    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-feacb8-6bc5dccfbf-44c2b"]
Nov 01 04:02:10.540225 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.540212    4400 cpu_manager.go:414] "containerMap: removing container" podUID="a6b65b4c-f4d1-49a4-a4d3-7ef365d8aac9" containerName="payload-pre-processing"
Nov 01 04:02:10.540269 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.540226    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6b65b4c-f4d1-49a4-a4d3-7ef365d8aac9" containerName="payload-pre-processing"
Nov 01 04:02:10.540269 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.540238    4400 cpu_manager.go:414] "containerMap: removing container" podUID="ed1ec2ab-443d-42d9-9221-20cdf668121a" containerName="payload-processing"
Nov 01 04:02:10.540269 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.540244    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed1ec2ab-443d-42d9-9221-20cdf668121a" containerName="payload-processing"
Nov 01 04:02:10.540269 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.540252    4400 cpu_manager.go:414] "containerMap: removing container" podUID="ac5c9c0b-3403-4f6a-b5f6-500a3df457dd" containerName="maas-api"
Nov 01 04:02:10.540269 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.540257    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac5c9c0b-3403-4f6a-b5f6-500a3df457dd" containerName="maas-api"
Nov 01 04:02:10.540269 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.540266    4400 cpu_manager.go:414] "containerMap: removing container" podUID="1b60b3a2-9941-49b1-a528-ee002c10a700" containerName="payload-pre-processing"
Nov 01 04:02:10.540269 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.540270    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b60b3a2-9941-49b1-a528-ee002c10a700" containerName="payload-pre-processing"
Nov 01 04:02:10.540533 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.540279    4400 cpu_manager.go:414] "containerMap: removing container" podUID="0e73051a-861a-4bb4-b3cf-b06d447768aa" containerName="maas-api"
Nov 01 04:02:10.540533 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.540283    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e73051a-861a-4bb4-b3cf-b06d447768aa" containerName="maas-api"
Nov 01 04:02:10.540533 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.540290    4400 cpu_manager.go:414] "containerMap: removing container" podUID="aa3e1de2-5fb8-4692-b87f-5b0cb323fe03" containerName="maas-api"
Nov 01 04:02:10.540533 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.540295    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa3e1de2-5fb8-4692-b87f-5b0cb323fe03" containerName="maas-api"
Nov 01 04:02:10.540533 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.540303    4400 cpu_manager.go:414] "containerMap: removing container" podUID="66ca3614-7687-403d-a5c0-359421d8584b" containerName="istio-proxy"
Nov 01 04:02:10.540533 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.540308    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66ca3614-7687-403d-a5c0-359421d8584b" containerName="istio-proxy"
Nov 01 04:02:10.540533 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.540315    4400 cpu_manager.go:414] "containerMap: removing container" podUID="6592a8ed-219c-460c-b354-e0e8331cfa64" containerName="maas-api"
Nov 01 04:02:10.540533 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.540319    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6592a8ed-219c-460c-b354-e0e8331cfa64" containerName="maas-api"
Nov 01 04:02:10.540533 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.540326    4400 cpu_manager.go:414] "containerMap: removing container" podUID="de71d35c-8a34-4a1c-96d3-7ae4aa24fee2" containerName="maas-api"
Nov 01 04:02:10.540533 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.540330    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de71d35c-8a34-4a1c-96d3-7ae4aa24fee2" containerName="maas-api"
Nov 01 04:02:10.540533 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.540336    4400 cpu_manager.go:414] "containerMap: removing container" podUID="a73d166a-a23f-4e1f-a653-4abf676d9acb" containerName="payload-pre-processing"
Nov 01 04:02:10.540533 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.540341    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a73d166a-a23f-4e1f-a653-4abf676d9acb" containerName="payload-pre-processing"
Nov 01 04:02:10.540533 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.540347    4400 cpu_manager.go:414] "containerMap: removing container" podUID="062959ee-ec7c-4385-98d5-cf4a240852e5" containerName="main"
Nov 01 04:02:10.540533 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.540351    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="062959ee-ec7c-4385-98d5-cf4a240852e5" containerName="main"
Nov 01 04:02:10.540533 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.540359    4400 cpu_manager.go:414] "containerMap: removing container" podUID="0a5c6392-1b3d-4823-a5fa-f91005df6759" containerName="payload-processing"
Nov 01 04:02:10.540533 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.540363    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a5c6392-1b3d-4823-a5fa-f91005df6759" containerName="payload-processing"
Nov 01 04:02:10.540533 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.540371    4400 cpu_manager.go:414] "containerMap: removing container" podUID="a617524f-b187-4692-aea7-89193c7bcad0" containerName="maas-api"
Nov 01 04:02:10.540533 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.540375    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a617524f-b187-4692-aea7-89193c7bcad0" containerName="maas-api"
Nov 01 04:02:10.540533 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.540381    4400 cpu_manager.go:414] "containerMap: removing container" podUID="466179e3-7600-464e-9f7c-ec099bc8086d" containerName="payload-processing"
Nov 01 04:02:10.540533 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.540385    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="466179e3-7600-464e-9f7c-ec099bc8086d" containerName="payload-processing"
Nov 01 04:02:10.540533 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.540394    4400 cpu_manager.go:414] "containerMap: removing container" podUID="21b19cc7-731a-45fb-b864-da76b9b41be0" containerName="maas-api"
Nov 01 04:02:10.540533 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.540399    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21b19cc7-731a-45fb-b864-da76b9b41be0" containerName="maas-api"
Nov 01 04:02:10.540533 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.540405    4400 cpu_manager.go:414] "containerMap: removing container" podUID="7179f114-d7bf-4d4d-9a12-4a88e9041272" containerName="maas-api"
Nov 01 04:02:10.540533 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.540414    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7179f114-d7bf-4d4d-9a12-4a88e9041272" containerName="maas-api"
Nov 01 04:02:10.540533 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.540421    4400 cpu_manager.go:414] "containerMap: removing container" podUID="92f2c828-dd25-4783-ab42-3840e6417e29" containerName="payload-processing"
Nov 01 04:02:10.540533 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.540425    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92f2c828-dd25-4783-ab42-3840e6417e29" containerName="payload-processing"
Nov 01 04:02:10.540533 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.540437    4400 cpu_manager.go:414] "containerMap: removing container" podUID="c0e53b77-f9a5-4c9b-9ccd-84d6d30c35e4" containerName="maas-api"
Nov 01 04:02:10.540533 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.540444    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0e53b77-f9a5-4c9b-9ccd-84d6d30c35e4" containerName="maas-api"
Nov 01 04:02:10.540533 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.540453    4400 cpu_manager.go:414] "containerMap: removing container" podUID="7159d3dd-647e-45b6-9f5b-1dbaecb70ad3" containerName="payload-pre-processing"
Nov 01 04:02:10.540533 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.540459    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7159d3dd-647e-45b6-9f5b-1dbaecb70ad3" containerName="payload-pre-processing"
Nov 01 04:02:10.540533 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.540469    4400 cpu_manager.go:414] "containerMap: removing container" podUID="58d4db89-1d9c-46f7-a944-43b865313c8c" containerName="payload-processing"
Nov 01 04:02:10.540533 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.540476    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58d4db89-1d9c-46f7-a944-43b865313c8c" containerName="payload-processing"
Nov 01 04:02:10.540533 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.540485    4400 cpu_manager.go:414] "containerMap: removing container" podUID="1c1fcbce-b87d-45df-877f-89394755bd40" containerName="payload-processing"
Nov 01 04:02:10.540533 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.540491    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c1fcbce-b87d-45df-877f-89394755bd40" containerName="payload-processing"
Nov 01 04:02:10.540533 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.540498    4400 cpu_manager.go:414] "containerMap: removing container" podUID="d2550fb6-41b8-4237-b186-237684022a2a" containerName="maas-api"
Nov 01 04:02:10.540533 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.540504    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2550fb6-41b8-4237-b186-237684022a2a" containerName="maas-api"
Nov 01 04:02:10.540533 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.540515    4400 cpu_manager.go:414] "containerMap: removing container" podUID="7c1732ba-87a1-43cf-92d6-5edf0869787f" containerName="payload-pre-processing"
Nov 01 04:02:10.540533 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.540522    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c1732ba-87a1-43cf-92d6-5edf0869787f" containerName="payload-pre-processing"
Nov 01 04:02:10.540533 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.540535    4400 cpu_manager.go:414] "containerMap: removing container" podUID="bc0b9a54-e2e5-44e2-bb9f-62949ff3db59" containerName="payload-pre-processing"
Nov 01 04:02:10.540533 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.540543    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc0b9a54-e2e5-44e2-bb9f-62949ff3db59" containerName="payload-pre-processing"
Nov 01 04:02:10.541804 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.540558    4400 cpu_manager.go:414] "containerMap: removing container" podUID="c6f13d0b-fb65-44b9-b627-ec07234e87bc" containerName="istio-proxy"
Nov 01 04:02:10.541804 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.540565    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6f13d0b-fb65-44b9-b627-ec07234e87bc" containerName="istio-proxy"
Nov 01 04:02:10.541804 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.540576    4400 cpu_manager.go:414] "containerMap: removing container" podUID="65358d08-83e6-45c5-a17e-f1dbde164345" containerName="main"
Nov 01 04:02:10.541804 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.540583    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65358d08-83e6-45c5-a17e-f1dbde164345" containerName="main"
Nov 01 04:02:10.541804 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.540600    4400 cpu_manager.go:414] "containerMap: removing container" podUID="dc04277f-2785-4c65-a353-4afca8198519" containerName="maas-api"
Nov 01 04:02:10.541804 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.540606    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc04277f-2785-4c65-a353-4afca8198519" containerName="maas-api"
Nov 01 04:02:10.541804 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.540617    4400 cpu_manager.go:414] "containerMap: removing container" podUID="51a974d4-ac5e-4dc5-b5b4-2fac0f648c06" containerName="istio-proxy"
Nov 01 04:02:10.541804 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.540621    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51a974d4-ac5e-4dc5-b5b4-2fac0f648c06" containerName="istio-proxy"
Nov 01 04:02:10.541804 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.540629    4400 cpu_manager.go:414] "containerMap: removing container" podUID="9cd440c6-7191-4fae-a806-f8728975b3cf" containerName="main"
Nov 01 04:02:10.541804 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.540633    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cd440c6-7191-4fae-a806-f8728975b3cf" containerName="main"
Nov 01 04:02:10.541804 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.540641    4400 cpu_manager.go:414] "containerMap: removing container" podUID="a0737af1-9631-4f69-ba0d-18e6cd2765d1" containerName="maas-api"
Nov 01 04:02:10.541804 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.540645    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0737af1-9631-4f69-ba0d-18e6cd2765d1" containerName="maas-api"
Nov 01 04:02:10.541804 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.540650    4400 cpu_manager.go:414] "containerMap: removing container" podUID="f7b0b2d1-ff0e-4bde-a832-430c7007749a" containerName="main"
Nov 01 04:02:10.541804 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.540654    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7b0b2d1-ff0e-4bde-a832-430c7007749a" containerName="main"
Nov 01 04:02:10.541804 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.540661    4400 cpu_manager.go:414] "containerMap: removing container" podUID="73e54a1c-e8c0-4edc-991e-94cf8d331ff5" containerName="maas-api"
Nov 01 04:02:10.541804 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.540665    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73e54a1c-e8c0-4edc-991e-94cf8d331ff5" containerName="maas-api"
Nov 01 04:02:10.541804 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.540673    4400 cpu_manager.go:414] "containerMap: removing container" podUID="925a7174-5bbd-4400-869a-c3b62c79199e" containerName="istio-proxy"
Nov 01 04:02:10.541804 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.540677    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="925a7174-5bbd-4400-869a-c3b62c79199e" containerName="istio-proxy"
Nov 01 04:02:10.541804 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.540798    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6f13d0b-fb65-44b9-b627-ec07234e87bc" containerName="istio-proxy"
Nov 01 04:02:10.541804 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.540806    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="65358d08-83e6-45c5-a17e-f1dbde164345" containerName="main"
Nov 01 04:02:10.541804 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.540813    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc04277f-2785-4c65-a353-4afca8198519" containerName="maas-api"
Nov 01 04:02:10.541804 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.540820    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="51a974d4-ac5e-4dc5-b5b4-2fac0f648c06" containerName="istio-proxy"
Nov 01 04:02:10.541804 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.540825    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cd440c6-7191-4fae-a806-f8728975b3cf" containerName="main"
Nov 01 04:02:10.541804 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.540830    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0737af1-9631-4f69-ba0d-18e6cd2765d1" containerName="maas-api"
Nov 01 04:02:10.541804 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.540835    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7b0b2d1-ff0e-4bde-a832-430c7007749a" containerName="main"
Nov 01 04:02:10.541804 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.540841    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="73e54a1c-e8c0-4edc-991e-94cf8d331ff5" containerName="maas-api"
Nov 01 04:02:10.541804 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.540846    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="925a7174-5bbd-4400-869a-c3b62c79199e" containerName="istio-proxy"
Nov 01 04:02:10.541804 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.540853    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6b65b4c-f4d1-49a4-a4d3-7ef365d8aac9" containerName="payload-pre-processing"
Nov 01 04:02:10.541804 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.540858    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed1ec2ab-443d-42d9-9221-20cdf668121a" containerName="payload-processing"
Nov 01 04:02:10.541804 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.540862    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac5c9c0b-3403-4f6a-b5f6-500a3df457dd" containerName="maas-api"
Nov 01 04:02:10.541804 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.540868    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b60b3a2-9941-49b1-a528-ee002c10a700" containerName="payload-pre-processing"
Nov 01 04:02:10.541804 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.540873    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e73051a-861a-4bb4-b3cf-b06d447768aa" containerName="maas-api"
Nov 01 04:02:10.541804 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.540878    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa3e1de2-5fb8-4692-b87f-5b0cb323fe03" containerName="maas-api"
Nov 01 04:02:10.541804 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.540888    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="66ca3614-7687-403d-a5c0-359421d8584b" containerName="istio-proxy"
Nov 01 04:02:10.541804 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.540894    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="6592a8ed-219c-460c-b354-e0e8331cfa64" containerName="maas-api"
Nov 01 04:02:10.541804 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.540900    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="de71d35c-8a34-4a1c-96d3-7ae4aa24fee2" containerName="maas-api"
Nov 01 04:02:10.541804 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.540906    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="a73d166a-a23f-4e1f-a653-4abf676d9acb" containerName="payload-pre-processing"
Nov 01 04:02:10.541804 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.540911    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="062959ee-ec7c-4385-98d5-cf4a240852e5" containerName="main"
Nov 01 04:02:10.541804 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.540916    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a5c6392-1b3d-4823-a5fa-f91005df6759" containerName="payload-processing"
Nov 01 04:02:10.541804 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.540923    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="a617524f-b187-4692-aea7-89193c7bcad0" containerName="maas-api"
Nov 01 04:02:10.541804 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.540928    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="466179e3-7600-464e-9f7c-ec099bc8086d" containerName="payload-processing"
Nov 01 04:02:10.541804 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.540935    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="21b19cc7-731a-45fb-b864-da76b9b41be0" containerName="maas-api"
Nov 01 04:02:10.541804 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.540941    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="7179f114-d7bf-4d4d-9a12-4a88e9041272" containerName="maas-api"
Nov 01 04:02:10.541804 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.540945    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="92f2c828-dd25-4783-ab42-3840e6417e29" containerName="payload-processing"
Nov 01 04:02:10.541804 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.540951    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0e53b77-f9a5-4c9b-9ccd-84d6d30c35e4" containerName="maas-api"
Nov 01 04:02:10.541804 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.540957    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="7159d3dd-647e-45b6-9f5b-1dbaecb70ad3" containerName="payload-pre-processing"
Nov 01 04:02:10.541804 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.540962    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="58d4db89-1d9c-46f7-a944-43b865313c8c" containerName="payload-processing"
Nov 01 04:02:10.541804 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.540968    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c1fcbce-b87d-45df-877f-89394755bd40" containerName="payload-processing"
Nov 01 04:02:10.541804 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.540972    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2550fb6-41b8-4237-b186-237684022a2a" containerName="maas-api"
Nov 01 04:02:10.541804 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.540977    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c1732ba-87a1-43cf-92d6-5edf0869787f" containerName="payload-pre-processing"
Nov 01 04:02:10.541804 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.540983    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc0b9a54-e2e5-44e2-bb9f-62949ff3db59" containerName="payload-pre-processing"
Nov 01 04:02:10.544140 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.544122    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-feacb8-6bc5dccfbf-44c2b"
Nov 01 04:02:10.600939 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.600907    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-feacb8-serving-cert\""
Nov 01 04:02:10.603887 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.603860    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-feacb8-6bc5dccfbf-44c2b"]
Nov 01 04:02:10.657527 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.657489    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/192e0f73-aa13-4508-9382-62d1ba125df7-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-feacb8-6bc5dccfbf-44c2b\" (UID: \"192e0f73-aa13-4508-9382-62d1ba125df7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-feacb8-6bc5dccfbf-44c2b"
Nov 01 04:02:10.657730 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.657612    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vk4v\" (UniqueName: \"kubernetes.io/projected/192e0f73-aa13-4508-9382-62d1ba125df7-kube-api-access-8vk4v\") pod \"maas-api-e2e-worker-single-main-feacb8-6bc5dccfbf-44c2b\" (UID: \"192e0f73-aa13-4508-9382-62d1ba125df7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-feacb8-6bc5dccfbf-44c2b"
Nov 01 04:02:10.657730 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.657634    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/192e0f73-aa13-4508-9382-62d1ba125df7-metrics-certs\") pod \"maas-api-e2e-worker-single-main-feacb8-6bc5dccfbf-44c2b\" (UID: \"192e0f73-aa13-4508-9382-62d1ba125df7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-feacb8-6bc5dccfbf-44c2b"
Nov 01 04:02:10.657809 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.657761    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/192e0f73-aa13-4508-9382-62d1ba125df7-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-feacb8-6bc5dccfbf-44c2b\" (UID: \"192e0f73-aa13-4508-9382-62d1ba125df7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-feacb8-6bc5dccfbf-44c2b"
Nov 01 04:02:10.759021 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.758982    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/192e0f73-aa13-4508-9382-62d1ba125df7-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-feacb8-6bc5dccfbf-44c2b\" (UID: \"192e0f73-aa13-4508-9382-62d1ba125df7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-feacb8-6bc5dccfbf-44c2b"
Nov 01 04:02:10.759242 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.759093    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vk4v\" (UniqueName: \"kubernetes.io/projected/192e0f73-aa13-4508-9382-62d1ba125df7-kube-api-access-8vk4v\") pod \"maas-api-e2e-worker-single-main-feacb8-6bc5dccfbf-44c2b\" (UID: \"192e0f73-aa13-4508-9382-62d1ba125df7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-feacb8-6bc5dccfbf-44c2b"
Nov 01 04:02:10.759242 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.759114    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/192e0f73-aa13-4508-9382-62d1ba125df7-metrics-certs\") pod \"maas-api-e2e-worker-single-main-feacb8-6bc5dccfbf-44c2b\" (UID: \"192e0f73-aa13-4508-9382-62d1ba125df7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-feacb8-6bc5dccfbf-44c2b"
Nov 01 04:02:10.759242 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.759164    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/192e0f73-aa13-4508-9382-62d1ba125df7-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-feacb8-6bc5dccfbf-44c2b\" (UID: \"192e0f73-aa13-4508-9382-62d1ba125df7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-feacb8-6bc5dccfbf-44c2b"
Nov 01 04:02:10.759354 ip-10-0-133-54 kubenswrapper[4400]: E1101 04:02:10.759258    4400 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-feacb8-serving-cert: secret "maas-api-e2e-worker-single-main-feacb8-serving-cert" not found
Nov 01 04:02:10.759354 ip-10-0-133-54 kubenswrapper[4400]: E1101 04:02:10.759327    4400 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/192e0f73-aa13-4508-9382-62d1ba125df7-maas-api-tls podName:192e0f73-aa13-4508-9382-62d1ba125df7 nodeName:}" failed. No retries permitted until 2026-11-01 04:02:11.259309586 +0000 UTC m=+3337.773955444 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/192e0f73-aa13-4508-9382-62d1ba125df7-maas-api-tls") pod "maas-api-e2e-worker-single-main-feacb8-6bc5dccfbf-44c2b" (UID: "192e0f73-aa13-4508-9382-62d1ba125df7") : secret "maas-api-e2e-worker-single-main-feacb8-serving-cert" not found
Nov 01 04:02:10.759793 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.759774    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/192e0f73-aa13-4508-9382-62d1ba125df7-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-feacb8-6bc5dccfbf-44c2b\" (UID: \"192e0f73-aa13-4508-9382-62d1ba125df7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-feacb8-6bc5dccfbf-44c2b"
Nov 01 04:02:10.761741 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.761724    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/192e0f73-aa13-4508-9382-62d1ba125df7-metrics-certs\") pod \"maas-api-e2e-worker-single-main-feacb8-6bc5dccfbf-44c2b\" (UID: \"192e0f73-aa13-4508-9382-62d1ba125df7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-feacb8-6bc5dccfbf-44c2b"
Nov 01 04:02:10.822219 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:10.822186    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vk4v\" (UniqueName: \"kubernetes.io/projected/192e0f73-aa13-4508-9382-62d1ba125df7-kube-api-access-8vk4v\") pod \"maas-api-e2e-worker-single-main-feacb8-6bc5dccfbf-44c2b\" (UID: \"192e0f73-aa13-4508-9382-62d1ba125df7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-feacb8-6bc5dccfbf-44c2b"
Nov 01 04:02:11.265076 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:11.264986    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/192e0f73-aa13-4508-9382-62d1ba125df7-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-feacb8-6bc5dccfbf-44c2b\" (UID: \"192e0f73-aa13-4508-9382-62d1ba125df7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-feacb8-6bc5dccfbf-44c2b"
Nov 01 04:02:11.267594 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:11.267561    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/192e0f73-aa13-4508-9382-62d1ba125df7-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-feacb8-6bc5dccfbf-44c2b\" (UID: \"192e0f73-aa13-4508-9382-62d1ba125df7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-feacb8-6bc5dccfbf-44c2b"
Nov 01 04:02:11.457514 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:11.457473    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-feacb8-6bc5dccfbf-44c2b"
Nov 01 04:02:11.836961 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:11.836936    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-feacb8-6bc5dccfbf-44c2b"]
Nov 01 04:02:11.838591 ip-10-0-133-54 kubenswrapper[4400]: W1101 04:02:11.838557    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod192e0f73_aa13_4508_9382_62d1ba125df7.slice/crio-786705378becbd23069ce128fb1fff3c5a9388da8ca87482d6359ddef8ffb6c0 WatchSource:0}: Error finding container 786705378becbd23069ce128fb1fff3c5a9388da8ca87482d6359ddef8ffb6c0: Status 404 returned error can't find the container with id 786705378becbd23069ce128fb1fff3c5a9388da8ca87482d6359ddef8ffb6c0
Nov 01 04:02:11.840129 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:11.840108    4400 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 04:02:12.109221 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:12.109136    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-feacb8-6bc5dccfbf-44c2b" event={"ID":"192e0f73-aa13-4508-9382-62d1ba125df7","Type":"ContainerStarted","Data":"786705378becbd23069ce128fb1fff3c5a9388da8ca87482d6359ddef8ffb6c0"}
Nov 01 04:02:13.116147 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:13.116115    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-feacb8-6bc5dccfbf-44c2b" event={"ID":"192e0f73-aa13-4508-9382-62d1ba125df7","Type":"ContainerStarted","Data":"9ca0157fb64bb3826a28d6aec6bc5db8f42c0f045a8238e2bf4be0c416441777"}
Nov 01 04:02:13.116566 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:13.116183    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-feacb8-6bc5dccfbf-44c2b"
Nov 01 04:02:13.237894 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:13.237832    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-feacb8-6bc5dccfbf-44c2b" podStartSLOduration=2.277098092 podStartE2EDuration="3.237813897s" totalImagesPullingTime="960.715805ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:02:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:02:13.23609238 +0000 UTC m=+3339.750738243" watchObservedRunningTime="2026-11-01 04:02:13.237813897 +0000 UTC m=+3339.752459761"
Nov 01 04:02:18.679405 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:18.679178    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 04:02:19.128492 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:19.128463    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-feacb8-6bc5dccfbf-44c2b"
Nov 01 04:02:31.729668 ip-10-0-133-54 kubenswrapper[4400]: E1101 04:02:31.729624    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 04:02:37.494492 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:02:37.494455    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 04:03:01.730285 ip-10-0-133-54 kubenswrapper[4400]: E1101 04:03:01.730201    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 04:03:03.405519 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:03:03.405487    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 04:03:31.729871 ip-10-0-133-54 kubenswrapper[4400]: E1101 04:03:31.729828    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 04:03:32.624648 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:03:32.624611    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 04:03:48.002762 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:03:48.002728    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-feacb8-6bc5dccfbf-44c2b"]
Nov 01 04:03:48.003182 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:03:48.002957    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-feacb8-6bc5dccfbf-44c2b" podUID="192e0f73-aa13-4508-9382-62d1ba125df7" containerName="maas-api" containerID="cri-o://9ca0157fb64bb3826a28d6aec6bc5db8f42c0f045a8238e2bf4be0c416441777" gracePeriod=30
Nov 01 04:03:49.123799 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:03:49.123761    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-feacb8-6bc5dccfbf-44c2b" podUID="192e0f73-aa13-4508-9382-62d1ba125df7" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.138:8443/health\": dial tcp 10.132.0.138:8443: connect: connection refused"
Nov 01 04:03:49.268883 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:03:49.268859    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-feacb8-6bc5dccfbf-44c2b"
Nov 01 04:03:49.328339 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:03:49.328301    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/192e0f73-aa13-4508-9382-62d1ba125df7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/192e0f73-aa13-4508-9382-62d1ba125df7-metrics-certs\") pod \"192e0f73-aa13-4508-9382-62d1ba125df7\" (UID: \"192e0f73-aa13-4508-9382-62d1ba125df7\") "
Nov 01 04:03:49.328339 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:03:49.328339    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/192e0f73-aa13-4508-9382-62d1ba125df7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/192e0f73-aa13-4508-9382-62d1ba125df7-maas-api-tls\") pod \"192e0f73-aa13-4508-9382-62d1ba125df7\" (UID: \"192e0f73-aa13-4508-9382-62d1ba125df7\") "
Nov 01 04:03:49.328539 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:03:49.328366    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/192e0f73-aa13-4508-9382-62d1ba125df7-kube-api-access-8vk4v\" (UniqueName: \"kubernetes.io/projected/192e0f73-aa13-4508-9382-62d1ba125df7-kube-api-access-8vk4v\") pod \"192e0f73-aa13-4508-9382-62d1ba125df7\" (UID: \"192e0f73-aa13-4508-9382-62d1ba125df7\") "
Nov 01 04:03:49.328539 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:03:49.328388    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/192e0f73-aa13-4508-9382-62d1ba125df7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/192e0f73-aa13-4508-9382-62d1ba125df7-openshift-service-ca\") pod \"192e0f73-aa13-4508-9382-62d1ba125df7\" (UID: \"192e0f73-aa13-4508-9382-62d1ba125df7\") "
Nov 01 04:03:49.328871 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:03:49.328847    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/192e0f73-aa13-4508-9382-62d1ba125df7-openshift-service-ca" pod "192e0f73-aa13-4508-9382-62d1ba125df7" (UID: "192e0f73-aa13-4508-9382-62d1ba125df7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:03:49.330783 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:03:49.330754    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/192e0f73-aa13-4508-9382-62d1ba125df7-maas-api-tls" pod "192e0f73-aa13-4508-9382-62d1ba125df7" (UID: "192e0f73-aa13-4508-9382-62d1ba125df7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:03:49.330888 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:03:49.330863    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/192e0f73-aa13-4508-9382-62d1ba125df7-kube-api-access-8vk4v" pod "192e0f73-aa13-4508-9382-62d1ba125df7" (UID: "192e0f73-aa13-4508-9382-62d1ba125df7"). InnerVolumeSpecName "kube-api-access-8vk4v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:03:49.330926 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:03:49.330880    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/192e0f73-aa13-4508-9382-62d1ba125df7-metrics-certs" pod "192e0f73-aa13-4508-9382-62d1ba125df7" (UID: "192e0f73-aa13-4508-9382-62d1ba125df7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:03:49.429553 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:03:49.429473    4400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/192e0f73-aa13-4508-9382-62d1ba125df7-metrics-certs\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:03:49.429553 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:03:49.429498    4400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/192e0f73-aa13-4508-9382-62d1ba125df7-maas-api-tls\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:03:49.429553 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:03:49.429510    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vk4v\" (UniqueName: \"kubernetes.io/projected/192e0f73-aa13-4508-9382-62d1ba125df7-kube-api-access-8vk4v\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:03:49.429553 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:03:49.429522    4400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/192e0f73-aa13-4508-9382-62d1ba125df7-openshift-service-ca\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:03:49.601005 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:03:49.600967    4400 generic.go:356] "Generic (PLEG): container finished" podID="192e0f73-aa13-4508-9382-62d1ba125df7" containerID="9ca0157fb64bb3826a28d6aec6bc5db8f42c0f045a8238e2bf4be0c416441777" exitCode=0
Nov 01 04:03:49.601181 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:03:49.601018    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-feacb8-6bc5dccfbf-44c2b"
Nov 01 04:03:49.601181 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:03:49.601024    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-feacb8-6bc5dccfbf-44c2b" event={"ID":"192e0f73-aa13-4508-9382-62d1ba125df7","Type":"ContainerDied","Data":"9ca0157fb64bb3826a28d6aec6bc5db8f42c0f045a8238e2bf4be0c416441777"}
Nov 01 04:03:49.601181 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:03:49.601079    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-feacb8-6bc5dccfbf-44c2b" event={"ID":"192e0f73-aa13-4508-9382-62d1ba125df7","Type":"ContainerDied","Data":"786705378becbd23069ce128fb1fff3c5a9388da8ca87482d6359ddef8ffb6c0"}
Nov 01 04:03:49.601181 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:03:49.601108    4400 scope.go:122] "RemoveContainer" containerID="9ca0157fb64bb3826a28d6aec6bc5db8f42c0f045a8238e2bf4be0c416441777"
Nov 01 04:03:49.611343 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:03:49.611324    4400 scope.go:122] "RemoveContainer" containerID="9ca0157fb64bb3826a28d6aec6bc5db8f42c0f045a8238e2bf4be0c416441777"
Nov 01 04:03:49.611598 ip-10-0-133-54 kubenswrapper[4400]: E1101 04:03:49.611579    4400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ca0157fb64bb3826a28d6aec6bc5db8f42c0f045a8238e2bf4be0c416441777\": container with ID starting with 9ca0157fb64bb3826a28d6aec6bc5db8f42c0f045a8238e2bf4be0c416441777 not found: ID does not exist" containerID="9ca0157fb64bb3826a28d6aec6bc5db8f42c0f045a8238e2bf4be0c416441777"
Nov 01 04:03:49.611657 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:03:49.611607    4400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ca0157fb64bb3826a28d6aec6bc5db8f42c0f045a8238e2bf4be0c416441777"} err="failed to get container status \"9ca0157fb64bb3826a28d6aec6bc5db8f42c0f045a8238e2bf4be0c416441777\": rpc error: code = NotFound desc = could not find container \"9ca0157fb64bb3826a28d6aec6bc5db8f42c0f045a8238e2bf4be0c416441777\": container with ID starting with 9ca0157fb64bb3826a28d6aec6bc5db8f42c0f045a8238e2bf4be0c416441777 not found: ID does not exist"
Nov 01 04:03:49.768924 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:03:49.768836    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-feacb8-6bc5dccfbf-44c2b"]
Nov 01 04:03:49.774993 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:03:49.774966    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-feacb8-6bc5dccfbf-44c2b"]
Nov 01 04:03:51.615234 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:03:51.615198    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="192e0f73-aa13-4508-9382-62d1ba125df7" path="/var/lib/kubelet/pods/192e0f73-aa13-4508-9382-62d1ba125df7/volumes"
Nov 01 04:03:52.726577 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:03:52.726532    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 04:04:01.729593 ip-10-0-133-54 kubenswrapper[4400]: E1101 04:04:01.729554    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 04:04:31.730186 ip-10-0-133-54 kubenswrapper[4400]: E1101 04:04:31.730139    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 04:05:01.730810 ip-10-0-133-54 kubenswrapper[4400]: E1101 04:05:01.730769    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 04:05:31.729447 ip-10-0-133-54 kubenswrapper[4400]: E1101 04:05:31.729407    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 04:05:45.114273 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:05:45.114182    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 04:06:01.730157 ip-10-0-133-54 kubenswrapper[4400]: E1101 04:06:01.730107    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 04:06:02.579376 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:06:02.579337    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 04:06:12.292147 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:06:12.292110    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 04:06:31.574276 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:06:31.574239    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 04:06:31.730244 ip-10-0-133-54 kubenswrapper[4400]: E1101 04:06:31.730208    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 04:06:42.230279 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:06:42.230243    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-077ac6-79fc79c4b-jq84x"]
Nov 01 04:06:42.232600 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:06:42.230734    4400 cpu_manager.go:414] "containerMap: removing container" podUID="192e0f73-aa13-4508-9382-62d1ba125df7" containerName="maas-api"
Nov 01 04:06:42.232600 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:06:42.230748    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="192e0f73-aa13-4508-9382-62d1ba125df7" containerName="maas-api"
Nov 01 04:06:42.232600 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:06:42.230840    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="192e0f73-aa13-4508-9382-62d1ba125df7" containerName="maas-api"
Nov 01 04:06:42.233831 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:06:42.233815    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-077ac6-79fc79c4b-jq84x"
Nov 01 04:06:42.287549 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:06:42.287513    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-077ac6-serving-cert\""
Nov 01 04:06:42.301138 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:06:42.301099    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5ghj\" (UniqueName: \"kubernetes.io/projected/31ffe144-97b2-4b7b-8f46-13fec704135e-kube-api-access-p5ghj\") pod \"maas-api-e2e-worker-single-main-077ac6-79fc79c4b-jq84x\" (UID: \"31ffe144-97b2-4b7b-8f46-13fec704135e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-077ac6-79fc79c4b-jq84x"
Nov 01 04:06:42.301300 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:06:42.301138    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31ffe144-97b2-4b7b-8f46-13fec704135e-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-077ac6-79fc79c4b-jq84x\" (UID: \"31ffe144-97b2-4b7b-8f46-13fec704135e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-077ac6-79fc79c4b-jq84x"
Nov 01 04:06:42.301355 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:06:42.301337    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31ffe144-97b2-4b7b-8f46-13fec704135e-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-077ac6-79fc79c4b-jq84x\" (UID: \"31ffe144-97b2-4b7b-8f46-13fec704135e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-077ac6-79fc79c4b-jq84x"
Nov 01 04:06:42.301408 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:06:42.301383    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31ffe144-97b2-4b7b-8f46-13fec704135e-metrics-certs\") pod \"maas-api-e2e-worker-single-main-077ac6-79fc79c4b-jq84x\" (UID: \"31ffe144-97b2-4b7b-8f46-13fec704135e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-077ac6-79fc79c4b-jq84x"
Nov 01 04:06:42.341002 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:06:42.340971    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-077ac6-79fc79c4b-jq84x"]
Nov 01 04:06:42.402159 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:06:42.402120    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5ghj\" (UniqueName: \"kubernetes.io/projected/31ffe144-97b2-4b7b-8f46-13fec704135e-kube-api-access-p5ghj\") pod \"maas-api-e2e-worker-single-main-077ac6-79fc79c4b-jq84x\" (UID: \"31ffe144-97b2-4b7b-8f46-13fec704135e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-077ac6-79fc79c4b-jq84x"
Nov 01 04:06:42.402337 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:06:42.402167    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31ffe144-97b2-4b7b-8f46-13fec704135e-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-077ac6-79fc79c4b-jq84x\" (UID: \"31ffe144-97b2-4b7b-8f46-13fec704135e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-077ac6-79fc79c4b-jq84x"
Nov 01 04:06:42.402337 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:06:42.402310    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31ffe144-97b2-4b7b-8f46-13fec704135e-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-077ac6-79fc79c4b-jq84x\" (UID: \"31ffe144-97b2-4b7b-8f46-13fec704135e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-077ac6-79fc79c4b-jq84x"
Nov 01 04:06:42.402337 ip-10-0-133-54 kubenswrapper[4400]: E1101 04:06:42.402318    4400 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-077ac6-serving-cert: secret "maas-api-e2e-worker-single-main-077ac6-serving-cert" not found
Nov 01 04:06:42.402485 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:06:42.402363    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31ffe144-97b2-4b7b-8f46-13fec704135e-metrics-certs\") pod \"maas-api-e2e-worker-single-main-077ac6-79fc79c4b-jq84x\" (UID: \"31ffe144-97b2-4b7b-8f46-13fec704135e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-077ac6-79fc79c4b-jq84x"
Nov 01 04:06:42.402485 ip-10-0-133-54 kubenswrapper[4400]: E1101 04:06:42.402382    4400 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31ffe144-97b2-4b7b-8f46-13fec704135e-maas-api-tls podName:31ffe144-97b2-4b7b-8f46-13fec704135e nodeName:}" failed. No retries permitted until 2026-11-01 04:06:42.902362485 +0000 UTC m=+3609.417008346 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/31ffe144-97b2-4b7b-8f46-13fec704135e-maas-api-tls") pod "maas-api-e2e-worker-single-main-077ac6-79fc79c4b-jq84x" (UID: "31ffe144-97b2-4b7b-8f46-13fec704135e") : secret "maas-api-e2e-worker-single-main-077ac6-serving-cert" not found
Nov 01 04:06:42.403003 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:06:42.402977    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31ffe144-97b2-4b7b-8f46-13fec704135e-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-077ac6-79fc79c4b-jq84x\" (UID: \"31ffe144-97b2-4b7b-8f46-13fec704135e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-077ac6-79fc79c4b-jq84x"
Nov 01 04:06:42.405059 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:06:42.405031    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31ffe144-97b2-4b7b-8f46-13fec704135e-metrics-certs\") pod \"maas-api-e2e-worker-single-main-077ac6-79fc79c4b-jq84x\" (UID: \"31ffe144-97b2-4b7b-8f46-13fec704135e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-077ac6-79fc79c4b-jq84x"
Nov 01 04:06:42.465217 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:06:42.465189    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5ghj\" (UniqueName: \"kubernetes.io/projected/31ffe144-97b2-4b7b-8f46-13fec704135e-kube-api-access-p5ghj\") pod \"maas-api-e2e-worker-single-main-077ac6-79fc79c4b-jq84x\" (UID: \"31ffe144-97b2-4b7b-8f46-13fec704135e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-077ac6-79fc79c4b-jq84x"
Nov 01 04:06:42.906969 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:06:42.906926    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31ffe144-97b2-4b7b-8f46-13fec704135e-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-077ac6-79fc79c4b-jq84x\" (UID: \"31ffe144-97b2-4b7b-8f46-13fec704135e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-077ac6-79fc79c4b-jq84x"
Nov 01 04:06:42.909431 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:06:42.909405    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31ffe144-97b2-4b7b-8f46-13fec704135e-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-077ac6-79fc79c4b-jq84x\" (UID: \"31ffe144-97b2-4b7b-8f46-13fec704135e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-077ac6-79fc79c4b-jq84x"
Nov 01 04:06:43.153167 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:06:43.153128    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-077ac6-79fc79c4b-jq84x"
Nov 01 04:06:43.531191 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:06:43.531163    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-077ac6-79fc79c4b-jq84x"]
Nov 01 04:06:43.532755 ip-10-0-133-54 kubenswrapper[4400]: W1101 04:06:43.532725    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31ffe144_97b2_4b7b_8f46_13fec704135e.slice/crio-35f737cadcf92bbc060dd562fb0948bb78ddcbead175d9d8bf86be0305f11797 WatchSource:0}: Error finding container 35f737cadcf92bbc060dd562fb0948bb78ddcbead175d9d8bf86be0305f11797: Status 404 returned error can't find the container with id 35f737cadcf92bbc060dd562fb0948bb78ddcbead175d9d8bf86be0305f11797
Nov 01 04:06:44.512414 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:06:44.512365    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-077ac6-79fc79c4b-jq84x" event={"ID":"31ffe144-97b2-4b7b-8f46-13fec704135e","Type":"ContainerStarted","Data":"a744df37207e5256f1efa44cb91ca538542bc9de63ddb3740beac4e45581acbe"}
Nov 01 04:06:44.512610 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:06:44.512568    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-077ac6-79fc79c4b-jq84x"
Nov 01 04:06:44.512610 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:06:44.512601    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-077ac6-79fc79c4b-jq84x" event={"ID":"31ffe144-97b2-4b7b-8f46-13fec704135e","Type":"ContainerStarted","Data":"35f737cadcf92bbc060dd562fb0948bb78ddcbead175d9d8bf86be0305f11797"}
Nov 01 04:06:44.632277 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:06:44.632228    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-077ac6-79fc79c4b-jq84x" podStartSLOduration=1.7643164900000001 podStartE2EDuration="2.632216112s" totalImagesPullingTime="867.899622ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:06:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:06:44.631955032 +0000 UTC m=+3611.146600894" watchObservedRunningTime="2026-11-01 04:06:44.632216112 +0000 UTC m=+3611.146861972"
Nov 01 04:06:50.416461 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:06:50.416425    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 04:06:50.523078 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:06:50.523047    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-077ac6-79fc79c4b-jq84x"
Nov 01 04:07:01.731083 ip-10-0-133-54 kubenswrapper[4400]: E1101 04:07:01.731046    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 04:07:09.093916 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:07:09.093876    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 04:07:31.729763 ip-10-0-133-54 kubenswrapper[4400]: E1101 04:07:31.729726    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 04:07:37.013918 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:07:37.013806    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 04:08:01.730221 ip-10-0-133-54 kubenswrapper[4400]: E1101 04:08:01.730181    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 04:08:02.905265 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:08:02.905232    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 04:08:18.980906 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:08:18.980868    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-077ac6-79fc79c4b-jq84x"]
Nov 01 04:08:18.981364 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:08:18.981064    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-077ac6-79fc79c4b-jq84x" podUID="31ffe144-97b2-4b7b-8f46-13fec704135e" containerName="maas-api" containerID="cri-o://a744df37207e5256f1efa44cb91ca538542bc9de63ddb3740beac4e45581acbe" gracePeriod=30
Nov 01 04:08:20.230942 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:08:20.230906    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-077ac6-79fc79c4b-jq84x"
Nov 01 04:08:20.324331 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:08:20.324229    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31ffe144-97b2-4b7b-8f46-13fec704135e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31ffe144-97b2-4b7b-8f46-13fec704135e-maas-api-tls\") pod \"31ffe144-97b2-4b7b-8f46-13fec704135e\" (UID: \"31ffe144-97b2-4b7b-8f46-13fec704135e\") "
Nov 01 04:08:20.324331 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:08:20.324299    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31ffe144-97b2-4b7b-8f46-13fec704135e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/31ffe144-97b2-4b7b-8f46-13fec704135e-metrics-certs\") pod \"31ffe144-97b2-4b7b-8f46-13fec704135e\" (UID: \"31ffe144-97b2-4b7b-8f46-13fec704135e\") "
Nov 01 04:08:20.324583 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:08:20.324334    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31ffe144-97b2-4b7b-8f46-13fec704135e-kube-api-access-p5ghj\" (UniqueName: \"kubernetes.io/projected/31ffe144-97b2-4b7b-8f46-13fec704135e-kube-api-access-p5ghj\") pod \"31ffe144-97b2-4b7b-8f46-13fec704135e\" (UID: \"31ffe144-97b2-4b7b-8f46-13fec704135e\") "
Nov 01 04:08:20.324583 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:08:20.324417    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31ffe144-97b2-4b7b-8f46-13fec704135e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31ffe144-97b2-4b7b-8f46-13fec704135e-openshift-service-ca\") pod \"31ffe144-97b2-4b7b-8f46-13fec704135e\" (UID: \"31ffe144-97b2-4b7b-8f46-13fec704135e\") "
Nov 01 04:08:20.324926 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:08:20.324893    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31ffe144-97b2-4b7b-8f46-13fec704135e-openshift-service-ca" pod "31ffe144-97b2-4b7b-8f46-13fec704135e" (UID: "31ffe144-97b2-4b7b-8f46-13fec704135e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:08:20.326710 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:08:20.326667    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31ffe144-97b2-4b7b-8f46-13fec704135e-metrics-certs" pod "31ffe144-97b2-4b7b-8f46-13fec704135e" (UID: "31ffe144-97b2-4b7b-8f46-13fec704135e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:08:20.326710 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:08:20.326675    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31ffe144-97b2-4b7b-8f46-13fec704135e-kube-api-access-p5ghj" pod "31ffe144-97b2-4b7b-8f46-13fec704135e" (UID: "31ffe144-97b2-4b7b-8f46-13fec704135e"). InnerVolumeSpecName "kube-api-access-p5ghj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:08:20.326848 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:08:20.326771    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31ffe144-97b2-4b7b-8f46-13fec704135e-maas-api-tls" pod "31ffe144-97b2-4b7b-8f46-13fec704135e" (UID: "31ffe144-97b2-4b7b-8f46-13fec704135e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:08:20.426148 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:08:20.426105    4400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31ffe144-97b2-4b7b-8f46-13fec704135e-maas-api-tls\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:08:20.426148 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:08:20.426145    4400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31ffe144-97b2-4b7b-8f46-13fec704135e-metrics-certs\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:08:20.426148 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:08:20.426157    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5ghj\" (UniqueName: \"kubernetes.io/projected/31ffe144-97b2-4b7b-8f46-13fec704135e-kube-api-access-p5ghj\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:08:20.426453 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:08:20.426169    4400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31ffe144-97b2-4b7b-8f46-13fec704135e-openshift-service-ca\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:08:21.015244 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:08:21.015206    4400 generic.go:356] "Generic (PLEG): container finished" podID="31ffe144-97b2-4b7b-8f46-13fec704135e" containerID="a744df37207e5256f1efa44cb91ca538542bc9de63ddb3740beac4e45581acbe" exitCode=0
Nov 01 04:08:21.015432 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:08:21.015259    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-077ac6-79fc79c4b-jq84x"
Nov 01 04:08:21.015432 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:08:21.015300    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-077ac6-79fc79c4b-jq84x" event={"ID":"31ffe144-97b2-4b7b-8f46-13fec704135e","Type":"ContainerDied","Data":"a744df37207e5256f1efa44cb91ca538542bc9de63ddb3740beac4e45581acbe"}
Nov 01 04:08:21.015432 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:08:21.015358    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-077ac6-79fc79c4b-jq84x" event={"ID":"31ffe144-97b2-4b7b-8f46-13fec704135e","Type":"ContainerDied","Data":"35f737cadcf92bbc060dd562fb0948bb78ddcbead175d9d8bf86be0305f11797"}
Nov 01 04:08:21.015432 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:08:21.015381    4400 scope.go:122] "RemoveContainer" containerID="a744df37207e5256f1efa44cb91ca538542bc9de63ddb3740beac4e45581acbe"
Nov 01 04:08:21.025717 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:08:21.025695    4400 scope.go:122] "RemoveContainer" containerID="a744df37207e5256f1efa44cb91ca538542bc9de63ddb3740beac4e45581acbe"
Nov 01 04:08:21.026039 ip-10-0-133-54 kubenswrapper[4400]: E1101 04:08:21.026020    4400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a744df37207e5256f1efa44cb91ca538542bc9de63ddb3740beac4e45581acbe\": container with ID starting with a744df37207e5256f1efa44cb91ca538542bc9de63ddb3740beac4e45581acbe not found: ID does not exist" containerID="a744df37207e5256f1efa44cb91ca538542bc9de63ddb3740beac4e45581acbe"
Nov 01 04:08:21.026093 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:08:21.026047    4400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a744df37207e5256f1efa44cb91ca538542bc9de63ddb3740beac4e45581acbe"} err="failed to get container status \"a744df37207e5256f1efa44cb91ca538542bc9de63ddb3740beac4e45581acbe\": rpc error: code = NotFound desc = could not find container \"a744df37207e5256f1efa44cb91ca538542bc9de63ddb3740beac4e45581acbe\": container with ID starting with a744df37207e5256f1efa44cb91ca538542bc9de63ddb3740beac4e45581acbe not found: ID does not exist"
Nov 01 04:08:21.185148 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:08:21.185084    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-077ac6-79fc79c4b-jq84x"]
Nov 01 04:08:21.236059 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:08:21.236025    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-077ac6-79fc79c4b-jq84x"]
Nov 01 04:08:21.613246 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:08:21.613212    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31ffe144-97b2-4b7b-8f46-13fec704135e" path="/var/lib/kubelet/pods/31ffe144-97b2-4b7b-8f46-13fec704135e/volumes"
Nov 01 04:08:23.808185 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:08:23.808151    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 04:08:31.730592 ip-10-0-133-54 kubenswrapper[4400]: E1101 04:08:31.730547    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 04:08:39.876370 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:08:39.876334    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 04:09:01.729181 ip-10-0-133-54 kubenswrapper[4400]: E1101 04:09:01.729142    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 04:09:18.346415 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:09:18.346377    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-d0082e19-kserve-75dd7dfcdf-jd4zd"]
Nov 01 04:09:18.347100 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:09:18.347079    4400 cpu_manager.go:414] "containerMap: removing container" podUID="31ffe144-97b2-4b7b-8f46-13fec704135e" containerName="maas-api"
Nov 01 04:09:18.347190 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:09:18.347102    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31ffe144-97b2-4b7b-8f46-13fec704135e" containerName="maas-api"
Nov 01 04:09:18.347247 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:09:18.347222    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="31ffe144-97b2-4b7b-8f46-13fec704135e" containerName="maas-api"
Nov 01 04:09:18.350242 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:09:18.350221    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-d0082e19-kserve-75dd7dfcdf-jd4zd"
Nov 01 04:09:18.445969 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:09:18.445934    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57a1b9ad-906d-422c-b3ca-0f9a8f5f560a-tls-certs\") pod \"e2e-same-id-b-d0082e19-kserve-75dd7dfcdf-jd4zd\" (UID: \"57a1b9ad-906d-422c-b3ca-0f9a8f5f560a\") " pod="llm/e2e-same-id-b-d0082e19-kserve-75dd7dfcdf-jd4zd"
Nov 01 04:09:18.446159 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:09:18.445993    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57a1b9ad-906d-422c-b3ca-0f9a8f5f560a-dshm\") pod \"e2e-same-id-b-d0082e19-kserve-75dd7dfcdf-jd4zd\" (UID: \"57a1b9ad-906d-422c-b3ca-0f9a8f5f560a\") " pod="llm/e2e-same-id-b-d0082e19-kserve-75dd7dfcdf-jd4zd"
Nov 01 04:09:18.446159 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:09:18.446086    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57a1b9ad-906d-422c-b3ca-0f9a8f5f560a-home\") pod \"e2e-same-id-b-d0082e19-kserve-75dd7dfcdf-jd4zd\" (UID: \"57a1b9ad-906d-422c-b3ca-0f9a8f5f560a\") " pod="llm/e2e-same-id-b-d0082e19-kserve-75dd7dfcdf-jd4zd"
Nov 01 04:09:18.446159 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:09:18.446105    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57a1b9ad-906d-422c-b3ca-0f9a8f5f560a-model-cache\") pod \"e2e-same-id-b-d0082e19-kserve-75dd7dfcdf-jd4zd\" (UID: \"57a1b9ad-906d-422c-b3ca-0f9a8f5f560a\") " pod="llm/e2e-same-id-b-d0082e19-kserve-75dd7dfcdf-jd4zd"
Nov 01 04:09:18.446285 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:09:18.446196    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mdkf\" (UniqueName: \"kubernetes.io/projected/57a1b9ad-906d-422c-b3ca-0f9a8f5f560a-kube-api-access-9mdkf\") pod \"e2e-same-id-b-d0082e19-kserve-75dd7dfcdf-jd4zd\" (UID: \"57a1b9ad-906d-422c-b3ca-0f9a8f5f560a\") " pod="llm/e2e-same-id-b-d0082e19-kserve-75dd7dfcdf-jd4zd"
Nov 01 04:09:18.446285 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:09:18.446235    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57a1b9ad-906d-422c-b3ca-0f9a8f5f560a-tmp-dir\") pod \"e2e-same-id-b-d0082e19-kserve-75dd7dfcdf-jd4zd\" (UID: \"57a1b9ad-906d-422c-b3ca-0f9a8f5f560a\") " pod="llm/e2e-same-id-b-d0082e19-kserve-75dd7dfcdf-jd4zd"
Nov 01 04:09:18.457376 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:09:18.457343    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-d0082e19-kserve-self-signed-certs\""
Nov 01 04:09:18.459492 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:09:18.459471    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-d0082e19-kserve-75dd7dfcdf-jd4zd"]
Nov 01 04:09:18.547116 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:09:18.547074    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57a1b9ad-906d-422c-b3ca-0f9a8f5f560a-model-cache\") pod \"e2e-same-id-b-d0082e19-kserve-75dd7dfcdf-jd4zd\" (UID: \"57a1b9ad-906d-422c-b3ca-0f9a8f5f560a\") " pod="llm/e2e-same-id-b-d0082e19-kserve-75dd7dfcdf-jd4zd"
Nov 01 04:09:18.547308 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:09:18.547176    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mdkf\" (UniqueName: \"kubernetes.io/projected/57a1b9ad-906d-422c-b3ca-0f9a8f5f560a-kube-api-access-9mdkf\") pod \"e2e-same-id-b-d0082e19-kserve-75dd7dfcdf-jd4zd\" (UID: \"57a1b9ad-906d-422c-b3ca-0f9a8f5f560a\") " pod="llm/e2e-same-id-b-d0082e19-kserve-75dd7dfcdf-jd4zd"
Nov 01 04:09:18.547308 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:09:18.547219    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57a1b9ad-906d-422c-b3ca-0f9a8f5f560a-tmp-dir\") pod \"e2e-same-id-b-d0082e19-kserve-75dd7dfcdf-jd4zd\" (UID: \"57a1b9ad-906d-422c-b3ca-0f9a8f5f560a\") " pod="llm/e2e-same-id-b-d0082e19-kserve-75dd7dfcdf-jd4zd"
Nov 01 04:09:18.547308 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:09:18.547262    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57a1b9ad-906d-422c-b3ca-0f9a8f5f560a-tls-certs\") pod \"e2e-same-id-b-d0082e19-kserve-75dd7dfcdf-jd4zd\" (UID: \"57a1b9ad-906d-422c-b3ca-0f9a8f5f560a\") " pod="llm/e2e-same-id-b-d0082e19-kserve-75dd7dfcdf-jd4zd"
Nov 01 04:09:18.547467 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:09:18.547319    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57a1b9ad-906d-422c-b3ca-0f9a8f5f560a-dshm\") pod \"e2e-same-id-b-d0082e19-kserve-75dd7dfcdf-jd4zd\" (UID: \"57a1b9ad-906d-422c-b3ca-0f9a8f5f560a\") " pod="llm/e2e-same-id-b-d0082e19-kserve-75dd7dfcdf-jd4zd"
Nov 01 04:09:18.547467 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:09:18.547343    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57a1b9ad-906d-422c-b3ca-0f9a8f5f560a-home\") pod \"e2e-same-id-b-d0082e19-kserve-75dd7dfcdf-jd4zd\" (UID: \"57a1b9ad-906d-422c-b3ca-0f9a8f5f560a\") " pod="llm/e2e-same-id-b-d0082e19-kserve-75dd7dfcdf-jd4zd"
Nov 01 04:09:18.547576 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:09:18.547500    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57a1b9ad-906d-422c-b3ca-0f9a8f5f560a-model-cache\") pod \"e2e-same-id-b-d0082e19-kserve-75dd7dfcdf-jd4zd\" (UID: \"57a1b9ad-906d-422c-b3ca-0f9a8f5f560a\") " pod="llm/e2e-same-id-b-d0082e19-kserve-75dd7dfcdf-jd4zd"
Nov 01 04:09:18.547695 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:09:18.547661    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57a1b9ad-906d-422c-b3ca-0f9a8f5f560a-home\") pod \"e2e-same-id-b-d0082e19-kserve-75dd7dfcdf-jd4zd\" (UID: \"57a1b9ad-906d-422c-b3ca-0f9a8f5f560a\") " pod="llm/e2e-same-id-b-d0082e19-kserve-75dd7dfcdf-jd4zd"
Nov 01 04:09:18.547776 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:09:18.547696    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57a1b9ad-906d-422c-b3ca-0f9a8f5f560a-tmp-dir\") pod \"e2e-same-id-b-d0082e19-kserve-75dd7dfcdf-jd4zd\" (UID: \"57a1b9ad-906d-422c-b3ca-0f9a8f5f560a\") " pod="llm/e2e-same-id-b-d0082e19-kserve-75dd7dfcdf-jd4zd"
Nov 01 04:09:18.550046 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:09:18.550006    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57a1b9ad-906d-422c-b3ca-0f9a8f5f560a-dshm\") pod \"e2e-same-id-b-d0082e19-kserve-75dd7dfcdf-jd4zd\" (UID: \"57a1b9ad-906d-422c-b3ca-0f9a8f5f560a\") " pod="llm/e2e-same-id-b-d0082e19-kserve-75dd7dfcdf-jd4zd"
Nov 01 04:09:18.550296 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:09:18.550268    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57a1b9ad-906d-422c-b3ca-0f9a8f5f560a-tls-certs\") pod \"e2e-same-id-b-d0082e19-kserve-75dd7dfcdf-jd4zd\" (UID: \"57a1b9ad-906d-422c-b3ca-0f9a8f5f560a\") " pod="llm/e2e-same-id-b-d0082e19-kserve-75dd7dfcdf-jd4zd"
Nov 01 04:09:18.609841 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:09:18.609776    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mdkf\" (UniqueName: \"kubernetes.io/projected/57a1b9ad-906d-422c-b3ca-0f9a8f5f560a-kube-api-access-9mdkf\") pod \"e2e-same-id-b-d0082e19-kserve-75dd7dfcdf-jd4zd\" (UID: \"57a1b9ad-906d-422c-b3ca-0f9a8f5f560a\") " pod="llm/e2e-same-id-b-d0082e19-kserve-75dd7dfcdf-jd4zd"
Nov 01 04:09:18.672423 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:09:18.672388    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-d0082e19-kserve-75dd7dfcdf-jd4zd"
Nov 01 04:09:19.070615 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:09:19.070591    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-d0082e19-kserve-75dd7dfcdf-jd4zd"]
Nov 01 04:09:19.072310 ip-10-0-133-54 kubenswrapper[4400]: W1101 04:09:19.072284    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod57a1b9ad_906d_422c_b3ca_0f9a8f5f560a.slice/crio-f9dff716d6b48874a244747ae2ce4a47883e274f9a2f1bbf716177693c793e93 WatchSource:0}: Error finding container f9dff716d6b48874a244747ae2ce4a47883e274f9a2f1bbf716177693c793e93: Status 404 returned error can't find the container with id f9dff716d6b48874a244747ae2ce4a47883e274f9a2f1bbf716177693c793e93
Nov 01 04:09:19.329152 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:09:19.329106    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-d0082e19-kserve-75dd7dfcdf-jd4zd" event={"ID":"57a1b9ad-906d-422c-b3ca-0f9a8f5f560a","Type":"ContainerStarted","Data":"c877f394e31091770437e307a60a87ac0d4308f496b2f9deca99fe99278f48c4"}
Nov 01 04:09:19.329152 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:09:19.329153    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-d0082e19-kserve-75dd7dfcdf-jd4zd" event={"ID":"57a1b9ad-906d-422c-b3ca-0f9a8f5f560a","Type":"ContainerStarted","Data":"f9dff716d6b48874a244747ae2ce4a47883e274f9a2f1bbf716177693c793e93"}
Nov 01 04:09:19.673562 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:09:19.673482    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-d0082e19-kserve-75dd7dfcdf-jd4zd"
Nov 01 04:09:23.119996 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:09:23.119942    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-d0082e19-kserve-75dd7dfcdf-jd4zd" podStartSLOduration=5.119927503 podStartE2EDuration="5.119927503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:09:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:09:19.470141296 +0000 UTC m=+3765.984787159" watchObservedRunningTime="2026-11-01 04:09:23.119927503 +0000 UTC m=+3769.634573365"
Nov 01 04:09:23.120406 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:09:23.120260    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 04:09:26.613710 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:09:26.612789    4400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-zkjpt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 04:09:26.613710 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:09:26.612869    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zkjpt" podUID="7204663c-ef51-407b-8b56-65e9e54a2d51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 04:09:28.672989 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:09:28.672957    4400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-d0082e19-kserve-75dd7dfcdf-jd4zd"
Nov 01 04:09:28.685544 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:09:28.685514    4400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-d0082e19-kserve-75dd7dfcdf-jd4zd"
Nov 01 04:09:29.415084 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:09:29.415052    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-d0082e19-kserve-75dd7dfcdf-jd4zd"
Nov 01 04:09:31.732776 ip-10-0-133-54 kubenswrapper[4400]: E1101 04:09:31.732736    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 04:09:53.143554 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:09:53.143519    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 04:10:01.729745 ip-10-0-133-54 kubenswrapper[4400]: E1101 04:10:01.729661    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 04:10:05.413533 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:10:05.413503    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 04:10:15.323221 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:10:15.323140    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-d0082e19-kserve-75dd7dfcdf-jd4zd"]
Nov 01 04:10:18.998747 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:10:18.998703    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 04:10:26.566281 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:10:26.566220    4400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-zkjpt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 04:10:26.567107 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:10:26.567071    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zkjpt" podUID="7204663c-ef51-407b-8b56-65e9e54a2d51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 04:10:28.763989 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:10:28.763950    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 04:10:30.343777 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:10:30.343734    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-d0082e19-kserve-75dd7dfcdf-jd4zd" podUID="57a1b9ad-906d-422c-b3ca-0f9a8f5f560a" containerName="main" containerID="cri-o://c877f394e31091770437e307a60a87ac0d4308f496b2f9deca99fe99278f48c4" gracePeriod=45
Nov 01 04:10:30.404311 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:10:30.404274    4400 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-b-d0082e19-kserve-75dd7dfcdf-jd4zd" podUID="57a1b9ad-906d-422c-b3ca-0f9a8f5f560a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.140:8000/ready\": dial tcp 10.132.0.140:8000: connect: connection refused"
Nov 01 04:10:30.604725 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:10:30.604649    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-d0082e19-kserve-75dd7dfcdf-jd4zd"
Nov 01 04:10:30.622060 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:10:30.622026    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57a1b9ad-906d-422c-b3ca-0f9a8f5f560a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57a1b9ad-906d-422c-b3ca-0f9a8f5f560a-tmp-dir\") pod \"57a1b9ad-906d-422c-b3ca-0f9a8f5f560a\" (UID: \"57a1b9ad-906d-422c-b3ca-0f9a8f5f560a\") "
Nov 01 04:10:30.622204 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:10:30.622068    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57a1b9ad-906d-422c-b3ca-0f9a8f5f560a-home\" (UniqueName: \"kubernetes.io/empty-dir/57a1b9ad-906d-422c-b3ca-0f9a8f5f560a-home\") pod \"57a1b9ad-906d-422c-b3ca-0f9a8f5f560a\" (UID: \"57a1b9ad-906d-422c-b3ca-0f9a8f5f560a\") "
Nov 01 04:10:30.622204 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:10:30.622144    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57a1b9ad-906d-422c-b3ca-0f9a8f5f560a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57a1b9ad-906d-422c-b3ca-0f9a8f5f560a-model-cache\") pod \"57a1b9ad-906d-422c-b3ca-0f9a8f5f560a\" (UID: \"57a1b9ad-906d-422c-b3ca-0f9a8f5f560a\") "
Nov 01 04:10:30.622321 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:10:30.622212    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57a1b9ad-906d-422c-b3ca-0f9a8f5f560a-kube-api-access-9mdkf\" (UniqueName: \"kubernetes.io/projected/57a1b9ad-906d-422c-b3ca-0f9a8f5f560a-kube-api-access-9mdkf\") pod \"57a1b9ad-906d-422c-b3ca-0f9a8f5f560a\" (UID: \"57a1b9ad-906d-422c-b3ca-0f9a8f5f560a\") "
Nov 01 04:10:30.622321 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:10:30.622265    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57a1b9ad-906d-422c-b3ca-0f9a8f5f560a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/57a1b9ad-906d-422c-b3ca-0f9a8f5f560a-dshm\") pod \"57a1b9ad-906d-422c-b3ca-0f9a8f5f560a\" (UID: \"57a1b9ad-906d-422c-b3ca-0f9a8f5f560a\") "
Nov 01 04:10:30.622321 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:10:30.622281    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57a1b9ad-906d-422c-b3ca-0f9a8f5f560a-tmp-dir" pod "57a1b9ad-906d-422c-b3ca-0f9a8f5f560a" (UID: "57a1b9ad-906d-422c-b3ca-0f9a8f5f560a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:10:30.622484 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:10:30.622365    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57a1b9ad-906d-422c-b3ca-0f9a8f5f560a-home" pod "57a1b9ad-906d-422c-b3ca-0f9a8f5f560a" (UID: "57a1b9ad-906d-422c-b3ca-0f9a8f5f560a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:10:30.622484 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:10:30.622404    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57a1b9ad-906d-422c-b3ca-0f9a8f5f560a-tls-certs\" (UniqueName: \"kubernetes.io/secret/57a1b9ad-906d-422c-b3ca-0f9a8f5f560a-tls-certs\") pod \"57a1b9ad-906d-422c-b3ca-0f9a8f5f560a\" (UID: \"57a1b9ad-906d-422c-b3ca-0f9a8f5f560a\") "
Nov 01 04:10:30.622484 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:10:30.622428    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57a1b9ad-906d-422c-b3ca-0f9a8f5f560a-model-cache" pod "57a1b9ad-906d-422c-b3ca-0f9a8f5f560a" (UID: "57a1b9ad-906d-422c-b3ca-0f9a8f5f560a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:10:30.622746 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:10:30.622725    4400 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57a1b9ad-906d-422c-b3ca-0f9a8f5f560a-tmp-dir\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:10:30.622746 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:10:30.622743    4400 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57a1b9ad-906d-422c-b3ca-0f9a8f5f560a-home\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:10:30.622932 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:10:30.622753    4400 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57a1b9ad-906d-422c-b3ca-0f9a8f5f560a-model-cache\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:10:30.624640 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:10:30.624616    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57a1b9ad-906d-422c-b3ca-0f9a8f5f560a-dshm" pod "57a1b9ad-906d-422c-b3ca-0f9a8f5f560a" (UID: "57a1b9ad-906d-422c-b3ca-0f9a8f5f560a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:10:30.624760 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:10:30.624678    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57a1b9ad-906d-422c-b3ca-0f9a8f5f560a-tls-certs" pod "57a1b9ad-906d-422c-b3ca-0f9a8f5f560a" (UID: "57a1b9ad-906d-422c-b3ca-0f9a8f5f560a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:10:30.624827 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:10:30.624802    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57a1b9ad-906d-422c-b3ca-0f9a8f5f560a-kube-api-access-9mdkf" pod "57a1b9ad-906d-422c-b3ca-0f9a8f5f560a" (UID: "57a1b9ad-906d-422c-b3ca-0f9a8f5f560a"). InnerVolumeSpecName "kube-api-access-9mdkf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:10:30.723786 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:10:30.723752    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mdkf\" (UniqueName: \"kubernetes.io/projected/57a1b9ad-906d-422c-b3ca-0f9a8f5f560a-kube-api-access-9mdkf\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:10:30.723786 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:10:30.723781    4400 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57a1b9ad-906d-422c-b3ca-0f9a8f5f560a-dshm\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:10:30.723786 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:10:30.723795    4400 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57a1b9ad-906d-422c-b3ca-0f9a8f5f560a-tls-certs\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:10:30.825708 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:10:30.825647    4400 generic.go:356] "Generic (PLEG): container finished" podID="57a1b9ad-906d-422c-b3ca-0f9a8f5f560a" containerID="c877f394e31091770437e307a60a87ac0d4308f496b2f9deca99fe99278f48c4" exitCode=0
Nov 01 04:10:30.825882 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:10:30.825728    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-d0082e19-kserve-75dd7dfcdf-jd4zd"
Nov 01 04:10:30.825882 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:10:30.825731    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-d0082e19-kserve-75dd7dfcdf-jd4zd" event={"ID":"57a1b9ad-906d-422c-b3ca-0f9a8f5f560a","Type":"ContainerDied","Data":"c877f394e31091770437e307a60a87ac0d4308f496b2f9deca99fe99278f48c4"}
Nov 01 04:10:30.825882 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:10:30.825781    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-d0082e19-kserve-75dd7dfcdf-jd4zd" event={"ID":"57a1b9ad-906d-422c-b3ca-0f9a8f5f560a","Type":"ContainerDied","Data":"f9dff716d6b48874a244747ae2ce4a47883e274f9a2f1bbf716177693c793e93"}
Nov 01 04:10:30.825882 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:10:30.825801    4400 scope.go:122] "RemoveContainer" containerID="c877f394e31091770437e307a60a87ac0d4308f496b2f9deca99fe99278f48c4"
Nov 01 04:10:30.837866 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:10:30.835823    4400 scope.go:122] "RemoveContainer" containerID="c877f394e31091770437e307a60a87ac0d4308f496b2f9deca99fe99278f48c4"
Nov 01 04:10:30.838128 ip-10-0-133-54 kubenswrapper[4400]: E1101 04:10:30.838105    4400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c877f394e31091770437e307a60a87ac0d4308f496b2f9deca99fe99278f48c4\": container with ID starting with c877f394e31091770437e307a60a87ac0d4308f496b2f9deca99fe99278f48c4 not found: ID does not exist" containerID="c877f394e31091770437e307a60a87ac0d4308f496b2f9deca99fe99278f48c4"
Nov 01 04:10:30.838194 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:10:30.838136    4400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c877f394e31091770437e307a60a87ac0d4308f496b2f9deca99fe99278f48c4"} err="failed to get container status \"c877f394e31091770437e307a60a87ac0d4308f496b2f9deca99fe99278f48c4\": rpc error: code = NotFound desc = could not find container \"c877f394e31091770437e307a60a87ac0d4308f496b2f9deca99fe99278f48c4\": container with ID starting with c877f394e31091770437e307a60a87ac0d4308f496b2f9deca99fe99278f48c4 not found: ID does not exist"
Nov 01 04:10:30.992741 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:10:30.992650    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-d0082e19-kserve-75dd7dfcdf-jd4zd"]
Nov 01 04:10:30.996639 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:10:30.996613    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-d0082e19-kserve-75dd7dfcdf-jd4zd"]
Nov 01 04:10:31.614025 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:10:31.613991    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57a1b9ad-906d-422c-b3ca-0f9a8f5f560a" path="/var/lib/kubelet/pods/57a1b9ad-906d-422c-b3ca-0f9a8f5f560a/volumes"
Nov 01 04:10:31.730817 ip-10-0-133-54 kubenswrapper[4400]: E1101 04:10:31.730780    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 04:10:38.592155 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:10:38.592111    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 04:11:01.730580 ip-10-0-133-54 kubenswrapper[4400]: E1101 04:11:01.730538    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 04:11:01.998409 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:01.998314    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 04:11:02.289946 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:02.289866    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-54d5899bf9-6xzng"]
Nov 01 04:11:02.290367 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:02.290353    4400 cpu_manager.go:414] "containerMap: removing container" podUID="57a1b9ad-906d-422c-b3ca-0f9a8f5f560a" containerName="main"
Nov 01 04:11:02.290409 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:02.290369    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57a1b9ad-906d-422c-b3ca-0f9a8f5f560a" containerName="main"
Nov 01 04:11:02.290462 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:02.290453    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="57a1b9ad-906d-422c-b3ca-0f9a8f5f560a" containerName="main"
Nov 01 04:11:02.293863 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:02.293839    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-54d5899bf9-6xzng"
Nov 01 04:11:02.388407 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:02.388363    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97f0f1f0-cc81-403e-8fe0-84f5ed50ed5a-openshift-service-ca\") pod \"maas-api-54d5899bf9-6xzng\" (UID: \"97f0f1f0-cc81-403e-8fe0-84f5ed50ed5a\") " pod="odh-ai-gateway-infra/maas-api-54d5899bf9-6xzng"
Nov 01 04:11:02.389038 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:02.389009    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97f0f1f0-cc81-403e-8fe0-84f5ed50ed5a-metrics-certs\") pod \"maas-api-54d5899bf9-6xzng\" (UID: \"97f0f1f0-cc81-403e-8fe0-84f5ed50ed5a\") " pod="odh-ai-gateway-infra/maas-api-54d5899bf9-6xzng"
Nov 01 04:11:02.389400 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:02.389382    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97f0f1f0-cc81-403e-8fe0-84f5ed50ed5a-maas-api-tls\") pod \"maas-api-54d5899bf9-6xzng\" (UID: \"97f0f1f0-cc81-403e-8fe0-84f5ed50ed5a\") " pod="odh-ai-gateway-infra/maas-api-54d5899bf9-6xzng"
Nov 01 04:11:02.389662 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:02.389642    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtccd\" (UniqueName: \"kubernetes.io/projected/97f0f1f0-cc81-403e-8fe0-84f5ed50ed5a-kube-api-access-wtccd\") pod \"maas-api-54d5899bf9-6xzng\" (UID: \"97f0f1f0-cc81-403e-8fe0-84f5ed50ed5a\") " pod="odh-ai-gateway-infra/maas-api-54d5899bf9-6xzng"
Nov 01 04:11:02.404570 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:02.404525    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-54d5899bf9-6xzng"]
Nov 01 04:11:02.490462 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:02.490424    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97f0f1f0-cc81-403e-8fe0-84f5ed50ed5a-maas-api-tls\") pod \"maas-api-54d5899bf9-6xzng\" (UID: \"97f0f1f0-cc81-403e-8fe0-84f5ed50ed5a\") " pod="odh-ai-gateway-infra/maas-api-54d5899bf9-6xzng"
Nov 01 04:11:02.490666 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:02.490484    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtccd\" (UniqueName: \"kubernetes.io/projected/97f0f1f0-cc81-403e-8fe0-84f5ed50ed5a-kube-api-access-wtccd\") pod \"maas-api-54d5899bf9-6xzng\" (UID: \"97f0f1f0-cc81-403e-8fe0-84f5ed50ed5a\") " pod="odh-ai-gateway-infra/maas-api-54d5899bf9-6xzng"
Nov 01 04:11:02.490666 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:02.490527    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97f0f1f0-cc81-403e-8fe0-84f5ed50ed5a-openshift-service-ca\") pod \"maas-api-54d5899bf9-6xzng\" (UID: \"97f0f1f0-cc81-403e-8fe0-84f5ed50ed5a\") " pod="odh-ai-gateway-infra/maas-api-54d5899bf9-6xzng"
Nov 01 04:11:02.490666 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:02.490556    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97f0f1f0-cc81-403e-8fe0-84f5ed50ed5a-metrics-certs\") pod \"maas-api-54d5899bf9-6xzng\" (UID: \"97f0f1f0-cc81-403e-8fe0-84f5ed50ed5a\") " pod="odh-ai-gateway-infra/maas-api-54d5899bf9-6xzng"
Nov 01 04:11:02.491334 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:02.491308    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97f0f1f0-cc81-403e-8fe0-84f5ed50ed5a-openshift-service-ca\") pod \"maas-api-54d5899bf9-6xzng\" (UID: \"97f0f1f0-cc81-403e-8fe0-84f5ed50ed5a\") " pod="odh-ai-gateway-infra/maas-api-54d5899bf9-6xzng"
Nov 01 04:11:02.493956 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:02.493905    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97f0f1f0-cc81-403e-8fe0-84f5ed50ed5a-metrics-certs\") pod \"maas-api-54d5899bf9-6xzng\" (UID: \"97f0f1f0-cc81-403e-8fe0-84f5ed50ed5a\") " pod="odh-ai-gateway-infra/maas-api-54d5899bf9-6xzng"
Nov 01 04:11:02.494155 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:02.493928    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97f0f1f0-cc81-403e-8fe0-84f5ed50ed5a-maas-api-tls\") pod \"maas-api-54d5899bf9-6xzng\" (UID: \"97f0f1f0-cc81-403e-8fe0-84f5ed50ed5a\") " pod="odh-ai-gateway-infra/maas-api-54d5899bf9-6xzng"
Nov 01 04:11:02.553204 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:02.553132    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtccd\" (UniqueName: \"kubernetes.io/projected/97f0f1f0-cc81-403e-8fe0-84f5ed50ed5a-kube-api-access-wtccd\") pod \"maas-api-54d5899bf9-6xzng\" (UID: \"97f0f1f0-cc81-403e-8fe0-84f5ed50ed5a\") " pod="odh-ai-gateway-infra/maas-api-54d5899bf9-6xzng"
Nov 01 04:11:02.606875 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:02.606838    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-54d5899bf9-6xzng"
Nov 01 04:11:02.986395 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:02.986365    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-54d5899bf9-6xzng"]
Nov 01 04:11:02.987458 ip-10-0-133-54 kubenswrapper[4400]: W1101 04:11:02.987431    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97f0f1f0_cc81_403e_8fe0_84f5ed50ed5a.slice/crio-25245ce5d2ed3ae375a120eb0615cb0820431efc9701616417342ee94bb04f5c WatchSource:0}: Error finding container 25245ce5d2ed3ae375a120eb0615cb0820431efc9701616417342ee94bb04f5c: Status 404 returned error can't find the container with id 25245ce5d2ed3ae375a120eb0615cb0820431efc9701616417342ee94bb04f5c
Nov 01 04:11:02.988913 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:02.988896    4400 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 04:11:02.998053 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:02.998014    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-54d5899bf9-6xzng" event={"ID":"97f0f1f0-cc81-403e-8fe0-84f5ed50ed5a","Type":"ContainerStarted","Data":"25245ce5d2ed3ae375a120eb0615cb0820431efc9701616417342ee94bb04f5c"}
Nov 01 04:11:04.004255 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:04.004220    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-54d5899bf9-6xzng" event={"ID":"97f0f1f0-cc81-403e-8fe0-84f5ed50ed5a","Type":"ContainerStarted","Data":"70dfd0aa87df2226cb17b6157407ee8a55b414af44b337611cceac2c7304745a"}
Nov 01 04:11:04.004776 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:04.004323    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-54d5899bf9-6xzng"
Nov 01 04:11:04.150929 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:04.150824    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-54d5899bf9-6xzng" podStartSLOduration=1.2745991 podStartE2EDuration="2.150809949s" totalImagesPullingTime="876.210849ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:11:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:11:04.149755425 +0000 UTC m=+3870.664401287" watchObservedRunningTime="2026-11-01 04:11:04.150809949 +0000 UTC m=+3870.665455811"
Nov 01 04:11:10.015277 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:10.015248    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-54d5899bf9-6xzng"
Nov 01 04:11:17.130243 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:17.130178    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ca6b5505-6f8585644d-82lq9"]
Nov 01 04:11:17.137116 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:17.137085    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca6b5505-6f8585644d-82lq9"
Nov 01 04:11:17.241955 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:17.241926    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-ca6b5505-serving-cert\""
Nov 01 04:11:17.244181 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:17.244151    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ca6b5505-6f8585644d-82lq9"]
Nov 01 04:11:17.252380 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:17.252355    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b033c224-2a4b-4344-80cf-0c7791b32248-maas-api-tls\") pod \"maas-api-e2e-ait-ca6b5505-6f8585644d-82lq9\" (UID: \"b033c224-2a4b-4344-80cf-0c7791b32248\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca6b5505-6f8585644d-82lq9"
Nov 01 04:11:17.252539 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:17.252510    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b033c224-2a4b-4344-80cf-0c7791b32248-openshift-service-ca\") pod \"maas-api-e2e-ait-ca6b5505-6f8585644d-82lq9\" (UID: \"b033c224-2a4b-4344-80cf-0c7791b32248\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca6b5505-6f8585644d-82lq9"
Nov 01 04:11:17.252618 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:17.252572    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b033c224-2a4b-4344-80cf-0c7791b32248-metrics-certs\") pod \"maas-api-e2e-ait-ca6b5505-6f8585644d-82lq9\" (UID: \"b033c224-2a4b-4344-80cf-0c7791b32248\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca6b5505-6f8585644d-82lq9"
Nov 01 04:11:17.252698 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:17.252670    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhr9j\" (UniqueName: \"kubernetes.io/projected/b033c224-2a4b-4344-80cf-0c7791b32248-kube-api-access-lhr9j\") pod \"maas-api-e2e-ait-ca6b5505-6f8585644d-82lq9\" (UID: \"b033c224-2a4b-4344-80cf-0c7791b32248\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca6b5505-6f8585644d-82lq9"
Nov 01 04:11:17.354142 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:17.354107    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhr9j\" (UniqueName: \"kubernetes.io/projected/b033c224-2a4b-4344-80cf-0c7791b32248-kube-api-access-lhr9j\") pod \"maas-api-e2e-ait-ca6b5505-6f8585644d-82lq9\" (UID: \"b033c224-2a4b-4344-80cf-0c7791b32248\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca6b5505-6f8585644d-82lq9"
Nov 01 04:11:17.354343 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:17.354179    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b033c224-2a4b-4344-80cf-0c7791b32248-maas-api-tls\") pod \"maas-api-e2e-ait-ca6b5505-6f8585644d-82lq9\" (UID: \"b033c224-2a4b-4344-80cf-0c7791b32248\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca6b5505-6f8585644d-82lq9"
Nov 01 04:11:17.354343 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:17.354241    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b033c224-2a4b-4344-80cf-0c7791b32248-openshift-service-ca\") pod \"maas-api-e2e-ait-ca6b5505-6f8585644d-82lq9\" (UID: \"b033c224-2a4b-4344-80cf-0c7791b32248\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca6b5505-6f8585644d-82lq9"
Nov 01 04:11:17.354343 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:17.354268    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b033c224-2a4b-4344-80cf-0c7791b32248-metrics-certs\") pod \"maas-api-e2e-ait-ca6b5505-6f8585644d-82lq9\" (UID: \"b033c224-2a4b-4344-80cf-0c7791b32248\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca6b5505-6f8585644d-82lq9"
Nov 01 04:11:17.354343 ip-10-0-133-54 kubenswrapper[4400]: E1101 04:11:17.354310    4400 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-ca6b5505-serving-cert: secret "maas-api-e2e-ait-ca6b5505-serving-cert" not found
Nov 01 04:11:17.354504 ip-10-0-133-54 kubenswrapper[4400]: E1101 04:11:17.354380    4400 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b033c224-2a4b-4344-80cf-0c7791b32248-maas-api-tls podName:b033c224-2a4b-4344-80cf-0c7791b32248 nodeName:}" failed. No retries permitted until 2026-11-01 04:11:17.854359726 +0000 UTC m=+3884.369005592 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b033c224-2a4b-4344-80cf-0c7791b32248-maas-api-tls") pod "maas-api-e2e-ait-ca6b5505-6f8585644d-82lq9" (UID: "b033c224-2a4b-4344-80cf-0c7791b32248") : secret "maas-api-e2e-ait-ca6b5505-serving-cert" not found
Nov 01 04:11:17.354937 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:17.354914    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b033c224-2a4b-4344-80cf-0c7791b32248-openshift-service-ca\") pod \"maas-api-e2e-ait-ca6b5505-6f8585644d-82lq9\" (UID: \"b033c224-2a4b-4344-80cf-0c7791b32248\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca6b5505-6f8585644d-82lq9"
Nov 01 04:11:17.356976 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:17.356955    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b033c224-2a4b-4344-80cf-0c7791b32248-metrics-certs\") pod \"maas-api-e2e-ait-ca6b5505-6f8585644d-82lq9\" (UID: \"b033c224-2a4b-4344-80cf-0c7791b32248\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca6b5505-6f8585644d-82lq9"
Nov 01 04:11:17.422777 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:17.422668    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhr9j\" (UniqueName: \"kubernetes.io/projected/b033c224-2a4b-4344-80cf-0c7791b32248-kube-api-access-lhr9j\") pod \"maas-api-e2e-ait-ca6b5505-6f8585644d-82lq9\" (UID: \"b033c224-2a4b-4344-80cf-0c7791b32248\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca6b5505-6f8585644d-82lq9"
Nov 01 04:11:17.860272 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:17.860229    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b033c224-2a4b-4344-80cf-0c7791b32248-maas-api-tls\") pod \"maas-api-e2e-ait-ca6b5505-6f8585644d-82lq9\" (UID: \"b033c224-2a4b-4344-80cf-0c7791b32248\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca6b5505-6f8585644d-82lq9"
Nov 01 04:11:17.862884 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:17.862854    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b033c224-2a4b-4344-80cf-0c7791b32248-maas-api-tls\") pod \"maas-api-e2e-ait-ca6b5505-6f8585644d-82lq9\" (UID: \"b033c224-2a4b-4344-80cf-0c7791b32248\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca6b5505-6f8585644d-82lq9"
Nov 01 04:11:18.051886 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:18.051840    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca6b5505-6f8585644d-82lq9"
Nov 01 04:11:18.427982 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:18.427942    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ca6b5505-6f8585644d-82lq9"]
Nov 01 04:11:18.430526 ip-10-0-133-54 kubenswrapper[4400]: W1101 04:11:18.430503    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb033c224_2a4b_4344_80cf_0c7791b32248.slice/crio-a978984df6fe4cb2238a1bcdc0352bac7fc0ef01ceb2b4bf67ef0a073fc0f3d3 WatchSource:0}: Error finding container a978984df6fe4cb2238a1bcdc0352bac7fc0ef01ceb2b4bf67ef0a073fc0f3d3: Status 404 returned error can't find the container with id a978984df6fe4cb2238a1bcdc0352bac7fc0ef01ceb2b4bf67ef0a073fc0f3d3
Nov 01 04:11:18.479149 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:18.479113    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7557fccb47-9l4f5"]
Nov 01 04:11:18.479377 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:18.479352    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-7557fccb47-9l4f5" podUID="7b2fe3be-c056-4ee9-a6bc-257f85273a05" containerName="maas-api" containerID="cri-o://c9470635d0c5a2a048de89756a521ccbb7de85eb81547ddddd6daebf8a68a9fa" gracePeriod=30
Nov 01 04:11:19.079304 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:19.079267    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca6b5505-6f8585644d-82lq9" event={"ID":"b033c224-2a4b-4344-80cf-0c7791b32248","Type":"ContainerStarted","Data":"a978984df6fe4cb2238a1bcdc0352bac7fc0ef01ceb2b4bf67ef0a073fc0f3d3"}
Nov 01 04:11:19.723061 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:19.723036    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7557fccb47-9l4f5"
Nov 01 04:11:19.890158 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:19.890119    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b2fe3be-c056-4ee9-a6bc-257f85273a05-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b2fe3be-c056-4ee9-a6bc-257f85273a05-openshift-service-ca\") pod \"7b2fe3be-c056-4ee9-a6bc-257f85273a05\" (UID: \"7b2fe3be-c056-4ee9-a6bc-257f85273a05\") "
Nov 01 04:11:19.890158 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:19.890165    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b2fe3be-c056-4ee9-a6bc-257f85273a05-kube-api-access-mhkpv\" (UniqueName: \"kubernetes.io/projected/7b2fe3be-c056-4ee9-a6bc-257f85273a05-kube-api-access-mhkpv\") pod \"7b2fe3be-c056-4ee9-a6bc-257f85273a05\" (UID: \"7b2fe3be-c056-4ee9-a6bc-257f85273a05\") "
Nov 01 04:11:19.890426 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:19.890202    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b2fe3be-c056-4ee9-a6bc-257f85273a05-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b2fe3be-c056-4ee9-a6bc-257f85273a05-maas-api-tls\") pod \"7b2fe3be-c056-4ee9-a6bc-257f85273a05\" (UID: \"7b2fe3be-c056-4ee9-a6bc-257f85273a05\") "
Nov 01 04:11:19.890426 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:19.890247    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b2fe3be-c056-4ee9-a6bc-257f85273a05-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b2fe3be-c056-4ee9-a6bc-257f85273a05-metrics-certs\") pod \"7b2fe3be-c056-4ee9-a6bc-257f85273a05\" (UID: \"7b2fe3be-c056-4ee9-a6bc-257f85273a05\") "
Nov 01 04:11:19.890611 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:19.890588    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b2fe3be-c056-4ee9-a6bc-257f85273a05-openshift-service-ca" pod "7b2fe3be-c056-4ee9-a6bc-257f85273a05" (UID: "7b2fe3be-c056-4ee9-a6bc-257f85273a05"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:11:19.892571 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:19.892536    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b2fe3be-c056-4ee9-a6bc-257f85273a05-maas-api-tls" pod "7b2fe3be-c056-4ee9-a6bc-257f85273a05" (UID: "7b2fe3be-c056-4ee9-a6bc-257f85273a05"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:11:19.892571 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:19.892560    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b2fe3be-c056-4ee9-a6bc-257f85273a05-metrics-certs" pod "7b2fe3be-c056-4ee9-a6bc-257f85273a05" (UID: "7b2fe3be-c056-4ee9-a6bc-257f85273a05"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:11:19.892765 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:19.892599    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b2fe3be-c056-4ee9-a6bc-257f85273a05-kube-api-access-mhkpv" pod "7b2fe3be-c056-4ee9-a6bc-257f85273a05" (UID: "7b2fe3be-c056-4ee9-a6bc-257f85273a05"). InnerVolumeSpecName "kube-api-access-mhkpv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:11:19.991275 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:19.991239    4400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b2fe3be-c056-4ee9-a6bc-257f85273a05-maas-api-tls\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:11:19.991275 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:19.991268    4400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b2fe3be-c056-4ee9-a6bc-257f85273a05-metrics-certs\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:11:19.991493 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:19.991300    4400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b2fe3be-c056-4ee9-a6bc-257f85273a05-openshift-service-ca\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:11:19.991493 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:19.991316    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhkpv\" (UniqueName: \"kubernetes.io/projected/7b2fe3be-c056-4ee9-a6bc-257f85273a05-kube-api-access-mhkpv\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:11:20.093111 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:20.093074    4400 generic.go:356] "Generic (PLEG): container finished" podID="7b2fe3be-c056-4ee9-a6bc-257f85273a05" containerID="c9470635d0c5a2a048de89756a521ccbb7de85eb81547ddddd6daebf8a68a9fa" exitCode=0
Nov 01 04:11:20.093321 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:20.093149    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7557fccb47-9l4f5"
Nov 01 04:11:20.093321 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:20.093150    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7557fccb47-9l4f5" event={"ID":"7b2fe3be-c056-4ee9-a6bc-257f85273a05","Type":"ContainerDied","Data":"c9470635d0c5a2a048de89756a521ccbb7de85eb81547ddddd6daebf8a68a9fa"}
Nov 01 04:11:20.093321 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:20.093248    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7557fccb47-9l4f5" event={"ID":"7b2fe3be-c056-4ee9-a6bc-257f85273a05","Type":"ContainerDied","Data":"a342f8228017e8b1fb03988a792d2744d6aa448abef593cd36f75a3420ffa991"}
Nov 01 04:11:20.093321 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:20.093277    4400 scope.go:122] "RemoveContainer" containerID="c9470635d0c5a2a048de89756a521ccbb7de85eb81547ddddd6daebf8a68a9fa"
Nov 01 04:11:20.095746 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:20.095673    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca6b5505-6f8585644d-82lq9" event={"ID":"b033c224-2a4b-4344-80cf-0c7791b32248","Type":"ContainerStarted","Data":"c801aa2569fe8858be6e24a7a38f7f1b6d60f4f04746387ce3ce780845ec5030"}
Nov 01 04:11:20.095946 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:20.095921    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca6b5505-6f8585644d-82lq9"
Nov 01 04:11:20.104623 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:20.104599    4400 scope.go:122] "RemoveContainer" containerID="c9470635d0c5a2a048de89756a521ccbb7de85eb81547ddddd6daebf8a68a9fa"
Nov 01 04:11:20.104952 ip-10-0-133-54 kubenswrapper[4400]: E1101 04:11:20.104932    4400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9470635d0c5a2a048de89756a521ccbb7de85eb81547ddddd6daebf8a68a9fa\": container with ID starting with c9470635d0c5a2a048de89756a521ccbb7de85eb81547ddddd6daebf8a68a9fa not found: ID does not exist" containerID="c9470635d0c5a2a048de89756a521ccbb7de85eb81547ddddd6daebf8a68a9fa"
Nov 01 04:11:20.105003 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:20.104961    4400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9470635d0c5a2a048de89756a521ccbb7de85eb81547ddddd6daebf8a68a9fa"} err="failed to get container status \"c9470635d0c5a2a048de89756a521ccbb7de85eb81547ddddd6daebf8a68a9fa\": rpc error: code = NotFound desc = could not find container \"c9470635d0c5a2a048de89756a521ccbb7de85eb81547ddddd6daebf8a68a9fa\": container with ID starting with c9470635d0c5a2a048de89756a521ccbb7de85eb81547ddddd6daebf8a68a9fa not found: ID does not exist"
Nov 01 04:11:20.270118 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:20.270018    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca6b5505-6f8585644d-82lq9" podStartSLOduration=2.471496548 podStartE2EDuration="3.270000131s" totalImagesPullingTime="798.503583ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:11:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:11:20.268757591 +0000 UTC m=+3886.783403451" watchObservedRunningTime="2026-11-01 04:11:20.270000131 +0000 UTC m=+3886.784645994"
Nov 01 04:11:20.382589 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:20.382557    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7557fccb47-9l4f5"]
Nov 01 04:11:20.388055 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:20.388027    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-7557fccb47-9l4f5"]
Nov 01 04:11:21.613043 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:21.613009    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b2fe3be-c056-4ee9-a6bc-257f85273a05" path="/var/lib/kubelet/pods/7b2fe3be-c056-4ee9-a6bc-257f85273a05/volumes"
Nov 01 04:11:25.419438 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:25.419401    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 04:11:26.107119 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:26.107090    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca6b5505-6f8585644d-82lq9"
Nov 01 04:11:31.750965 ip-10-0-133-54 kubenswrapper[4400]: E1101 04:11:31.750926    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 04:11:56.357499 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:56.357464    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ca6b5505-6f8585644d-82lq9"]
Nov 01 04:11:56.358000 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:56.357706    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca6b5505-6f8585644d-82lq9" podUID="b033c224-2a4b-4344-80cf-0c7791b32248" containerName="maas-api" containerID="cri-o://c801aa2569fe8858be6e24a7a38f7f1b6d60f4f04746387ce3ce780845ec5030" gracePeriod=30
Nov 01 04:11:56.629203 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:56.629178    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca6b5505-6f8585644d-82lq9"
Nov 01 04:11:56.781630 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:56.781575    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b033c224-2a4b-4344-80cf-0c7791b32248-kube-api-access-lhr9j\" (UniqueName: \"kubernetes.io/projected/b033c224-2a4b-4344-80cf-0c7791b32248-kube-api-access-lhr9j\") pod \"b033c224-2a4b-4344-80cf-0c7791b32248\" (UID: \"b033c224-2a4b-4344-80cf-0c7791b32248\") "
Nov 01 04:11:56.781846 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:56.781798    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b033c224-2a4b-4344-80cf-0c7791b32248-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b033c224-2a4b-4344-80cf-0c7791b32248-maas-api-tls\") pod \"b033c224-2a4b-4344-80cf-0c7791b32248\" (UID: \"b033c224-2a4b-4344-80cf-0c7791b32248\") "
Nov 01 04:11:56.781846 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:56.781833    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b033c224-2a4b-4344-80cf-0c7791b32248-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b033c224-2a4b-4344-80cf-0c7791b32248-metrics-certs\") pod \"b033c224-2a4b-4344-80cf-0c7791b32248\" (UID: \"b033c224-2a4b-4344-80cf-0c7791b32248\") "
Nov 01 04:11:56.781969 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:56.781867    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b033c224-2a4b-4344-80cf-0c7791b32248-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b033c224-2a4b-4344-80cf-0c7791b32248-openshift-service-ca\") pod \"b033c224-2a4b-4344-80cf-0c7791b32248\" (UID: \"b033c224-2a4b-4344-80cf-0c7791b32248\") "
Nov 01 04:11:56.782391 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:56.782360    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b033c224-2a4b-4344-80cf-0c7791b32248-openshift-service-ca" pod "b033c224-2a4b-4344-80cf-0c7791b32248" (UID: "b033c224-2a4b-4344-80cf-0c7791b32248"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:11:56.784290 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:56.784261    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b033c224-2a4b-4344-80cf-0c7791b32248-maas-api-tls" pod "b033c224-2a4b-4344-80cf-0c7791b32248" (UID: "b033c224-2a4b-4344-80cf-0c7791b32248"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:11:56.784290 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:56.784275    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b033c224-2a4b-4344-80cf-0c7791b32248-kube-api-access-lhr9j" pod "b033c224-2a4b-4344-80cf-0c7791b32248" (UID: "b033c224-2a4b-4344-80cf-0c7791b32248"). InnerVolumeSpecName "kube-api-access-lhr9j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:11:56.784490 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:56.784417    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b033c224-2a4b-4344-80cf-0c7791b32248-metrics-certs" pod "b033c224-2a4b-4344-80cf-0c7791b32248" (UID: "b033c224-2a4b-4344-80cf-0c7791b32248"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:11:56.884229 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:56.884069    4400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b033c224-2a4b-4344-80cf-0c7791b32248-maas-api-tls\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:11:56.884229 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:56.884106    4400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b033c224-2a4b-4344-80cf-0c7791b32248-metrics-certs\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:11:56.884229 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:56.884122    4400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b033c224-2a4b-4344-80cf-0c7791b32248-openshift-service-ca\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:11:56.884229 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:56.884136    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhr9j\" (UniqueName: \"kubernetes.io/projected/b033c224-2a4b-4344-80cf-0c7791b32248-kube-api-access-lhr9j\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:11:57.301807 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:57.301715    4400 generic.go:356] "Generic (PLEG): container finished" podID="b033c224-2a4b-4344-80cf-0c7791b32248" containerID="c801aa2569fe8858be6e24a7a38f7f1b6d60f4f04746387ce3ce780845ec5030" exitCode=0
Nov 01 04:11:57.301807 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:57.301761    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca6b5505-6f8585644d-82lq9" event={"ID":"b033c224-2a4b-4344-80cf-0c7791b32248","Type":"ContainerDied","Data":"c801aa2569fe8858be6e24a7a38f7f1b6d60f4f04746387ce3ce780845ec5030"}
Nov 01 04:11:57.302057 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:57.301811    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca6b5505-6f8585644d-82lq9" event={"ID":"b033c224-2a4b-4344-80cf-0c7791b32248","Type":"ContainerDied","Data":"a978984df6fe4cb2238a1bcdc0352bac7fc0ef01ceb2b4bf67ef0a073fc0f3d3"}
Nov 01 04:11:57.302057 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:57.301838    4400 scope.go:122] "RemoveContainer" containerID="c801aa2569fe8858be6e24a7a38f7f1b6d60f4f04746387ce3ce780845ec5030"
Nov 01 04:11:57.302057 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:57.301856    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca6b5505-6f8585644d-82lq9"
Nov 01 04:11:57.315216 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:57.315192    4400 scope.go:122] "RemoveContainer" containerID="c801aa2569fe8858be6e24a7a38f7f1b6d60f4f04746387ce3ce780845ec5030"
Nov 01 04:11:57.315560 ip-10-0-133-54 kubenswrapper[4400]: E1101 04:11:57.315532    4400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c801aa2569fe8858be6e24a7a38f7f1b6d60f4f04746387ce3ce780845ec5030\": container with ID starting with c801aa2569fe8858be6e24a7a38f7f1b6d60f4f04746387ce3ce780845ec5030 not found: ID does not exist" containerID="c801aa2569fe8858be6e24a7a38f7f1b6d60f4f04746387ce3ce780845ec5030"
Nov 01 04:11:57.315711 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:57.315574    4400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c801aa2569fe8858be6e24a7a38f7f1b6d60f4f04746387ce3ce780845ec5030"} err="failed to get container status \"c801aa2569fe8858be6e24a7a38f7f1b6d60f4f04746387ce3ce780845ec5030\": rpc error: code = NotFound desc = could not find container \"c801aa2569fe8858be6e24a7a38f7f1b6d60f4f04746387ce3ce780845ec5030\": container with ID starting with c801aa2569fe8858be6e24a7a38f7f1b6d60f4f04746387ce3ce780845ec5030 not found: ID does not exist"
Nov 01 04:11:57.472074 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:57.472033    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ca6b5505-6f8585644d-82lq9"]
Nov 01 04:11:57.476605 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:57.476575    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ca6b5505-6f8585644d-82lq9"]
Nov 01 04:11:57.613506 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:11:57.613476    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b033c224-2a4b-4344-80cf-0c7791b32248" path="/var/lib/kubelet/pods/b033c224-2a4b-4344-80cf-0c7791b32248/volumes"
Nov 01 04:12:00.816256 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:12:00.816222    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 04:12:01.729849 ip-10-0-133-54 kubenswrapper[4400]: E1101 04:12:01.729816    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 04:12:18.623306 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:12:18.623272    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-84f12f-6b96644849-fvdmr"]
Nov 01 04:12:18.623848 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:12:18.623767    4400 cpu_manager.go:414] "containerMap: removing container" podUID="7b2fe3be-c056-4ee9-a6bc-257f85273a05" containerName="maas-api"
Nov 01 04:12:18.623848 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:12:18.623782    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b2fe3be-c056-4ee9-a6bc-257f85273a05" containerName="maas-api"
Nov 01 04:12:18.623848 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:12:18.623793    4400 cpu_manager.go:414] "containerMap: removing container" podUID="b033c224-2a4b-4344-80cf-0c7791b32248" containerName="maas-api"
Nov 01 04:12:18.623848 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:12:18.623799    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b033c224-2a4b-4344-80cf-0c7791b32248" containerName="maas-api"
Nov 01 04:12:18.623987 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:12:18.623891    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b2fe3be-c056-4ee9-a6bc-257f85273a05" containerName="maas-api"
Nov 01 04:12:18.623987 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:12:18.623899    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="b033c224-2a4b-4344-80cf-0c7791b32248" containerName="maas-api"
Nov 01 04:12:18.626993 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:12:18.626970    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-84f12f-6b96644849-fvdmr"
Nov 01 04:12:18.683533 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:12:18.683498    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-authcfg-http-84f12f-serving-cert\""
Nov 01 04:12:18.707055 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:12:18.707020    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d97be95-6efe-4683-9e54-7829190d0b50-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-84f12f-6b96644849-fvdmr\" (UID: \"7d97be95-6efe-4683-9e54-7829190d0b50\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-84f12f-6b96644849-fvdmr"
Nov 01 04:12:18.707226 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:12:18.707074    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d97be95-6efe-4683-9e54-7829190d0b50-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-84f12f-6b96644849-fvdmr\" (UID: \"7d97be95-6efe-4683-9e54-7829190d0b50\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-84f12f-6b96644849-fvdmr"
Nov 01 04:12:18.707271 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:12:18.707223    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cskl2\" (UniqueName: \"kubernetes.io/projected/7d97be95-6efe-4683-9e54-7829190d0b50-kube-api-access-cskl2\") pod \"maas-api-e2e-authcfg-http-84f12f-6b96644849-fvdmr\" (UID: \"7d97be95-6efe-4683-9e54-7829190d0b50\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-84f12f-6b96644849-fvdmr"
Nov 01 04:12:18.707335 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:12:18.707321    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d97be95-6efe-4683-9e54-7829190d0b50-metrics-certs\") pod \"maas-api-e2e-authcfg-http-84f12f-6b96644849-fvdmr\" (UID: \"7d97be95-6efe-4683-9e54-7829190d0b50\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-84f12f-6b96644849-fvdmr"
Nov 01 04:12:18.736927 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:12:18.736897    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-84f12f-6b96644849-fvdmr"]
Nov 01 04:12:18.808122 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:12:18.808089    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d97be95-6efe-4683-9e54-7829190d0b50-metrics-certs\") pod \"maas-api-e2e-authcfg-http-84f12f-6b96644849-fvdmr\" (UID: \"7d97be95-6efe-4683-9e54-7829190d0b50\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-84f12f-6b96644849-fvdmr"
Nov 01 04:12:18.808324 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:12:18.808160    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d97be95-6efe-4683-9e54-7829190d0b50-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-84f12f-6b96644849-fvdmr\" (UID: \"7d97be95-6efe-4683-9e54-7829190d0b50\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-84f12f-6b96644849-fvdmr"
Nov 01 04:12:18.808324 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:12:18.808186    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d97be95-6efe-4683-9e54-7829190d0b50-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-84f12f-6b96644849-fvdmr\" (UID: \"7d97be95-6efe-4683-9e54-7829190d0b50\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-84f12f-6b96644849-fvdmr"
Nov 01 04:12:18.808324 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:12:18.808239    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cskl2\" (UniqueName: \"kubernetes.io/projected/7d97be95-6efe-4683-9e54-7829190d0b50-kube-api-access-cskl2\") pod \"maas-api-e2e-authcfg-http-84f12f-6b96644849-fvdmr\" (UID: \"7d97be95-6efe-4683-9e54-7829190d0b50\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-84f12f-6b96644849-fvdmr"
Nov 01 04:12:18.808508 ip-10-0-133-54 kubenswrapper[4400]: E1101 04:12:18.808324    4400 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-authcfg-http-84f12f-serving-cert: secret "maas-api-e2e-authcfg-http-84f12f-serving-cert" not found
Nov 01 04:12:18.808508 ip-10-0-133-54 kubenswrapper[4400]: E1101 04:12:18.808403    4400 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d97be95-6efe-4683-9e54-7829190d0b50-maas-api-tls podName:7d97be95-6efe-4683-9e54-7829190d0b50 nodeName:}" failed. No retries permitted until 2026-11-01 04:12:19.308384072 +0000 UTC m=+3945.823029928 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7d97be95-6efe-4683-9e54-7829190d0b50-maas-api-tls") pod "maas-api-e2e-authcfg-http-84f12f-6b96644849-fvdmr" (UID: "7d97be95-6efe-4683-9e54-7829190d0b50") : secret "maas-api-e2e-authcfg-http-84f12f-serving-cert" not found
Nov 01 04:12:18.808868 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:12:18.808849    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d97be95-6efe-4683-9e54-7829190d0b50-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-84f12f-6b96644849-fvdmr\" (UID: \"7d97be95-6efe-4683-9e54-7829190d0b50\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-84f12f-6b96644849-fvdmr"
Nov 01 04:12:18.810915 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:12:18.810890    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d97be95-6efe-4683-9e54-7829190d0b50-metrics-certs\") pod \"maas-api-e2e-authcfg-http-84f12f-6b96644849-fvdmr\" (UID: \"7d97be95-6efe-4683-9e54-7829190d0b50\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-84f12f-6b96644849-fvdmr"
Nov 01 04:12:18.876335 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:12:18.876255    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cskl2\" (UniqueName: \"kubernetes.io/projected/7d97be95-6efe-4683-9e54-7829190d0b50-kube-api-access-cskl2\") pod \"maas-api-e2e-authcfg-http-84f12f-6b96644849-fvdmr\" (UID: \"7d97be95-6efe-4683-9e54-7829190d0b50\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-84f12f-6b96644849-fvdmr"
Nov 01 04:12:19.314076 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:12:19.313975    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d97be95-6efe-4683-9e54-7829190d0b50-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-84f12f-6b96644849-fvdmr\" (UID: \"7d97be95-6efe-4683-9e54-7829190d0b50\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-84f12f-6b96644849-fvdmr"
Nov 01 04:12:19.316595 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:12:19.316567    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d97be95-6efe-4683-9e54-7829190d0b50-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-84f12f-6b96644849-fvdmr\" (UID: \"7d97be95-6efe-4683-9e54-7829190d0b50\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-84f12f-6b96644849-fvdmr"
Nov 01 04:12:19.539998 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:12:19.539963    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-84f12f-6b96644849-fvdmr"
Nov 01 04:12:19.916363 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:12:19.916300    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-84f12f-6b96644849-fvdmr"]
Nov 01 04:12:19.916836 ip-10-0-133-54 kubenswrapper[4400]: W1101 04:12:19.916799    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d97be95_6efe_4683_9e54_7829190d0b50.slice/crio-1d5381dca06d451e360e394239b2c525123bf19252379ce959d5ac2ba9b39dbf WatchSource:0}: Error finding container 1d5381dca06d451e360e394239b2c525123bf19252379ce959d5ac2ba9b39dbf: Status 404 returned error can't find the container with id 1d5381dca06d451e360e394239b2c525123bf19252379ce959d5ac2ba9b39dbf
Nov 01 04:12:20.421253 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:12:20.421214    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-84f12f-6b96644849-fvdmr" event={"ID":"7d97be95-6efe-4683-9e54-7829190d0b50","Type":"ContainerStarted","Data":"1d5381dca06d451e360e394239b2c525123bf19252379ce959d5ac2ba9b39dbf"}
Nov 01 04:12:21.430813 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:12:21.430764    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-84f12f-6b96644849-fvdmr" event={"ID":"7d97be95-6efe-4683-9e54-7829190d0b50","Type":"ContainerStarted","Data":"6f15daaffb5197621a78a14247427abaf9ed69ecda3452e86c3135607266a2e6"}
Nov 01 04:12:21.431455 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:12:21.431432    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-84f12f-6b96644849-fvdmr"
Nov 01 04:12:21.556917 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:12:21.556866    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-84f12f-6b96644849-fvdmr" podStartSLOduration=2.658225943 podStartE2EDuration="3.556851618s" totalImagesPullingTime="898.625675ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:12:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:12:21.554897166 +0000 UTC m=+3948.069543029" watchObservedRunningTime="2026-11-01 04:12:21.556851618 +0000 UTC m=+3948.071497480"
Nov 01 04:12:26.867503 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:12:26.867462    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 04:12:27.440875 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:12:27.440848    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-84f12f-6b96644849-fvdmr"
Nov 01 04:12:31.730809 ip-10-0-133-54 kubenswrapper[4400]: E1101 04:12:31.730771    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 04:13:01.730364 ip-10-0-133-54 kubenswrapper[4400]: E1101 04:13:01.730320    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 04:13:31.730229 ip-10-0-133-54 kubenswrapper[4400]: E1101 04:13:31.730148    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 04:14:01.730925 ip-10-0-133-54 kubenswrapper[4400]: E1101 04:14:01.730882    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 04:14:19.749441 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:14:19.749403    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-84f12f-6b96644849-fvdmr"]
Nov 01 04:14:19.749891 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:14:19.749622    4400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-84f12f-6b96644849-fvdmr" podUID="7d97be95-6efe-4683-9e54-7829190d0b50" containerName="maas-api" containerID="cri-o://6f15daaffb5197621a78a14247427abaf9ed69ecda3452e86c3135607266a2e6" gracePeriod=30
Nov 01 04:14:20.027225 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:14:20.027197    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-84f12f-6b96644849-fvdmr"
Nov 01 04:14:20.031927 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:14:20.031895    4400 generic.go:356] "Generic (PLEG): container finished" podID="7d97be95-6efe-4683-9e54-7829190d0b50" containerID="6f15daaffb5197621a78a14247427abaf9ed69ecda3452e86c3135607266a2e6" exitCode=0
Nov 01 04:14:20.032061 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:14:20.031940    4400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-84f12f-6b96644849-fvdmr"
Nov 01 04:14:20.032061 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:14:20.031977    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-84f12f-6b96644849-fvdmr" event={"ID":"7d97be95-6efe-4683-9e54-7829190d0b50","Type":"ContainerDied","Data":"6f15daaffb5197621a78a14247427abaf9ed69ecda3452e86c3135607266a2e6"}
Nov 01 04:14:20.032061 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:14:20.032029    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-84f12f-6b96644849-fvdmr" event={"ID":"7d97be95-6efe-4683-9e54-7829190d0b50","Type":"ContainerDied","Data":"1d5381dca06d451e360e394239b2c525123bf19252379ce959d5ac2ba9b39dbf"}
Nov 01 04:14:20.032061 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:14:20.032052    4400 scope.go:122] "RemoveContainer" containerID="6f15daaffb5197621a78a14247427abaf9ed69ecda3452e86c3135607266a2e6"
Nov 01 04:14:20.041131 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:14:20.041112    4400 scope.go:122] "RemoveContainer" containerID="6f15daaffb5197621a78a14247427abaf9ed69ecda3452e86c3135607266a2e6"
Nov 01 04:14:20.041390 ip-10-0-133-54 kubenswrapper[4400]: E1101 04:14:20.041372    4400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f15daaffb5197621a78a14247427abaf9ed69ecda3452e86c3135607266a2e6\": container with ID starting with 6f15daaffb5197621a78a14247427abaf9ed69ecda3452e86c3135607266a2e6 not found: ID does not exist" containerID="6f15daaffb5197621a78a14247427abaf9ed69ecda3452e86c3135607266a2e6"
Nov 01 04:14:20.041458 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:14:20.041402    4400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f15daaffb5197621a78a14247427abaf9ed69ecda3452e86c3135607266a2e6"} err="failed to get container status \"6f15daaffb5197621a78a14247427abaf9ed69ecda3452e86c3135607266a2e6\": rpc error: code = NotFound desc = could not find container \"6f15daaffb5197621a78a14247427abaf9ed69ecda3452e86c3135607266a2e6\": container with ID starting with 6f15daaffb5197621a78a14247427abaf9ed69ecda3452e86c3135607266a2e6 not found: ID does not exist"
Nov 01 04:14:20.194766 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:14:20.194731    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d97be95-6efe-4683-9e54-7829190d0b50-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d97be95-6efe-4683-9e54-7829190d0b50-metrics-certs\") pod \"7d97be95-6efe-4683-9e54-7829190d0b50\" (UID: \"7d97be95-6efe-4683-9e54-7829190d0b50\") "
Nov 01 04:14:20.194943 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:14:20.194913    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d97be95-6efe-4683-9e54-7829190d0b50-kube-api-access-cskl2\" (UniqueName: \"kubernetes.io/projected/7d97be95-6efe-4683-9e54-7829190d0b50-kube-api-access-cskl2\") pod \"7d97be95-6efe-4683-9e54-7829190d0b50\" (UID: \"7d97be95-6efe-4683-9e54-7829190d0b50\") "
Nov 01 04:14:20.194943 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:14:20.194936    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d97be95-6efe-4683-9e54-7829190d0b50-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d97be95-6efe-4683-9e54-7829190d0b50-openshift-service-ca\") pod \"7d97be95-6efe-4683-9e54-7829190d0b50\" (UID: \"7d97be95-6efe-4683-9e54-7829190d0b50\") "
Nov 01 04:14:20.195061 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:14:20.194968    4400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d97be95-6efe-4683-9e54-7829190d0b50-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d97be95-6efe-4683-9e54-7829190d0b50-maas-api-tls\") pod \"7d97be95-6efe-4683-9e54-7829190d0b50\" (UID: \"7d97be95-6efe-4683-9e54-7829190d0b50\") "
Nov 01 04:14:20.195348 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:14:20.195314    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d97be95-6efe-4683-9e54-7829190d0b50-openshift-service-ca" pod "7d97be95-6efe-4683-9e54-7829190d0b50" (UID: "7d97be95-6efe-4683-9e54-7829190d0b50"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:14:20.197093 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:14:20.197069    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d97be95-6efe-4683-9e54-7829190d0b50-metrics-certs" pod "7d97be95-6efe-4683-9e54-7829190d0b50" (UID: "7d97be95-6efe-4683-9e54-7829190d0b50"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:14:20.197212 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:14:20.197176    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d97be95-6efe-4683-9e54-7829190d0b50-maas-api-tls" pod "7d97be95-6efe-4683-9e54-7829190d0b50" (UID: "7d97be95-6efe-4683-9e54-7829190d0b50"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:14:20.197269 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:14:20.197240    4400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d97be95-6efe-4683-9e54-7829190d0b50-kube-api-access-cskl2" pod "7d97be95-6efe-4683-9e54-7829190d0b50" (UID: "7d97be95-6efe-4683-9e54-7829190d0b50"). InnerVolumeSpecName "kube-api-access-cskl2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:14:20.296269 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:14:20.296189    4400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cskl2\" (UniqueName: \"kubernetes.io/projected/7d97be95-6efe-4683-9e54-7829190d0b50-kube-api-access-cskl2\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:14:20.296269 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:14:20.296217    4400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d97be95-6efe-4683-9e54-7829190d0b50-openshift-service-ca\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:14:20.296269 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:14:20.296228    4400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d97be95-6efe-4683-9e54-7829190d0b50-maas-api-tls\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:14:20.296269 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:14:20.296238    4400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d97be95-6efe-4683-9e54-7829190d0b50-metrics-certs\") on node \"ip-10-0-133-54.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:14:20.455448 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:14:20.455410    4400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-84f12f-6b96644849-fvdmr"]
Nov 01 04:14:20.460091 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:14:20.460062    4400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-84f12f-6b96644849-fvdmr"]
Nov 01 04:14:21.613311 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:14:21.613267    4400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d97be95-6efe-4683-9e54-7829190d0b50" path="/var/lib/kubelet/pods/7d97be95-6efe-4683-9e54-7829190d0b50/volumes"
Nov 01 04:14:25.082277 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:14:25.082239    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 04:14:31.730861 ip-10-0-133-54 kubenswrapper[4400]: E1101 04:14:31.730821    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 04:14:36.882021 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:14:36.881982    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 04:14:48.491927 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:14:48.491840    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 04:15:01.730525 ip-10-0-133-54 kubenswrapper[4400]: E1101 04:15:01.730480    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 04:15:09.798173 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:15:09.798137    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 04:15:31.730016 ip-10-0-133-54 kubenswrapper[4400]: E1101 04:15:31.729973    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 04:15:42.506333 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:15:42.506300    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mpdqs"]
Nov 01 04:15:53.316381 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:15:53.316341    4400 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-67978ff98b-l5wnn_a86d4816-aeb1-4bbf-8b17-07a0ff269577/manager/0.log"
Nov 01 04:15:54.158906 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:15:54.158877    4400 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-s65d6_6e617fe1-f094-4315-8c52-0a4d6f7fc809/manager/0.log"
Nov 01 04:15:55.278016 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:15:55.277983    4400 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-sz4xt_ab0fbe1a-6400-45cc-9a94-83a4b65049a4/server/0.log"
Nov 01 04:15:55.525938 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:15:55.525907    4400 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-qh87r_50cf97f8-3174-4880-a97a-5da66a3ccf7a/manager/0.log"
Nov 01 04:15:56.672768 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:15:56.672733    4400 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-54d5899bf9-6xzng_97f0f1f0-cc81-403e-8fe0-84f5ed50ed5a/maas-api/0.log"
Nov 01 04:16:00.919422 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:16:00.919389    4400 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-mpdqs_63495340-52fe-4035-825d-50250e797b66/limitador/0.log"
Nov 01 04:16:01.730527 ip-10-0-133-54 kubenswrapper[4400]: E1101 04:16:01.730483    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 04:16:02.209047 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:16:02.209007    4400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-lw9fr_0fca1311-9bbd-4b5d-a996-2f67742bedb3/discovery/0.log"
Nov 01 04:16:02.822170 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:16:02.822135    4400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d5d9754f-zcj7p_121dbd3a-6f50-4e38-8e72-85297471ca08/kube-auth-proxy/0.log"
Nov 01 04:16:03.060241 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:16:03.060209    4400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7c5b899d5f-8xrcf_fa4d3ec1-8b26-4c06-8290-87c055cdca0a/istio-proxy/0.log"
Nov 01 04:16:04.361787 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:16:04.361747    4400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7c5b899d5f-rtwgm_b9547cd7-2c26-4515-91fe-8aebb8f8b263/istio-proxy/0.log"
Nov 01 04:16:05.490839 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:16:05.490802    4400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7c5b899d5f-zkjpt_7204663c-ef51-407b-8b56-65e9e54a2d51/istio-proxy/0.log"
Nov 01 04:16:06.080022 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:16:06.079984    4400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-z8nnz_555aa9f6-e7e9-4e90-b3a3-1749c88e35ac/payload-pre-processing/0.log"
Nov 01 04:16:06.320312 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:16:06.320283    4400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-26mft_3ffcb7e5-2dfc-4b83-ab39-85bbb773329f/payload-processing/0.log"
Nov 01 04:16:07.546098 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:16:07.546064    4400 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-j6ptb_94e47bc1-fd92-4865-a6b7-b7ab4617b348/main/0.log"
Nov 01 04:16:08.410899 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:16:08.410861    4400 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-2g58t_a15e6f20-9998-40fb-9474-ad3d5af53823/main/0.log"
Nov 01 04:16:08.678357 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:16:08.678274    4400 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88th2j_feec6c6c-1074-485e-88fe-f27a3b1606bb/main/0.log"
Nov 01 04:16:09.220662 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:16:09.220630    4400 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-hzp9k_cdde4990-3c7e-4c5f-b374-2dd5d4fb9ff4/main/0.log"
Nov 01 04:16:19.874017 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:16:19.873938    4400 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-67978ff98b-l5wnn_a86d4816-aeb1-4bbf-8b17-07a0ff269577/manager/0.log"
Nov 01 04:16:20.553736 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:16:20.553701    4400 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-s65d6_6e617fe1-f094-4315-8c52-0a4d6f7fc809/manager/0.log"
Nov 01 04:16:21.549276 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:16:21.549244    4400 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-sz4xt_ab0fbe1a-6400-45cc-9a94-83a4b65049a4/server/0.log"
Nov 01 04:16:21.780864 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:16:21.780833    4400 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-qh87r_50cf97f8-3174-4880-a97a-5da66a3ccf7a/manager/0.log"
Nov 01 04:16:22.662137 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:16:22.662102    4400 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-54d5899bf9-6xzng_97f0f1f0-cc81-403e-8fe0-84f5ed50ed5a/maas-api/0.log"
Nov 01 04:16:26.571407 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:16:26.571375    4400 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-mpdqs_63495340-52fe-4035-825d-50250e797b66/limitador/0.log"
Nov 01 04:16:27.691006 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:16:27.690969    4400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-lw9fr_0fca1311-9bbd-4b5d-a996-2f67742bedb3/discovery/0.log"
Nov 01 04:16:28.281348 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:16:28.281324    4400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d5d9754f-zcj7p_121dbd3a-6f50-4e38-8e72-85297471ca08/kube-auth-proxy/0.log"
Nov 01 04:16:28.522967 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:16:28.522933    4400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7c5b899d5f-8xrcf_fa4d3ec1-8b26-4c06-8290-87c055cdca0a/istio-proxy/0.log"
Nov 01 04:16:29.816317 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:16:29.816288    4400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7c5b899d5f-rtwgm_b9547cd7-2c26-4515-91fe-8aebb8f8b263/istio-proxy/0.log"
Nov 01 04:16:31.006470 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:16:31.006434    4400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7c5b899d5f-zkjpt_7204663c-ef51-407b-8b56-65e9e54a2d51/istio-proxy/0.log"
Nov 01 04:16:31.611229 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:16:31.611199    4400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-z8nnz_555aa9f6-e7e9-4e90-b3a3-1749c88e35ac/payload-pre-processing/0.log"
Nov 01 04:16:31.731341 ip-10-0-133-54 kubenswrapper[4400]: E1101 04:16:31.731295    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 04:16:31.838373 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:16:31.838344    4400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-26mft_3ffcb7e5-2dfc-4b83-ab39-85bbb773329f/payload-processing/0.log"
Nov 01 04:16:32.753746 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:16:32.753706    4400 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-j6ptb_94e47bc1-fd92-4865-a6b7-b7ab4617b348/main/0.log"
Nov 01 04:16:33.608411 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:16:33.608377    4400 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-2g58t_a15e6f20-9998-40fb-9474-ad3d5af53823/main/0.log"
Nov 01 04:16:33.887931 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:16:33.887850    4400 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88th2j_feec6c6c-1074-485e-88fe-f27a3b1606bb/main/0.log"
Nov 01 04:16:34.441612 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:16:34.441583    4400 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-hzp9k_cdde4990-3c7e-4c5f-b374-2dd5d4fb9ff4/main/0.log"
Nov 01 04:17:01.730352 ip-10-0-133-54 kubenswrapper[4400]: E1101 04:17:01.730311    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 04:17:07.472824 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:17:07.472791    4400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-26mft_3ffcb7e5-2dfc-4b83-ab39-85bbb773329f/payload-processing/0.log"
Nov 01 04:17:07.693060 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:17:07.693035    4400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-26mft_3ffcb7e5-2dfc-4b83-ab39-85bbb773329f/payload-processing/0.log"
Nov 01 04:17:31.730618 ip-10-0-133-54 kubenswrapper[4400]: E1101 04:17:31.730577    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 04:17:50.371980 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:17:50.371905    4400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-26mft_3ffcb7e5-2dfc-4b83-ab39-85bbb773329f/payload-processing/0.log"
Nov 01 04:17:50.610113 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:17:50.610083    4400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-26mft_3ffcb7e5-2dfc-4b83-ab39-85bbb773329f/payload-processing/0.log"
Nov 01 04:18:01.565447 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:18:01.565410    4400 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-rrdzd_0303d6c6-c4ec-4403-af74-f4bcb5cbf2a4/global-pull-secret-syncer/0.log"
Nov 01 04:18:01.730125 ip-10-0-133-54 kubenswrapper[4400]: E1101 04:18:01.730090    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 04:18:04.469053 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:18:04.469024    4400 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-fh7sf_bd8260bb-3811-4cdc-91d0-c6ddc693891e/konnectivity-agent/0.log"
Nov 01 04:18:04.855907 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:18:04.855876    4400 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-54.us-west-1.compute.internal_8cf98fd570993ef6ee2b41a9af4899c8/haproxy/0.log"
Nov 01 04:18:10.695258 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:18:10.695226    4400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-p847s/perf-node-gather-daemonset-p44f7"]
Nov 01 04:18:10.695806 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:18:10.695790    4400 cpu_manager.go:414] "containerMap: removing container" podUID="7d97be95-6efe-4683-9e54-7829190d0b50" containerName="maas-api"
Nov 01 04:18:10.695865 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:18:10.695808    4400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d97be95-6efe-4683-9e54-7829190d0b50" containerName="maas-api"
Nov 01 04:18:10.695912 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:18:10.695902    4400 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d97be95-6efe-4683-9e54-7829190d0b50" containerName="maas-api"
Nov 01 04:18:10.699091 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:18:10.699076    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-p847s/perf-node-gather-daemonset-p44f7"
Nov 01 04:18:10.754008 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:18:10.753968    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-p847s\"/\"kube-root-ca.crt\""
Nov 01 04:18:10.804838 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:18:10.804807    4400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-p847s\"/\"openshift-service-ca.crt\""
Nov 01 04:18:10.805004 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:18:10.804874    4400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-p847s\"/\"default-dockercfg-d54w4\""
Nov 01 04:18:10.807043 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:18:10.807022    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-p847s/perf-node-gather-daemonset-p44f7"]
Nov 01 04:18:10.886069 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:18:10.886033    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d4edbb37-6cac-4370-bd73-eb06cb1d8996-podres\") pod \"perf-node-gather-daemonset-p44f7\" (UID: \"d4edbb37-6cac-4370-bd73-eb06cb1d8996\") " pod="openshift-must-gather-p847s/perf-node-gather-daemonset-p44f7"
Nov 01 04:18:10.886269 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:18:10.886161    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d4edbb37-6cac-4370-bd73-eb06cb1d8996-sys\") pod \"perf-node-gather-daemonset-p44f7\" (UID: \"d4edbb37-6cac-4370-bd73-eb06cb1d8996\") " pod="openshift-must-gather-p847s/perf-node-gather-daemonset-p44f7"
Nov 01 04:18:10.886269 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:18:10.886198    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d4edbb37-6cac-4370-bd73-eb06cb1d8996-lib-modules\") pod \"perf-node-gather-daemonset-p44f7\" (UID: \"d4edbb37-6cac-4370-bd73-eb06cb1d8996\") " pod="openshift-must-gather-p847s/perf-node-gather-daemonset-p44f7"
Nov 01 04:18:10.886269 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:18:10.886253    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d4edbb37-6cac-4370-bd73-eb06cb1d8996-proc\") pod \"perf-node-gather-daemonset-p44f7\" (UID: \"d4edbb37-6cac-4370-bd73-eb06cb1d8996\") " pod="openshift-must-gather-p847s/perf-node-gather-daemonset-p44f7"
Nov 01 04:18:10.886417 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:18:10.886370    4400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsgk9\" (UniqueName: \"kubernetes.io/projected/d4edbb37-6cac-4370-bd73-eb06cb1d8996-kube-api-access-hsgk9\") pod \"perf-node-gather-daemonset-p44f7\" (UID: \"d4edbb37-6cac-4370-bd73-eb06cb1d8996\") " pod="openshift-must-gather-p847s/perf-node-gather-daemonset-p44f7"
Nov 01 04:18:10.987492 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:18:10.987393    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d4edbb37-6cac-4370-bd73-eb06cb1d8996-podres\") pod \"perf-node-gather-daemonset-p44f7\" (UID: \"d4edbb37-6cac-4370-bd73-eb06cb1d8996\") " pod="openshift-must-gather-p847s/perf-node-gather-daemonset-p44f7"
Nov 01 04:18:10.987492 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:18:10.987453    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d4edbb37-6cac-4370-bd73-eb06cb1d8996-sys\") pod \"perf-node-gather-daemonset-p44f7\" (UID: \"d4edbb37-6cac-4370-bd73-eb06cb1d8996\") " pod="openshift-must-gather-p847s/perf-node-gather-daemonset-p44f7"
Nov 01 04:18:10.987492 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:18:10.987482    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d4edbb37-6cac-4370-bd73-eb06cb1d8996-lib-modules\") pod \"perf-node-gather-daemonset-p44f7\" (UID: \"d4edbb37-6cac-4370-bd73-eb06cb1d8996\") " pod="openshift-must-gather-p847s/perf-node-gather-daemonset-p44f7"
Nov 01 04:18:10.987844 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:18:10.987506    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d4edbb37-6cac-4370-bd73-eb06cb1d8996-proc\") pod \"perf-node-gather-daemonset-p44f7\" (UID: \"d4edbb37-6cac-4370-bd73-eb06cb1d8996\") " pod="openshift-must-gather-p847s/perf-node-gather-daemonset-p44f7"
Nov 01 04:18:10.987844 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:18:10.987541    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d4edbb37-6cac-4370-bd73-eb06cb1d8996-podres\") pod \"perf-node-gather-daemonset-p44f7\" (UID: \"d4edbb37-6cac-4370-bd73-eb06cb1d8996\") " pod="openshift-must-gather-p847s/perf-node-gather-daemonset-p44f7"
Nov 01 04:18:10.987844 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:18:10.987602    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d4edbb37-6cac-4370-bd73-eb06cb1d8996-sys\") pod \"perf-node-gather-daemonset-p44f7\" (UID: \"d4edbb37-6cac-4370-bd73-eb06cb1d8996\") " pod="openshift-must-gather-p847s/perf-node-gather-daemonset-p44f7"
Nov 01 04:18:10.987844 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:18:10.987615    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d4edbb37-6cac-4370-bd73-eb06cb1d8996-proc\") pod \"perf-node-gather-daemonset-p44f7\" (UID: \"d4edbb37-6cac-4370-bd73-eb06cb1d8996\") " pod="openshift-must-gather-p847s/perf-node-gather-daemonset-p44f7"
Nov 01 04:18:10.987844 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:18:10.987625    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d4edbb37-6cac-4370-bd73-eb06cb1d8996-lib-modules\") pod \"perf-node-gather-daemonset-p44f7\" (UID: \"d4edbb37-6cac-4370-bd73-eb06cb1d8996\") " pod="openshift-must-gather-p847s/perf-node-gather-daemonset-p44f7"
Nov 01 04:18:10.987844 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:18:10.987657    4400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsgk9\" (UniqueName: \"kubernetes.io/projected/d4edbb37-6cac-4370-bd73-eb06cb1d8996-kube-api-access-hsgk9\") pod \"perf-node-gather-daemonset-p44f7\" (UID: \"d4edbb37-6cac-4370-bd73-eb06cb1d8996\") " pod="openshift-must-gather-p847s/perf-node-gather-daemonset-p44f7"
Nov 01 04:18:11.049519 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:18:11.049483    4400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsgk9\" (UniqueName: \"kubernetes.io/projected/d4edbb37-6cac-4370-bd73-eb06cb1d8996-kube-api-access-hsgk9\") pod \"perf-node-gather-daemonset-p44f7\" (UID: \"d4edbb37-6cac-4370-bd73-eb06cb1d8996\") " pod="openshift-must-gather-p847s/perf-node-gather-daemonset-p44f7"
Nov 01 04:18:11.311454 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:18:11.311361    4400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-p847s/perf-node-gather-daemonset-p44f7"
Nov 01 04:18:11.687326 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:18:11.687295    4400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-p847s/perf-node-gather-daemonset-p44f7"]
Nov 01 04:18:11.689382 ip-10-0-133-54 kubenswrapper[4400]: W1101 04:18:11.689350    4400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd4edbb37_6cac_4370_bd73_eb06cb1d8996.slice/crio-695df098b89436cd7ba3d58d41c6150254e2958b46da6e0b9f7fe2faa08e23ec WatchSource:0}: Error finding container 695df098b89436cd7ba3d58d41c6150254e2958b46da6e0b9f7fe2faa08e23ec: Status 404 returned error can't find the container with id 695df098b89436cd7ba3d58d41c6150254e2958b46da6e0b9f7fe2faa08e23ec
Nov 01 04:18:12.309427 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:18:12.309388    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-p847s/perf-node-gather-daemonset-p44f7" event={"ID":"d4edbb37-6cac-4370-bd73-eb06cb1d8996","Type":"ContainerStarted","Data":"66cff4550c2e12e90763a8663a0d01612a386d94f61f805ec15e41c4c880d565"}
Nov 01 04:18:12.309858 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:18:12.309450    4400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-p847s/perf-node-gather-daemonset-p44f7" event={"ID":"d4edbb37-6cac-4370-bd73-eb06cb1d8996","Type":"ContainerStarted","Data":"695df098b89436cd7ba3d58d41c6150254e2958b46da6e0b9f7fe2faa08e23ec"}
Nov 01 04:18:12.309858 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:18:12.309526    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-p847s/perf-node-gather-daemonset-p44f7"
Nov 01 04:18:12.430355 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:18:12.430297    4400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-p847s/perf-node-gather-daemonset-p44f7" podStartSLOduration=2.430282663 podStartE2EDuration="2.430282663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 04:18:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:18:12.427936111 +0000 UTC m=+4298.942581972" watchObservedRunningTime="2026-11-01 04:18:12.430282663 +0000 UTC m=+4298.944928531"
Nov 01 04:18:12.444087 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:18:12.444060    4400 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-mpdqs_63495340-52fe-4035-825d-50250e797b66/limitador/0.log"
Nov 01 04:18:15.547521 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:18:15.547483    4400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_395c33e8-a15b-42bc-b556-bf8e9b2a6d64/alertmanager/0.log"
Nov 01 04:18:15.963457 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:18:15.963426    4400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_395c33e8-a15b-42bc-b556-bf8e9b2a6d64/config-reloader/0.log"
Nov 01 04:18:16.378410 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:18:16.378380    4400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_395c33e8-a15b-42bc-b556-bf8e9b2a6d64/kube-rbac-proxy-web/0.log"
Nov 01 04:18:16.794840 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:18:16.794763    4400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_395c33e8-a15b-42bc-b556-bf8e9b2a6d64/kube-rbac-proxy/0.log"
Nov 01 04:18:17.210658 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:18:17.210618    4400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_395c33e8-a15b-42bc-b556-bf8e9b2a6d64/kube-rbac-proxy-metric/0.log"
Nov 01 04:18:17.627047 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:18:17.627023    4400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_395c33e8-a15b-42bc-b556-bf8e9b2a6d64/prom-label-proxy/0.log"
Nov 01 04:18:18.043492 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:18:18.043412    4400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_395c33e8-a15b-42bc-b556-bf8e9b2a6d64/init-config-reloader/0.log"
Nov 01 04:18:18.325876 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:18:18.325849    4400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-p847s/perf-node-gather-daemonset-p44f7"
Nov 01 04:18:18.993032 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:18:18.992999    4400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-ncr5j_d7f2c2ca-b2a2-4a31-987a-e7049fa74f2d/kube-state-metrics/0.log"
Nov 01 04:18:19.444598 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:18:19.444570    4400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-ncr5j_d7f2c2ca-b2a2-4a31-987a-e7049fa74f2d/kube-rbac-proxy-main/0.log"
Nov 01 04:18:19.866017 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:18:19.865994    4400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-ncr5j_d7f2c2ca-b2a2-4a31-987a-e7049fa74f2d/kube-rbac-proxy-self/0.log"
Nov 01 04:18:20.288269 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:18:20.288184    4400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5845695f9b-7vfc4_147c6bba-08a4-4956-9854-c93fa11bbd15/metrics-server/0.log"
Nov 01 04:18:20.707819 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:18:20.707790    4400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-l5ng6_32b3b49b-3371-4455-a4c5-a4152e66cc7d/monitoring-plugin/0.log"
Nov 01 04:18:22.386100 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:18:22.386063    4400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dw9pj_4b8bb646-3c9e-4885-abe6-8dec55c060a0/node-exporter/0.log"
Nov 01 04:18:22.804209 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:18:22.804117    4400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dw9pj_4b8bb646-3c9e-4885-abe6-8dec55c060a0/kube-rbac-proxy/0.log"
Nov 01 04:18:23.218755 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:18:23.218726    4400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dw9pj_4b8bb646-3c9e-4885-abe6-8dec55c060a0/init-textfile/0.log"
Nov 01 04:18:24.903602 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:18:24.903572    4400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-6nx7f_74fb2a00-d669-4759-a1bf-6af8425e1f71/kube-rbac-proxy-main/0.log"
Nov 01 04:18:25.317462 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:18:25.317387    4400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-6nx7f_74fb2a00-d669-4759-a1bf-6af8425e1f71/kube-rbac-proxy-self/0.log"
Nov 01 04:18:25.742098 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:18:25.742065    4400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-6nx7f_74fb2a00-d669-4759-a1bf-6af8425e1f71/openshift-state-metrics/0.log"
Nov 01 04:18:29.118248 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:18:29.118217    4400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-ssff4_cb9062e4-c82c-473d-9243-aea4fa1fa169/prometheus-operator/0.log"
Nov 01 04:18:29.527143 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:18:29.527063    4400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-ssff4_cb9062e4-c82c-473d-9243-aea4fa1fa169/kube-rbac-proxy/0.log"
Nov 01 04:18:30.365609 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:18:30.365580    4400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6cc696c965-lpr8d_a18ff3a3-2339-4b95-8239-6bea17756751/telemeter-client/0.log"
Nov 01 04:18:30.780112 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:18:30.780022    4400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6cc696c965-lpr8d_a18ff3a3-2339-4b95-8239-6bea17756751/reload/0.log"
Nov 01 04:18:31.195142 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:18:31.195111    4400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6cc696c965-lpr8d_a18ff3a3-2339-4b95-8239-6bea17756751/kube-rbac-proxy/0.log"
Nov 01 04:18:31.623268 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:18:31.623235    4400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-59d787964f-6q8gp_570761ed-076a-4c28-a14b-46471838f93c/thanos-query/0.log"
Nov 01 04:18:31.730575 ip-10-0-133-54 kubenswrapper[4400]: E1101 04:18:31.730544    4400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vv92" containerName="extractor"
Nov 01 04:18:32.038703 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:18:32.038585    4400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-59d787964f-6q8gp_570761ed-076a-4c28-a14b-46471838f93c/kube-rbac-proxy-web/0.log"
Nov 01 04:18:32.456515 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:18:32.456476    4400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-59d787964f-6q8gp_570761ed-076a-4c28-a14b-46471838f93c/kube-rbac-proxy/0.log"
Nov 01 04:18:32.872418 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:18:32.872378    4400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-59d787964f-6q8gp_570761ed-076a-4c28-a14b-46471838f93c/prom-label-proxy/0.log"
Nov 01 04:18:33.289030 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:18:33.288948    4400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-59d787964f-6q8gp_570761ed-076a-4c28-a14b-46471838f93c/kube-rbac-proxy-rules/0.log"
Nov 01 04:18:33.749229 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:18:33.749167    4400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-59d787964f-6q8gp_570761ed-076a-4c28-a14b-46471838f93c/kube-rbac-proxy-metrics/0.log"
Nov 01 04:18:34.493187 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:18:34.493147    4400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-lw9fr_0fca1311-9bbd-4b5d-a996-2f67742bedb3/discovery/0.log"
Nov 01 04:18:34.814001 ip-10-0-133-54 kubenswrapper[4400]: I1101 04:18:34.813917    4400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d5d9754f-zcj7p_121dbd3a-6f50-4e38-8e72-85297471ca08/kube-auth-proxy/0.log"