Nov 02 19:27:32.794340 ip-10-0-133-9 systemd[1]: Starting Kubernetes Kubelet...
Nov 02 19:27:33.486371 ip-10-0-133-9 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 19:27:33.486371 ip-10-0-133-9 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 02 19:27:33.486371 ip-10-0-133-9 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 19:27:33.486371 ip-10-0-133-9 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 19:27:33.489789 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.489700    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 19:27:33.489789 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.489785    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 19:27:33.489789 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.489791    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 19:27:33.489789 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.489797    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 19:27:33.489986 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.489801    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 19:27:33.489986 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.489805    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 19:27:33.489986 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.489809    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 19:27:33.489986 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.489813    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 19:27:33.489986 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.489817    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 19:27:33.489986 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.489820    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 19:27:33.489986 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.489824    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 19:27:33.489986 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.489828    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 19:27:33.489986 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.489832    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 19:27:33.489986 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.489836    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 19:27:33.489986 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.489839    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 19:27:33.490673 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.490660    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 19:27:33.490673 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.490675    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 19:27:33.490773 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.490681    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 19:27:33.490773 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.490685    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 19:27:33.490773 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.490690    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 19:27:33.490773 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.490695    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 19:27:33.490773 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.490706    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 19:27:33.490773 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.490711    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 19:27:33.490773 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.490716    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 19:27:33.490773 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.490720    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 19:27:33.490773 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.490726    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 19:27:33.490773 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.490730    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 19:27:33.490773 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.490735    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 19:27:33.490773 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.490739    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 19:27:33.490773 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.490743    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 19:27:33.490773 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.490747    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 19:27:33.490773 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.490752    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 19:27:33.490773 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.490756    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 19:27:33.490773 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.490761    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 19:27:33.490773 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.490765    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 19:27:33.490773 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.490770    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 19:27:33.490773 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.490774    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 19:27:33.491609 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.490778    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 19:27:33.491609 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.490782    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 19:27:33.491609 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.490786    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 19:27:33.491609 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.490790    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 19:27:33.491609 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.490794    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 19:27:33.491609 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.490799    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 19:27:33.491609 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.490803    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 19:27:33.491609 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.490807    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 19:27:33.491609 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.490812    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 19:27:33.491609 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.490818    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 19:27:33.491609 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.490822    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 19:27:33.491609 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.490826    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 19:27:33.491609 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.490830    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 19:27:33.491609 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.490834    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 19:27:33.491609 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.490838    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 19:27:33.491609 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.490842    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 19:27:33.491609 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.490847    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 19:27:33.491609 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.490851    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 19:27:33.491609 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.490855    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 19:27:33.492453 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.490859    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 19:27:33.492453 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.490864    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 19:27:33.492453 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.490868    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 19:27:33.492453 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.490872    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 19:27:33.492453 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.490876    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 19:27:33.492453 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.490880    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 19:27:33.492453 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.490886    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 19:27:33.492453 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.490890    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 19:27:33.492453 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.490894    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 19:27:33.492453 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.490898    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 19:27:33.492453 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.490902    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 19:27:33.492453 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.490907    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 19:27:33.492453 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.490911    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 19:27:33.492453 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.490915    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 19:27:33.492453 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.490920    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 19:27:33.492453 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.490925    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 19:27:33.492453 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.490930    2579 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 19:27:33.492453 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.490935    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 19:27:33.492453 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.490939    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 19:27:33.492453 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.490944    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 19:27:33.493308 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.490948    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 19:27:33.493308 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.490953    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 19:27:33.493308 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.490957    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 19:27:33.493308 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.490961    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 19:27:33.493308 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.490966    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 19:27:33.493308 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.490970    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 19:27:33.493308 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.490975    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 19:27:33.493308 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.490980    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 19:27:33.493308 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.490984    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 19:27:33.493308 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.490988    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 19:27:33.493308 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.490992    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 19:27:33.493308 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.490996    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 19:27:33.493308 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.491001    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 19:27:33.493308 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.491005    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 19:27:33.493308 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.491023    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 19:27:33.493308 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.491028    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 19:27:33.493308 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.491032    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 19:27:33.493308 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.491036    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 19:27:33.493308 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.491040    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 19:27:33.493308 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.491045    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 19:27:33.494178 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.491049    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 19:27:33.494178 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.491053    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 19:27:33.494178 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.491057    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 19:27:33.494178 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.491061    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 19:27:33.494178 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.491067    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 19:27:33.494178 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.491077    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 19:27:33.494178 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.491084    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 19:27:33.494178 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.491090    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 19:27:33.494178 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.491095    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 19:27:33.494178 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.491101    2579 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 19:27:33.494178 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492406    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 19:27:33.494178 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492418    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 19:27:33.494178 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492424    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 19:27:33.494178 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492430    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 19:27:33.494178 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492435    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 19:27:33.494178 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492440    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 19:27:33.494178 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492444    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 19:27:33.494178 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492448    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 19:27:33.494178 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492452    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 19:27:33.494720 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492456    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 19:27:33.494720 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492460    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 19:27:33.494720 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492464    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 19:27:33.494720 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492468    2579 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 19:27:33.494720 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492472    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 19:27:33.494720 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492476    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 19:27:33.494720 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492480    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 19:27:33.494720 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492484    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 19:27:33.494720 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492488    2579 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 19:27:33.494720 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492492    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 19:27:33.494720 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492497    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 19:27:33.494720 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492502    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 19:27:33.494720 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492506    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 19:27:33.494720 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492510    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 19:27:33.494720 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492515    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 19:27:33.494720 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492519    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 19:27:33.494720 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492525    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 19:27:33.494720 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492529    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 19:27:33.494720 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492533    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 19:27:33.494720 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492538    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 19:27:33.495243 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492543    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 19:27:33.495243 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492547    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 19:27:33.495243 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492552    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 19:27:33.495243 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492557    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 19:27:33.495243 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492561    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 19:27:33.495243 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492566    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 19:27:33.495243 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492570    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 19:27:33.495243 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492574    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 19:27:33.495243 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492579    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 19:27:33.495243 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492583    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 19:27:33.495243 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492587    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 19:27:33.495243 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492592    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 19:27:33.495243 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492596    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 19:27:33.495243 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492600    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 19:27:33.495243 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492604    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 19:27:33.495243 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492608    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 19:27:33.495243 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492612    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 19:27:33.495243 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492616    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 19:27:33.495243 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492621    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 19:27:33.495243 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492625    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 19:27:33.496121 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492629    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 19:27:33.496121 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492634    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 19:27:33.496121 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492638    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 19:27:33.496121 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492643    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 19:27:33.496121 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492647    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 19:27:33.496121 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492652    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 19:27:33.496121 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492658    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 19:27:33.496121 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492666    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 19:27:33.496121 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492671    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 19:27:33.496121 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492675    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 19:27:33.496121 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492680    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 19:27:33.496121 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492685    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 19:27:33.496121 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492690    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 19:27:33.496121 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492694    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 19:27:33.496121 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492699    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 19:27:33.496121 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492703    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 19:27:33.496121 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492708    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 19:27:33.496121 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492712    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 19:27:33.496121 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492717    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 19:27:33.496983 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492721    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 19:27:33.496983 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492727    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 19:27:33.496983 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492730    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 19:27:33.496983 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492735    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 19:27:33.496983 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492739    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 19:27:33.496983 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492743    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 19:27:33.496983 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492747    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 19:27:33.496983 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492751    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 19:27:33.496983 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492756    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 19:27:33.496983 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492760    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 19:27:33.496983 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492764    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 19:27:33.496983 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492770    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 19:27:33.496983 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492774    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 19:27:33.496983 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492778    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 19:27:33.496983 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492782    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 19:27:33.496983 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492787    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 19:27:33.496983 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492791    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 19:27:33.496983 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492795    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 19:27:33.496983 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492800    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 19:27:33.497688 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492804    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 19:27:33.497688 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492808    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 19:27:33.497688 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492812    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 19:27:33.497688 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492816    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 19:27:33.497688 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492820    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 19:27:33.497688 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492825    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 19:27:33.497688 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492829    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 19:27:33.497688 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492834    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 19:27:33.497688 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492838    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 19:27:33.497688 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492842    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 19:27:33.497688 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492847    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 19:27:33.497688 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492852    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 19:27:33.497688 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492856    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 19:27:33.497688 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492860    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 19:27:33.497688 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492865    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 19:27:33.497688 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492872    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 19:27:33.497688 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492880    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 19:27:33.497688 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492884    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 19:27:33.497688 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.492889    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 19:27:33.498268 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.493712    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 19:27:33.498268 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.493721    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 19:27:33.498268 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.493725    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 19:27:33.498268 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.493730    2579 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 19:27:33.498268 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.493734    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 19:27:33.498268 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.493738    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 19:27:33.498268 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.493742    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 19:27:33.498268 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.493747    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 19:27:33.498268 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.493751    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 19:27:33.498268 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.493755    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 19:27:33.498268 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.493761    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 19:27:33.498268 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.493766    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 19:27:33.498268 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.493770    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 19:27:33.498268 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.493774    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 19:27:33.498268 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.493778    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 19:27:33.498268 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.493782    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 19:27:33.498268 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.493787    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 19:27:33.498268 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.493792    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 19:27:33.498268 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.493796    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 19:27:33.498268 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.493800    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 19:27:33.498864 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.493804    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 19:27:33.498864 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.493808    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 19:27:33.498864 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.493813    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 19:27:33.498864 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.493817    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 19:27:33.498864 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.493823    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 19:27:33.498864 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.493827    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 19:27:33.498864 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.493831    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 19:27:33.498864 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.493836    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 19:27:33.498864 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.493840    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 19:27:33.498864 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.493844    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 19:27:33.498864 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.493848    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 19:27:33.498864 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.493853    2579 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 19:27:33.498864 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.493856    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 19:27:33.498864 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.493861    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 19:27:33.498864 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.493873    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 19:27:33.498864 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.493877    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 19:27:33.498864 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.493882    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 19:27:33.498864 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.493886    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 19:27:33.498864 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.493890    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 19:27:33.498864 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.493894    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 19:27:33.499369 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.493898    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 19:27:33.499369 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.493903    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 19:27:33.499369 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.493908    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 19:27:33.499369 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.493912    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 19:27:33.499369 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.493916    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 19:27:33.499369 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.493921    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 19:27:33.499369 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.493925    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 19:27:33.499369 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.493929    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 19:27:33.499369 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.493934    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 19:27:33.499369 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.493938    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 19:27:33.499369 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.493942    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 19:27:33.499369 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.493946    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 19:27:33.499369 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.493950    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 19:27:33.499369 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.493955    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 19:27:33.499369 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.493960    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 19:27:33.499369 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.493964    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 19:27:33.499369 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.493968    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 19:27:33.499369 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.493974    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 19:27:33.499369 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.493978    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 19:27:33.499911 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.493984    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 19:27:33.499911 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.493988    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 19:27:33.499911 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.493993    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 19:27:33.499911 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.493997    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 19:27:33.499911 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.494001    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 19:27:33.499911 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.494005    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 19:27:33.499911 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.494029    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 19:27:33.499911 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.494036    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 19:27:33.499911 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.494042    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 19:27:33.499911 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.494050    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 19:27:33.499911 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.494054    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 19:27:33.499911 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.494059    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 19:27:33.499911 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.494063    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 19:27:33.499911 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.494068    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 19:27:33.499911 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.494073    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 19:27:33.499911 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.494077    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 19:27:33.499911 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.494082    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 19:27:33.499911 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.494087    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 19:27:33.499911 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.494090    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 19:27:33.500378 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.494095    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 19:27:33.500378 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.494099    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 19:27:33.500378 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.494103    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 19:27:33.500378 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.494107    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 19:27:33.500378 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.494111    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 19:27:33.500378 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.494115    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 19:27:33.500378 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.494119    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 19:27:33.500378 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.494124    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 19:27:33.500378 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.494128    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 19:27:33.500378 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.494132    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 19:27:33.500378 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.494136    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 19:27:33.500378 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.494140    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 19:27:33.500378 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.494145    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 19:27:33.500378 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.494150    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 19:27:33.500378 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.494154    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 19:27:33.500378 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.494158    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 19:27:33.500378 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.494162    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 19:27:33.500378 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.494166    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 19:27:33.500378 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.494170    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 19:27:33.500859 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.494176    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 19:27:33.500859 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.494180    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 19:27:33.500859 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.494185    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 19:27:33.500859 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.494189    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 19:27:33.500859 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.494193    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 19:27:33.500859 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.494197    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 19:27:33.500859 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.494201    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 19:27:33.500859 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.494205    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 19:27:33.500859 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.494212    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 19:27:33.500859 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495211    2579 flags.go:64] FLAG: --address="0.0.0.0"
Nov 02 19:27:33.500859 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495225    2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 02 19:27:33.500859 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495235    2579 flags.go:64] FLAG: --anonymous-auth="true"
Nov 02 19:27:33.500859 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495242    2579 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 02 19:27:33.500859 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495249    2579 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 02 19:27:33.500859 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495254    2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 02 19:27:33.500859 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495261    2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 02 19:27:33.500859 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495268    2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 02 19:27:33.500859 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495273    2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 02 19:27:33.500859 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495278    2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 02 19:27:33.500859 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495284    2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 02 19:27:33.500859 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495290    2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 02 19:27:33.500859 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495295    2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 02 19:27:33.501396 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495299    2579 flags.go:64] FLAG: --cgroup-root=""
Nov 02 19:27:33.501396 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495304    2579 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 02 19:27:33.501396 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495309    2579 flags.go:64] FLAG: --client-ca-file=""
Nov 02 19:27:33.501396 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495316    2579 flags.go:64] FLAG: --cloud-provider="external"
Nov 02 19:27:33.501396 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495323    2579 flags.go:64] FLAG: --cluster-dns="[]"
Nov 02 19:27:33.501396 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495329    2579 flags.go:64] FLAG: --cluster-domain=""
Nov 02 19:27:33.501396 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495333    2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 02 19:27:33.501396 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495338    2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 02 19:27:33.501396 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495343    2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 02 19:27:33.501396 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495349    2579 flags.go:64] FLAG: --container-log-max-files="5"
Nov 02 19:27:33.501396 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495355    2579 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 02 19:27:33.501396 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495360    2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 02 19:27:33.501396 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495365    2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 02 19:27:33.501396 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495371    2579 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 02 19:27:33.501396 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495375    2579 flags.go:64] FLAG: --contention-profiling="false"
Nov 02 19:27:33.501396 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495380    2579 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 02 19:27:33.501396 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495385    2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 02 19:27:33.501396 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495390    2579 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 02 19:27:33.501396 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495395    2579 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 02 19:27:33.501396 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495402    2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 02 19:27:33.501396 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495407    2579 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 02 19:27:33.501396 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495412    2579 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 02 19:27:33.501396 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495417    2579 flags.go:64] FLAG: --enable-load-reader="false"
Nov 02 19:27:33.501396 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495421    2579 flags.go:64] FLAG: --enable-server="true"
Nov 02 19:27:33.501396 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495426    2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 02 19:27:33.502025 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495433    2579 flags.go:64] FLAG: --event-burst="100"
Nov 02 19:27:33.502025 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495438    2579 flags.go:64] FLAG: --event-qps="50"
Nov 02 19:27:33.502025 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495443    2579 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 02 19:27:33.502025 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495448    2579 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 02 19:27:33.502025 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495453    2579 flags.go:64] FLAG: --eviction-hard=""
Nov 02 19:27:33.502025 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495459    2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 02 19:27:33.502025 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495464    2579 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 02 19:27:33.502025 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495469    2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 02 19:27:33.502025 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495474    2579 flags.go:64] FLAG: --eviction-soft=""
Nov 02 19:27:33.502025 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495479    2579 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 02 19:27:33.502025 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495486    2579 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 02 19:27:33.502025 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495491    2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 02 19:27:33.502025 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495496    2579 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 02 19:27:33.502025 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495501    2579 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 02 19:27:33.502025 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495506    2579 flags.go:64] FLAG: --fail-swap-on="true"
Nov 02 19:27:33.502025 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495511    2579 flags.go:64] FLAG: --feature-gates=""
Nov 02 19:27:33.502025 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495517    2579 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 02 19:27:33.502025 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495523    2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 02 19:27:33.502025 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495528    2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 02 19:27:33.502025 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495534    2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 02 19:27:33.502025 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495539    2579 flags.go:64] FLAG: --healthz-port="10248"
Nov 02 19:27:33.502025 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495544    2579 flags.go:64] FLAG: --help="false"
Nov 02 19:27:33.502025 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495549    2579 flags.go:64] FLAG: --hostname-override="ip-10-0-133-9.us-west-2.compute.internal"
Nov 02 19:27:33.502025 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495554    2579 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 02 19:27:33.502025 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495559    2579 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 02 19:27:33.502984 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495563    2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 02 19:27:33.502984 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495569    2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 02 19:27:33.502984 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495575    2579 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 02 19:27:33.502984 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495579    2579 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 02 19:27:33.502984 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495585    2579 flags.go:64] FLAG: --image-service-endpoint=""
Nov 02 19:27:33.502984 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495590    2579 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 02 19:27:33.502984 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495594    2579 flags.go:64] FLAG: --kube-api-burst="100"
Nov 02 19:27:33.502984 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495599    2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 02 19:27:33.502984 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495604    2579 flags.go:64] FLAG: --kube-api-qps="50"
Nov 02 19:27:33.502984 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495609    2579 flags.go:64] FLAG: --kube-reserved=""
Nov 02 19:27:33.502984 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495614    2579 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 02 19:27:33.502984 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495618    2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 02 19:27:33.502984 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495623    2579 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 02 19:27:33.502984 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495628    2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 02 19:27:33.502984 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495633    2579 flags.go:64] FLAG: --lock-file=""
Nov 02 19:27:33.502984 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495637    2579 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 02 19:27:33.502984 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495642    2579 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 02 19:27:33.502984 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495649    2579 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 02 19:27:33.502984 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495658    2579 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 02 19:27:33.502984 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495663    2579 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 02 19:27:33.502984 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495669    2579 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 02 19:27:33.502984 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495674    2579 flags.go:64] FLAG: --logging-format="text"
Nov 02 19:27:33.502984 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495679    2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 02 19:27:33.504000 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495684    2579 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 02 19:27:33.504000 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495689    2579 flags.go:64] FLAG: --manifest-url=""
Nov 02 19:27:33.504000 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495694    2579 flags.go:64] FLAG: --manifest-url-header=""
Nov 02 19:27:33.504000 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495702    2579 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 02 19:27:33.504000 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495707    2579 flags.go:64] FLAG: --max-open-files="1000000"
Nov 02 19:27:33.504000 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495714    2579 flags.go:64] FLAG: --max-pods="110"
Nov 02 19:27:33.504000 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495719    2579 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 02 19:27:33.504000 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495723    2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 02 19:27:33.504000 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495728    2579 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 02 19:27:33.504000 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495733    2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 02 19:27:33.504000 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495738    2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 02 19:27:33.504000 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495743    2579 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 02 19:27:33.504000 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495748    2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 02 19:27:33.504000 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495759    2579 flags.go:64] FLAG: --node-status-max-images="50"
Nov 02 19:27:33.504000 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495764    2579 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 02 19:27:33.504000 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495769    2579 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 02 19:27:33.504000 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495774    2579 flags.go:64] FLAG: --pod-cidr=""
Nov 02 19:27:33.504000 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495779    2579 flags.go:64] FLAG: --pod-manifest-path=""
Nov 02 19:27:33.504000 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495783    2579 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 02 19:27:33.504000 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495789    2579 flags.go:64] FLAG: --pods-per-core="0"
Nov 02 19:27:33.504000 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495793    2579 flags.go:64] FLAG: --port="10250"
Nov 02 19:27:33.504000 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495798    2579 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 02 19:27:33.504000 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495803    2579 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0b653f3ce8ab2eeb9"
Nov 02 19:27:33.504000 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495808    2579 flags.go:64] FLAG: --qos-reserved=""
Nov 02 19:27:33.504000 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495813    2579 flags.go:64] FLAG: --read-only-port="10255"
Nov 02 19:27:33.504000 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495818    2579 flags.go:64] FLAG: --register-node="true"
Nov 02 19:27:33.505149 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495824    2579 flags.go:64] FLAG: --register-with-taints=""
Nov 02 19:27:33.505149 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495838    2579 flags.go:64] FLAG: --registry-burst="10"
Nov 02 19:27:33.505149 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495842    2579 flags.go:64] FLAG: --registry-qps="5"
Nov 02 19:27:33.505149 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495847    2579 flags.go:64] FLAG: --reserved-cpus=""
Nov 02 19:27:33.505149 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495853    2579 flags.go:64] FLAG: --reserved-memory=""
Nov 02 19:27:33.505149 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495859    2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 02 19:27:33.505149 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495864    2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 02 19:27:33.505149 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495869    2579 flags.go:64] FLAG: --rotate-certificates="false"
Nov 02 19:27:33.505149 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495873    2579 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 02 19:27:33.505149 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495878    2579 flags.go:64] FLAG: --runonce="false"
Nov 02 19:27:33.505149 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495883    2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 02 19:27:33.505149 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495889    2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 02 19:27:33.505149 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495893    2579 flags.go:64] FLAG: --seccomp-default="false"
Nov 02 19:27:33.505149 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495898    2579 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 02 19:27:33.505149 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495903    2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 02 19:27:33.505149 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495908    2579 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 02 19:27:33.505149 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495913    2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 02 19:27:33.505149 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495918    2579 flags.go:64] FLAG: --storage-driver-password="root"
Nov 02 19:27:33.505149 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495922    2579 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 02 19:27:33.505149 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495927    2579 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 02 19:27:33.505149 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495932    2579 flags.go:64] FLAG: --storage-driver-user="root"
Nov 02 19:27:33.505149 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495936    2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 02 19:27:33.505149 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495941    2579 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 02 19:27:33.505149 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495946    2579 flags.go:64] FLAG: --system-cgroups=""
Nov 02 19:27:33.505149 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495950    2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 02 19:27:33.506237 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495959    2579 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 02 19:27:33.506237 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495964    2579 flags.go:64] FLAG: --tls-cert-file=""
Nov 02 19:27:33.506237 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495968    2579 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 02 19:27:33.506237 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495978    2579 flags.go:64] FLAG: --tls-min-version=""
Nov 02 19:27:33.506237 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495983    2579 flags.go:64] FLAG: --tls-private-key-file=""
Nov 02 19:27:33.506237 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495987    2579 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 02 19:27:33.506237 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495992    2579 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 02 19:27:33.506237 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.495999    2579 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 02 19:27:33.506237 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.496004    2579 flags.go:64] FLAG: --v="2"
Nov 02 19:27:33.506237 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.496025    2579 flags.go:64] FLAG: --version="false"
Nov 02 19:27:33.506237 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.496032    2579 flags.go:64] FLAG: --vmodule=""
Nov 02 19:27:33.506237 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.496039    2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 02 19:27:33.506237 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.496045    2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 02 19:27:33.506237 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496258    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 19:27:33.506237 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496267    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 19:27:33.506237 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496272    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 19:27:33.506237 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496276    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 19:27:33.506237 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496283    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 19:27:33.506237 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496289    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 19:27:33.506237 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496294    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 19:27:33.506237 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496298    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 19:27:33.506237 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496303    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 19:27:33.507153 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496307    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 19:27:33.507153 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496311    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 19:27:33.507153 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496315    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 19:27:33.507153 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496320    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 19:27:33.507153 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496324    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 19:27:33.507153 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496329    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 19:27:33.507153 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496333    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 19:27:33.507153 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496337    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 19:27:33.507153 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496342    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 19:27:33.507153 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496346    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 19:27:33.507153 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496351    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 19:27:33.507153 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496355    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 19:27:33.507153 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496360    2579 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 19:27:33.507153 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496364    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 19:27:33.507153 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496369    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 19:27:33.507153 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496373    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 19:27:33.507153 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496378    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 19:27:33.507153 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496384    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 19:27:33.507153 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496389    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 19:27:33.507153 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496393    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 19:27:33.507635 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496398    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 19:27:33.507635 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496402    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 19:27:33.507635 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496406    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 19:27:33.507635 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496411    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 19:27:33.507635 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496418    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 19:27:33.507635 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496422    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 19:27:33.507635 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496426    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 19:27:33.507635 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496431    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 19:27:33.507635 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496435    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 19:27:33.507635 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496440    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 19:27:33.507635 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496444    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 19:27:33.507635 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496448    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 19:27:33.507635 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496452    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 19:27:33.507635 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496456    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 19:27:33.507635 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496461    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 19:27:33.507635 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496465    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 19:27:33.507635 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496469    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 19:27:33.507635 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496474    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 19:27:33.507635 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496478    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 19:27:33.507635 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496482    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 19:27:33.509044 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496486    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 19:27:33.509044 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496491    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 19:27:33.509044 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496495    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 19:27:33.509044 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496499    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 19:27:33.509044 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496504    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 19:27:33.509044 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496509    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 19:27:33.509044 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496513    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 19:27:33.509044 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496517    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 19:27:33.509044 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496521    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 19:27:33.509044 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496527    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 19:27:33.509044 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496531    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 19:27:33.509044 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496536    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 19:27:33.509044 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496540    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 19:27:33.509044 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496547    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 19:27:33.509044 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496553    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 19:27:33.509044 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496557    2579 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 19:27:33.509044 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496564    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 19:27:33.509044 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496568    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 19:27:33.509044 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496572    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 19:27:33.509533 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496576    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 19:27:33.509533 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496598    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 19:27:33.509533 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496603    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 19:27:33.509533 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496607    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 19:27:33.509533 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496611    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 19:27:33.509533 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496616    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 19:27:33.509533 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496620    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 19:27:33.509533 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496624    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 19:27:33.509533 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496628    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 19:27:33.509533 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496632    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 19:27:33.509533 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496637    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 19:27:33.509533 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496641    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 19:27:33.509533 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496645    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 19:27:33.509533 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496650    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 19:27:33.509533 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496654    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 19:27:33.509533 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496658    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 19:27:33.509533 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496663    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 19:27:33.509533 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496667    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 19:27:33.509533 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496671    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 19:27:33.509987 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496676    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 19:27:33.509987 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496680    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 19:27:33.509987 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496684    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 19:27:33.509987 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496690    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 19:27:33.509987 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496694    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 19:27:33.509987 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496699    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 19:27:33.509987 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496703    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 19:27:33.509987 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496708    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 19:27:33.509987 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496712    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 19:27:33.509987 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496717    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 19:27:33.509987 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496722    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 19:27:33.509987 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496727    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 19:27:33.509987 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496731    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 19:27:33.509987 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496736    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 19:27:33.509987 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496740    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 19:27:33.509987 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496745    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 19:27:33.509987 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496749    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 19:27:33.509987 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496754    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 19:27:33.509987 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496758    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 19:27:33.510459 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.496800    2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 19:27:33.510459 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496854    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 19:27:33.510459 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496861    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 19:27:33.510459 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496867    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 19:27:33.510459 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496872    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 19:27:33.510459 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496876    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 19:27:33.510459 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496880    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 19:27:33.510459 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496885    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 19:27:33.510459 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496889    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 19:27:33.510459 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496893    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 19:27:33.510459 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496897    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 19:27:33.510459 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496902    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 19:27:33.510459 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496906    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 19:27:33.510459 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496910    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 19:27:33.510459 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496914    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 19:27:33.510459 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496919    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 19:27:33.510835 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496923    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 19:27:33.510835 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496930    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 19:27:33.510835 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496934    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 19:27:33.510835 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496938    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 19:27:33.510835 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496942    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 19:27:33.510835 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496947    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 19:27:33.510835 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496952    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 19:27:33.510835 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496959    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 19:27:33.510835 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496963    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 19:27:33.510835 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496968    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 19:27:33.510835 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496975    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 19:27:33.510835 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496981    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 19:27:33.510835 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496985    2579 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 19:27:33.510835 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496991    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 19:27:33.510835 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.496995    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 19:27:33.510835 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.497000    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 19:27:33.510835 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.497005    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 19:27:33.510835 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.497031    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 19:27:33.510835 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.497037    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 19:27:33.511312 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.497042    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 19:27:33.511312 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.497047    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 19:27:33.511312 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.497051    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 19:27:33.511312 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.497055    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 19:27:33.511312 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.497059    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 19:27:33.511312 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.497063    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 19:27:33.511312 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.497067    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 19:27:33.511312 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.497072    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 19:27:33.511312 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.497076    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 19:27:33.511312 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.497080    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 19:27:33.511312 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.497085    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 19:27:33.511312 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.497090    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 19:27:33.511312 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.497094    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 19:27:33.511312 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.497098    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 19:27:33.511312 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.497102    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 19:27:33.511312 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.497106    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 19:27:33.511312 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.497110    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 19:27:33.511312 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.497115    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 19:27:33.511312 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.497119    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 19:27:33.511758 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.497123    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 19:27:33.511758 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.497129    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 19:27:33.511758 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.497133    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 19:27:33.511758 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.497138    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 19:27:33.511758 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.497143    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 19:27:33.511758 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.497149    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 19:27:33.511758 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.497153    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 19:27:33.511758 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.497158    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 19:27:33.511758 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.497162    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 19:27:33.511758 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.497167    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 19:27:33.511758 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.497171    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 19:27:33.511758 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.497175    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 19:27:33.511758 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.497179    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 19:27:33.511758 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.497183    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 19:27:33.511758 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.497188    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 19:27:33.511758 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.497192    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 19:27:33.511758 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.497196    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 19:27:33.511758 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.497200    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 19:27:33.511758 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.497204    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 19:27:33.511758 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.497208    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 19:27:33.512294 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.497214    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 19:27:33.512294 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.497221    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 19:27:33.512294 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.497225    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 19:27:33.512294 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.497230    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 19:27:33.512294 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.497235    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 19:27:33.512294 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.497239    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 19:27:33.512294 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.497242    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 19:27:33.512294 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.497247    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 19:27:33.512294 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.497251    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 19:27:33.512294 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.497256    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 19:27:33.512294 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.497260    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 19:27:33.512294 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.497264    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 19:27:33.512294 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.497268    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 19:27:33.512294 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.497274    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 19:27:33.512294 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.497278    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 19:27:33.512294 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.497282    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 19:27:33.512294 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.497286    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 19:27:33.512294 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.497291    2579 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 19:27:33.512294 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.497296    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 19:27:33.512294 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.497300    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 19:27:33.512757 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.497304    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 19:27:33.512757 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.497308    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 19:27:33.512757 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.497312    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 19:27:33.512757 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.497316    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 19:27:33.512757 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.497322    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 19:27:33.512757 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.497326    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 19:27:33.512757 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.497331    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 19:27:33.512757 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.497335    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 19:27:33.512757 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.497339    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 19:27:33.512757 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.497343    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 19:27:33.512757 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.497347    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 19:27:33.512757 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.497352    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 19:27:33.512757 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.497356    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 19:27:33.512757 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.505624    2579 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 02 19:27:33.512757 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.505642    2579 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 02 19:27:33.512757 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.505664    2579 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 02 19:27:33.512757 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.505671    2579 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 02 19:27:33.512757 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506243    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 19:27:33.512757 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506255    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 19:27:33.512757 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506268    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 19:27:33.513373 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506274    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 19:27:33.513373 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506278    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 19:27:33.513373 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506283    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 19:27:33.513373 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506288    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 19:27:33.513373 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506292    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 19:27:33.513373 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506298    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 19:27:33.513373 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506302    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 19:27:33.513373 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506307    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 19:27:33.513373 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506311    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 19:27:33.513373 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506316    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 19:27:33.513373 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506321    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 19:27:33.513373 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506325    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 19:27:33.513373 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506337    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 19:27:33.513373 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506343    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 19:27:33.513373 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506348    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 19:27:33.513373 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506354    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 19:27:33.513373 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506361    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 19:27:33.513373 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506370    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 19:27:33.513373 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506377    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 19:27:33.513373 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506382    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 19:27:33.513857 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506389    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 19:27:33.513857 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506394    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 19:27:33.513857 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506401    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 19:27:33.513857 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506405    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 19:27:33.513857 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506412    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 19:27:33.513857 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506422    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 19:27:33.513857 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506458    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 19:27:33.513857 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506464    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 19:27:33.513857 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506468    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 19:27:33.513857 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506473    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 19:27:33.513857 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506477    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 19:27:33.513857 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506482    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 19:27:33.513857 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506486    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 19:27:33.513857 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506491    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 19:27:33.513857 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506495    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 19:27:33.513857 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506500    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 19:27:33.513857 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506505    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 19:27:33.513857 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506509    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 19:27:33.513857 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506518    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 19:27:33.513857 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506523    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 19:27:33.514661 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506527    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 19:27:33.514661 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506532    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 19:27:33.514661 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506537    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 19:27:33.514661 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506541    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 19:27:33.514661 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506546    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 19:27:33.514661 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506550    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 19:27:33.514661 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506555    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 19:27:33.514661 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506559    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 19:27:33.514661 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506564    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 19:27:33.514661 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506568    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 19:27:33.514661 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506578    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 19:27:33.514661 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506582    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 19:27:33.514661 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506587    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 19:27:33.514661 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506591    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 19:27:33.514661 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506596    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 19:27:33.514661 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506600    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 19:27:33.514661 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506605    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 19:27:33.514661 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506610    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 19:27:33.514661 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506614    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 19:27:33.514661 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506652    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 19:27:33.515323 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506660    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 19:27:33.515323 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506667    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 19:27:33.515323 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506672    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 19:27:33.515323 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506682    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 19:27:33.515323 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506687    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 19:27:33.515323 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506691    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 19:27:33.515323 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506696    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 19:27:33.515323 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506700    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 19:27:33.515323 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506705    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 19:27:33.515323 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506710    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 19:27:33.515323 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506714    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 19:27:33.515323 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506719    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 19:27:33.515323 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506723    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 19:27:33.515323 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506727    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 19:27:33.515323 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506732    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 19:27:33.515323 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506743    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 19:27:33.515323 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506747    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 19:27:33.515323 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506752    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 19:27:33.515323 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506756    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 19:27:33.515803 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506760    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 19:27:33.515803 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506765    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 19:27:33.515803 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506769    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 19:27:33.515803 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506774    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 19:27:33.515803 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506779    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 19:27:33.515803 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506783    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 19:27:33.515803 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506787    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 19:27:33.515803 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506792    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 19:27:33.515803 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506796    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 19:27:33.515803 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506806    2579 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 19:27:33.515803 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506810    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 19:27:33.515803 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506815    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 19:27:33.515803 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506819    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 19:27:33.515803 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506851    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 19:27:33.515803 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506856    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 19:27:33.515803 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506861    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 19:27:33.515803 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506866    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 19:27:33.515803 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506898    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 19:27:33.515803 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506905    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 19:27:33.516428 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506910    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 19:27:33.516428 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506914    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 19:27:33.516428 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506918    2579 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 19:27:33.516428 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506928    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 19:27:33.516428 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.506932    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 19:27:33.516428 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.507106    2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 19:27:33.516428 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507388    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 19:27:33.516428 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507395    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 19:27:33.516428 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507399    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 19:27:33.516428 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507402    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 19:27:33.516428 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507405    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 19:27:33.516428 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507408    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 19:27:33.516428 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507411    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 19:27:33.516428 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507413    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 19:27:33.516428 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507416    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 19:27:33.516428 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507419    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 19:27:33.516948 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507422    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 19:27:33.516948 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507424    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 19:27:33.516948 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507427    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 19:27:33.516948 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507429    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 19:27:33.516948 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507432    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 19:27:33.516948 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507435    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 19:27:33.516948 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507437    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 19:27:33.516948 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507440    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 19:27:33.516948 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507442    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 19:27:33.516948 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507445    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 19:27:33.516948 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507448    2579 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 19:27:33.516948 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507451    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 19:27:33.516948 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507453    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 19:27:33.516948 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507456    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 19:27:33.516948 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507458    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 19:27:33.516948 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507461    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 19:27:33.516948 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507464    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 19:27:33.516948 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507466    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 19:27:33.516948 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507469    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 19:27:33.516948 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507472    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 19:27:33.517445 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507476    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 19:27:33.517445 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507481    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 19:27:33.517445 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507486    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 19:27:33.517445 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507489    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 19:27:33.517445 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507492    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 19:27:33.517445 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507494    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 19:27:33.517445 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507497    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 19:27:33.517445 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507499    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 19:27:33.517445 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507502    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 19:27:33.517445 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507505    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 19:27:33.517445 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507508    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 19:27:33.517445 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507510    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 19:27:33.517445 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507513    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 19:27:33.517445 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507515    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 19:27:33.517445 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507518    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 19:27:33.517445 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507521    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 19:27:33.517445 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507523    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 19:27:33.517445 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507526    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 19:27:33.517445 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507529    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 19:27:33.517897 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507531    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 19:27:33.517897 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507534    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 19:27:33.517897 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507536    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 19:27:33.517897 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507539    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 19:27:33.517897 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507542    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 19:27:33.517897 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507545    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 19:27:33.517897 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507548    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 19:27:33.517897 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507550    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 19:27:33.517897 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507553    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 19:27:33.517897 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507555    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 19:27:33.517897 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507558    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 19:27:33.517897 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507560    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 19:27:33.517897 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507563    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 19:27:33.517897 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507566    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 19:27:33.517897 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507569    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 19:27:33.517897 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507572    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 19:27:33.517897 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507575    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 19:27:33.517897 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507577    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 19:27:33.517897 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507580    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 19:27:33.517897 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507583    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 19:27:33.518508 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507585    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 19:27:33.518508 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507588    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 19:27:33.518508 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507591    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 19:27:33.518508 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507593    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 19:27:33.518508 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507596    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 19:27:33.518508 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507598    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 19:27:33.518508 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507601    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 19:27:33.518508 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507603    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 19:27:33.518508 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507606    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 19:27:33.518508 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507609    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 19:27:33.518508 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507611    2579 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 19:27:33.518508 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507614    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 19:27:33.518508 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507616    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 19:27:33.518508 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507619    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 19:27:33.518508 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507621    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 19:27:33.518508 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507624    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 19:27:33.518508 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507626    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 19:27:33.518508 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507629    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 19:27:33.518508 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507632    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 19:27:33.518508 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507634    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 19:27:33.519034 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507637    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 19:27:33.519034 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507640    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 19:27:33.519034 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507642    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 19:27:33.519034 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507646    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 19:27:33.519034 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507650    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 19:27:33.519034 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507653    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 19:27:33.519034 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507656    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 19:27:33.519034 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507659    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 19:27:33.519034 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507662    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 19:27:33.519034 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507664    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 19:27:33.519034 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507667    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 19:27:33.519034 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507669    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 19:27:33.519034 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507672    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 19:27:33.519034 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507675    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 19:27:33.519034 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507677    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 19:27:33.519034 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507680    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 19:27:33.519034 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507682    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 19:27:33.519469 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.507708    2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 19:27:33.519469 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507740    2579 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 19:27:33.519469 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507744    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 19:27:33.519469 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507746    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 19:27:33.519469 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507749    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 19:27:33.519469 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507752    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 19:27:33.519469 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507755    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 19:27:33.519469 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507758    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 19:27:33.519469 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507760    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 19:27:33.519469 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507763    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 19:27:33.519469 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507766    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 19:27:33.519469 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507769    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 19:27:33.519469 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507774    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 19:27:33.519469 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507777    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 19:27:33.519469 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507781    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 19:27:33.519469 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507784    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 19:27:33.519842 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507787    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 19:27:33.519842 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507790    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 19:27:33.519842 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507792    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 19:27:33.519842 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507795    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 19:27:33.519842 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507798    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 19:27:33.519842 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507801    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 19:27:33.519842 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507804    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 19:27:33.519842 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507807    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 19:27:33.519842 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507810    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 19:27:33.519842 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507812    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 19:27:33.519842 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507815    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 19:27:33.519842 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507818    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 19:27:33.519842 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507820    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 19:27:33.519842 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507823    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 19:27:33.519842 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507825    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 19:27:33.519842 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507828    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 19:27:33.519842 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507831    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 19:27:33.519842 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507834    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 19:27:33.519842 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507836    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 19:27:33.519842 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507841    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 19:27:33.520449 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507843    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 19:27:33.520449 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507846    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 19:27:33.520449 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507848    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 19:27:33.520449 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507851    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 19:27:33.520449 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507853    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 19:27:33.520449 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507856    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 19:27:33.520449 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507858    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 19:27:33.520449 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507861    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 19:27:33.520449 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507864    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 19:27:33.520449 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507867    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 19:27:33.520449 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507869    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 19:27:33.520449 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507872    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 19:27:33.520449 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507874    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 19:27:33.520449 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507877    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 19:27:33.520449 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507879    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 19:27:33.520449 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507882    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 19:27:33.520449 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507885    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 19:27:33.520449 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507888    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 19:27:33.520449 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507891    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 19:27:33.520449 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507893    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 19:27:33.520904 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507896    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 19:27:33.520904 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507899    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 19:27:33.520904 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507901    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 19:27:33.520904 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507904    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 19:27:33.520904 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507907    2579 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 19:27:33.520904 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507909    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 19:27:33.520904 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507912    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 19:27:33.520904 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507914    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 19:27:33.520904 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507917    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 19:27:33.520904 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507919    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 19:27:33.520904 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507922    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 19:27:33.520904 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507925    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 19:27:33.520904 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507928    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 19:27:33.520904 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507930    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 19:27:33.520904 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507933    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 19:27:33.520904 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507935    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 19:27:33.520904 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507938    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 19:27:33.520904 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507942    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 19:27:33.520904 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507945    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 19:27:33.521363 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507948    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 19:27:33.521363 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507950    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 19:27:33.521363 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507956    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 19:27:33.521363 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507958    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 19:27:33.521363 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507962    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 19:27:33.521363 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507965    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 19:27:33.521363 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507967    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 19:27:33.521363 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507970    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 19:27:33.521363 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507972    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 19:27:33.521363 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507975    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 19:27:33.521363 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507977    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 19:27:33.521363 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507980    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 19:27:33.521363 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507982    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 19:27:33.521363 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507985    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 19:27:33.521363 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507988    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 19:27:33.521363 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507990    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 19:27:33.521363 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507993    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 19:27:33.521363 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507995    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 19:27:33.521363 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.507998    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 19:27:33.521810 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.508000    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 19:27:33.521810 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.508003    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 19:27:33.521810 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.508005    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 19:27:33.521810 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.508008    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 19:27:33.521810 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.508025    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 19:27:33.521810 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.508029    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 19:27:33.521810 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.508031    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 19:27:33.521810 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.508034    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 19:27:33.521810 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.508037    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 19:27:33.521810 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.508040    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 19:27:33.521810 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.508042    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 19:27:33.521810 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.508046    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 19:27:33.521810 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:33.508048    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 19:27:33.521810 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.508107    2579 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 02 19:27:33.521810 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.510919    2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 02 19:27:33.521810 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.511843    2579 server.go:1014] "Starting client certificate rotation"
Nov 02 19:27:33.521810 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.511946    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 02 19:27:33.521810 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.512932    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 02 19:27:33.522277 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.514721    2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 19:27:33.522277 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.515845    2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 19:27:33.523758 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.523744    2579 log.go:25] "Validated CRI v1 runtime API"
Nov 02 19:27:33.527511 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.527498    2579 log.go:25] "Validated CRI v1 image API"
Nov 02 19:27:33.528353 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.528332    2579 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 02 19:27:33.532903 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.532879    2579 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 96a9f595-0df5-4f03-8f0c-c6da713f0add:/dev/nvme0n1p4 d0dc1a70-3182-4b90-b28d-cb9e1470b581:/dev/nvme0n1p3]
Nov 02 19:27:33.532971 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.532901    2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 02 19:27:33.538823 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.538713    2579 manager.go:217] Machine: {Timestamp:2026-11-02 19:27:33.537260289 +0000 UTC m=+0.565534598 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100029 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec26c1c90f4d11743645d2d184f7ad79 SystemUUID:ec26c1c9-0f4d-1174-3645-d2d184f7ad79 BootID:8ee7c652-3485-4821-8e0e-6dd701b91d80 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:de:70:42:9d Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:de:70:42:9d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:86:47:23:d9:6a:02 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 02 19:27:33.538823 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.538822    2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 02 19:27:33.538933 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.538895    2579 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 02 19:27:33.541544 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.541520    2579 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 02 19:27:33.541692 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.541546    2579 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-9.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 02 19:27:33.541733 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.541694    2579 topology_manager.go:143] "Creating topology manager with none policy"
Nov 02 19:27:33.541733 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.541702    2579 container_manager_linux.go:311] "Creating device plugin manager"
Nov 02 19:27:33.541785 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.541738    2579 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 19:27:33.541785 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.541755    2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 19:27:33.542530 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.542519    2579 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 02 19:27:33.545622 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.545610    2579 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 02 19:27:33.545739 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.545730    2579 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 02 19:27:33.546436 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.546426    2579 kubelet.go:493] "Attempting to sync node with API server"
Nov 02 19:27:33.546469 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.546443    2579 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 02 19:27:33.546469 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.546460    2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 02 19:27:33.546534 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.546491    2579 kubelet.go:405] "Adding apiserver pod source"
Nov 02 19:27:33.546534 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.546499    2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 02 19:27:33.547041 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.547024    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 19:27:33.549714 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.549691    2579 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 02 19:27:33.555035 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.555004    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 19:27:33.555965 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.555952    2579 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 02 19:27:33.556027 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.555983    2579 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 02 19:27:33.556152 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.556141    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 02 19:27:33.556196 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.556156    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 02 19:27:33.556196 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.556164    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 02 19:27:33.556196 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.556170    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 02 19:27:33.556196 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.556176    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 02 19:27:33.556196 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.556181    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 02 19:27:33.556717 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.556709    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 02 19:27:33.556748 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.556718    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 02 19:27:33.556748 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.556726    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 02 19:27:33.556748 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.556732    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 02 19:27:33.556748 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.556740    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 02 19:27:33.556748 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.556748    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 02 19:27:33.558258 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.558248    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 02 19:27:33.558258 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.558258    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 02 19:27:33.562925 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.562913    2579 server.go:1263] "Started kubelet"
Nov 02 19:27:33.562990 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.562965    2579 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 02 19:27:33.563102 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.563052    2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 02 19:27:33.563154 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.563121    2579 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 02 19:27:33.563646 ip-10-0-133-9 systemd[1]: Started Kubernetes Kubelet.
Nov 02 19:27:33.565036 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.564030    2579 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 02 19:27:33.566331 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.565461    2579 server.go:317] "Adding debug handlers to kubelet server"
Nov 02 19:27:33.569455 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.569436    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 02 19:27:33.570202 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:33.570172    2579 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 02 19:27:33.570202 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.570200    2579 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 02 19:27:33.570333 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:33.570314    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-9.us-west-2.compute.internal\" not found"
Nov 02 19:27:33.570874 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.570851    2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 02 19:27:33.570874 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.570861    2579 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 02 19:27:33.571021 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.570886    2579 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 02 19:27:33.571021 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.570980    2579 reconstruct.go:97] "Volume reconstruction finished"
Nov 02 19:27:33.571021 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.570989    2579 reconciler.go:29] "Reconciler: start to sync state"
Nov 02 19:27:33.571021 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.570996    2579 factory.go:55] Registering systemd factory
Nov 02 19:27:33.571021 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.571023    2579 factory.go:223] Registration of the systemd container factory successfully
Nov 02 19:27:33.571257 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.571244    2579 factory.go:153] Registering CRI-O factory
Nov 02 19:27:33.571257 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.571259    2579 factory.go:223] Registration of the crio container factory successfully
Nov 02 19:27:33.571352 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.571294    2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 02 19:27:33.571352 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.571306    2579 factory.go:103] Registering Raw factory
Nov 02 19:27:33.571352 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.571317    2579 manager.go:1199] Started watching for new ooms in manager
Nov 02 19:27:33.572209 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.572188    2579 manager.go:319] Starting recovery of all containers
Nov 02 19:27:33.577584 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.577554    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 02 19:27:33.584033 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.583968    2579 manager.go:324] Recovery completed
Nov 02 19:27:33.590536 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.590518    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 19:27:33.592770 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.592744    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-9.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 19:27:33.592849 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.592788    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-9.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 19:27:33.592849 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.592808    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-9.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 19:27:33.593379 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.593366    2579 cpu_manager.go:226] "Starting" policy="none"
Nov 02 19:27:33.593450 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.593378    2579 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 02 19:27:33.593450 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.593400    2579 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 02 19:27:33.593450 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.593418    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 19:27:33.594153 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.594135    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-9.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 19:27:33.594223 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.594165    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-9.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 19:27:33.594223 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.594176    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-9.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 19:27:33.594958 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.594942    2579 policy_none.go:50] "Start"
Nov 02 19:27:33.594958 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.594958    2579 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 02 19:27:33.595089 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.594971    2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 02 19:27:33.596314 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.596303    2579 policy_none.go:44] "Start"
Nov 02 19:27:33.634666 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.634540    2579 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 02 19:27:33.640001 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.639987    2579 manager.go:341] "Starting Device Plugin manager"
Nov 02 19:27:33.640105 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:33.640039    2579 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 02 19:27:33.640105 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.640051    2579 server.go:87] "Starting device plugin registration server"
Nov 02 19:27:33.640284 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.640271    2579 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 02 19:27:33.640338 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.640285    2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 02 19:27:33.640427 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.640411    2579 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 02 19:27:33.640495 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.640483    2579 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 02 19:27:33.640495 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.640495    2579 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 02 19:27:33.641166 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:33.641091    2579 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 02 19:27:33.641166 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:33.641150    2579 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-9.us-west-2.compute.internal\" not found"
Nov 02 19:27:33.703382 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:33.703348    2579 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-9.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Nov 02 19:27:33.703382 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.703354    2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-9.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 02 19:27:33.740985 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.740960    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 19:27:33.741908 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.741888    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-9.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 19:27:33.742001 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.741923    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-9.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 19:27:33.742001 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.741946    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-9.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 19:27:33.742001 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.741990    2579 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-9.us-west-2.compute.internal"
Nov 02 19:27:33.749403 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:33.748580    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-9.us-west-2.compute.internal.18e45109aad42f08  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-9.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-9.us-west-2.compute.internal,},FirstTimestamp:2026-11-02 19:27:33.562887944 +0000 UTC m=+0.591162254,LastTimestamp:2026-11-02 19:27:33.562887944 +0000 UTC m=+0.591162254,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-9.us-west-2.compute.internal,}"
Nov 02 19:27:33.812433 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.796554    2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 02 19:27:33.812433 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.796629    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hkkd8"
Nov 02 19:27:33.846866 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.846810    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 02 19:27:33.846866 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.846835    2579 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 02 19:27:33.846866 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.846853    2579 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 02 19:27:33.847086 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:33.846891    2579 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Nov 02 19:27:33.866159 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.866141    2579 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 19:27:33.947711 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.947649    2579 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-9.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-9.us-west-2.compute.internal"]
Nov 02 19:27:33.947872 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.947757    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 19:27:33.948837 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.948812    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-9.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 19:27:33.948958 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.948849    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-9.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 19:27:33.948958 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.948860    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-9.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 19:27:33.950118 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.950106    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 19:27:33.950710 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.950693    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-9.us-west-2.compute.internal"
Nov 02 19:27:33.950785 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.950726    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 19:27:33.950896 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.950883    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-9.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 19:27:33.950937 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.950911    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-9.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 19:27:33.950937 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.950923    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-9.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 19:27:33.952309 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.952296    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-9.us-west-2.compute.internal"
Nov 02 19:27:33.952360 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.952322    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 19:27:33.952486 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.952473    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-9.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 19:27:33.952531 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.952503    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-9.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 19:27:33.952531 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.952519    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-9.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 19:27:33.952984 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.952967    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-9.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 19:27:33.953054 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.952999    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-9.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 19:27:33.953054 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.953041    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-9.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 19:27:33.964133 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.964117    2579 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 19:27:33.964666 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.964644    2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 19:27:33.965499 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.965486    2579 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 19:27:33.965911 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.965895    2579 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-133-9.us-west-2.compute.internal"
Nov 02 19:27:33.965954 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:33.965928    2579 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-9.us-west-2.compute.internal\": node \"ip-10-0-133-9.us-west-2.compute.internal\" not found"
Nov 02 19:27:33.970493 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.970477    2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-9.us-west-2.compute.internal"
Nov 02 19:27:33.972589 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.972573    2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-9.us-west-2.compute.internal"
Nov 02 19:27:33.973902 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.973884    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b70d1060da282fd74b3ebbf01b381326-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-9.us-west-2.compute.internal\" (UID: \"b70d1060da282fd74b3ebbf01b381326\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-9.us-west-2.compute.internal"
Nov 02 19:27:33.973964 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.973909    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b70d1060da282fd74b3ebbf01b381326-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-9.us-west-2.compute.internal\" (UID: \"b70d1060da282fd74b3ebbf01b381326\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-9.us-west-2.compute.internal"
Nov 02 19:27:33.973964 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.973927    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d6e1f694aea2cef1694653a075ee3dd5-config\") pod \"kube-apiserver-proxy-ip-10-0-133-9.us-west-2.compute.internal\" (UID: \"d6e1f694aea2cef1694653a075ee3dd5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-9.us-west-2.compute.internal"
Nov 02 19:27:33.976142 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:33.976130    2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-9.us-west-2.compute.internal"
Nov 02 19:27:34.031114 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.031094    2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 19:27:34.074981 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.074954    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b70d1060da282fd74b3ebbf01b381326-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-9.us-west-2.compute.internal\" (UID: \"b70d1060da282fd74b3ebbf01b381326\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-9.us-west-2.compute.internal"
Nov 02 19:27:34.075104 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.074985    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b70d1060da282fd74b3ebbf01b381326-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-9.us-west-2.compute.internal\" (UID: \"b70d1060da282fd74b3ebbf01b381326\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-9.us-west-2.compute.internal"
Nov 02 19:27:34.075104 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.075003    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d6e1f694aea2cef1694653a075ee3dd5-config\") pod \"kube-apiserver-proxy-ip-10-0-133-9.us-west-2.compute.internal\" (UID: \"d6e1f694aea2cef1694653a075ee3dd5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-9.us-west-2.compute.internal"
Nov 02 19:27:34.075104 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.075084    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d6e1f694aea2cef1694653a075ee3dd5-config\") pod \"kube-apiserver-proxy-ip-10-0-133-9.us-west-2.compute.internal\" (UID: \"d6e1f694aea2cef1694653a075ee3dd5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-9.us-west-2.compute.internal"
Nov 02 19:27:34.075216 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.075084    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b70d1060da282fd74b3ebbf01b381326-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-9.us-west-2.compute.internal\" (UID: \"b70d1060da282fd74b3ebbf01b381326\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-9.us-west-2.compute.internal"
Nov 02 19:27:34.075216 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.075085    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b70d1060da282fd74b3ebbf01b381326-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-9.us-west-2.compute.internal\" (UID: \"b70d1060da282fd74b3ebbf01b381326\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-9.us-west-2.compute.internal"
Nov 02 19:27:34.077871 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.077854    2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 19:27:34.079475 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.079461    2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 19:27:34.079539 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:34.079500    2579 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-133-9.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-9.us-west-2.compute.internal"
Nov 02 19:27:34.079539 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.079512    2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-9.us-west-2.compute.internal"
Nov 02 19:27:34.081608 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.081591    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-9.us-west-2.compute.internal"
Nov 02 19:27:34.138392 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.138331    2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 19:27:34.138392 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:34.138372    2579 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-133-9.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-9.us-west-2.compute.internal"
Nov 02 19:27:34.335836 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.335794    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-9.us-west-2.compute.internal"
Nov 02 19:27:34.512211 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.512100    2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 02 19:27:34.512760 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.512291    2579 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 02 19:27:34.512760 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.512316    2579 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 02 19:27:34.512760 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.512376    2579 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 02 19:27:34.512760 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:34.512356    2579 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a15d747bf66324e00b1608078aae7d90-1121cf0d51ffc33d.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-133-9.us-west-2.compute.internal.18e45109ac9cd1ed\": read tcp 10.0.133.9:57254->13.58.97.234:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-133-9.us-west-2.compute.internal.18e45109ac9cd1ed  default   4996 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-9.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-133-9.us-west-2.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-133-9.us-west-2.compute.internal,},FirstTimestamp:2026-11-02 19:27:33 +0000 UTC,LastTimestamp:2026-11-02 19:27:33.741952006 +0000 UTC m=+0.770226318,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-9.us-west-2.compute.internal,}"
Nov 02 19:27:34.547678 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.547641    2579 apiserver.go:52] "Watching apiserver"
Nov 02 19:27:34.571209 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.571169    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 02 19:27:34.802003 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.801901    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 19:22:34 +0000 UTC" deadline="2028-07-16 04:22:10.048435902 +0000 UTC"
Nov 02 19:27:34.802003 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.801945    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14912h54m35.246494196s"
Nov 02 19:27:34.864400 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.864361    2579 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 02 19:27:34.864737 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.864716    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-f6r6d","openshift-ovn-kubernetes/ovnkube-node-vst5r","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5wvz","openshift-image-registry/node-ca-hrhpj","openshift-multus/multus-qhswb","openshift-network-diagnostics/network-check-target-rhdd6","openshift-network-operator/iptables-alerter-cmk66","openshift-dns/node-resolver-r9fcz","kube-system/kube-apiserver-proxy-ip-10-0-133-9.us-west-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-9.us-west-2.compute.internal","kube-system/global-pull-secret-syncer-4b2pr","kube-system/konnectivity-agent-jgfjj","openshift-cluster-node-tuning-operator/tuned-nvdtg","openshift-multus/multus-additional-cni-plugins-4ddbs"]
Nov 02 19:27:34.866379 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.866360    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f6r6d"
Nov 02 19:27:34.866477 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:34.866457    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f6r6d" podUID="17fe5072-f4c5-4075-8614-4229ed05d4b2"
Nov 02 19:27:34.867546 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.867529    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:27:34.868739 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.868716    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5wvz"
Nov 02 19:27:34.869900 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.869883    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hrhpj"
Nov 02 19:27:34.871173 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.871157    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qhswb"
Nov 02 19:27:34.872440 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.872420    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rhdd6"
Nov 02 19:27:34.872516 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:34.872489    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rhdd6" podUID="31a69a0e-cc9c-47ad-b577-36b50d61d213"
Nov 02 19:27:34.873750 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.873734    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cmk66"
Nov 02 19:27:34.875069 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.875053    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-r9fcz"
Nov 02 19:27:34.876306 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.876294    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4b2pr"
Nov 02 19:27:34.876367 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:34.876351    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4b2pr" podUID="0b2d7938-0c97-4f07-a7d2-b5c35d2bc8f9"
Nov 02 19:27:34.877399 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.877386    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jgfjj"
Nov 02 19:27:34.878701 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.878685    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nvdtg"
Nov 02 19:27:34.880317 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.880302    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4ddbs"
Nov 02 19:27:34.880377 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.880362    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8756b800-1b4e-41da-934a-25c6b8ee738c-var-lib-openvswitch\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:27:34.880416 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.880386    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8756b800-1b4e-41da-934a-25c6b8ee738c-log-socket\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:27:34.880416 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.880403    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-828qc\" (UniqueName: \"kubernetes.io/projected/8756b800-1b4e-41da-934a-25c6b8ee738c-kube-api-access-828qc\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:27:34.880508 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.880420    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/74106611-359e-4353-b113-70521144bed1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-f5wvz\" (UID: \"74106611-359e-4353-b113-70521144bed1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5wvz"
Nov 02 19:27:34.880508 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.880441    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8756b800-1b4e-41da-934a-25c6b8ee738c-run-ovn\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:27:34.880508 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.880458    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9b7f05bf-12cd-4aad-b4ee-8b517e901ca6-host-run-k8s-cni-cncf-io\") pod \"multus-qhswb\" (UID: \"9b7f05bf-12cd-4aad-b4ee-8b517e901ca6\") " pod="openshift-multus/multus-qhswb"
Nov 02 19:27:34.880623 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.880519    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9b7f05bf-12cd-4aad-b4ee-8b517e901ca6-host-run-netns\") pod \"multus-qhswb\" (UID: \"9b7f05bf-12cd-4aad-b4ee-8b517e901ca6\") " pod="openshift-multus/multus-qhswb"
Nov 02 19:27:34.880623 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.880557    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9b7f05bf-12cd-4aad-b4ee-8b517e901ca6-host-var-lib-cni-multus\") pod \"multus-qhswb\" (UID: \"9b7f05bf-12cd-4aad-b4ee-8b517e901ca6\") " pod="openshift-multus/multus-qhswb"
Nov 02 19:27:34.880623 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.880599    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9b7f05bf-12cd-4aad-b4ee-8b517e901ca6-host-var-lib-kubelet\") pod \"multus-qhswb\" (UID: \"9b7f05bf-12cd-4aad-b4ee-8b517e901ca6\") " pod="openshift-multus/multus-qhswb"
Nov 02 19:27:34.880714 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.880638    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9b7f05bf-12cd-4aad-b4ee-8b517e901ca6-etc-kubernetes\") pod \"multus-qhswb\" (UID: \"9b7f05bf-12cd-4aad-b4ee-8b517e901ca6\") " pod="openshift-multus/multus-qhswb"
Nov 02 19:27:34.880714 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.880673    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccf8l\" (UniqueName: \"kubernetes.io/projected/31a69a0e-cc9c-47ad-b577-36b50d61d213-kube-api-access-ccf8l\") pod \"network-check-target-rhdd6\" (UID: \"31a69a0e-cc9c-47ad-b577-36b50d61d213\") " pod="openshift-network-diagnostics/network-check-target-rhdd6"
Nov 02 19:27:34.880714 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.880707    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8756b800-1b4e-41da-934a-25c6b8ee738c-run-systemd\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:27:34.880806 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.880730    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8756b800-1b4e-41da-934a-25c6b8ee738c-node-log\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:27:34.880806 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.880751    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8756b800-1b4e-41da-934a-25c6b8ee738c-host-run-ovn-kubernetes\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:27:34.880806 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.880784    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8756b800-1b4e-41da-934a-25c6b8ee738c-ovn-node-metrics-cert\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:27:34.880897 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.880808    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgbgt\" (UniqueName: \"kubernetes.io/projected/74106611-359e-4353-b113-70521144bed1-kube-api-access-pgbgt\") pod \"aws-ebs-csi-driver-node-f5wvz\" (UID: \"74106611-359e-4353-b113-70521144bed1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5wvz"
Nov 02 19:27:34.880897 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.880833    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b6ad2c98-52b5-496e-9824-614a2e509a8a-serviceca\") pod \"node-ca-hrhpj\" (UID: \"b6ad2c98-52b5-496e-9824-614a2e509a8a\") " pod="openshift-image-registry/node-ca-hrhpj"
Nov 02 19:27:34.880897 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.880866    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9b7f05bf-12cd-4aad-b4ee-8b517e901ca6-hostroot\") pod \"multus-qhswb\" (UID: \"9b7f05bf-12cd-4aad-b4ee-8b517e901ca6\") " pod="openshift-multus/multus-qhswb"
Nov 02 19:27:34.880897 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.880892    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17fe5072-f4c5-4075-8614-4229ed05d4b2-metrics-certs\") pod \"network-metrics-daemon-f6r6d\" (UID: \"17fe5072-f4c5-4075-8614-4229ed05d4b2\") " pod="openshift-multus/network-metrics-daemon-f6r6d"
Nov 02 19:27:34.881016 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.880912    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbxts\" (UniqueName: \"kubernetes.io/projected/17fe5072-f4c5-4075-8614-4229ed05d4b2-kube-api-access-lbxts\") pod \"network-metrics-daemon-f6r6d\" (UID: \"17fe5072-f4c5-4075-8614-4229ed05d4b2\") " pod="openshift-multus/network-metrics-daemon-f6r6d"
Nov 02 19:27:34.881016 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.880934    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8756b800-1b4e-41da-934a-25c6b8ee738c-run-openvswitch\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:27:34.881016 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.880976    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/74106611-359e-4353-b113-70521144bed1-etc-selinux\") pod \"aws-ebs-csi-driver-node-f5wvz\" (UID: \"74106611-359e-4353-b113-70521144bed1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5wvz"
Nov 02 19:27:34.881016 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.881000    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9b7f05bf-12cd-4aad-b4ee-8b517e901ca6-multus-socket-dir-parent\") pod \"multus-qhswb\" (UID: \"9b7f05bf-12cd-4aad-b4ee-8b517e901ca6\") " pod="openshift-multus/multus-qhswb"
Nov 02 19:27:34.881153 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.881040    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2d956\" (UniqueName: \"kubernetes.io/projected/9b7f05bf-12cd-4aad-b4ee-8b517e901ca6-kube-api-access-2d956\") pod \"multus-qhswb\" (UID: \"9b7f05bf-12cd-4aad-b4ee-8b517e901ca6\") " pod="openshift-multus/multus-qhswb"
Nov 02 19:27:34.881153 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.881065    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8756b800-1b4e-41da-934a-25c6b8ee738c-etc-openvswitch\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:27:34.881153 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.881087    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8756b800-1b4e-41da-934a-25c6b8ee738c-ovnkube-script-lib\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:27:34.881153 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.881121    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/74106611-359e-4353-b113-70521144bed1-socket-dir\") pod \"aws-ebs-csi-driver-node-f5wvz\" (UID: \"74106611-359e-4353-b113-70521144bed1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5wvz"
Nov 02 19:27:34.881153 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.881136    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b6ad2c98-52b5-496e-9824-614a2e509a8a-host\") pod \"node-ca-hrhpj\" (UID: \"b6ad2c98-52b5-496e-9824-614a2e509a8a\") " pod="openshift-image-registry/node-ca-hrhpj"
Nov 02 19:27:34.881153 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.881151    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxwsm\" (UniqueName: \"kubernetes.io/projected/b6ad2c98-52b5-496e-9824-614a2e509a8a-kube-api-access-sxwsm\") pod \"node-ca-hrhpj\" (UID: \"b6ad2c98-52b5-496e-9824-614a2e509a8a\") " pod="openshift-image-registry/node-ca-hrhpj"
Nov 02 19:27:34.881331 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.881167    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9b7f05bf-12cd-4aad-b4ee-8b517e901ca6-system-cni-dir\") pod \"multus-qhswb\" (UID: \"9b7f05bf-12cd-4aad-b4ee-8b517e901ca6\") " pod="openshift-multus/multus-qhswb"
Nov 02 19:27:34.881331 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.881207    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9b7f05bf-12cd-4aad-b4ee-8b517e901ca6-multus-conf-dir\") pod \"multus-qhswb\" (UID: \"9b7f05bf-12cd-4aad-b4ee-8b517e901ca6\") " pod="openshift-multus/multus-qhswb"
Nov 02 19:27:34.881331 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.881240    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8756b800-1b4e-41da-934a-25c6b8ee738c-host-cni-netd\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:27:34.881331 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.881261    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8756b800-1b4e-41da-934a-25c6b8ee738c-env-overrides\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:27:34.881331 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.881276    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/74106611-359e-4353-b113-70521144bed1-sys-fs\") pod \"aws-ebs-csi-driver-node-f5wvz\" (UID: \"74106611-359e-4353-b113-70521144bed1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5wvz"
Nov 02 19:27:34.881331 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.881292    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9b7f05bf-12cd-4aad-b4ee-8b517e901ca6-cnibin\") pod \"multus-qhswb\" (UID: \"9b7f05bf-12cd-4aad-b4ee-8b517e901ca6\") " pod="openshift-multus/multus-qhswb"
Nov 02 19:27:34.881331 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.881315    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8756b800-1b4e-41da-934a-25c6b8ee738c-host-slash\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:27:34.881531 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.881334    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8756b800-1b4e-41da-934a-25c6b8ee738c-host-run-netns\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:27:34.881531 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.881372    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8756b800-1b4e-41da-934a-25c6b8ee738c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:27:34.881531 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.881406    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9b7f05bf-12cd-4aad-b4ee-8b517e901ca6-multus-cni-dir\") pod \"multus-qhswb\" (UID: \"9b7f05bf-12cd-4aad-b4ee-8b517e901ca6\") " pod="openshift-multus/multus-qhswb"
Nov 02 19:27:34.881531 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.881435    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9b7f05bf-12cd-4aad-b4ee-8b517e901ca6-cni-binary-copy\") pod \"multus-qhswb\" (UID: \"9b7f05bf-12cd-4aad-b4ee-8b517e901ca6\") " pod="openshift-multus/multus-qhswb"
Nov 02 19:27:34.881652 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.881533    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9b7f05bf-12cd-4aad-b4ee-8b517e901ca6-host-var-lib-cni-bin\") pod \"multus-qhswb\" (UID: \"9b7f05bf-12cd-4aad-b4ee-8b517e901ca6\") " pod="openshift-multus/multus-qhswb"
Nov 02 19:27:34.881652 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.881564    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8756b800-1b4e-41da-934a-25c6b8ee738c-host-kubelet\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:27:34.881652 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.881591    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8756b800-1b4e-41da-934a-25c6b8ee738c-host-cni-bin\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:27:34.881652 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.881632    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8756b800-1b4e-41da-934a-25c6b8ee738c-ovnkube-config\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:27:34.881797 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.881665    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/74106611-359e-4353-b113-70521144bed1-registration-dir\") pod \"aws-ebs-csi-driver-node-f5wvz\" (UID: \"74106611-359e-4353-b113-70521144bed1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5wvz"
Nov 02 19:27:34.881797 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.881697    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/74106611-359e-4353-b113-70521144bed1-device-dir\") pod \"aws-ebs-csi-driver-node-f5wvz\" (UID: \"74106611-359e-4353-b113-70521144bed1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5wvz"
Nov 02 19:27:34.881797 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.881725    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9b7f05bf-12cd-4aad-b4ee-8b517e901ca6-multus-daemon-config\") pod \"multus-qhswb\" (UID: \"9b7f05bf-12cd-4aad-b4ee-8b517e901ca6\") " pod="openshift-multus/multus-qhswb"
Nov 02 19:27:34.881797 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.881747    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9b7f05bf-12cd-4aad-b4ee-8b517e901ca6-host-run-multus-certs\") pod \"multus-qhswb\" (UID: \"9b7f05bf-12cd-4aad-b4ee-8b517e901ca6\") " pod="openshift-multus/multus-qhswb"
Nov 02 19:27:34.881797 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.881788    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8756b800-1b4e-41da-934a-25c6b8ee738c-systemd-units\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:27:34.927488 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.927463    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 02 19:27:34.927488 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.927461    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 02 19:27:34.927655 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.927491    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 02 19:27:34.927655 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.927500    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 02 19:27:34.927655 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.927523    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6bb8t\""
Nov 02 19:27:34.927655 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.927486    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 02 19:27:34.927787 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.927736    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 02 19:27:34.928111 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.928096    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 02 19:27:34.928185 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.928172    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 02 19:27:34.928256 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.928243    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 02 19:27:34.928305 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.928248    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-nnqx8\""
Nov 02 19:27:34.929367 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.929354    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-gsgfx\""
Nov 02 19:27:34.929548 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.929534    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 02 19:27:34.929548 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.929536    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 02 19:27:34.929643 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.929557    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 02 19:27:34.932866 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.930561    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 02 19:27:34.932866 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.930873    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 02 19:27:34.932866 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.931117    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-lnn7q\""
Nov 02 19:27:34.932866 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.931235    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 02 19:27:34.972325 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.972300    2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 02 19:27:34.981625 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.981601    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 02 19:27:34.982342 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.982320    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343-system-cni-dir\") pod \"multus-additional-cni-plugins-4ddbs\" (UID: \"4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343\") " pod="openshift-multus/multus-additional-cni-plugins-4ddbs"
Nov 02 19:27:34.982423 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.982354    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8756b800-1b4e-41da-934a-25c6b8ee738c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:27:34.982423 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.982379    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9b7f05bf-12cd-4aad-b4ee-8b517e901ca6-multus-cni-dir\") pod \"multus-qhswb\" (UID: \"9b7f05bf-12cd-4aad-b4ee-8b517e901ca6\") " pod="openshift-multus/multus-qhswb"
Nov 02 19:27:34.982540 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.982441    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9b7f05bf-12cd-4aad-b4ee-8b517e901ca6-host-var-lib-cni-bin\") pod \"multus-qhswb\" (UID: \"9b7f05bf-12cd-4aad-b4ee-8b517e901ca6\") " pod="openshift-multus/multus-qhswb"
Nov 02 19:27:34.982540 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.982506    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b6cb1c47-e00c-484d-bf34-445b9380553c-iptables-alerter-script\") pod \"iptables-alerter-cmk66\" (UID: \"b6cb1c47-e00c-484d-bf34-445b9380553c\") " pod="openshift-network-operator/iptables-alerter-cmk66"
Nov 02 19:27:34.982540 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.982527    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8756b800-1b4e-41da-934a-25c6b8ee738c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:27:34.982662 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.982537    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a8e20267-1b77-4eb3-b940-a29d79b0ec19-run\") pod \"tuned-nvdtg\" (UID: \"a8e20267-1b77-4eb3-b940-a29d79b0ec19\") " pod="openshift-cluster-node-tuning-operator/tuned-nvdtg"
Nov 02 19:27:34.982662 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.982565    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a8e20267-1b77-4eb3-b940-a29d79b0ec19-tmp\") pod \"tuned-nvdtg\" (UID: \"a8e20267-1b77-4eb3-b940-a29d79b0ec19\") " pod="openshift-cluster-node-tuning-operator/tuned-nvdtg"
Nov 02 19:27:34.982662 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.982595    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9b7f05bf-12cd-4aad-b4ee-8b517e901ca6-multus-cni-dir\") pod \"multus-qhswb\" (UID: \"9b7f05bf-12cd-4aad-b4ee-8b517e901ca6\") " pod="openshift-multus/multus-qhswb"
Nov 02 19:27:34.982662 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.982599    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9b7f05bf-12cd-4aad-b4ee-8b517e901ca6-host-var-lib-cni-bin\") pod \"multus-qhswb\" (UID: \"9b7f05bf-12cd-4aad-b4ee-8b517e901ca6\") " pod="openshift-multus/multus-qhswb"
Nov 02 19:27:34.982662 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.982650    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8756b800-1b4e-41da-934a-25c6b8ee738c-host-kubelet\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:27:34.982864 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.982689    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8756b800-1b4e-41da-934a-25c6b8ee738c-host-cni-bin\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:27:34.982864 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.982709    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8756b800-1b4e-41da-934a-25c6b8ee738c-host-kubelet\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:27:34.982864 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.982741    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/74106611-359e-4353-b113-70521144bed1-registration-dir\") pod \"aws-ebs-csi-driver-node-f5wvz\" (UID: \"74106611-359e-4353-b113-70521144bed1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5wvz"
Nov 02 19:27:34.982864 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.982761    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8756b800-1b4e-41da-934a-25c6b8ee738c-host-cni-bin\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:27:34.982864 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.982805    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/74106611-359e-4353-b113-70521144bed1-device-dir\") pod \"aws-ebs-csi-driver-node-f5wvz\" (UID: \"74106611-359e-4353-b113-70521144bed1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5wvz"
Nov 02 19:27:34.982864 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.982816    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/74106611-359e-4353-b113-70521144bed1-registration-dir\") pod \"aws-ebs-csi-driver-node-f5wvz\" (UID: \"74106611-359e-4353-b113-70521144bed1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5wvz"
Nov 02 19:27:34.982864 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.982861    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9b7f05bf-12cd-4aad-b4ee-8b517e901ca6-multus-daemon-config\") pod \"multus-qhswb\" (UID: \"9b7f05bf-12cd-4aad-b4ee-8b517e901ca6\") " pod="openshift-multus/multus-qhswb"
Nov 02 19:27:34.983202 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.982895    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/74106611-359e-4353-b113-70521144bed1-device-dir\") pod \"aws-ebs-csi-driver-node-f5wvz\" (UID: \"74106611-359e-4353-b113-70521144bed1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5wvz"
Nov 02 19:27:34.983202 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.982933    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9b7f05bf-12cd-4aad-b4ee-8b517e901ca6-host-run-multus-certs\") pod \"multus-qhswb\" (UID: \"9b7f05bf-12cd-4aad-b4ee-8b517e901ca6\") " pod="openshift-multus/multus-qhswb"
Nov 02 19:27:34.983202 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.983031    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a8e20267-1b77-4eb3-b940-a29d79b0ec19-etc-sysconfig\") pod \"tuned-nvdtg\" (UID: \"a8e20267-1b77-4eb3-b940-a29d79b0ec19\") " pod="openshift-cluster-node-tuning-operator/tuned-nvdtg"
Nov 02 19:27:34.983202 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.983052    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9b7f05bf-12cd-4aad-b4ee-8b517e901ca6-host-run-multus-certs\") pod \"multus-qhswb\" (UID: \"9b7f05bf-12cd-4aad-b4ee-8b517e901ca6\") " pod="openshift-multus/multus-qhswb"
Nov 02 19:27:34.983202 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.983082    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a8e20267-1b77-4eb3-b940-a29d79b0ec19-etc-sysctl-conf\") pod \"tuned-nvdtg\" (UID: \"a8e20267-1b77-4eb3-b940-a29d79b0ec19\") " pod="openshift-cluster-node-tuning-operator/tuned-nvdtg"
Nov 02 19:27:34.983202 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.983127    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8756b800-1b4e-41da-934a-25c6b8ee738c-systemd-units\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:27:34.983202 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.983165    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8756b800-1b4e-41da-934a-25c6b8ee738c-var-lib-openvswitch\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:27:34.983656 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.983192    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8756b800-1b4e-41da-934a-25c6b8ee738c-log-socket\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:27:34.983656 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.983276    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8756b800-1b4e-41da-934a-25c6b8ee738c-log-socket\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:27:34.983656 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.983281    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd804657-39fd-4ae9-899d-cd1f1b62717e-tmp-dir\") pod \"node-resolver-r9fcz\" (UID: \"dd804657-39fd-4ae9-899d-cd1f1b62717e\") " pod="openshift-dns/node-resolver-r9fcz"
Nov 02 19:27:34.983656 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.983341    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8756b800-1b4e-41da-934a-25c6b8ee738c-systemd-units\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:27:34.983656 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.983370    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8756b800-1b4e-41da-934a-25c6b8ee738c-var-lib-openvswitch\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:27:34.983656 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.983430    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a8e20267-1b77-4eb3-b940-a29d79b0ec19-etc-modprobe-d\") pod \"tuned-nvdtg\" (UID: \"a8e20267-1b77-4eb3-b940-a29d79b0ec19\") " pod="openshift-cluster-node-tuning-operator/tuned-nvdtg"
Nov 02 19:27:34.983656 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.983457    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a8e20267-1b77-4eb3-b940-a29d79b0ec19-sys\") pod \"tuned-nvdtg\" (UID: \"a8e20267-1b77-4eb3-b940-a29d79b0ec19\") " pod="openshift-cluster-node-tuning-operator/tuned-nvdtg"
Nov 02 19:27:34.983656 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.983539    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343-cni-binary-copy\") pod \"multus-additional-cni-plugins-4ddbs\" (UID: \"4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343\") " pod="openshift-multus/multus-additional-cni-plugins-4ddbs"
Nov 02 19:27:34.983656 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.983544    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9b7f05bf-12cd-4aad-b4ee-8b517e901ca6-multus-daemon-config\") pod \"multus-qhswb\" (UID: \"9b7f05bf-12cd-4aad-b4ee-8b517e901ca6\") " pod="openshift-multus/multus-qhswb"
Nov 02 19:27:34.983656 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.983604    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4ddbs\" (UID: \"4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343\") " pod="openshift-multus/multus-additional-cni-plugins-4ddbs"
Nov 02 19:27:34.983656 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.983639    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9b7f05bf-12cd-4aad-b4ee-8b517e901ca6-host-run-k8s-cni-cncf-io\") pod \"multus-qhswb\" (UID: \"9b7f05bf-12cd-4aad-b4ee-8b517e901ca6\") " pod="openshift-multus/multus-qhswb"
Nov 02 19:27:34.984136 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.983666    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9b7f05bf-12cd-4aad-b4ee-8b517e901ca6-host-run-netns\") pod \"multus-qhswb\" (UID: \"9b7f05bf-12cd-4aad-b4ee-8b517e901ca6\") " pod="openshift-multus/multus-qhswb"
Nov 02 19:27:34.984136 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.983696    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9b7f05bf-12cd-4aad-b4ee-8b517e901ca6-host-run-netns\") pod \"multus-qhswb\" (UID: \"9b7f05bf-12cd-4aad-b4ee-8b517e901ca6\") " pod="openshift-multus/multus-qhswb"
Nov 02 19:27:34.984136 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.983711    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9b7f05bf-12cd-4aad-b4ee-8b517e901ca6-host-run-k8s-cni-cncf-io\") pod \"multus-qhswb\" (UID: \"9b7f05bf-12cd-4aad-b4ee-8b517e901ca6\") " pod="openshift-multus/multus-qhswb"
Nov 02 19:27:34.984136 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.983735    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9b7f05bf-12cd-4aad-b4ee-8b517e901ca6-host-var-lib-kubelet\") pod \"multus-qhswb\" (UID: \"9b7f05bf-12cd-4aad-b4ee-8b517e901ca6\") " pod="openshift-multus/multus-qhswb"
Nov 02 19:27:34.984136 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.983770    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccf8l\" (UniqueName: \"kubernetes.io/projected/31a69a0e-cc9c-47ad-b577-36b50d61d213-kube-api-access-ccf8l\") pod \"network-check-target-rhdd6\" (UID: \"31a69a0e-cc9c-47ad-b577-36b50d61d213\") " pod="openshift-network-diagnostics/network-check-target-rhdd6"
Nov 02 19:27:34.984136 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.983798    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0b2d7938-0c97-4f07-a7d2-b5c35d2bc8f9-dbus\") pod \"global-pull-secret-syncer-4b2pr\" (UID: \"0b2d7938-0c97-4f07-a7d2-b5c35d2bc8f9\") " pod="kube-system/global-pull-secret-syncer-4b2pr"
Nov 02 19:27:34.984136 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.983835    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9b7f05bf-12cd-4aad-b4ee-8b517e901ca6-host-var-lib-kubelet\") pod \"multus-qhswb\" (UID: \"9b7f05bf-12cd-4aad-b4ee-8b517e901ca6\") " pod="openshift-multus/multus-qhswb"
Nov 02 19:27:34.984136 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.983864    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8756b800-1b4e-41da-934a-25c6b8ee738c-run-systemd\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:27:34.984136 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.983925    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgbgt\" (UniqueName: \"kubernetes.io/projected/74106611-359e-4353-b113-70521144bed1-kube-api-access-pgbgt\") pod \"aws-ebs-csi-driver-node-f5wvz\" (UID: \"74106611-359e-4353-b113-70521144bed1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5wvz"
Nov 02 19:27:34.984136 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.983946    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8756b800-1b4e-41da-934a-25c6b8ee738c-run-systemd\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:27:34.984136 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.983976    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9b7f05bf-12cd-4aad-b4ee-8b517e901ca6-hostroot\") pod \"multus-qhswb\" (UID: \"9b7f05bf-12cd-4aad-b4ee-8b517e901ca6\") " pod="openshift-multus/multus-qhswb"
Nov 02 19:27:34.984136 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.984004    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9b7f05bf-12cd-4aad-b4ee-8b517e901ca6-hostroot\") pod \"multus-qhswb\" (UID: \"9b7f05bf-12cd-4aad-b4ee-8b517e901ca6\") " pod="openshift-multus/multus-qhswb"
Nov 02 19:27:34.984136 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.984029    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcpxd\" (UniqueName: \"kubernetes.io/projected/dd804657-39fd-4ae9-899d-cd1f1b62717e-kube-api-access-bcpxd\") pod \"node-resolver-r9fcz\" (UID: \"dd804657-39fd-4ae9-899d-cd1f1b62717e\") " pod="openshift-dns/node-resolver-r9fcz"
Nov 02 19:27:34.984136 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.984063    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17fe5072-f4c5-4075-8614-4229ed05d4b2-metrics-certs\") pod \"network-metrics-daemon-f6r6d\" (UID: \"17fe5072-f4c5-4075-8614-4229ed05d4b2\") " pod="openshift-multus/network-metrics-daemon-f6r6d"
Nov 02 19:27:34.984136 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.984090    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8756b800-1b4e-41da-934a-25c6b8ee738c-run-openvswitch\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:27:34.984136 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.984121    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/74106611-359e-4353-b113-70521144bed1-etc-selinux\") pod \"aws-ebs-csi-driver-node-f5wvz\" (UID: \"74106611-359e-4353-b113-70521144bed1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5wvz"
Nov 02 19:27:34.984136 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.984145    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9b7f05bf-12cd-4aad-b4ee-8b517e901ca6-multus-socket-dir-parent\") pod \"multus-qhswb\" (UID: \"9b7f05bf-12cd-4aad-b4ee-8b517e901ca6\") " pod="openshift-multus/multus-qhswb"
Nov 02 19:27:34.984945 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.984167    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d956\" (UniqueName: \"kubernetes.io/projected/9b7f05bf-12cd-4aad-b4ee-8b517e901ca6-kube-api-access-2d956\") pod \"multus-qhswb\" (UID: \"9b7f05bf-12cd-4aad-b4ee-8b517e901ca6\") " pod="openshift-multus/multus-qhswb"
Nov 02 19:27:34.984945 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:34.984187    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 19:27:34.984945 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.984190    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8756b800-1b4e-41da-934a-25c6b8ee738c-etc-openvswitch\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:27:34.984945 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.984214    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9b7f05bf-12cd-4aad-b4ee-8b517e901ca6-system-cni-dir\") pod \"multus-qhswb\" (UID: \"9b7f05bf-12cd-4aad-b4ee-8b517e901ca6\") " pod="openshift-multus/multus-qhswb"
Nov 02 19:27:34.984945 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.984242    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b6cb1c47-e00c-484d-bf34-445b9380553c-host-slash\") pod \"iptables-alerter-cmk66\" (UID: \"b6cb1c47-e00c-484d-bf34-445b9380553c\") " pod="openshift-network-operator/iptables-alerter-cmk66"
Nov 02 19:27:34.984945 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.984267    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b2d7938-0c97-4f07-a7d2-b5c35d2bc8f9-original-pull-secret\") pod \"global-pull-secret-syncer-4b2pr\" (UID: \"0b2d7938-0c97-4f07-a7d2-b5c35d2bc8f9\") " pod="kube-system/global-pull-secret-syncer-4b2pr"
Nov 02 19:27:34.984945 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.984286    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8756b800-1b4e-41da-934a-25c6b8ee738c-etc-openvswitch\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:27:34.984945 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.984297    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9b7f05bf-12cd-4aad-b4ee-8b517e901ca6-system-cni-dir\") pod \"multus-qhswb\" (UID: \"9b7f05bf-12cd-4aad-b4ee-8b517e901ca6\") " pod="openshift-multus/multus-qhswb"
Nov 02 19:27:34.984945 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:34.984309    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17fe5072-f4c5-4075-8614-4229ed05d4b2-metrics-certs podName:17fe5072-f4c5-4075-8614-4229ed05d4b2 nodeName:}" failed. No retries permitted until 2026-11-02 19:27:35.484285193 +0000 UTC m=+2.512559508 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/17fe5072-f4c5-4075-8614-4229ed05d4b2-metrics-certs") pod "network-metrics-daemon-f6r6d" (UID: "17fe5072-f4c5-4075-8614-4229ed05d4b2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 19:27:34.984945 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.984323    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8756b800-1b4e-41da-934a-25c6b8ee738c-run-openvswitch\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:27:34.984945 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.984328    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/74106611-359e-4353-b113-70521144bed1-etc-selinux\") pod \"aws-ebs-csi-driver-node-f5wvz\" (UID: \"74106611-359e-4353-b113-70521144bed1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5wvz"
Nov 02 19:27:34.984945 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.984338    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a8e20267-1b77-4eb3-b940-a29d79b0ec19-host\") pod \"tuned-nvdtg\" (UID: \"a8e20267-1b77-4eb3-b940-a29d79b0ec19\") " pod="openshift-cluster-node-tuning-operator/tuned-nvdtg"
Nov 02 19:27:34.984945 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.984384    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343-cnibin\") pod \"multus-additional-cni-plugins-4ddbs\" (UID: \"4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343\") " pod="openshift-multus/multus-additional-cni-plugins-4ddbs"
Nov 02 19:27:34.984945 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.984473    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8756b800-1b4e-41da-934a-25c6b8ee738c-host-cni-netd\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:27:34.984945 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.984480    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9b7f05bf-12cd-4aad-b4ee-8b517e901ca6-multus-socket-dir-parent\") pod \"multus-qhswb\" (UID: \"9b7f05bf-12cd-4aad-b4ee-8b517e901ca6\") " pod="openshift-multus/multus-qhswb"
Nov 02 19:27:34.984945 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.984547    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3fabcb64-faf2-49f9-813a-ddafde0fd159-agent-certs\") pod \"konnectivity-agent-jgfjj\" (UID: \"3fabcb64-faf2-49f9-813a-ddafde0fd159\") " pod="kube-system/konnectivity-agent-jgfjj"
Nov 02 19:27:34.984945 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.984592    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8756b800-1b4e-41da-934a-25c6b8ee738c-host-cni-netd\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:27:34.985764 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.984595    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a8e20267-1b77-4eb3-b940-a29d79b0ec19-lib-modules\") pod \"tuned-nvdtg\" (UID: \"a8e20267-1b77-4eb3-b940-a29d79b0ec19\") " pod="openshift-cluster-node-tuning-operator/tuned-nvdtg"
Nov 02 19:27:34.985764 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.984630    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a8e20267-1b77-4eb3-b940-a29d79b0ec19-var-lib-kubelet\") pod \"tuned-nvdtg\" (UID: \"a8e20267-1b77-4eb3-b940-a29d79b0ec19\") " pod="openshift-cluster-node-tuning-operator/tuned-nvdtg"
Nov 02 19:27:34.985764 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.984658    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mptpf\" (UniqueName: \"kubernetes.io/projected/4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343-kube-api-access-mptpf\") pod \"multus-additional-cni-plugins-4ddbs\" (UID: \"4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343\") " pod="openshift-multus/multus-additional-cni-plugins-4ddbs"
Nov 02 19:27:34.985764 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.984690    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8756b800-1b4e-41da-934a-25c6b8ee738c-host-slash\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:27:34.985764 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.984750    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8756b800-1b4e-41da-934a-25c6b8ee738c-host-slash\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:27:34.985764 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.984759    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8756b800-1b4e-41da-934a-25c6b8ee738c-host-run-netns\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:27:34.985764 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.984807    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9b7f05bf-12cd-4aad-b4ee-8b517e901ca6-cni-binary-copy\") pod \"multus-qhswb\" (UID: \"9b7f05bf-12cd-4aad-b4ee-8b517e901ca6\") " pod="openshift-multus/multus-qhswb"
Nov 02 19:27:34.985764 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.984818    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8756b800-1b4e-41da-934a-25c6b8ee738c-host-run-netns\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:27:34.985764 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.984827    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4sz9\" (UniqueName: \"kubernetes.io/projected/b6cb1c47-e00c-484d-bf34-445b9380553c-kube-api-access-l4sz9\") pod \"iptables-alerter-cmk66\" (UID: \"b6cb1c47-e00c-484d-bf34-445b9380553c\") " pod="openshift-network-operator/iptables-alerter-cmk66"
Nov 02 19:27:34.985764 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.984845    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a8e20267-1b77-4eb3-b940-a29d79b0ec19-etc-sysctl-d\") pod \"tuned-nvdtg\" (UID: \"a8e20267-1b77-4eb3-b940-a29d79b0ec19\") " pod="openshift-cluster-node-tuning-operator/tuned-nvdtg"
Nov 02 19:27:34.985764 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.984860    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4ddbs\" (UID: \"4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343\") " pod="openshift-multus/multus-additional-cni-plugins-4ddbs"
Nov 02 19:27:34.985764 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.984876    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8756b800-1b4e-41da-934a-25c6b8ee738c-ovnkube-config\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:27:34.985764 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.984892    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3fabcb64-faf2-49f9-813a-ddafde0fd159-konnectivity-ca\") pod \"konnectivity-agent-jgfjj\" (UID: \"3fabcb64-faf2-49f9-813a-ddafde0fd159\") " pod="kube-system/konnectivity-agent-jgfjj"
Nov 02 19:27:34.985764 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.984934    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-828qc\" (UniqueName: \"kubernetes.io/projected/8756b800-1b4e-41da-934a-25c6b8ee738c-kube-api-access-828qc\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:27:34.985764 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.984969    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/74106611-359e-4353-b113-70521144bed1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-f5wvz\" (UID: \"74106611-359e-4353-b113-70521144bed1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5wvz"
Nov 02 19:27:34.985764 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.984999    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/dd804657-39fd-4ae9-899d-cd1f1b62717e-hosts-file\") pod \"node-resolver-r9fcz\" (UID: \"dd804657-39fd-4ae9-899d-cd1f1b62717e\") " pod="openshift-dns/node-resolver-r9fcz"
Nov 02 19:27:34.985764 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.985083    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8756b800-1b4e-41da-934a-25c6b8ee738c-run-ovn\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:27:34.986409 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.985138    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8756b800-1b4e-41da-934a-25c6b8ee738c-run-ovn\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:27:34.986409 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.985137    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9b7f05bf-12cd-4aad-b4ee-8b517e901ca6-host-var-lib-cni-multus\") pod \"multus-qhswb\" (UID: \"9b7f05bf-12cd-4aad-b4ee-8b517e901ca6\") " pod="openshift-multus/multus-qhswb"
Nov 02 19:27:34.986409 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.985158    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/74106611-359e-4353-b113-70521144bed1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-f5wvz\" (UID: \"74106611-359e-4353-b113-70521144bed1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5wvz"
Nov 02 19:27:34.986409 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.985190    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9b7f05bf-12cd-4aad-b4ee-8b517e901ca6-host-var-lib-cni-multus\") pod \"multus-qhswb\" (UID: \"9b7f05bf-12cd-4aad-b4ee-8b517e901ca6\") " pod="openshift-multus/multus-qhswb"
Nov 02 19:27:34.986409 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.985219    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9b7f05bf-12cd-4aad-b4ee-8b517e901ca6-etc-kubernetes\") pod \"multus-qhswb\" (UID: \"9b7f05bf-12cd-4aad-b4ee-8b517e901ca6\") " pod="openshift-multus/multus-qhswb"
Nov 02 19:27:34.986409 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.985260    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4ddbs\" (UID: \"4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343\") " pod="openshift-multus/multus-additional-cni-plugins-4ddbs"
Nov 02 19:27:34.986409 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.985291    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8756b800-1b4e-41da-934a-25c6b8ee738c-node-log\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:27:34.986409 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.985296    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9b7f05bf-12cd-4aad-b4ee-8b517e901ca6-etc-kubernetes\") pod \"multus-qhswb\" (UID: \"9b7f05bf-12cd-4aad-b4ee-8b517e901ca6\") " pod="openshift-multus/multus-qhswb"
Nov 02 19:27:34.986409 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.985297    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9b7f05bf-12cd-4aad-b4ee-8b517e901ca6-cni-binary-copy\") pod \"multus-qhswb\" (UID: \"9b7f05bf-12cd-4aad-b4ee-8b517e901ca6\") " pod="openshift-multus/multus-qhswb"
Nov 02 19:27:34.986409 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.985317    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8756b800-1b4e-41da-934a-25c6b8ee738c-host-run-ovn-kubernetes\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:27:34.986409 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.985349    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8756b800-1b4e-41da-934a-25c6b8ee738c-host-run-ovn-kubernetes\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:27:34.986409 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.985344    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8756b800-1b4e-41da-934a-25c6b8ee738c-ovn-node-metrics-cert\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:27:34.986409 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.985382    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b6ad2c98-52b5-496e-9824-614a2e509a8a-serviceca\") pod \"node-ca-hrhpj\" (UID: \"b6ad2c98-52b5-496e-9824-614a2e509a8a\") " pod="openshift-image-registry/node-ca-hrhpj"
Nov 02 19:27:34.986409 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.985386    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8756b800-1b4e-41da-934a-25c6b8ee738c-node-log\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:27:34.986409 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.985408    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0b2d7938-0c97-4f07-a7d2-b5c35d2bc8f9-kubelet-config\") pod \"global-pull-secret-syncer-4b2pr\" (UID: \"0b2d7938-0c97-4f07-a7d2-b5c35d2bc8f9\") " pod="kube-system/global-pull-secret-syncer-4b2pr"
Nov 02 19:27:34.986409 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.985433    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a8e20267-1b77-4eb3-b940-a29d79b0ec19-etc-tuned\") pod \"tuned-nvdtg\" (UID: \"a8e20267-1b77-4eb3-b940-a29d79b0ec19\") " pod="openshift-cluster-node-tuning-operator/tuned-nvdtg"
Nov 02 19:27:34.986409 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.985434    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8756b800-1b4e-41da-934a-25c6b8ee738c-ovnkube-config\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:27:34.986970 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.985469    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbxts\" (UniqueName: \"kubernetes.io/projected/17fe5072-f4c5-4075-8614-4229ed05d4b2-kube-api-access-lbxts\") pod \"network-metrics-daemon-f6r6d\" (UID: \"17fe5072-f4c5-4075-8614-4229ed05d4b2\") " pod="openshift-multus/network-metrics-daemon-f6r6d"
Nov 02 19:27:34.986970 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.985497    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a8e20267-1b77-4eb3-b940-a29d79b0ec19-etc-kubernetes\") pod \"tuned-nvdtg\" (UID: \"a8e20267-1b77-4eb3-b940-a29d79b0ec19\") " pod="openshift-cluster-node-tuning-operator/tuned-nvdtg"
Nov 02 19:27:34.986970 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.985524    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8756b800-1b4e-41da-934a-25c6b8ee738c-ovnkube-script-lib\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:27:34.986970 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.985568    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/74106611-359e-4353-b113-70521144bed1-socket-dir\") pod \"aws-ebs-csi-driver-node-f5wvz\" (UID: \"74106611-359e-4353-b113-70521144bed1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5wvz"
Nov 02 19:27:34.986970 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.985596    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b6ad2c98-52b5-496e-9824-614a2e509a8a-host\") pod \"node-ca-hrhpj\" (UID: \"b6ad2c98-52b5-496e-9824-614a2e509a8a\") " pod="openshift-image-registry/node-ca-hrhpj"
Nov 02 19:27:34.986970 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.985623    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxwsm\" (UniqueName: \"kubernetes.io/projected/b6ad2c98-52b5-496e-9824-614a2e509a8a-kube-api-access-sxwsm\") pod \"node-ca-hrhpj\" (UID: \"b6ad2c98-52b5-496e-9824-614a2e509a8a\") " pod="openshift-image-registry/node-ca-hrhpj"
Nov 02 19:27:34.986970 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.985648    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9b7f05bf-12cd-4aad-b4ee-8b517e901ca6-multus-conf-dir\") pod \"multus-qhswb\" (UID: \"9b7f05bf-12cd-4aad-b4ee-8b517e901ca6\") " pod="openshift-multus/multus-qhswb"
Nov 02 19:27:34.986970 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.985674    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a8e20267-1b77-4eb3-b940-a29d79b0ec19-etc-systemd\") pod \"tuned-nvdtg\" (UID: \"a8e20267-1b77-4eb3-b940-a29d79b0ec19\") " pod="openshift-cluster-node-tuning-operator/tuned-nvdtg"
Nov 02 19:27:34.986970 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.985727    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/74106611-359e-4353-b113-70521144bed1-socket-dir\") pod \"aws-ebs-csi-driver-node-f5wvz\" (UID: \"74106611-359e-4353-b113-70521144bed1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5wvz"
Nov 02 19:27:34.986970 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.985725    2579 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 02 19:27:34.986970 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.985768    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b6ad2c98-52b5-496e-9824-614a2e509a8a-serviceca\") pod \"node-ca-hrhpj\" (UID: \"b6ad2c98-52b5-496e-9824-614a2e509a8a\") " pod="openshift-image-registry/node-ca-hrhpj"
Nov 02 19:27:34.986970 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.985776    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b6ad2c98-52b5-496e-9824-614a2e509a8a-host\") pod \"node-ca-hrhpj\" (UID: \"b6ad2c98-52b5-496e-9824-614a2e509a8a\") " pod="openshift-image-registry/node-ca-hrhpj"
Nov 02 19:27:34.986970 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.985745    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9b7f05bf-12cd-4aad-b4ee-8b517e901ca6-multus-conf-dir\") pod \"multus-qhswb\" (UID: \"9b7f05bf-12cd-4aad-b4ee-8b517e901ca6\") " pod="openshift-multus/multus-qhswb"
Nov 02 19:27:34.986970 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.985882    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jxxh\" (UniqueName: \"kubernetes.io/projected/a8e20267-1b77-4eb3-b940-a29d79b0ec19-kube-api-access-7jxxh\") pod \"tuned-nvdtg\" (UID: \"a8e20267-1b77-4eb3-b940-a29d79b0ec19\") " pod="openshift-cluster-node-tuning-operator/tuned-nvdtg"
Nov 02 19:27:34.986970 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.985931    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8756b800-1b4e-41da-934a-25c6b8ee738c-env-overrides\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:27:34.986970 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.985975    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/74106611-359e-4353-b113-70521144bed1-sys-fs\") pod \"aws-ebs-csi-driver-node-f5wvz\" (UID: \"74106611-359e-4353-b113-70521144bed1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5wvz"
Nov 02 19:27:34.986970 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.986003    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9b7f05bf-12cd-4aad-b4ee-8b517e901ca6-cnibin\") pod \"multus-qhswb\" (UID: \"9b7f05bf-12cd-4aad-b4ee-8b517e901ca6\") " pod="openshift-multus/multus-qhswb"
Nov 02 19:27:34.986970 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.986069    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/74106611-359e-4353-b113-70521144bed1-sys-fs\") pod \"aws-ebs-csi-driver-node-f5wvz\" (UID: \"74106611-359e-4353-b113-70521144bed1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5wvz"
Nov 02 19:27:34.987563 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.986115    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9b7f05bf-12cd-4aad-b4ee-8b517e901ca6-cnibin\") pod \"multus-qhswb\" (UID: \"9b7f05bf-12cd-4aad-b4ee-8b517e901ca6\") " pod="openshift-multus/multus-qhswb"
Nov 02 19:27:34.987563 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.986126    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8756b800-1b4e-41da-934a-25c6b8ee738c-ovnkube-script-lib\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:27:34.987563 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.986274    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8756b800-1b4e-41da-934a-25c6b8ee738c-env-overrides\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:27:34.989435 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:34.989417    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8756b800-1b4e-41da-934a-25c6b8ee738c-ovn-node-metrics-cert\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:27:35.033174 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.033145    2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 02 19:27:35.033299 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.033235    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-2pvjv"
Nov 02 19:27:35.036641 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.036583    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 02 19:27:35.036641 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.036583    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-sp8wk\""
Nov 02 19:27:35.036641 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.036632    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 02 19:27:35.036865 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.036642    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-wxdrr\""
Nov 02 19:27:35.036865 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.036672    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-t4nhx\""
Nov 02 19:27:35.036865 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.036757    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 02 19:27:35.036865 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.036763    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 02 19:27:35.036865 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.036759    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 02 19:27:35.036865 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.036780    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 02 19:27:35.036865 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.036829    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 02 19:27:35.036865 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.036836    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 02 19:27:35.036865 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.036836    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 02 19:27:35.036865 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.036850    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9t85s\""
Nov 02 19:27:35.036865 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.036854    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 02 19:27:35.036865 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.036861    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 02 19:27:35.037248 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.036876    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-bg56f\""
Nov 02 19:27:35.047585 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:35.047567    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 19:27:35.047629 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:35.047588    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 19:27:35.047629 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:35.047599    2579 projected.go:196] Error preparing data for projected volume kube-api-access-ccf8l for pod openshift-network-diagnostics/network-check-target-rhdd6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 19:27:35.047694 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:35.047671    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/31a69a0e-cc9c-47ad-b577-36b50d61d213-kube-api-access-ccf8l podName:31a69a0e-cc9c-47ad-b577-36b50d61d213 nodeName:}" failed. No retries permitted until 2026-11-02 19:27:35.547655291 +0000 UTC m=+2.575929604 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-ccf8l" (UniqueName: "kubernetes.io/projected/31a69a0e-cc9c-47ad-b577-36b50d61d213-kube-api-access-ccf8l") pod "network-check-target-rhdd6" (UID: "31a69a0e-cc9c-47ad-b577-36b50d61d213") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 19:27:35.050239 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.050213    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2d956\" (UniqueName: \"kubernetes.io/projected/9b7f05bf-12cd-4aad-b4ee-8b517e901ca6-kube-api-access-2d956\") pod \"multus-qhswb\" (UID: \"9b7f05bf-12cd-4aad-b4ee-8b517e901ca6\") " pod="openshift-multus/multus-qhswb"
Nov 02 19:27:35.050349 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.050298    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgbgt\" (UniqueName: \"kubernetes.io/projected/74106611-359e-4353-b113-70521144bed1-kube-api-access-pgbgt\") pod \"aws-ebs-csi-driver-node-f5wvz\" (UID: \"74106611-359e-4353-b113-70521144bed1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5wvz"
Nov 02 19:27:35.050543 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.050525    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbxts\" (UniqueName: \"kubernetes.io/projected/17fe5072-f4c5-4075-8614-4229ed05d4b2-kube-api-access-lbxts\") pod \"network-metrics-daemon-f6r6d\" (UID: \"17fe5072-f4c5-4075-8614-4229ed05d4b2\") " pod="openshift-multus/network-metrics-daemon-f6r6d"
Nov 02 19:27:35.050602 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.050536    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-828qc\" (UniqueName: \"kubernetes.io/projected/8756b800-1b4e-41da-934a-25c6b8ee738c-kube-api-access-828qc\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:27:35.054900 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.054839    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxwsm\" (UniqueName: \"kubernetes.io/projected/b6ad2c98-52b5-496e-9824-614a2e509a8a-kube-api-access-sxwsm\") pod \"node-ca-hrhpj\" (UID: \"b6ad2c98-52b5-496e-9824-614a2e509a8a\") " pod="openshift-image-registry/node-ca-hrhpj"
Nov 02 19:27:35.086678 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.086638    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4ddbs\" (UID: \"4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343\") " pod="openshift-multus/multus-additional-cni-plugins-4ddbs"
Nov 02 19:27:35.086808 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.086711    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0b2d7938-0c97-4f07-a7d2-b5c35d2bc8f9-kubelet-config\") pod \"global-pull-secret-syncer-4b2pr\" (UID: \"0b2d7938-0c97-4f07-a7d2-b5c35d2bc8f9\") " pod="kube-system/global-pull-secret-syncer-4b2pr"
Nov 02 19:27:35.086808 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.086740    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a8e20267-1b77-4eb3-b940-a29d79b0ec19-etc-tuned\") pod \"tuned-nvdtg\" (UID: \"a8e20267-1b77-4eb3-b940-a29d79b0ec19\") " pod="openshift-cluster-node-tuning-operator/tuned-nvdtg"
Nov 02 19:27:35.086808 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.086764    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a8e20267-1b77-4eb3-b940-a29d79b0ec19-etc-kubernetes\") pod \"tuned-nvdtg\" (UID: \"a8e20267-1b77-4eb3-b940-a29d79b0ec19\") " pod="openshift-cluster-node-tuning-operator/tuned-nvdtg"
Nov 02 19:27:35.086999 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.086836    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a8e20267-1b77-4eb3-b940-a29d79b0ec19-etc-kubernetes\") pod \"tuned-nvdtg\" (UID: \"a8e20267-1b77-4eb3-b940-a29d79b0ec19\") " pod="openshift-cluster-node-tuning-operator/tuned-nvdtg"
Nov 02 19:27:35.086999 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.086848    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a8e20267-1b77-4eb3-b940-a29d79b0ec19-etc-systemd\") pod \"tuned-nvdtg\" (UID: \"a8e20267-1b77-4eb3-b940-a29d79b0ec19\") " pod="openshift-cluster-node-tuning-operator/tuned-nvdtg"
Nov 02 19:27:35.086999 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.086881    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jxxh\" (UniqueName: \"kubernetes.io/projected/a8e20267-1b77-4eb3-b940-a29d79b0ec19-kube-api-access-7jxxh\") pod \"tuned-nvdtg\" (UID: \"a8e20267-1b77-4eb3-b940-a29d79b0ec19\") " pod="openshift-cluster-node-tuning-operator/tuned-nvdtg"
Nov 02 19:27:35.086999 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.086850    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0b2d7938-0c97-4f07-a7d2-b5c35d2bc8f9-kubelet-config\") pod \"global-pull-secret-syncer-4b2pr\" (UID: \"0b2d7938-0c97-4f07-a7d2-b5c35d2bc8f9\") " pod="kube-system/global-pull-secret-syncer-4b2pr"
Nov 02 19:27:35.086999 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.086899    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a8e20267-1b77-4eb3-b940-a29d79b0ec19-etc-systemd\") pod \"tuned-nvdtg\" (UID: \"a8e20267-1b77-4eb3-b940-a29d79b0ec19\") " pod="openshift-cluster-node-tuning-operator/tuned-nvdtg"
Nov 02 19:27:35.086999 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.086911    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343-system-cni-dir\") pod \"multus-additional-cni-plugins-4ddbs\" (UID: \"4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343\") " pod="openshift-multus/multus-additional-cni-plugins-4ddbs"
Nov 02 19:27:35.086999 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.086949    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343-system-cni-dir\") pod \"multus-additional-cni-plugins-4ddbs\" (UID: \"4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343\") " pod="openshift-multus/multus-additional-cni-plugins-4ddbs"
Nov 02 19:27:35.086999 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.086958    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b6cb1c47-e00c-484d-bf34-445b9380553c-iptables-alerter-script\") pod \"iptables-alerter-cmk66\" (UID: \"b6cb1c47-e00c-484d-bf34-445b9380553c\") " pod="openshift-network-operator/iptables-alerter-cmk66"
Nov 02 19:27:35.086999 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.086989    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a8e20267-1b77-4eb3-b940-a29d79b0ec19-run\") pod \"tuned-nvdtg\" (UID: \"a8e20267-1b77-4eb3-b940-a29d79b0ec19\") " pod="openshift-cluster-node-tuning-operator/tuned-nvdtg"
Nov 02 19:27:35.087502 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.087013    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a8e20267-1b77-4eb3-b940-a29d79b0ec19-tmp\") pod \"tuned-nvdtg\" (UID: \"a8e20267-1b77-4eb3-b940-a29d79b0ec19\") " pod="openshift-cluster-node-tuning-operator/tuned-nvdtg"
Nov 02 19:27:35.087502 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.087062    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a8e20267-1b77-4eb3-b940-a29d79b0ec19-etc-sysconfig\") pod \"tuned-nvdtg\" (UID: \"a8e20267-1b77-4eb3-b940-a29d79b0ec19\") " pod="openshift-cluster-node-tuning-operator/tuned-nvdtg"
Nov 02 19:27:35.087502 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.087080    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a8e20267-1b77-4eb3-b940-a29d79b0ec19-run\") pod \"tuned-nvdtg\" (UID: \"a8e20267-1b77-4eb3-b940-a29d79b0ec19\") " pod="openshift-cluster-node-tuning-operator/tuned-nvdtg"
Nov 02 19:27:35.087502 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.087087    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a8e20267-1b77-4eb3-b940-a29d79b0ec19-etc-sysctl-conf\") pod \"tuned-nvdtg\" (UID: \"a8e20267-1b77-4eb3-b940-a29d79b0ec19\") " pod="openshift-cluster-node-tuning-operator/tuned-nvdtg"
Nov 02 19:27:35.087502 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.087116    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd804657-39fd-4ae9-899d-cd1f1b62717e-tmp-dir\") pod \"node-resolver-r9fcz\" (UID: \"dd804657-39fd-4ae9-899d-cd1f1b62717e\") " pod="openshift-dns/node-resolver-r9fcz"
Nov 02 19:27:35.087502 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.087139    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a8e20267-1b77-4eb3-b940-a29d79b0ec19-etc-sysconfig\") pod \"tuned-nvdtg\" (UID: \"a8e20267-1b77-4eb3-b940-a29d79b0ec19\") " pod="openshift-cluster-node-tuning-operator/tuned-nvdtg"
Nov 02 19:27:35.087502 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.087142    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a8e20267-1b77-4eb3-b940-a29d79b0ec19-etc-modprobe-d\") pod \"tuned-nvdtg\" (UID: \"a8e20267-1b77-4eb3-b940-a29d79b0ec19\") " pod="openshift-cluster-node-tuning-operator/tuned-nvdtg"
Nov 02 19:27:35.087502 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.087199    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a8e20267-1b77-4eb3-b940-a29d79b0ec19-sys\") pod \"tuned-nvdtg\" (UID: \"a8e20267-1b77-4eb3-b940-a29d79b0ec19\") " pod="openshift-cluster-node-tuning-operator/tuned-nvdtg"
Nov 02 19:27:35.087502 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.087223    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343-cni-binary-copy\") pod \"multus-additional-cni-plugins-4ddbs\" (UID: \"4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343\") " pod="openshift-multus/multus-additional-cni-plugins-4ddbs"
Nov 02 19:27:35.087502 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.087247    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4ddbs\" (UID: \"4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343\") " pod="openshift-multus/multus-additional-cni-plugins-4ddbs"
Nov 02 19:27:35.087502 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.087256    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a8e20267-1b77-4eb3-b940-a29d79b0ec19-etc-modprobe-d\") pod \"tuned-nvdtg\" (UID: \"a8e20267-1b77-4eb3-b940-a29d79b0ec19\") " pod="openshift-cluster-node-tuning-operator/tuned-nvdtg"
Nov 02 19:27:35.087502 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.087288    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0b2d7938-0c97-4f07-a7d2-b5c35d2bc8f9-dbus\") pod \"global-pull-secret-syncer-4b2pr\" (UID: \"0b2d7938-0c97-4f07-a7d2-b5c35d2bc8f9\") " pod="kube-system/global-pull-secret-syncer-4b2pr"
Nov 02 19:27:35.087502 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.087322    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a8e20267-1b77-4eb3-b940-a29d79b0ec19-etc-sysctl-conf\") pod \"tuned-nvdtg\" (UID: \"a8e20267-1b77-4eb3-b940-a29d79b0ec19\") " pod="openshift-cluster-node-tuning-operator/tuned-nvdtg"
Nov 02 19:27:35.087502 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.087326    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a8e20267-1b77-4eb3-b940-a29d79b0ec19-sys\") pod \"tuned-nvdtg\" (UID: \"a8e20267-1b77-4eb3-b940-a29d79b0ec19\") " pod="openshift-cluster-node-tuning-operator/tuned-nvdtg"
Nov 02 19:27:35.087502 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.087350    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcpxd\" (UniqueName: \"kubernetes.io/projected/dd804657-39fd-4ae9-899d-cd1f1b62717e-kube-api-access-bcpxd\") pod \"node-resolver-r9fcz\" (UID: \"dd804657-39fd-4ae9-899d-cd1f1b62717e\") " pod="openshift-dns/node-resolver-r9fcz"
Nov 02 19:27:35.087502 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.087393    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b6cb1c47-e00c-484d-bf34-445b9380553c-host-slash\") pod \"iptables-alerter-cmk66\" (UID: \"b6cb1c47-e00c-484d-bf34-445b9380553c\") " pod="openshift-network-operator/iptables-alerter-cmk66"
Nov 02 19:27:35.087502 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.087421    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b2d7938-0c97-4f07-a7d2-b5c35d2bc8f9-original-pull-secret\") pod \"global-pull-secret-syncer-4b2pr\" (UID: \"0b2d7938-0c97-4f07-a7d2-b5c35d2bc8f9\") " pod="kube-system/global-pull-secret-syncer-4b2pr"
Nov 02 19:27:35.087502 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.087445    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a8e20267-1b77-4eb3-b940-a29d79b0ec19-host\") pod \"tuned-nvdtg\" (UID: \"a8e20267-1b77-4eb3-b940-a29d79b0ec19\") " pod="openshift-cluster-node-tuning-operator/tuned-nvdtg"
Nov 02 19:27:35.088252 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.087466    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343-cnibin\") pod \"multus-additional-cni-plugins-4ddbs\" (UID: \"4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343\") " pod="openshift-multus/multus-additional-cni-plugins-4ddbs"
Nov 02 19:27:35.088252 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.087490    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3fabcb64-faf2-49f9-813a-ddafde0fd159-agent-certs\") pod \"konnectivity-agent-jgfjj\" (UID: \"3fabcb64-faf2-49f9-813a-ddafde0fd159\") " pod="kube-system/konnectivity-agent-jgfjj"
Nov 02 19:27:35.088252 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.087505    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a8e20267-1b77-4eb3-b940-a29d79b0ec19-lib-modules\") pod \"tuned-nvdtg\" (UID: \"a8e20267-1b77-4eb3-b940-a29d79b0ec19\") " pod="openshift-cluster-node-tuning-operator/tuned-nvdtg"
Nov 02 19:27:35.088252 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.087524    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a8e20267-1b77-4eb3-b940-a29d79b0ec19-var-lib-kubelet\") pod \"tuned-nvdtg\" (UID: \"a8e20267-1b77-4eb3-b940-a29d79b0ec19\") " pod="openshift-cluster-node-tuning-operator/tuned-nvdtg"
Nov 02 19:27:35.088252 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.087548    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mptpf\" (UniqueName: \"kubernetes.io/projected/4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343-kube-api-access-mptpf\") pod \"multus-additional-cni-plugins-4ddbs\" (UID: \"4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343\") " pod="openshift-multus/multus-additional-cni-plugins-4ddbs"
Nov 02 19:27:35.088252 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.087568    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0b2d7938-0c97-4f07-a7d2-b5c35d2bc8f9-dbus\") pod \"global-pull-secret-syncer-4b2pr\" (UID: \"0b2d7938-0c97-4f07-a7d2-b5c35d2bc8f9\") " pod="kube-system/global-pull-secret-syncer-4b2pr"
Nov 02 19:27:35.088252 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.087610    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343-cnibin\") pod \"multus-additional-cni-plugins-4ddbs\" (UID: \"4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343\") " pod="openshift-multus/multus-additional-cni-plugins-4ddbs"
Nov 02 19:27:35.088252 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.087629    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a8e20267-1b77-4eb3-b940-a29d79b0ec19-var-lib-kubelet\") pod \"tuned-nvdtg\" (UID: \"a8e20267-1b77-4eb3-b940-a29d79b0ec19\") " pod="openshift-cluster-node-tuning-operator/tuned-nvdtg"
Nov 02 19:27:35.088252 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:35.087714    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 19:27:35.088252 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:35.087764    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b2d7938-0c97-4f07-a7d2-b5c35d2bc8f9-original-pull-secret podName:0b2d7938-0c97-4f07-a7d2-b5c35d2bc8f9 nodeName:}" failed. No retries permitted until 2026-11-02 19:27:35.587749866 +0000 UTC m=+2.616024182 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0b2d7938-0c97-4f07-a7d2-b5c35d2bc8f9-original-pull-secret") pod "global-pull-secret-syncer-4b2pr" (UID: "0b2d7938-0c97-4f07-a7d2-b5c35d2bc8f9") : object "kube-system"/"original-pull-secret" not registered
Nov 02 19:27:35.088252 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.087798    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4sz9\" (UniqueName: \"kubernetes.io/projected/b6cb1c47-e00c-484d-bf34-445b9380553c-kube-api-access-l4sz9\") pod \"iptables-alerter-cmk66\" (UID: \"b6cb1c47-e00c-484d-bf34-445b9380553c\") " pod="openshift-network-operator/iptables-alerter-cmk66"
Nov 02 19:27:35.088252 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.087820    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a8e20267-1b77-4eb3-b940-a29d79b0ec19-host\") pod \"tuned-nvdtg\" (UID: \"a8e20267-1b77-4eb3-b940-a29d79b0ec19\") " pod="openshift-cluster-node-tuning-operator/tuned-nvdtg"
Nov 02 19:27:35.088252 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.087806    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b6cb1c47-e00c-484d-bf34-445b9380553c-host-slash\") pod \"iptables-alerter-cmk66\" (UID: \"b6cb1c47-e00c-484d-bf34-445b9380553c\") " pod="openshift-network-operator/iptables-alerter-cmk66"
Nov 02 19:27:35.088252 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.087843    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a8e20267-1b77-4eb3-b940-a29d79b0ec19-etc-sysctl-d\") pod \"tuned-nvdtg\" (UID: \"a8e20267-1b77-4eb3-b940-a29d79b0ec19\") " pod="openshift-cluster-node-tuning-operator/tuned-nvdtg"
Nov 02 19:27:35.088252 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.087870    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4ddbs\" (UID: \"4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343\") " pod="openshift-multus/multus-additional-cni-plugins-4ddbs"
Nov 02 19:27:35.088252 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.087904    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3fabcb64-faf2-49f9-813a-ddafde0fd159-konnectivity-ca\") pod \"konnectivity-agent-jgfjj\" (UID: \"3fabcb64-faf2-49f9-813a-ddafde0fd159\") " pod="kube-system/konnectivity-agent-jgfjj"
Nov 02 19:27:35.088252 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.087937    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/dd804657-39fd-4ae9-899d-cd1f1b62717e-hosts-file\") pod \"node-resolver-r9fcz\" (UID: \"dd804657-39fd-4ae9-899d-cd1f1b62717e\") " pod="openshift-dns/node-resolver-r9fcz"
Nov 02 19:27:35.089018 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.087942    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a8e20267-1b77-4eb3-b940-a29d79b0ec19-lib-modules\") pod \"tuned-nvdtg\" (UID: \"a8e20267-1b77-4eb3-b940-a29d79b0ec19\") " pod="openshift-cluster-node-tuning-operator/tuned-nvdtg"
Nov 02 19:27:35.089018 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.087988    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a8e20267-1b77-4eb3-b940-a29d79b0ec19-etc-sysctl-d\") pod \"tuned-nvdtg\" (UID: \"a8e20267-1b77-4eb3-b940-a29d79b0ec19\") " pod="openshift-cluster-node-tuning-operator/tuned-nvdtg"
Nov 02 19:27:35.089018 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.088043    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/dd804657-39fd-4ae9-899d-cd1f1b62717e-hosts-file\") pod \"node-resolver-r9fcz\" (UID: \"dd804657-39fd-4ae9-899d-cd1f1b62717e\") " pod="openshift-dns/node-resolver-r9fcz"
Nov 02 19:27:35.089018 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.088737    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4ddbs\" (UID: \"4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343\") " pod="openshift-multus/multus-additional-cni-plugins-4ddbs"
Nov 02 19:27:35.089018 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.088759    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd804657-39fd-4ae9-899d-cd1f1b62717e-tmp-dir\") pod \"node-resolver-r9fcz\" (UID: \"dd804657-39fd-4ae9-899d-cd1f1b62717e\") " pod="openshift-dns/node-resolver-r9fcz"
Nov 02 19:27:35.089018 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.088890    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4ddbs\" (UID: \"4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343\") " pod="openshift-multus/multus-additional-cni-plugins-4ddbs"
Nov 02 19:27:35.089018 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.088942    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b6cb1c47-e00c-484d-bf34-445b9380553c-iptables-alerter-script\") pod \"iptables-alerter-cmk66\" (UID: \"b6cb1c47-e00c-484d-bf34-445b9380553c\") " pod="openshift-network-operator/iptables-alerter-cmk66"
Nov 02 19:27:35.089334 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.089113    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343-cni-binary-copy\") pod \"multus-additional-cni-plugins-4ddbs\" (UID: \"4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343\") " pod="openshift-multus/multus-additional-cni-plugins-4ddbs"
Nov 02 19:27:35.089876 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.089841    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a8e20267-1b77-4eb3-b940-a29d79b0ec19-etc-tuned\") pod \"tuned-nvdtg\" (UID: \"a8e20267-1b77-4eb3-b940-a29d79b0ec19\") " pod="openshift-cluster-node-tuning-operator/tuned-nvdtg"
Nov 02 19:27:35.090012 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.089991    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3fabcb64-faf2-49f9-813a-ddafde0fd159-konnectivity-ca\") pod \"konnectivity-agent-jgfjj\" (UID: \"3fabcb64-faf2-49f9-813a-ddafde0fd159\") " pod="kube-system/konnectivity-agent-jgfjj"
Nov 02 19:27:35.090233 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.090212    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4ddbs\" (UID: \"4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343\") " pod="openshift-multus/multus-additional-cni-plugins-4ddbs"
Nov 02 19:27:35.090978 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.090962    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a8e20267-1b77-4eb3-b940-a29d79b0ec19-tmp\") pod \"tuned-nvdtg\" (UID: \"a8e20267-1b77-4eb3-b940-a29d79b0ec19\") " pod="openshift-cluster-node-tuning-operator/tuned-nvdtg"
Nov 02 19:27:35.091522 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.091506    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3fabcb64-faf2-49f9-813a-ddafde0fd159-agent-certs\") pod \"konnectivity-agent-jgfjj\" (UID: \"3fabcb64-faf2-49f9-813a-ddafde0fd159\") " pod="kube-system/konnectivity-agent-jgfjj"
Nov 02 19:27:35.151997 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.151963    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcpxd\" (UniqueName: \"kubernetes.io/projected/dd804657-39fd-4ae9-899d-cd1f1b62717e-kube-api-access-bcpxd\") pod \"node-resolver-r9fcz\" (UID: \"dd804657-39fd-4ae9-899d-cd1f1b62717e\") " pod="openshift-dns/node-resolver-r9fcz"
Nov 02 19:27:35.152157 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.152063    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jxxh\" (UniqueName: \"kubernetes.io/projected/a8e20267-1b77-4eb3-b940-a29d79b0ec19-kube-api-access-7jxxh\") pod \"tuned-nvdtg\" (UID: \"a8e20267-1b77-4eb3-b940-a29d79b0ec19\") " pod="openshift-cluster-node-tuning-operator/tuned-nvdtg"
Nov 02 19:27:35.152581 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.152560    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mptpf\" (UniqueName: \"kubernetes.io/projected/4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343-kube-api-access-mptpf\") pod \"multus-additional-cni-plugins-4ddbs\" (UID: \"4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343\") " pod="openshift-multus/multus-additional-cni-plugins-4ddbs"
Nov 02 19:27:35.152751 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.152736    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4sz9\" (UniqueName: \"kubernetes.io/projected/b6cb1c47-e00c-484d-bf34-445b9380553c-kube-api-access-l4sz9\") pod \"iptables-alerter-cmk66\" (UID: \"b6cb1c47-e00c-484d-bf34-445b9380553c\") " pod="openshift-network-operator/iptables-alerter-cmk66"
Nov 02 19:27:35.211342 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.211310    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:27:35.221931 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.221907    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5wvz"
Nov 02 19:27:35.239307 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.239274    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hrhpj"
Nov 02 19:27:35.245815 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.245204    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qhswb"
Nov 02 19:27:35.264197 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.264172    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cmk66"
Nov 02 19:27:35.281202 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.281175    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-r9fcz"
Nov 02 19:27:35.289212 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.289187    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jgfjj"
Nov 02 19:27:35.312238 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.312173    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nvdtg"
Nov 02 19:27:35.321944 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.321923    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4ddbs"
Nov 02 19:27:35.331350 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:35.331319    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8e20267_1b77_4eb3_b940_a29d79b0ec19.slice/crio-fb5ca3da175e0696fd67e5eb8ef89c5fda5b3c1bd9de15eed4d4daa2348aa87c WatchSource:0}: Error finding container fb5ca3da175e0696fd67e5eb8ef89c5fda5b3c1bd9de15eed4d4daa2348aa87c: Status 404 returned error can't find the container with id fb5ca3da175e0696fd67e5eb8ef89c5fda5b3c1bd9de15eed4d4daa2348aa87c
Nov 02 19:27:35.331812 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:35.331790    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6e1f694aea2cef1694653a075ee3dd5.slice/crio-ba7396663395b30e8caee684e3b103368f2aa74a904ffab9058964d5bf446d10 WatchSource:0}: Error finding container ba7396663395b30e8caee684e3b103368f2aa74a904ffab9058964d5bf446d10: Status 404 returned error can't find the container with id ba7396663395b30e8caee684e3b103368f2aa74a904ffab9058964d5bf446d10
Nov 02 19:27:35.333484 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:35.333461    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c8ad35a_7977_4e7a_8eb0_a4c8d1ce2343.slice/crio-667d2758ccf9eaad27f48fb50e5834c02305aa559406fa46b65f01390ffe20bd WatchSource:0}: Error finding container 667d2758ccf9eaad27f48fb50e5834c02305aa559406fa46b65f01390ffe20bd: Status 404 returned error can't find the container with id 667d2758ccf9eaad27f48fb50e5834c02305aa559406fa46b65f01390ffe20bd
Nov 02 19:27:35.340166 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.340148    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 19:27:35.374738 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:35.374707    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb70d1060da282fd74b3ebbf01b381326.slice/crio-7fe4319957955c98ed7c6909bd338655b884e2d6cb6f89741f4252df918c536f WatchSource:0}: Error finding container 7fe4319957955c98ed7c6909bd338655b884e2d6cb6f89741f4252df918c536f: Status 404 returned error can't find the container with id 7fe4319957955c98ed7c6909bd338655b884e2d6cb6f89741f4252df918c536f
Nov 02 19:27:35.490901 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.490855    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17fe5072-f4c5-4075-8614-4229ed05d4b2-metrics-certs\") pod \"network-metrics-daemon-f6r6d\" (UID: \"17fe5072-f4c5-4075-8614-4229ed05d4b2\") " pod="openshift-multus/network-metrics-daemon-f6r6d"
Nov 02 19:27:35.491095 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:35.491033    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 19:27:35.491145 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:35.491102    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17fe5072-f4c5-4075-8614-4229ed05d4b2-metrics-certs podName:17fe5072-f4c5-4075-8614-4229ed05d4b2 nodeName:}" failed. No retries permitted until 2026-11-02 19:27:36.491087529 +0000 UTC m=+3.519361831 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/17fe5072-f4c5-4075-8614-4229ed05d4b2-metrics-certs") pod "network-metrics-daemon-f6r6d" (UID: "17fe5072-f4c5-4075-8614-4229ed05d4b2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 19:27:35.594903 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.592516    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccf8l\" (UniqueName: \"kubernetes.io/projected/31a69a0e-cc9c-47ad-b577-36b50d61d213-kube-api-access-ccf8l\") pod \"network-check-target-rhdd6\" (UID: \"31a69a0e-cc9c-47ad-b577-36b50d61d213\") " pod="openshift-network-diagnostics/network-check-target-rhdd6"
Nov 02 19:27:35.594903 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.592682    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b2d7938-0c97-4f07-a7d2-b5c35d2bc8f9-original-pull-secret\") pod \"global-pull-secret-syncer-4b2pr\" (UID: \"0b2d7938-0c97-4f07-a7d2-b5c35d2bc8f9\") " pod="kube-system/global-pull-secret-syncer-4b2pr"
Nov 02 19:27:35.594903 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:35.592880    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 19:27:35.594903 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:35.592955    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b2d7938-0c97-4f07-a7d2-b5c35d2bc8f9-original-pull-secret podName:0b2d7938-0c97-4f07-a7d2-b5c35d2bc8f9 nodeName:}" failed. No retries permitted until 2026-11-02 19:27:36.592936161 +0000 UTC m=+3.621210476 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0b2d7938-0c97-4f07-a7d2-b5c35d2bc8f9-original-pull-secret") pod "global-pull-secret-syncer-4b2pr" (UID: "0b2d7938-0c97-4f07-a7d2-b5c35d2bc8f9") : object "kube-system"/"original-pull-secret" not registered
Nov 02 19:27:35.594903 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:35.593041    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 19:27:35.594903 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:35.593057    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 19:27:35.594903 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:35.593070    2579 projected.go:196] Error preparing data for projected volume kube-api-access-ccf8l for pod openshift-network-diagnostics/network-check-target-rhdd6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 19:27:35.594903 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:35.593142    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/31a69a0e-cc9c-47ad-b577-36b50d61d213-kube-api-access-ccf8l podName:31a69a0e-cc9c-47ad-b577-36b50d61d213 nodeName:}" failed. No retries permitted until 2026-11-02 19:27:36.593129446 +0000 UTC m=+3.621403763 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-ccf8l" (UniqueName: "kubernetes.io/projected/31a69a0e-cc9c-47ad-b577-36b50d61d213-kube-api-access-ccf8l") pod "network-check-target-rhdd6" (UID: "31a69a0e-cc9c-47ad-b577-36b50d61d213") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 19:27:35.734571 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.734537    2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 19:27:35.789910 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.789876    2579 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 19:27:35.852579 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.852473    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-9.us-west-2.compute.internal" event={"ID":"d6e1f694aea2cef1694653a075ee3dd5","Type":"ContainerStarted","Data":"ba7396663395b30e8caee684e3b103368f2aa74a904ffab9058964d5bf446d10"}
Nov 02 19:27:35.853505 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.853486    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-9.us-west-2.compute.internal" event={"ID":"b70d1060da282fd74b3ebbf01b381326","Type":"ContainerStarted","Data":"7fe4319957955c98ed7c6909bd338655b884e2d6cb6f89741f4252df918c536f"}
Nov 02 19:27:35.854602 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.854579    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4ddbs" event={"ID":"4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343","Type":"ContainerStarted","Data":"667d2758ccf9eaad27f48fb50e5834c02305aa559406fa46b65f01390ffe20bd"}
Nov 02 19:27:35.855590 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:35.855570    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nvdtg" event={"ID":"a8e20267-1b77-4eb3-b940-a29d79b0ec19","Type":"ContainerStarted","Data":"fb5ca3da175e0696fd67e5eb8ef89c5fda5b3c1bd9de15eed4d4daa2348aa87c"}
Nov 02 19:27:36.035624 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:36.035569    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 19:22:35 +0000 UTC" deadline="2028-05-17 18:54:45.261933047 +0000 UTC"
Nov 02 19:27:36.035624 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:36.035619    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13487h27m9.226317333s"
Nov 02 19:27:36.046312 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:36.046287    2579 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 19:27:36.064473 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:36.064443    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b7f05bf_12cd_4aad_b4ee_8b517e901ca6.slice/crio-abfbe7f65e87c82cfa535f8e1f3445cb20d888ecbb7f8abb697df2085a6a3c73 WatchSource:0}: Error finding container abfbe7f65e87c82cfa535f8e1f3445cb20d888ecbb7f8abb697df2085a6a3c73: Status 404 returned error can't find the container with id abfbe7f65e87c82cfa535f8e1f3445cb20d888ecbb7f8abb697df2085a6a3c73
Nov 02 19:27:36.156866 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:36.156838    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8756b800_1b4e_41da_934a_25c6b8ee738c.slice/crio-655d515e9caa08a107af1d948364e4afa72c74189600ae462600c0de1ee66879 WatchSource:0}: Error finding container 655d515e9caa08a107af1d948364e4afa72c74189600ae462600c0de1ee66879: Status 404 returned error can't find the container with id 655d515e9caa08a107af1d948364e4afa72c74189600ae462600c0de1ee66879
Nov 02 19:27:36.157306 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:36.157283    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd804657_39fd_4ae9_899d_cd1f1b62717e.slice/crio-4ce5d46f7311979bfa15d5f696ef882348ffbf80aa8c5acf5c07b8316af4865d WatchSource:0}: Error finding container 4ce5d46f7311979bfa15d5f696ef882348ffbf80aa8c5acf5c07b8316af4865d: Status 404 returned error can't find the container with id 4ce5d46f7311979bfa15d5f696ef882348ffbf80aa8c5acf5c07b8316af4865d
Nov 02 19:27:36.200309 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:36.200277    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6cb1c47_e00c_484d_bf34_445b9380553c.slice/crio-1e383e5f91ee92619c7178649dc6b6d776a9eb3b2e97103bccfdca415dab6887 WatchSource:0}: Error finding container 1e383e5f91ee92619c7178649dc6b6d776a9eb3b2e97103bccfdca415dab6887: Status 404 returned error can't find the container with id 1e383e5f91ee92619c7178649dc6b6d776a9eb3b2e97103bccfdca415dab6887
Nov 02 19:27:36.213998 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:36.213962    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74106611_359e_4353_b113_70521144bed1.slice/crio-2fddafd70439b8e26feb1fff6e97dcb631902ca9cb63bf44a13df77960ad2022 WatchSource:0}: Error finding container 2fddafd70439b8e26feb1fff6e97dcb631902ca9cb63bf44a13df77960ad2022: Status 404 returned error can't find the container with id 2fddafd70439b8e26feb1fff6e97dcb631902ca9cb63bf44a13df77960ad2022
Nov 02 19:27:36.220303 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:36.220278    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fabcb64_faf2_49f9_813a_ddafde0fd159.slice/crio-7655f51aa5471290e3a2ff847995a116fe03885514a5c839f545a75f27a82c0e WatchSource:0}: Error finding container 7655f51aa5471290e3a2ff847995a116fe03885514a5c839f545a75f27a82c0e: Status 404 returned error can't find the container with id 7655f51aa5471290e3a2ff847995a116fe03885514a5c839f545a75f27a82c0e
Nov 02 19:27:36.228610 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:27:36.228586    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6ad2c98_52b5_496e_9824_614a2e509a8a.slice/crio-29b253e3dff4d86876a235f3c9b0a36d3815e1fb4be3fbdc60497516afaf9b34 WatchSource:0}: Error finding container 29b253e3dff4d86876a235f3c9b0a36d3815e1fb4be3fbdc60497516afaf9b34: Status 404 returned error can't find the container with id 29b253e3dff4d86876a235f3c9b0a36d3815e1fb4be3fbdc60497516afaf9b34
Nov 02 19:27:36.501099 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:36.500991    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17fe5072-f4c5-4075-8614-4229ed05d4b2-metrics-certs\") pod \"network-metrics-daemon-f6r6d\" (UID: \"17fe5072-f4c5-4075-8614-4229ed05d4b2\") " pod="openshift-multus/network-metrics-daemon-f6r6d"
Nov 02 19:27:36.501278 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:36.501159    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 19:27:36.501278 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:36.501225    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17fe5072-f4c5-4075-8614-4229ed05d4b2-metrics-certs podName:17fe5072-f4c5-4075-8614-4229ed05d4b2 nodeName:}" failed. No retries permitted until 2026-11-02 19:27:38.501204155 +0000 UTC m=+5.529478469 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/17fe5072-f4c5-4075-8614-4229ed05d4b2-metrics-certs") pod "network-metrics-daemon-f6r6d" (UID: "17fe5072-f4c5-4075-8614-4229ed05d4b2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 19:27:36.602193 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:36.602161    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccf8l\" (UniqueName: \"kubernetes.io/projected/31a69a0e-cc9c-47ad-b577-36b50d61d213-kube-api-access-ccf8l\") pod \"network-check-target-rhdd6\" (UID: \"31a69a0e-cc9c-47ad-b577-36b50d61d213\") " pod="openshift-network-diagnostics/network-check-target-rhdd6"
Nov 02 19:27:36.602850 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:36.602216    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b2d7938-0c97-4f07-a7d2-b5c35d2bc8f9-original-pull-secret\") pod \"global-pull-secret-syncer-4b2pr\" (UID: \"0b2d7938-0c97-4f07-a7d2-b5c35d2bc8f9\") " pod="kube-system/global-pull-secret-syncer-4b2pr"
Nov 02 19:27:36.602850 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:36.602328    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 19:27:36.602850 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:36.602377    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b2d7938-0c97-4f07-a7d2-b5c35d2bc8f9-original-pull-secret podName:0b2d7938-0c97-4f07-a7d2-b5c35d2bc8f9 nodeName:}" failed. No retries permitted until 2026-11-02 19:27:38.602364518 +0000 UTC m=+5.630638831 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0b2d7938-0c97-4f07-a7d2-b5c35d2bc8f9-original-pull-secret") pod "global-pull-secret-syncer-4b2pr" (UID: "0b2d7938-0c97-4f07-a7d2-b5c35d2bc8f9") : object "kube-system"/"original-pull-secret" not registered
Nov 02 19:27:36.602850 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:36.602437    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 19:27:36.602850 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:36.602449    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 19:27:36.602850 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:36.602458    2579 projected.go:196] Error preparing data for projected volume kube-api-access-ccf8l for pod openshift-network-diagnostics/network-check-target-rhdd6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 19:27:36.602850 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:36.602481    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/31a69a0e-cc9c-47ad-b577-36b50d61d213-kube-api-access-ccf8l podName:31a69a0e-cc9c-47ad-b577-36b50d61d213 nodeName:}" failed. No retries permitted until 2026-11-02 19:27:38.602474183 +0000 UTC m=+5.630748495 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-ccf8l" (UniqueName: "kubernetes.io/projected/31a69a0e-cc9c-47ad-b577-36b50d61d213-kube-api-access-ccf8l") pod "network-check-target-rhdd6" (UID: "31a69a0e-cc9c-47ad-b577-36b50d61d213") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 19:27:36.847649 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:36.847561    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4b2pr"
Nov 02 19:27:36.847813 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:36.847732    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4b2pr" podUID="0b2d7938-0c97-4f07-a7d2-b5c35d2bc8f9"
Nov 02 19:27:36.847885 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:36.847838    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f6r6d"
Nov 02 19:27:36.847976 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:36.847953    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f6r6d" podUID="17fe5072-f4c5-4075-8614-4229ed05d4b2"
Nov 02 19:27:36.848071 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:36.848030    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rhdd6"
Nov 02 19:27:36.848141 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:36.848122    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rhdd6" podUID="31a69a0e-cc9c-47ad-b577-36b50d61d213"
Nov 02 19:27:36.859417 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:36.859349    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hrhpj" event={"ID":"b6ad2c98-52b5-496e-9824-614a2e509a8a","Type":"ContainerStarted","Data":"29b253e3dff4d86876a235f3c9b0a36d3815e1fb4be3fbdc60497516afaf9b34"}
Nov 02 19:27:36.862233 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:36.862107    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jgfjj" event={"ID":"3fabcb64-faf2-49f9-813a-ddafde0fd159","Type":"ContainerStarted","Data":"7655f51aa5471290e3a2ff847995a116fe03885514a5c839f545a75f27a82c0e"}
Nov 02 19:27:36.864220 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:36.864192    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cmk66" event={"ID":"b6cb1c47-e00c-484d-bf34-445b9380553c","Type":"ContainerStarted","Data":"1e383e5f91ee92619c7178649dc6b6d776a9eb3b2e97103bccfdca415dab6887"}
Nov 02 19:27:36.866154 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:36.866129    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-r9fcz" event={"ID":"dd804657-39fd-4ae9-899d-cd1f1b62717e","Type":"ContainerStarted","Data":"4ce5d46f7311979bfa15d5f696ef882348ffbf80aa8c5acf5c07b8316af4865d"}
Nov 02 19:27:36.868152 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:36.868126    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vst5r" event={"ID":"8756b800-1b4e-41da-934a-25c6b8ee738c","Type":"ContainerStarted","Data":"655d515e9caa08a107af1d948364e4afa72c74189600ae462600c0de1ee66879"}
Nov 02 19:27:36.870158 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:36.870136    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qhswb" event={"ID":"9b7f05bf-12cd-4aad-b4ee-8b517e901ca6","Type":"ContainerStarted","Data":"abfbe7f65e87c82cfa535f8e1f3445cb20d888ecbb7f8abb697df2085a6a3c73"}
Nov 02 19:27:36.872910 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:36.872846    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5wvz" event={"ID":"74106611-359e-4353-b113-70521144bed1","Type":"ContainerStarted","Data":"2fddafd70439b8e26feb1fff6e97dcb631902ca9cb63bf44a13df77960ad2022"}
Nov 02 19:27:37.035899 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:37.035853    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 19:22:35 +0000 UTC" deadline="2028-05-03 19:26:24.732877991 +0000 UTC"
Nov 02 19:27:37.035899 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:37.035897    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13151h58m47.696985615s"
Nov 02 19:27:38.520523 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:38.519901    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17fe5072-f4c5-4075-8614-4229ed05d4b2-metrics-certs\") pod \"network-metrics-daemon-f6r6d\" (UID: \"17fe5072-f4c5-4075-8614-4229ed05d4b2\") " pod="openshift-multus/network-metrics-daemon-f6r6d"
Nov 02 19:27:38.520523 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:38.520092    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 19:27:38.520964 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:38.520595    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17fe5072-f4c5-4075-8614-4229ed05d4b2-metrics-certs podName:17fe5072-f4c5-4075-8614-4229ed05d4b2 nodeName:}" failed. No retries permitted until 2026-11-02 19:27:42.52057224 +0000 UTC m=+9.548846559 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/17fe5072-f4c5-4075-8614-4229ed05d4b2-metrics-certs") pod "network-metrics-daemon-f6r6d" (UID: "17fe5072-f4c5-4075-8614-4229ed05d4b2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 19:27:38.621156 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:38.621118    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccf8l\" (UniqueName: \"kubernetes.io/projected/31a69a0e-cc9c-47ad-b577-36b50d61d213-kube-api-access-ccf8l\") pod \"network-check-target-rhdd6\" (UID: \"31a69a0e-cc9c-47ad-b577-36b50d61d213\") " pod="openshift-network-diagnostics/network-check-target-rhdd6"
Nov 02 19:27:38.621375 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:38.621192    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b2d7938-0c97-4f07-a7d2-b5c35d2bc8f9-original-pull-secret\") pod \"global-pull-secret-syncer-4b2pr\" (UID: \"0b2d7938-0c97-4f07-a7d2-b5c35d2bc8f9\") " pod="kube-system/global-pull-secret-syncer-4b2pr"
Nov 02 19:27:38.621375 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:38.621334    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 19:27:38.621499 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:38.621400    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b2d7938-0c97-4f07-a7d2-b5c35d2bc8f9-original-pull-secret podName:0b2d7938-0c97-4f07-a7d2-b5c35d2bc8f9 nodeName:}" failed. No retries permitted until 2026-11-02 19:27:42.621380648 +0000 UTC m=+9.649654957 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0b2d7938-0c97-4f07-a7d2-b5c35d2bc8f9-original-pull-secret") pod "global-pull-secret-syncer-4b2pr" (UID: "0b2d7938-0c97-4f07-a7d2-b5c35d2bc8f9") : object "kube-system"/"original-pull-secret" not registered
Nov 02 19:27:38.621499 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:38.621494    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 19:27:38.621622 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:38.621537    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 19:27:38.621622 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:38.621553    2579 projected.go:196] Error preparing data for projected volume kube-api-access-ccf8l for pod openshift-network-diagnostics/network-check-target-rhdd6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 19:27:38.621724 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:38.621622    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/31a69a0e-cc9c-47ad-b577-36b50d61d213-kube-api-access-ccf8l podName:31a69a0e-cc9c-47ad-b577-36b50d61d213 nodeName:}" failed. No retries permitted until 2026-11-02 19:27:42.621608602 +0000 UTC m=+9.649882906 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-ccf8l" (UniqueName: "kubernetes.io/projected/31a69a0e-cc9c-47ad-b577-36b50d61d213-kube-api-access-ccf8l") pod "network-check-target-rhdd6" (UID: "31a69a0e-cc9c-47ad-b577-36b50d61d213") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 19:27:38.847666 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:38.847550    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4b2pr"
Nov 02 19:27:38.847854 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:38.847691    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4b2pr" podUID="0b2d7938-0c97-4f07-a7d2-b5c35d2bc8f9"
Nov 02 19:27:38.847854 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:38.847775    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f6r6d"
Nov 02 19:27:38.847974 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:38.847861    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f6r6d" podUID="17fe5072-f4c5-4075-8614-4229ed05d4b2"
Nov 02 19:27:38.847974 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:38.847921    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rhdd6"
Nov 02 19:27:38.848103 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:38.847994    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rhdd6" podUID="31a69a0e-cc9c-47ad-b577-36b50d61d213"
Nov 02 19:27:38.901754 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:38.901710    2579 generic.go:356] "Generic (PLEG): container finished" podID="4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343" containerID="ba119f338d444c26564d72c637569e929c193f3fbd586e6c6820ac8c6b9c2cbe" exitCode=0
Nov 02 19:27:38.901957 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:38.901779    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4ddbs" event={"ID":"4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343","Type":"ContainerDied","Data":"ba119f338d444c26564d72c637569e929c193f3fbd586e6c6820ac8c6b9c2cbe"}
Nov 02 19:27:39.912732 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:39.912155    2579 generic.go:356] "Generic (PLEG): container finished" podID="b70d1060da282fd74b3ebbf01b381326" containerID="99b56a4cd13b0048eb354a2432abd2878b8ad0ec5cc126fcd16fbf66fac0abd8" exitCode=0
Nov 02 19:27:39.912732 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:39.912207    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-9.us-west-2.compute.internal" event={"ID":"b70d1060da282fd74b3ebbf01b381326","Type":"ContainerDied","Data":"99b56a4cd13b0048eb354a2432abd2878b8ad0ec5cc126fcd16fbf66fac0abd8"}
Nov 02 19:27:40.847461 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:40.847422    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f6r6d"
Nov 02 19:27:40.847723 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:40.847583    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f6r6d" podUID="17fe5072-f4c5-4075-8614-4229ed05d4b2"
Nov 02 19:27:40.848047 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:40.848026    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rhdd6"
Nov 02 19:27:40.848168 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:40.848149    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rhdd6" podUID="31a69a0e-cc9c-47ad-b577-36b50d61d213"
Nov 02 19:27:40.848312 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:40.848217    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4b2pr"
Nov 02 19:27:40.848312 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:40.848289    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4b2pr" podUID="0b2d7938-0c97-4f07-a7d2-b5c35d2bc8f9"
Nov 02 19:27:40.923157 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:40.923122    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-9.us-west-2.compute.internal" event={"ID":"b70d1060da282fd74b3ebbf01b381326","Type":"ContainerStarted","Data":"562ff5488cc749de466161bb83aad8ad4a17a89c784bb1ed1eda42d7d6e55a34"}
Nov 02 19:27:40.929127 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:40.929027    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-9.us-west-2.compute.internal" event={"ID":"d6e1f694aea2cef1694653a075ee3dd5","Type":"ContainerStarted","Data":"530bfca33e295dd1d1a0b0e78873d32784ad7abf446824f3bd8f99ccf04ff7a3"}
Nov 02 19:27:41.179102 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:41.178971    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-9.us-west-2.compute.internal" podStartSLOduration=7.178951497 podStartE2EDuration="7.178951497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 19:27:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:27:41.052275777 +0000 UTC m=+8.080550101" watchObservedRunningTime="2026-11-02 19:27:41.178951497 +0000 UTC m=+8.207225821"
Nov 02 19:27:41.934539 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:41.934085    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jgfjj" event={"ID":"3fabcb64-faf2-49f9-813a-ddafde0fd159","Type":"ContainerStarted","Data":"4dbf245920b1fed0c3b198d33ace350407ce0eebab0c11afed285e5069c41d35"}
Nov 02 19:27:42.055044 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:42.052028    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-9.us-west-2.compute.internal" podStartSLOduration=8.051993414 podStartE2EDuration="8.051993414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 19:27:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:27:41.180258171 +0000 UTC m=+8.208532496" watchObservedRunningTime="2026-11-02 19:27:42.051993414 +0000 UTC m=+9.080267734"
Nov 02 19:27:42.561577 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:42.561542    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17fe5072-f4c5-4075-8614-4229ed05d4b2-metrics-certs\") pod \"network-metrics-daemon-f6r6d\" (UID: \"17fe5072-f4c5-4075-8614-4229ed05d4b2\") " pod="openshift-multus/network-metrics-daemon-f6r6d"
Nov 02 19:27:42.561722 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:42.561703    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 19:27:42.561812 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:42.561768    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17fe5072-f4c5-4075-8614-4229ed05d4b2-metrics-certs podName:17fe5072-f4c5-4075-8614-4229ed05d4b2 nodeName:}" failed. No retries permitted until 2026-11-02 19:27:50.561750411 +0000 UTC m=+17.590024723 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/17fe5072-f4c5-4075-8614-4229ed05d4b2-metrics-certs") pod "network-metrics-daemon-f6r6d" (UID: "17fe5072-f4c5-4075-8614-4229ed05d4b2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 19:27:42.662201 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:42.662125    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccf8l\" (UniqueName: \"kubernetes.io/projected/31a69a0e-cc9c-47ad-b577-36b50d61d213-kube-api-access-ccf8l\") pod \"network-check-target-rhdd6\" (UID: \"31a69a0e-cc9c-47ad-b577-36b50d61d213\") " pod="openshift-network-diagnostics/network-check-target-rhdd6"
Nov 02 19:27:42.662201 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:42.662182    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b2d7938-0c97-4f07-a7d2-b5c35d2bc8f9-original-pull-secret\") pod \"global-pull-secret-syncer-4b2pr\" (UID: \"0b2d7938-0c97-4f07-a7d2-b5c35d2bc8f9\") " pod="kube-system/global-pull-secret-syncer-4b2pr"
Nov 02 19:27:42.662404 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:42.662273    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 19:27:42.662404 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:42.662316    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b2d7938-0c97-4f07-a7d2-b5c35d2bc8f9-original-pull-secret podName:0b2d7938-0c97-4f07-a7d2-b5c35d2bc8f9 nodeName:}" failed. No retries permitted until 2026-11-02 19:27:50.662304611 +0000 UTC m=+17.690578921 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0b2d7938-0c97-4f07-a7d2-b5c35d2bc8f9-original-pull-secret") pod "global-pull-secret-syncer-4b2pr" (UID: "0b2d7938-0c97-4f07-a7d2-b5c35d2bc8f9") : object "kube-system"/"original-pull-secret" not registered
Nov 02 19:27:42.662404 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:42.662378    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 19:27:42.662404 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:42.662392    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 19:27:42.662404 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:42.662404    2579 projected.go:196] Error preparing data for projected volume kube-api-access-ccf8l for pod openshift-network-diagnostics/network-check-target-rhdd6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 19:27:42.662650 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:42.662427    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/31a69a0e-cc9c-47ad-b577-36b50d61d213-kube-api-access-ccf8l podName:31a69a0e-cc9c-47ad-b577-36b50d61d213 nodeName:}" failed. No retries permitted until 2026-11-02 19:27:50.662420182 +0000 UTC m=+17.690694492 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-ccf8l" (UniqueName: "kubernetes.io/projected/31a69a0e-cc9c-47ad-b577-36b50d61d213-kube-api-access-ccf8l") pod "network-check-target-rhdd6" (UID: "31a69a0e-cc9c-47ad-b577-36b50d61d213") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 19:27:42.849351 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:42.847514    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4b2pr"
Nov 02 19:27:42.849351 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:42.847642    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4b2pr" podUID="0b2d7938-0c97-4f07-a7d2-b5c35d2bc8f9"
Nov 02 19:27:42.849351 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:42.847708    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f6r6d"
Nov 02 19:27:42.849351 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:42.847794    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f6r6d" podUID="17fe5072-f4c5-4075-8614-4229ed05d4b2"
Nov 02 19:27:42.849351 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:42.847838    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rhdd6"
Nov 02 19:27:42.849351 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:42.847902    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rhdd6" podUID="31a69a0e-cc9c-47ad-b577-36b50d61d213"
Nov 02 19:27:42.938060 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:42.937972    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hrhpj" event={"ID":"b6ad2c98-52b5-496e-9824-614a2e509a8a","Type":"ContainerStarted","Data":"ecb1893fb62e129e98649f47afeda81723ffdccc923b4c8d4209dfbdb8cb01c6"}
Nov 02 19:27:43.055671 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:43.055615    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-jgfjj" podStartSLOduration=4.071980378 podStartE2EDuration="9.055595608s" totalImagesPullingTime="4.98361523s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:27:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:27:42.052642208 +0000 UTC m=+9.080916530" watchObservedRunningTime="2026-11-02 19:27:43.055595608 +0000 UTC m=+10.083869928"
Nov 02 19:27:43.056307 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:43.056260    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-hrhpj" podStartSLOduration=2.91583436 podStartE2EDuration="9.056248269s" totalImagesPullingTime="6.140413909s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 19:27:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:27:43.05552193 +0000 UTC m=+10.083796251" watchObservedRunningTime="2026-11-02 19:27:43.056248269 +0000 UTC m=+10.084522591"
Nov 02 19:27:43.941921 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:43.941828    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5wvz" event={"ID":"74106611-359e-4353-b113-70521144bed1","Type":"ContainerStarted","Data":"2a0f2dffb0f8130b3b9dedf09ff15cd5bd205378e6ecaf456b853ab21eace18b"}
Nov 02 19:27:44.847052 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:44.847006    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4b2pr"
Nov 02 19:27:44.847176 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:44.847148    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4b2pr" podUID="0b2d7938-0c97-4f07-a7d2-b5c35d2bc8f9"
Nov 02 19:27:44.847258 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:44.847242    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f6r6d"
Nov 02 19:27:44.847356 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:44.847336    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f6r6d" podUID="17fe5072-f4c5-4075-8614-4229ed05d4b2"
Nov 02 19:27:44.847413 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:44.847404    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rhdd6"
Nov 02 19:27:44.847493 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:44.847473    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rhdd6" podUID="31a69a0e-cc9c-47ad-b577-36b50d61d213"
Nov 02 19:27:44.866608 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:44.865869    2579 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 02 19:27:44.946966 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:44.946356    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qhswb" event={"ID":"9b7f05bf-12cd-4aad-b4ee-8b517e901ca6","Type":"ContainerStarted","Data":"aa67d72adcc6ac966af305c6d3772318b75b6fbf250327fa231c1f6470efdb64"}
Nov 02 19:27:44.951228 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:44.950666    2579 generic.go:356] "Generic (PLEG): container finished" podID="4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343" containerID="6a6ee7801ec9837ab0021f979afba29017c79437d8dbb1f14d56b4453b2afc46" exitCode=0
Nov 02 19:27:44.951228 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:44.950806    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4ddbs" event={"ID":"4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343","Type":"ContainerDied","Data":"6a6ee7801ec9837ab0021f979afba29017c79437d8dbb1f14d56b4453b2afc46"}
Nov 02 19:27:44.953826 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:44.953754    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5wvz" event={"ID":"74106611-359e-4353-b113-70521144bed1","Type":"ContainerStarted","Data":"55a0c39718906793234be4985abacda4f5b647734c2fa42e2b446ed6e44a3ee7"}
Nov 02 19:27:45.067533 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:45.067271    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-qhswb" podStartSLOduration=2.336608028 podStartE2EDuration="11.067255543s" totalImagesPullingTime="8.730647515s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 19:27:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:27:45.067170286 +0000 UTC m=+12.095444606" watchObservedRunningTime="2026-11-02 19:27:45.067255543 +0000 UTC m=+12.095529862"
Nov 02 19:27:45.647780 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:45.647465    2579 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T19:27:44.865887006Z","UUID":"dacf16ac-42bd-4185-8039-a94b837fa707","Handler":null,"Name":"","Endpoint":""}
Nov 02 19:27:45.649029 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:45.648988    2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 02 19:27:45.649157 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:45.649042    2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 02 19:27:45.959967 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:45.959893    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-r9fcz" event={"ID":"dd804657-39fd-4ae9-899d-cd1f1b62717e","Type":"ContainerStarted","Data":"e90742d63d5b1f6f8c96e3adb793922fcb7d1e39ada36586756625d079a6b453"}
Nov 02 19:27:46.081373 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:46.081312    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-r9fcz" podStartSLOduration=3.06053393 podStartE2EDuration="12.081293922s" totalImagesPullingTime="9.020759992s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 19:27:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:27:46.080243494 +0000 UTC m=+13.108517814" watchObservedRunningTime="2026-11-02 19:27:46.081293922 +0000 UTC m=+13.109568242"
Nov 02 19:27:46.320601 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:46.320567    2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-jgfjj"
Nov 02 19:27:46.321226 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:46.321202    2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-jgfjj"
Nov 02 19:27:46.847964 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:46.847881    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rhdd6"
Nov 02 19:27:46.847964 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:46.847916    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4b2pr"
Nov 02 19:27:46.848188 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:46.848000    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rhdd6" podUID="31a69a0e-cc9c-47ad-b577-36b50d61d213"
Nov 02 19:27:46.848188 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:46.848076    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f6r6d"
Nov 02 19:27:46.848188 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:46.848179    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f6r6d" podUID="17fe5072-f4c5-4075-8614-4229ed05d4b2"
Nov 02 19:27:46.848356 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:46.848229    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4b2pr" podUID="0b2d7938-0c97-4f07-a7d2-b5c35d2bc8f9"
Nov 02 19:27:46.962825 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:46.962788    2579 generic.go:356] "Generic (PLEG): container finished" podID="4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343" containerID="54528f4bec65f2f6c39231332edad67d23fbc44865fa24e2af6fc22662971c17" exitCode=0
Nov 02 19:27:46.963483 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:46.962870    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4ddbs" event={"ID":"4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343","Type":"ContainerDied","Data":"54528f4bec65f2f6c39231332edad67d23fbc44865fa24e2af6fc22662971c17"}
Nov 02 19:27:46.965765 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:46.965714    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nvdtg" event={"ID":"a8e20267-1b77-4eb3-b940-a29d79b0ec19","Type":"ContainerStarted","Data":"069f95463990e0f99cb1d99974c786a9b67bcadb09c2dfa5983899bc20f0cc18"}
Nov 02 19:27:46.969418 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:46.968854    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5wvz" event={"ID":"74106611-359e-4353-b113-70521144bed1","Type":"ContainerStarted","Data":"b1799b9421b474c0e967600482fa7ae6a7625411326be30b7bab0c289ed7f982"}
Nov 02 19:27:46.970768 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:46.970736    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cmk66" event={"ID":"b6cb1c47-e00c-484d-bf34-445b9380553c","Type":"ContainerStarted","Data":"a22aa91bf1f7318bb5ff8c99397b5df6d9f042b4562e54cc05366804ed9fbb70"}
Nov 02 19:27:46.971205 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:46.970957    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-jgfjj"
Nov 02 19:27:46.971393 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:46.971376    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-jgfjj"
Nov 02 19:27:47.206278 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:47.206175    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-nvdtg" podStartSLOduration=1.997090562 podStartE2EDuration="13.206156615s" totalImagesPullingTime="11.209066053s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 19:27:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:27:47.205777955 +0000 UTC m=+14.234052276" watchObservedRunningTime="2026-11-02 19:27:47.206156615 +0000 UTC m=+14.234430937"
Nov 02 19:27:47.326162 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:47.326108    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5wvz" podStartSLOduration=3.500381296 podStartE2EDuration="13.326091124s" totalImagesPullingTime="9.825709828s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 19:27:34 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:27:47.325560503 +0000 UTC m=+14.353834823" watchObservedRunningTime="2026-11-02 19:27:47.326091124 +0000 UTC m=+14.354365444"
Nov 02 19:27:47.564242 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:47.564150    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-cmk66" podStartSLOduration=4.755451699 podStartE2EDuration="13.564135688s" totalImagesPullingTime="8.808683989s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 19:27:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:27:47.563388902 +0000 UTC m=+14.591663221" watchObservedRunningTime="2026-11-02 19:27:47.564135688 +0000 UTC m=+14.592410008"
Nov 02 19:27:48.847592 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:48.847517    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f6r6d"
Nov 02 19:27:48.848297 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:48.847632    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f6r6d" podUID="17fe5072-f4c5-4075-8614-4229ed05d4b2"
Nov 02 19:27:48.848297 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:48.847962    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rhdd6"
Nov 02 19:27:48.848297 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:48.848053    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rhdd6" podUID="31a69a0e-cc9c-47ad-b577-36b50d61d213"
Nov 02 19:27:48.848297 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:48.848113    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4b2pr"
Nov 02 19:27:48.848297 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:48.848164    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4b2pr" podUID="0b2d7938-0c97-4f07-a7d2-b5c35d2bc8f9"
Nov 02 19:27:48.977660 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:48.977625    2579 generic.go:356] "Generic (PLEG): container finished" podID="4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343" containerID="45648625589c2bcc99e47f800b0d4569857fd84049cfa03c5744fc75a23269ba" exitCode=0
Nov 02 19:27:48.977843 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:48.977718    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4ddbs" event={"ID":"4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343","Type":"ContainerDied","Data":"45648625589c2bcc99e47f800b0d4569857fd84049cfa03c5744fc75a23269ba"}
Nov 02 19:27:50.624810 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:50.624566    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17fe5072-f4c5-4075-8614-4229ed05d4b2-metrics-certs\") pod \"network-metrics-daemon-f6r6d\" (UID: \"17fe5072-f4c5-4075-8614-4229ed05d4b2\") " pod="openshift-multus/network-metrics-daemon-f6r6d"
Nov 02 19:27:50.625291 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:50.624737    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 19:27:50.625291 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:50.624900    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17fe5072-f4c5-4075-8614-4229ed05d4b2-metrics-certs podName:17fe5072-f4c5-4075-8614-4229ed05d4b2 nodeName:}" failed. No retries permitted until 2026-11-02 19:28:06.624884927 +0000 UTC m=+33.653159244 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/17fe5072-f4c5-4075-8614-4229ed05d4b2-metrics-certs") pod "network-metrics-daemon-f6r6d" (UID: "17fe5072-f4c5-4075-8614-4229ed05d4b2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 19:27:50.725264 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:50.725227    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccf8l\" (UniqueName: \"kubernetes.io/projected/31a69a0e-cc9c-47ad-b577-36b50d61d213-kube-api-access-ccf8l\") pod \"network-check-target-rhdd6\" (UID: \"31a69a0e-cc9c-47ad-b577-36b50d61d213\") " pod="openshift-network-diagnostics/network-check-target-rhdd6"
Nov 02 19:27:50.725424 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:50.725282    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b2d7938-0c97-4f07-a7d2-b5c35d2bc8f9-original-pull-secret\") pod \"global-pull-secret-syncer-4b2pr\" (UID: \"0b2d7938-0c97-4f07-a7d2-b5c35d2bc8f9\") " pod="kube-system/global-pull-secret-syncer-4b2pr"
Nov 02 19:27:50.725424 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:50.725396    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 19:27:50.725513 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:50.725426    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 19:27:50.725513 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:50.725446    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b2d7938-0c97-4f07-a7d2-b5c35d2bc8f9-original-pull-secret podName:0b2d7938-0c97-4f07-a7d2-b5c35d2bc8f9 nodeName:}" failed. No retries permitted until 2026-11-02 19:28:06.725429948 +0000 UTC m=+33.753704260 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0b2d7938-0c97-4f07-a7d2-b5c35d2bc8f9-original-pull-secret") pod "global-pull-secret-syncer-4b2pr" (UID: "0b2d7938-0c97-4f07-a7d2-b5c35d2bc8f9") : object "kube-system"/"original-pull-secret" not registered
Nov 02 19:27:50.725513 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:50.725451    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 19:27:50.725513 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:50.725465    2579 projected.go:196] Error preparing data for projected volume kube-api-access-ccf8l for pod openshift-network-diagnostics/network-check-target-rhdd6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 19:27:50.725513 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:50.725513    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/31a69a0e-cc9c-47ad-b577-36b50d61d213-kube-api-access-ccf8l podName:31a69a0e-cc9c-47ad-b577-36b50d61d213 nodeName:}" failed. No retries permitted until 2026-11-02 19:28:06.725496676 +0000 UTC m=+33.753770987 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-ccf8l" (UniqueName: "kubernetes.io/projected/31a69a0e-cc9c-47ad-b577-36b50d61d213-kube-api-access-ccf8l") pod "network-check-target-rhdd6" (UID: "31a69a0e-cc9c-47ad-b577-36b50d61d213") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 19:27:50.847409 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:50.847380    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f6r6d"
Nov 02 19:27:50.847575 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:50.847379    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4b2pr"
Nov 02 19:27:50.847575 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:50.847504    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f6r6d" podUID="17fe5072-f4c5-4075-8614-4229ed05d4b2"
Nov 02 19:27:50.847575 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:50.847536    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rhdd6"
Nov 02 19:27:50.847742 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:50.847601    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4b2pr" podUID="0b2d7938-0c97-4f07-a7d2-b5c35d2bc8f9"
Nov 02 19:27:50.847742 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:50.847660    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rhdd6" podUID="31a69a0e-cc9c-47ad-b577-36b50d61d213"
Nov 02 19:27:52.848167 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:52.847930    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4b2pr"
Nov 02 19:27:52.848518 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:52.848005    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rhdd6"
Nov 02 19:27:52.848518 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:52.848191    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4b2pr" podUID="0b2d7938-0c97-4f07-a7d2-b5c35d2bc8f9"
Nov 02 19:27:52.848518 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:52.848274    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rhdd6" podUID="31a69a0e-cc9c-47ad-b577-36b50d61d213"
Nov 02 19:27:52.848518 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:52.848006    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f6r6d"
Nov 02 19:27:52.848518 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:52.848350    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f6r6d" podUID="17fe5072-f4c5-4075-8614-4229ed05d4b2"
Nov 02 19:27:52.994950 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:52.994915    2579 generic.go:356] "Generic (PLEG): container finished" podID="4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343" containerID="67a7e23ec25a1a59a219af07c8fc672d978f64a76f0065c10597f72db2606392" exitCode=0
Nov 02 19:27:52.995137 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:52.994989    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4ddbs" event={"ID":"4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343","Type":"ContainerDied","Data":"67a7e23ec25a1a59a219af07c8fc672d978f64a76f0065c10597f72db2606392"}
Nov 02 19:27:52.997878 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:52.997751    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vst5r" event={"ID":"8756b800-1b4e-41da-934a-25c6b8ee738c","Type":"ContainerStarted","Data":"20f2b6b0dcfc959f8018c6b975d1e8981ef3590a5c8c05cb5eda8dc1344ed4ef"}
Nov 02 19:27:52.997878 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:52.997805    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vst5r" event={"ID":"8756b800-1b4e-41da-934a-25c6b8ee738c","Type":"ContainerStarted","Data":"9d369b3db34d1d1b702d071463961c8ae360724776d01603245b835300e984f2"}
Nov 02 19:27:52.997878 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:52.997835    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vst5r" event={"ID":"8756b800-1b4e-41da-934a-25c6b8ee738c","Type":"ContainerStarted","Data":"68f10f42ae65c5b597c3abe4585a3e45ff27df06d635767d146cd7218c5eff39"}
Nov 02 19:27:52.997878 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:52.997862    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vst5r" event={"ID":"8756b800-1b4e-41da-934a-25c6b8ee738c","Type":"ContainerStarted","Data":"874c98cdd1079d2da4cc6d92c8d81c7016422696b23856eb8efef85952623994"}
Nov 02 19:27:52.998030 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:52.997888    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vst5r" event={"ID":"8756b800-1b4e-41da-934a-25c6b8ee738c","Type":"ContainerStarted","Data":"a53ba0c8e38bd48c4a782f2f4a015aa9892663ff743e4273c785a71edc794594"}
Nov 02 19:27:52.998030 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:52.997946    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vst5r" event={"ID":"8756b800-1b4e-41da-934a-25c6b8ee738c","Type":"ContainerStarted","Data":"d0a9cd6e215bb5cab4f887cf06da004ab9a0b49db53b1a35086edee502743f54"}
Nov 02 19:27:54.002670 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:54.002633    2579 generic.go:356] "Generic (PLEG): container finished" podID="4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343" containerID="77beba51e1d986850256508f7319d9d84cce1764452332f2a5952d9b32e61435" exitCode=0
Nov 02 19:27:54.003154 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:54.002689    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4ddbs" event={"ID":"4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343","Type":"ContainerDied","Data":"77beba51e1d986850256508f7319d9d84cce1764452332f2a5952d9b32e61435"}
Nov 02 19:27:54.847976 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:54.847895    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rhdd6"
Nov 02 19:27:54.847976 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:54.847972    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f6r6d"
Nov 02 19:27:54.848181 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:54.848058    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4b2pr"
Nov 02 19:27:54.848181 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:54.848064    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rhdd6" podUID="31a69a0e-cc9c-47ad-b577-36b50d61d213"
Nov 02 19:27:54.848181 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:54.848120    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4b2pr" podUID="0b2d7938-0c97-4f07-a7d2-b5c35d2bc8f9"
Nov 02 19:27:54.848314 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:54.848231    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f6r6d" podUID="17fe5072-f4c5-4075-8614-4229ed05d4b2"
Nov 02 19:27:55.008281 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:55.008244    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4ddbs" event={"ID":"4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343","Type":"ContainerStarted","Data":"f0d2e315c92cf3c56f62ec411751a956447a1c82aba180adda5ec0517b36a315"}
Nov 02 19:27:55.011213 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:55.011186    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vst5r" event={"ID":"8756b800-1b4e-41da-934a-25c6b8ee738c","Type":"ContainerStarted","Data":"d6738c0eeec50ec0af935abc1c2def01cc2c9a13138d40852cdd2c17bc0b3df2"}
Nov 02 19:27:55.131003 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:55.130911    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-4ddbs" podStartSLOduration=8.000843797 podStartE2EDuration="21.130899131s" totalImagesPullingTime="13.130055334s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 19:27:34 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:27:55.130176035 +0000 UTC m=+22.158450365" watchObservedRunningTime="2026-11-02 19:27:55.130899131 +0000 UTC m=+22.159173450"
Nov 02 19:27:56.848115 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:56.848074    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f6r6d"
Nov 02 19:27:56.848115 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:56.848097    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4b2pr"
Nov 02 19:27:56.848897 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:56.848100    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rhdd6"
Nov 02 19:27:56.848897 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:56.848206    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f6r6d" podUID="17fe5072-f4c5-4075-8614-4229ed05d4b2"
Nov 02 19:27:56.848897 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:56.848323    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4b2pr" podUID="0b2d7938-0c97-4f07-a7d2-b5c35d2bc8f9"
Nov 02 19:27:56.848897 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:56.848387    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rhdd6" podUID="31a69a0e-cc9c-47ad-b577-36b50d61d213"
Nov 02 19:27:57.018136 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:57.018053    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vst5r" event={"ID":"8756b800-1b4e-41da-934a-25c6b8ee738c","Type":"ContainerStarted","Data":"4f3e64cd645d7b240963979c1d793d4555b67d55a22aee8a53d41ca5363ab74b"}
Nov 02 19:27:57.018278 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:57.018248    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:27:57.018333 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:57.018321    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:27:57.018378 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:57.018373    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:27:57.033192 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:57.033169    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:27:57.033339 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:57.033268    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:27:57.305577 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:57.305482    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-vst5r" podStartSLOduration=7.270826993 podStartE2EDuration="23.305469277s" totalImagesPullingTime="16.034642284s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 19:27:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:27:57.252054599 +0000 UTC m=+24.280328915" watchObservedRunningTime="2026-11-02 19:27:57.305469277 +0000 UTC m=+24.333743617"
Nov 02 19:27:58.847822 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:58.847789    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4b2pr"
Nov 02 19:27:58.848344 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:58.847897    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f6r6d"
Nov 02 19:27:58.848344 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:58.847910    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4b2pr" podUID="0b2d7938-0c97-4f07-a7d2-b5c35d2bc8f9"
Nov 02 19:27:58.848344 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:58.847928    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rhdd6"
Nov 02 19:27:58.848344 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:58.848003    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f6r6d" podUID="17fe5072-f4c5-4075-8614-4229ed05d4b2"
Nov 02 19:27:58.848344 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:58.848091    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rhdd6" podUID="31a69a0e-cc9c-47ad-b577-36b50d61d213"
Nov 02 19:27:59.336089 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:59.335829    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-f6r6d"]
Nov 02 19:27:59.336219 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:59.336103    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f6r6d"
Nov 02 19:27:59.336219 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:59.336200    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f6r6d" podUID="17fe5072-f4c5-4075-8614-4229ed05d4b2"
Nov 02 19:27:59.339008 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:59.338986    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rhdd6"]
Nov 02 19:27:59.339172 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:59.339075    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rhdd6"
Nov 02 19:27:59.339172 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:59.339149    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rhdd6" podUID="31a69a0e-cc9c-47ad-b577-36b50d61d213"
Nov 02 19:27:59.339575 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:59.339554    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4b2pr"]
Nov 02 19:27:59.339655 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:27:59.339640    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4b2pr"
Nov 02 19:27:59.339751 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:27:59.339733    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4b2pr" podUID="0b2d7938-0c97-4f07-a7d2-b5c35d2bc8f9"
Nov 02 19:28:00.847766 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:00.847738    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f6r6d"
Nov 02 19:28:00.848153 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:00.847738    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4b2pr"
Nov 02 19:28:00.848153 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:28:00.847844    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f6r6d" podUID="17fe5072-f4c5-4075-8614-4229ed05d4b2"
Nov 02 19:28:00.848153 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:00.847738    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rhdd6"
Nov 02 19:28:00.848153 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:28:00.847906    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4b2pr" podUID="0b2d7938-0c97-4f07-a7d2-b5c35d2bc8f9"
Nov 02 19:28:00.848153 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:28:00.847966    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rhdd6" podUID="31a69a0e-cc9c-47ad-b577-36b50d61d213"
Nov 02 19:28:02.847873 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:02.847840    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rhdd6"
Nov 02 19:28:02.847873 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:02.847857    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4b2pr"
Nov 02 19:28:02.848333 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:28:02.847947    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rhdd6" podUID="31a69a0e-cc9c-47ad-b577-36b50d61d213"
Nov 02 19:28:02.848333 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:02.848001    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f6r6d"
Nov 02 19:28:02.848333 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:28:02.848090    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4b2pr" podUID="0b2d7938-0c97-4f07-a7d2-b5c35d2bc8f9"
Nov 02 19:28:02.848333 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:28:02.848161    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f6r6d" podUID="17fe5072-f4c5-4075-8614-4229ed05d4b2"
Nov 02 19:28:04.683710 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:04.683688    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-9.us-west-2.compute.internal" event="NodeReady"
Nov 02 19:28:04.684096 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:04.683816    2579 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 02 19:28:04.846938 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:04.846905    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-qj8mr"]
Nov 02 19:28:04.885966 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:04.885933    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-sw9mp"]
Nov 02 19:28:04.886152 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:04.886061    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rhdd6"
Nov 02 19:28:04.886152 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:04.886117    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4b2pr"
Nov 02 19:28:04.886250 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:04.886192    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f6r6d"
Nov 02 19:28:04.886419 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:04.886395    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qj8mr"
Nov 02 19:28:04.896006 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:04.895793    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qj8mr"]
Nov 02 19:28:04.896006 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:04.895904    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-sw9mp"
Nov 02 19:28:04.902519 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:04.902498    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-sw9mp"]
Nov 02 19:28:04.942654 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:04.942574    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 02 19:28:04.942654 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:04.942580    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 02 19:28:04.942654 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:04.942646    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 02 19:28:04.942874 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:04.942665    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-n2mjf\""
Nov 02 19:28:04.942874 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:04.942674    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 02 19:28:04.942874 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:04.942679    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 02 19:28:04.942874 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:04.942655    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-dc6gd\""
Nov 02 19:28:04.942874 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:04.942742    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-fbvgr\""
Nov 02 19:28:04.942874 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:04.942795    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 02 19:28:04.943070 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:04.942894    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-c8nwr\""
Nov 02 19:28:05.028584 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:05.028553    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dfb225e8-fabd-49fb-9763-4e41bf6ae7de-metrics-tls\") pod \"dns-default-qj8mr\" (UID: \"dfb225e8-fabd-49fb-9763-4e41bf6ae7de\") " pod="openshift-dns/dns-default-qj8mr"
Nov 02 19:28:05.028584 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:05.028584    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgbw6\" (UniqueName: \"kubernetes.io/projected/5d436341-2000-4c2b-87ac-7b67e4cdb909-kube-api-access-dgbw6\") pod \"ingress-canary-sw9mp\" (UID: \"5d436341-2000-4c2b-87ac-7b67e4cdb909\") " pod="openshift-ingress-canary/ingress-canary-sw9mp"
Nov 02 19:28:05.028796 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:05.028620    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfb225e8-fabd-49fb-9763-4e41bf6ae7de-tmp-dir\") pod \"dns-default-qj8mr\" (UID: \"dfb225e8-fabd-49fb-9763-4e41bf6ae7de\") " pod="openshift-dns/dns-default-qj8mr"
Nov 02 19:28:05.028796 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:05.028690    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5d436341-2000-4c2b-87ac-7b67e4cdb909-cert\") pod \"ingress-canary-sw9mp\" (UID: \"5d436341-2000-4c2b-87ac-7b67e4cdb909\") " pod="openshift-ingress-canary/ingress-canary-sw9mp"
Nov 02 19:28:05.028796 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:05.028787    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgmdg\" (UniqueName: \"kubernetes.io/projected/dfb225e8-fabd-49fb-9763-4e41bf6ae7de-kube-api-access-rgmdg\") pod \"dns-default-qj8mr\" (UID: \"dfb225e8-fabd-49fb-9763-4e41bf6ae7de\") " pod="openshift-dns/dns-default-qj8mr"
Nov 02 19:28:05.028895 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:05.028807    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/dfb225e8-fabd-49fb-9763-4e41bf6ae7de-config-volume\") pod \"dns-default-qj8mr\" (UID: \"dfb225e8-fabd-49fb-9763-4e41bf6ae7de\") " pod="openshift-dns/dns-default-qj8mr"
Nov 02 19:28:05.043628 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:05.043587    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 02 19:28:05.043628 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:05.043628    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 02 19:28:05.043819 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:05.043648    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-fzxnx\""
Nov 02 19:28:05.043819 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:05.043655    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 02 19:28:05.129773 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:05.129739    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dfb225e8-fabd-49fb-9763-4e41bf6ae7de-metrics-tls\") pod \"dns-default-qj8mr\" (UID: \"dfb225e8-fabd-49fb-9763-4e41bf6ae7de\") " pod="openshift-dns/dns-default-qj8mr"
Nov 02 19:28:05.129773 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:05.129774    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgbw6\" (UniqueName: \"kubernetes.io/projected/5d436341-2000-4c2b-87ac-7b67e4cdb909-kube-api-access-dgbw6\") pod \"ingress-canary-sw9mp\" (UID: \"5d436341-2000-4c2b-87ac-7b67e4cdb909\") " pod="openshift-ingress-canary/ingress-canary-sw9mp"
Nov 02 19:28:05.129984 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:28:05.129860    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 19:28:05.129984 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:05.129890    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfb225e8-fabd-49fb-9763-4e41bf6ae7de-tmp-dir\") pod \"dns-default-qj8mr\" (UID: \"dfb225e8-fabd-49fb-9763-4e41bf6ae7de\") " pod="openshift-dns/dns-default-qj8mr"
Nov 02 19:28:05.129984 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:05.129910    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5d436341-2000-4c2b-87ac-7b67e4cdb909-cert\") pod \"ingress-canary-sw9mp\" (UID: \"5d436341-2000-4c2b-87ac-7b67e4cdb909\") " pod="openshift-ingress-canary/ingress-canary-sw9mp"
Nov 02 19:28:05.129984 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:28:05.129927    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfb225e8-fabd-49fb-9763-4e41bf6ae7de-metrics-tls podName:dfb225e8-fabd-49fb-9763-4e41bf6ae7de nodeName:}" failed. No retries permitted until 2026-11-02 19:28:05.629911532 +0000 UTC m=+32.658185844 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dfb225e8-fabd-49fb-9763-4e41bf6ae7de-metrics-tls") pod "dns-default-qj8mr" (UID: "dfb225e8-fabd-49fb-9763-4e41bf6ae7de") : secret "dns-default-metrics-tls" not found
Nov 02 19:28:05.129984 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:05.129966    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgmdg\" (UniqueName: \"kubernetes.io/projected/dfb225e8-fabd-49fb-9763-4e41bf6ae7de-kube-api-access-rgmdg\") pod \"dns-default-qj8mr\" (UID: \"dfb225e8-fabd-49fb-9763-4e41bf6ae7de\") " pod="openshift-dns/dns-default-qj8mr"
Nov 02 19:28:05.130297 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:05.130001    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/dfb225e8-fabd-49fb-9763-4e41bf6ae7de-config-volume\") pod \"dns-default-qj8mr\" (UID: \"dfb225e8-fabd-49fb-9763-4e41bf6ae7de\") " pod="openshift-dns/dns-default-qj8mr"
Nov 02 19:28:05.130297 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:05.130290    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfb225e8-fabd-49fb-9763-4e41bf6ae7de-tmp-dir\") pod \"dns-default-qj8mr\" (UID: \"dfb225e8-fabd-49fb-9763-4e41bf6ae7de\") " pod="openshift-dns/dns-default-qj8mr"
Nov 02 19:28:05.130610 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:05.130594    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/dfb225e8-fabd-49fb-9763-4e41bf6ae7de-config-volume\") pod \"dns-default-qj8mr\" (UID: \"dfb225e8-fabd-49fb-9763-4e41bf6ae7de\") " pod="openshift-dns/dns-default-qj8mr"
Nov 02 19:28:05.133273 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:05.133257    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5d436341-2000-4c2b-87ac-7b67e4cdb909-cert\") pod \"ingress-canary-sw9mp\" (UID: \"5d436341-2000-4c2b-87ac-7b67e4cdb909\") " pod="openshift-ingress-canary/ingress-canary-sw9mp"
Nov 02 19:28:05.190558 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:05.190524    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgbw6\" (UniqueName: \"kubernetes.io/projected/5d436341-2000-4c2b-87ac-7b67e4cdb909-kube-api-access-dgbw6\") pod \"ingress-canary-sw9mp\" (UID: \"5d436341-2000-4c2b-87ac-7b67e4cdb909\") " pod="openshift-ingress-canary/ingress-canary-sw9mp"
Nov 02 19:28:05.190694 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:05.190682    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgmdg\" (UniqueName: \"kubernetes.io/projected/dfb225e8-fabd-49fb-9763-4e41bf6ae7de-kube-api-access-rgmdg\") pod \"dns-default-qj8mr\" (UID: \"dfb225e8-fabd-49fb-9763-4e41bf6ae7de\") " pod="openshift-dns/dns-default-qj8mr"
Nov 02 19:28:05.284397 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:05.284312    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-sw9mp"
Nov 02 19:28:05.634370 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:05.634280    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dfb225e8-fabd-49fb-9763-4e41bf6ae7de-metrics-tls\") pod \"dns-default-qj8mr\" (UID: \"dfb225e8-fabd-49fb-9763-4e41bf6ae7de\") " pod="openshift-dns/dns-default-qj8mr"
Nov 02 19:28:05.634544 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:28:05.634462    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 19:28:05.634600 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:28:05.634546    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfb225e8-fabd-49fb-9763-4e41bf6ae7de-metrics-tls podName:dfb225e8-fabd-49fb-9763-4e41bf6ae7de nodeName:}" failed. No retries permitted until 2026-11-02 19:28:06.634525041 +0000 UTC m=+33.662799360 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dfb225e8-fabd-49fb-9763-4e41bf6ae7de-metrics-tls") pod "dns-default-qj8mr" (UID: "dfb225e8-fabd-49fb-9763-4e41bf6ae7de") : secret "dns-default-metrics-tls" not found
Nov 02 19:28:05.695735 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:28:05.695709    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d436341_2000_4c2b_87ac_7b67e4cdb909.slice/crio-b2cb51f772139f867440341c3eabff8fe9e6429b771fbd5c866d0cfe265a74a2 WatchSource:0}: Error finding container b2cb51f772139f867440341c3eabff8fe9e6429b771fbd5c866d0cfe265a74a2: Status 404 returned error can't find the container with id b2cb51f772139f867440341c3eabff8fe9e6429b771fbd5c866d0cfe265a74a2
Nov 02 19:28:05.698324 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:05.698295    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-sw9mp"]
Nov 02 19:28:06.037894 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:06.037857    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-sw9mp" event={"ID":"5d436341-2000-4c2b-87ac-7b67e4cdb909","Type":"ContainerStarted","Data":"b2cb51f772139f867440341c3eabff8fe9e6429b771fbd5c866d0cfe265a74a2"}
Nov 02 19:28:06.641625 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:06.641584    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dfb225e8-fabd-49fb-9763-4e41bf6ae7de-metrics-tls\") pod \"dns-default-qj8mr\" (UID: \"dfb225e8-fabd-49fb-9763-4e41bf6ae7de\") " pod="openshift-dns/dns-default-qj8mr"
Nov 02 19:28:06.641796 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:06.641632    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17fe5072-f4c5-4075-8614-4229ed05d4b2-metrics-certs\") pod \"network-metrics-daemon-f6r6d\" (UID: \"17fe5072-f4c5-4075-8614-4229ed05d4b2\") " pod="openshift-multus/network-metrics-daemon-f6r6d"
Nov 02 19:28:06.642281 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:28:06.641944    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 19:28:06.642281 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:28:06.642054    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfb225e8-fabd-49fb-9763-4e41bf6ae7de-metrics-tls podName:dfb225e8-fabd-49fb-9763-4e41bf6ae7de nodeName:}" failed. No retries permitted until 2026-11-02 19:28:08.642008484 +0000 UTC m=+35.670282805 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dfb225e8-fabd-49fb-9763-4e41bf6ae7de-metrics-tls") pod "dns-default-qj8mr" (UID: "dfb225e8-fabd-49fb-9763-4e41bf6ae7de") : secret "dns-default-metrics-tls" not found
Nov 02 19:28:06.642281 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:28:06.642101    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 19:28:06.642281 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:28:06.642151    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17fe5072-f4c5-4075-8614-4229ed05d4b2-metrics-certs podName:17fe5072-f4c5-4075-8614-4229ed05d4b2 nodeName:}" failed. No retries permitted until 2026-11-02 19:28:38.642138893 +0000 UTC m=+65.670413204 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/17fe5072-f4c5-4075-8614-4229ed05d4b2-metrics-certs") pod "network-metrics-daemon-f6r6d" (UID: "17fe5072-f4c5-4075-8614-4229ed05d4b2") : secret "metrics-daemon-secret" not found
Nov 02 19:28:06.743084 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:06.743039    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccf8l\" (UniqueName: \"kubernetes.io/projected/31a69a0e-cc9c-47ad-b577-36b50d61d213-kube-api-access-ccf8l\") pod \"network-check-target-rhdd6\" (UID: \"31a69a0e-cc9c-47ad-b577-36b50d61d213\") " pod="openshift-network-diagnostics/network-check-target-rhdd6"
Nov 02 19:28:06.743498 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:06.743134    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b2d7938-0c97-4f07-a7d2-b5c35d2bc8f9-original-pull-secret\") pod \"global-pull-secret-syncer-4b2pr\" (UID: \"0b2d7938-0c97-4f07-a7d2-b5c35d2bc8f9\") " pod="kube-system/global-pull-secret-syncer-4b2pr"
Nov 02 19:28:06.745724 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:06.745685    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b2d7938-0c97-4f07-a7d2-b5c35d2bc8f9-original-pull-secret\") pod \"global-pull-secret-syncer-4b2pr\" (UID: \"0b2d7938-0c97-4f07-a7d2-b5c35d2bc8f9\") " pod="kube-system/global-pull-secret-syncer-4b2pr"
Nov 02 19:28:06.745919 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:06.745900    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccf8l\" (UniqueName: \"kubernetes.io/projected/31a69a0e-cc9c-47ad-b577-36b50d61d213-kube-api-access-ccf8l\") pod \"network-check-target-rhdd6\" (UID: \"31a69a0e-cc9c-47ad-b577-36b50d61d213\") " pod="openshift-network-diagnostics/network-check-target-rhdd6"
Nov 02 19:28:06.762235 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:06.762210    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4b2pr"
Nov 02 19:28:07.021604 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:07.021571    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rhdd6"
Nov 02 19:28:07.151785 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:28:07.151752    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b2d7938_0c97_4f07_a7d2_b5c35d2bc8f9.slice/crio-34ffb39bf1d7fea750bf39bc7b22f4ef70166d88d57a034b4fa561753454caee WatchSource:0}: Error finding container 34ffb39bf1d7fea750bf39bc7b22f4ef70166d88d57a034b4fa561753454caee: Status 404 returned error can't find the container with id 34ffb39bf1d7fea750bf39bc7b22f4ef70166d88d57a034b4fa561753454caee
Nov 02 19:28:07.154740 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:07.154635    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4b2pr"]
Nov 02 19:28:07.413469 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:28:07.413432    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31a69a0e_cc9c_47ad_b577_36b50d61d213.slice/crio-e49208b3830b22cd3c99cbb381515dcd9235cf0a71aff04e049925e4863af5f6 WatchSource:0}: Error finding container e49208b3830b22cd3c99cbb381515dcd9235cf0a71aff04e049925e4863af5f6: Status 404 returned error can't find the container with id e49208b3830b22cd3c99cbb381515dcd9235cf0a71aff04e049925e4863af5f6
Nov 02 19:28:07.415294 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:07.415214    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rhdd6"]
Nov 02 19:28:08.043978 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:08.043936    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rhdd6" event={"ID":"31a69a0e-cc9c-47ad-b577-36b50d61d213","Type":"ContainerStarted","Data":"e49208b3830b22cd3c99cbb381515dcd9235cf0a71aff04e049925e4863af5f6"}
Nov 02 19:28:08.045170 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:08.045140    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4b2pr" event={"ID":"0b2d7938-0c97-4f07-a7d2-b5c35d2bc8f9","Type":"ContainerStarted","Data":"34ffb39bf1d7fea750bf39bc7b22f4ef70166d88d57a034b4fa561753454caee"}
Nov 02 19:28:08.656201 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:08.656165    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dfb225e8-fabd-49fb-9763-4e41bf6ae7de-metrics-tls\") pod \"dns-default-qj8mr\" (UID: \"dfb225e8-fabd-49fb-9763-4e41bf6ae7de\") " pod="openshift-dns/dns-default-qj8mr"
Nov 02 19:28:08.656393 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:28:08.656337    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 19:28:08.656457 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:28:08.656399    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfb225e8-fabd-49fb-9763-4e41bf6ae7de-metrics-tls podName:dfb225e8-fabd-49fb-9763-4e41bf6ae7de nodeName:}" failed. No retries permitted until 2026-11-02 19:28:12.656381831 +0000 UTC m=+39.684656141 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dfb225e8-fabd-49fb-9763-4e41bf6ae7de-metrics-tls") pod "dns-default-qj8mr" (UID: "dfb225e8-fabd-49fb-9763-4e41bf6ae7de") : secret "dns-default-metrics-tls" not found
Nov 02 19:28:09.914155 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:09.914124    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-bh65q"]
Nov 02 19:28:09.937133 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:09.937111    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bh65q"
Nov 02 19:28:10.047076 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:10.046998    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bh65q"]
Nov 02 19:28:10.052214 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:10.052175    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-sw9mp" event={"ID":"5d436341-2000-4c2b-87ac-7b67e4cdb909","Type":"ContainerStarted","Data":"d382e40a54157d0db1ed545395c6c934c8f4c05364528d7b6a951df400846311"}
Nov 02 19:28:10.062160 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:10.062139    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 02 19:28:10.062160 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:10.062139    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 02 19:28:10.062307 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:10.062181    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 02 19:28:10.074746 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:10.074721    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bh65q\" (UID: \"0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2\") " pod="openshift-insights/insights-runtime-extractor-bh65q"
Nov 02 19:28:10.074872 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:10.074764    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bh65q\" (UID: \"0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2\") " pod="openshift-insights/insights-runtime-extractor-bh65q"
Nov 02 19:28:10.074872 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:10.074851    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2g6f\" (UniqueName: \"kubernetes.io/projected/0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2-kube-api-access-j2g6f\") pod \"insights-runtime-extractor-bh65q\" (UID: \"0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2\") " pod="openshift-insights/insights-runtime-extractor-bh65q"
Nov 02 19:28:10.074979 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:10.074882    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2-crio-socket\") pod \"insights-runtime-extractor-bh65q\" (UID: \"0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2\") " pod="openshift-insights/insights-runtime-extractor-bh65q"
Nov 02 19:28:10.074979 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:10.074919    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2-data-volume\") pod \"insights-runtime-extractor-bh65q\" (UID: \"0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2\") " pod="openshift-insights/insights-runtime-extractor-bh65q"
Nov 02 19:28:10.099634 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:10.099613    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 02 19:28:10.115678 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:10.115657    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-k9tfs\""
Nov 02 19:28:10.152318 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:10.152298    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-xvzjr-ggg8w"]
Nov 02 19:28:10.175819 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:10.175794    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bh65q\" (UID: \"0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2\") " pod="openshift-insights/insights-runtime-extractor-bh65q"
Nov 02 19:28:10.175930 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:10.175834    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bh65q\" (UID: \"0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2\") " pod="openshift-insights/insights-runtime-extractor-bh65q"
Nov 02 19:28:10.175930 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:10.175867    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2g6f\" (UniqueName: \"kubernetes.io/projected/0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2-kube-api-access-j2g6f\") pod \"insights-runtime-extractor-bh65q\" (UID: \"0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2\") " pod="openshift-insights/insights-runtime-extractor-bh65q"
Nov 02 19:28:10.175930 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:10.175897    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2-crio-socket\") pod \"insights-runtime-extractor-bh65q\" (UID: \"0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2\") " pod="openshift-insights/insights-runtime-extractor-bh65q"
Nov 02 19:28:10.175930 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:28:10.175924    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 02 19:28:10.176155 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:10.175936    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2-data-volume\") pod \"insights-runtime-extractor-bh65q\" (UID: \"0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2\") " pod="openshift-insights/insights-runtime-extractor-bh65q"
Nov 02 19:28:10.176155 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:28:10.176055    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2-kube-rbac-proxy-cm podName:0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2 nodeName:}" failed. No retries permitted until 2026-11-02 19:28:10.676036792 +0000 UTC m=+37.704311094 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bh65q" (UID: "0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2") : configmap "kube-rbac-proxy" not found
Nov 02 19:28:10.176155 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:28:10.176152    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 19:28:10.176330 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:28:10.176199    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2-insights-runtime-extractor-tls podName:0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2 nodeName:}" failed. No retries permitted until 2026-11-02 19:28:10.676186128 +0000 UTC m=+37.704460445 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bh65q" (UID: "0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2") : secret "insights-runtime-extractor-tls" not found
Nov 02 19:28:10.176330 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:10.176257    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2-crio-socket\") pod \"insights-runtime-extractor-bh65q\" (UID: \"0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2\") " pod="openshift-insights/insights-runtime-extractor-bh65q"
Nov 02 19:28:10.176496 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:10.176478    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2-data-volume\") pod \"insights-runtime-extractor-bh65q\" (UID: \"0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2\") " pod="openshift-insights/insights-runtime-extractor-bh65q"
Nov 02 19:28:10.182337 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:10.182320    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-xvzjr-ggg8w"
Nov 02 19:28:10.209041 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:10.208974    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-sw9mp" podStartSLOduration=2.608943762 podStartE2EDuration="6.208957164s" totalImagesPullingTime="3.600013402s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:28:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:28:10.186786246 +0000 UTC m=+37.215060567" watchObservedRunningTime="2026-11-02 19:28:10.208957164 +0000 UTC m=+37.237231485"
Nov 02 19:28:10.209248 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:10.209225    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-xvzjr-ggg8w"]
Nov 02 19:28:10.239235 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:10.239209    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2g6f\" (UniqueName: \"kubernetes.io/projected/0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2-kube-api-access-j2g6f\") pod \"insights-runtime-extractor-bh65q\" (UID: \"0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2\") " pod="openshift-insights/insights-runtime-extractor-bh65q"
Nov 02 19:28:10.261235 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:10.261192    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 02 19:28:10.276704 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:10.276676    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4874b345-eac7-4694-8351-d24085bf180a-service-ca-bundle\") pod \"periodic-gathering-xvzjr-ggg8w\" (UID: \"4874b345-eac7-4694-8351-d24085bf180a\") " pod="openshift-insights/periodic-gathering-xvzjr-ggg8w"
Nov 02 19:28:10.276829 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:10.276714    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jz8qt\" (UniqueName: \"kubernetes.io/projected/4874b345-eac7-4694-8351-d24085bf180a-kube-api-access-jz8qt\") pod \"periodic-gathering-xvzjr-ggg8w\" (UID: \"4874b345-eac7-4694-8351-d24085bf180a\") " pod="openshift-insights/periodic-gathering-xvzjr-ggg8w"
Nov 02 19:28:10.276881 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:10.276828    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4874b345-eac7-4694-8351-d24085bf180a-archives-path\") pod \"periodic-gathering-xvzjr-ggg8w\" (UID: \"4874b345-eac7-4694-8351-d24085bf180a\") " pod="openshift-insights/periodic-gathering-xvzjr-ggg8w"
Nov 02 19:28:10.276881 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:10.276867    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4874b345-eac7-4694-8351-d24085bf180a-trusted-ca-bundle\") pod \"periodic-gathering-xvzjr-ggg8w\" (UID: \"4874b345-eac7-4694-8351-d24085bf180a\") " pod="openshift-insights/periodic-gathering-xvzjr-ggg8w"
Nov 02 19:28:10.315673 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:10.315639    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-rnxqc\""
Nov 02 19:28:10.315673 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:10.315646    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 02 19:28:10.378108 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:10.378074    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4874b345-eac7-4694-8351-d24085bf180a-service-ca-bundle\") pod \"periodic-gathering-xvzjr-ggg8w\" (UID: \"4874b345-eac7-4694-8351-d24085bf180a\") " pod="openshift-insights/periodic-gathering-xvzjr-ggg8w"
Nov 02 19:28:10.378250 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:10.378115    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jz8qt\" (UniqueName: \"kubernetes.io/projected/4874b345-eac7-4694-8351-d24085bf180a-kube-api-access-jz8qt\") pod \"periodic-gathering-xvzjr-ggg8w\" (UID: \"4874b345-eac7-4694-8351-d24085bf180a\") " pod="openshift-insights/periodic-gathering-xvzjr-ggg8w"
Nov 02 19:28:10.378250 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:10.378164    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4874b345-eac7-4694-8351-d24085bf180a-archives-path\") pod \"periodic-gathering-xvzjr-ggg8w\" (UID: \"4874b345-eac7-4694-8351-d24085bf180a\") " pod="openshift-insights/periodic-gathering-xvzjr-ggg8w"
Nov 02 19:28:10.378250 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:10.378189    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4874b345-eac7-4694-8351-d24085bf180a-trusted-ca-bundle\") pod \"periodic-gathering-xvzjr-ggg8w\" (UID: \"4874b345-eac7-4694-8351-d24085bf180a\") " pod="openshift-insights/periodic-gathering-xvzjr-ggg8w"
Nov 02 19:28:10.378597 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:10.378570    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4874b345-eac7-4694-8351-d24085bf180a-archives-path\") pod \"periodic-gathering-xvzjr-ggg8w\" (UID: \"4874b345-eac7-4694-8351-d24085bf180a\") " pod="openshift-insights/periodic-gathering-xvzjr-ggg8w"
Nov 02 19:28:10.378947 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:10.378895    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4874b345-eac7-4694-8351-d24085bf180a-service-ca-bundle\") pod \"periodic-gathering-xvzjr-ggg8w\" (UID: \"4874b345-eac7-4694-8351-d24085bf180a\") " pod="openshift-insights/periodic-gathering-xvzjr-ggg8w"
Nov 02 19:28:10.379149 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:10.379122    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4874b345-eac7-4694-8351-d24085bf180a-trusted-ca-bundle\") pod \"periodic-gathering-xvzjr-ggg8w\" (UID: \"4874b345-eac7-4694-8351-d24085bf180a\") " pod="openshift-insights/periodic-gathering-xvzjr-ggg8w"
Nov 02 19:28:10.471817 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:10.471757    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jz8qt\" (UniqueName: \"kubernetes.io/projected/4874b345-eac7-4694-8351-d24085bf180a-kube-api-access-jz8qt\") pod \"periodic-gathering-xvzjr-ggg8w\" (UID: \"4874b345-eac7-4694-8351-d24085bf180a\") " pod="openshift-insights/periodic-gathering-xvzjr-ggg8w"
Nov 02 19:28:10.514680 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:10.514650    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-xvzjr-ggg8w"
Nov 02 19:28:10.539479 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:10.539452    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-zqz7w"]
Nov 02 19:28:10.563110 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:10.563084    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-zqz7w"
Nov 02 19:28:10.593649 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:10.593622    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-zqz7w"]
Nov 02 19:28:10.618774 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:10.618746    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 02 19:28:10.618904 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:10.618812    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 02 19:28:10.618904 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:10.618819    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-m9rmj\""
Nov 02 19:28:10.680733 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:10.680110    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bh65q\" (UID: \"0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2\") " pod="openshift-insights/insights-runtime-extractor-bh65q"
Nov 02 19:28:10.680733 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:10.680156    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bh65q\" (UID: \"0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2\") " pod="openshift-insights/insights-runtime-extractor-bh65q"
Nov 02 19:28:10.680733 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:10.680190    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcgjq\" (UniqueName: \"kubernetes.io/projected/54c1de40-ce9d-4d7d-82ca-c5fa41c97e97-kube-api-access-fcgjq\") pod \"migrator-66df5d496b-zqz7w\" (UID: \"54c1de40-ce9d-4d7d-82ca-c5fa41c97e97\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-zqz7w"
Nov 02 19:28:10.680733 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:28:10.680337    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 02 19:28:10.680733 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:28:10.680382    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2-kube-rbac-proxy-cm podName:0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2 nodeName:}" failed. No retries permitted until 2026-11-02 19:28:11.680368913 +0000 UTC m=+38.708643223 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bh65q" (UID: "0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2") : configmap "kube-rbac-proxy" not found
Nov 02 19:28:10.680733 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:28:10.680665    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 19:28:10.680733 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:28:10.680700    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2-insights-runtime-extractor-tls podName:0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2 nodeName:}" failed. No retries permitted until 2026-11-02 19:28:11.680689071 +0000 UTC m=+38.708963381 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bh65q" (UID: "0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2") : secret "insights-runtime-extractor-tls" not found
Nov 02 19:28:10.781368 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:10.781297    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcgjq\" (UniqueName: \"kubernetes.io/projected/54c1de40-ce9d-4d7d-82ca-c5fa41c97e97-kube-api-access-fcgjq\") pod \"migrator-66df5d496b-zqz7w\" (UID: \"54c1de40-ce9d-4d7d-82ca-c5fa41c97e97\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-zqz7w"
Nov 02 19:28:10.841616 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:10.841578    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcgjq\" (UniqueName: \"kubernetes.io/projected/54c1de40-ce9d-4d7d-82ca-c5fa41c97e97-kube-api-access-fcgjq\") pod \"migrator-66df5d496b-zqz7w\" (UID: \"54c1de40-ce9d-4d7d-82ca-c5fa41c97e97\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-zqz7w"
Nov 02 19:28:10.896776 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:10.896745    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-zqz7w"
Nov 02 19:28:10.903981 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:28:10.903953    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4874b345_eac7_4694_8351_d24085bf180a.slice/crio-c8861ce03831186f570d157c4d607cc643f2909b90a3a4e9d33fcf0b3dec3570 WatchSource:0}: Error finding container c8861ce03831186f570d157c4d607cc643f2909b90a3a4e9d33fcf0b3dec3570: Status 404 returned error can't find the container with id c8861ce03831186f570d157c4d607cc643f2909b90a3a4e9d33fcf0b3dec3570
Nov 02 19:28:10.906160 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:10.906102    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-xvzjr-ggg8w"]
Nov 02 19:28:11.056912 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:11.056827    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-xvzjr-ggg8w" event={"ID":"4874b345-eac7-4694-8351-d24085bf180a","Type":"ContainerStarted","Data":"c8861ce03831186f570d157c4d607cc643f2909b90a3a4e9d33fcf0b3dec3570"}
Nov 02 19:28:11.292584 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:28:11.292550    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54c1de40_ce9d_4d7d_82ca_c5fa41c97e97.slice/crio-205f9a1cdfee951b8ab921391eb6de239e18105e14febb2b60c181b59f644e32 WatchSource:0}: Error finding container 205f9a1cdfee951b8ab921391eb6de239e18105e14febb2b60c181b59f644e32: Status 404 returned error can't find the container with id 205f9a1cdfee951b8ab921391eb6de239e18105e14febb2b60c181b59f644e32
Nov 02 19:28:11.295551 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:11.295522    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-zqz7w"]
Nov 02 19:28:11.490028 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:11.489987    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-h5fc9"]
Nov 02 19:28:11.509969 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:11.509942    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-h5fc9"
Nov 02 19:28:11.546690 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:11.546658    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-h5fc9"]
Nov 02 19:28:11.565868 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:11.565841    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Nov 02 19:28:11.565868 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:11.565851    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Nov 02 19:28:11.566097 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:11.565873    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Nov 02 19:28:11.566097 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:11.565900    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-g68w8\""
Nov 02 19:28:11.566097 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:11.565916    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Nov 02 19:28:11.690216 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:11.690176    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bh65q\" (UID: \"0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2\") " pod="openshift-insights/insights-runtime-extractor-bh65q"
Nov 02 19:28:11.690375 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:11.690274    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/28b76ecd-8da7-428f-a825-323670ae5395-signing-cabundle\") pod \"service-ca-5fffc97bdf-h5fc9\" (UID: \"28b76ecd-8da7-428f-a825-323670ae5395\") " pod="openshift-service-ca/service-ca-5fffc97bdf-h5fc9"
Nov 02 19:28:11.690375 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:11.690317    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n45ql\" (UniqueName: \"kubernetes.io/projected/28b76ecd-8da7-428f-a825-323670ae5395-kube-api-access-n45ql\") pod \"service-ca-5fffc97bdf-h5fc9\" (UID: \"28b76ecd-8da7-428f-a825-323670ae5395\") " pod="openshift-service-ca/service-ca-5fffc97bdf-h5fc9"
Nov 02 19:28:11.690375 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:11.690355    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/28b76ecd-8da7-428f-a825-323670ae5395-signing-key\") pod \"service-ca-5fffc97bdf-h5fc9\" (UID: \"28b76ecd-8da7-428f-a825-323670ae5395\") " pod="openshift-service-ca/service-ca-5fffc97bdf-h5fc9"
Nov 02 19:28:11.690508 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:11.690389    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bh65q\" (UID: \"0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2\") " pod="openshift-insights/insights-runtime-extractor-bh65q"
Nov 02 19:28:11.690508 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:28:11.690484    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 02 19:28:11.690589 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:28:11.690545    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2-kube-rbac-proxy-cm podName:0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2 nodeName:}" failed. No retries permitted until 2026-11-02 19:28:13.69052673 +0000 UTC m=+40.718801051 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bh65q" (UID: "0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2") : configmap "kube-rbac-proxy" not found
Nov 02 19:28:11.690654 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:28:11.690633    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 19:28:11.690692 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:28:11.690667    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2-insights-runtime-extractor-tls podName:0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2 nodeName:}" failed. No retries permitted until 2026-11-02 19:28:13.690656666 +0000 UTC m=+40.718930977 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bh65q" (UID: "0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2") : secret "insights-runtime-extractor-tls" not found
Nov 02 19:28:11.791303 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:11.791276    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/28b76ecd-8da7-428f-a825-323670ae5395-signing-cabundle\") pod \"service-ca-5fffc97bdf-h5fc9\" (UID: \"28b76ecd-8da7-428f-a825-323670ae5395\") " pod="openshift-service-ca/service-ca-5fffc97bdf-h5fc9"
Nov 02 19:28:11.791409 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:11.791321    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n45ql\" (UniqueName: \"kubernetes.io/projected/28b76ecd-8da7-428f-a825-323670ae5395-kube-api-access-n45ql\") pod \"service-ca-5fffc97bdf-h5fc9\" (UID: \"28b76ecd-8da7-428f-a825-323670ae5395\") " pod="openshift-service-ca/service-ca-5fffc97bdf-h5fc9"
Nov 02 19:28:11.791409 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:11.791355    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/28b76ecd-8da7-428f-a825-323670ae5395-signing-key\") pod \"service-ca-5fffc97bdf-h5fc9\" (UID: \"28b76ecd-8da7-428f-a825-323670ae5395\") " pod="openshift-service-ca/service-ca-5fffc97bdf-h5fc9"
Nov 02 19:28:11.792467 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:11.792416    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/28b76ecd-8da7-428f-a825-323670ae5395-signing-cabundle\") pod \"service-ca-5fffc97bdf-h5fc9\" (UID: \"28b76ecd-8da7-428f-a825-323670ae5395\") " pod="openshift-service-ca/service-ca-5fffc97bdf-h5fc9"
Nov 02 19:28:11.793861 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:11.793838    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/28b76ecd-8da7-428f-a825-323670ae5395-signing-key\") pod \"service-ca-5fffc97bdf-h5fc9\" (UID: \"28b76ecd-8da7-428f-a825-323670ae5395\") " pod="openshift-service-ca/service-ca-5fffc97bdf-h5fc9"
Nov 02 19:28:11.852303 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:11.852278    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n45ql\" (UniqueName: \"kubernetes.io/projected/28b76ecd-8da7-428f-a825-323670ae5395-kube-api-access-n45ql\") pod \"service-ca-5fffc97bdf-h5fc9\" (UID: \"28b76ecd-8da7-428f-a825-323670ae5395\") " pod="openshift-service-ca/service-ca-5fffc97bdf-h5fc9"
Nov 02 19:28:12.062206 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:12.062126    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rhdd6" event={"ID":"31a69a0e-cc9c-47ad-b577-36b50d61d213","Type":"ContainerStarted","Data":"522b0448f1c15dccd09627f810ff00842f115f8c793b15597a4a6a8bea0f3986"}
Nov 02 19:28:12.062627 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:12.062215    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-rhdd6"
Nov 02 19:28:12.068986 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:12.064894    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-zqz7w" event={"ID":"54c1de40-ce9d-4d7d-82ca-c5fa41c97e97","Type":"ContainerStarted","Data":"205f9a1cdfee951b8ab921391eb6de239e18105e14febb2b60c181b59f644e32"}
Nov 02 19:28:12.141241 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:12.141211    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-h5fc9"
Nov 02 19:28:12.181286 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:12.181211    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-rhdd6" podStartSLOduration=33.895276508 podStartE2EDuration="38.181192591s" totalImagesPullingTime="4.285916083s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:27:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:28:12.18046202 +0000 UTC m=+39.208736339" watchObservedRunningTime="2026-11-02 19:28:12.181192591 +0000 UTC m=+39.209466910"
Nov 02 19:28:12.530119 ip-10-0-133-9 kubenswrapper[2579]: W1102 19:28:12.530086    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28b76ecd_8da7_428f_a825_323670ae5395.slice/crio-625b7eb489e12552e74b6ff8823e0dffd3c14fb16c148ef7f9f401a4327db444 WatchSource:0}: Error finding container 625b7eb489e12552e74b6ff8823e0dffd3c14fb16c148ef7f9f401a4327db444: Status 404 returned error can't find the container with id 625b7eb489e12552e74b6ff8823e0dffd3c14fb16c148ef7f9f401a4327db444
Nov 02 19:28:12.533457 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:12.533384    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-h5fc9"]
Nov 02 19:28:12.697522 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:12.697492    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dfb225e8-fabd-49fb-9763-4e41bf6ae7de-metrics-tls\") pod \"dns-default-qj8mr\" (UID: \"dfb225e8-fabd-49fb-9763-4e41bf6ae7de\") " pod="openshift-dns/dns-default-qj8mr"
Nov 02 19:28:12.697689 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:28:12.697632    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 19:28:12.697689 ip-10-0-133-9 kubenswrapper[2579]: E1102 19:28:12.697688    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfb225e8-fabd-49fb-9763-4e41bf6ae7de-metrics-tls podName:dfb225e8-fabd-49fb-9763-4e41bf6ae7de nodeName:}" failed. No retries permitted until 2026-11-02 19:28:20.697674058 +0000 UTC m=+47.725948377 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dfb225e8-fabd-49fb-9763-4e41bf6ae7de-metrics-tls") pod "dns-default-qj8mr" (UID: "dfb225e8-fabd-49fb-9763-4e41bf6ae7de") : secret "dns-default-metrics-tls" not found
Nov 02 19:28:13.069554 ip-10-0-133-9 kubenswrapper[2579]: I1102 19:28:13.069526    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-h5fc9" event={"ID":"28b76ecd-8da7-428f-a825-323670ae5395","Type":"ContainerStarted","Data":"625b7eb489e12552e74b6ff8823e0dffd3c14fb16c148ef7f9f401a4327db444"}
Nov 02 19:28:13.315477 ip-10-0-133-9 systemd[1]: Stopping Kubernetes Kubelet...
Nov 02 19:28:13.331452 ip-10-0-133-9 systemd[1]: kubelet.service: Deactivated successfully.
Nov 02 19:28:13.331681 ip-10-0-133-9 systemd[1]: Stopped Kubernetes Kubelet.
Nov 02 19:28:13.331739 ip-10-0-133-9 systemd[1]: kubelet.service: Consumed 2.296s CPU time.
Nov 02 19:28:13.344537 ip-10-0-133-9 systemd[1]: Starting Kubernetes Kubelet...
Nov 02 19:28:13.448730 ip-10-0-133-9 kubenswrapper[4463]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 19:28:13.448730 ip-10-0-133-9 kubenswrapper[4463]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 02 19:28:13.448730 ip-10-0-133-9 kubenswrapper[4463]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 19:28:13.448730 ip-10-0-133-9 kubenswrapper[4463]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 19:28:13.451193 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451102    4463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 19:28:13.451193 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451191    4463 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 19:28:13.451193 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451198    4463 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 19:28:13.451193 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451202    4463 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 19:28:13.451402 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451207    4463 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 19:28:13.451402 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451212    4463 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 19:28:13.451402 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451216    4463 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 19:28:13.451402 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451221    4463 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 19:28:13.451402 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451224    4463 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 19:28:13.451402 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451226    4463 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 19:28:13.451402 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451230    4463 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 19:28:13.451402 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451233    4463 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 19:28:13.451402 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451238    4463 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 19:28:13.451402 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451244    4463 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 19:28:13.451402 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451248    4463 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 19:28:13.451402 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451252    4463 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 19:28:13.451402 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451257    4463 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 19:28:13.451402 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451261    4463 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 19:28:13.451402 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451265    4463 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 19:28:13.451402 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451270    4463 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 19:28:13.451402 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451274    4463 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 19:28:13.451402 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451278    4463 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 19:28:13.451402 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451283    4463 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 19:28:13.452088 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451286    4463 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 19:28:13.452088 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451289    4463 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 19:28:13.452088 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451291    4463 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 19:28:13.452088 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451298    4463 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 19:28:13.452088 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451301    4463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 19:28:13.452088 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451304    4463 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 19:28:13.452088 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451307    4463 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 19:28:13.452088 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451309    4463 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 19:28:13.452088 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451312    4463 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 19:28:13.452088 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451314    4463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 19:28:13.452088 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451317    4463 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 19:28:13.452088 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451320    4463 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 19:28:13.452088 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451322    4463 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 19:28:13.452088 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451325    4463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 19:28:13.452088 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451329    4463 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 19:28:13.452088 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451331    4463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 19:28:13.452088 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451334    4463 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 19:28:13.452088 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451337    4463 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 19:28:13.452088 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451339    4463 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 19:28:13.452802 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451342    4463 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 19:28:13.452802 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451345    4463 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 19:28:13.452802 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451347    4463 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 19:28:13.452802 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451350    4463 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 19:28:13.452802 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451353    4463 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 19:28:13.452802 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451356    4463 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 19:28:13.452802 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451359    4463 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 19:28:13.452802 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451361    4463 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 19:28:13.452802 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451364    4463 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 19:28:13.452802 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451367    4463 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 19:28:13.452802 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451370    4463 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 19:28:13.452802 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451372    4463 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 19:28:13.452802 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451375    4463 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 19:28:13.452802 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451377    4463 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 19:28:13.452802 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451380    4463 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 19:28:13.452802 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451382    4463 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 19:28:13.452802 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451385    4463 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 19:28:13.452802 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451390    4463 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 19:28:13.452802 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451396    4463 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 19:28:13.452802 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451400    4463 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 19:28:13.453508 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451405    4463 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 19:28:13.453508 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451413    4463 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 19:28:13.453508 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451420    4463 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 19:28:13.453508 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451425    4463 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 19:28:13.453508 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451430    4463 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 19:28:13.453508 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451435    4463 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 19:28:13.453508 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451440    4463 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 19:28:13.453508 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451444    4463 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 19:28:13.453508 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451447    4463 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 19:28:13.453508 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451450    4463 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 19:28:13.453508 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451453    4463 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 19:28:13.453508 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451456    4463 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 19:28:13.453508 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451460    4463 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 19:28:13.453508 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451464    4463 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 19:28:13.453508 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451469    4463 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 19:28:13.453508 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451474    4463 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 19:28:13.453508 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451479    4463 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 19:28:13.453508 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451483    4463 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 19:28:13.453508 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451487    4463 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 19:28:13.454132 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451492    4463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 19:28:13.454132 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451496    4463 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 19:28:13.454132 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451501    4463 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 19:28:13.454132 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451505    4463 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 19:28:13.454132 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451510    4463 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 19:28:13.454132 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451513    4463 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 19:28:13.454132 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451515    4463 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 19:28:13.454132 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451518    4463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 19:28:13.454132 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451521    4463 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 19:28:13.454132 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451524    4463 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 19:28:13.454132 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451526    4463 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 19:28:13.454132 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451529    4463 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 19:28:13.454132 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451531    4463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 19:28:13.454132 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451534    4463 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 19:28:13.454132 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451538    4463 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 19:28:13.454132 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451542    4463 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 19:28:13.454132 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451545    4463 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 19:28:13.454132 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451547    4463 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 19:28:13.454132 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451550    4463 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 19:28:13.454132 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451554    4463 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 19:28:13.454801 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451558    4463 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 19:28:13.454801 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451562    4463 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 19:28:13.454801 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451566    4463 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 19:28:13.454801 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451570    4463 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 19:28:13.454801 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.451574    4463 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 19:28:13.454801 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452547    4463 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 19:28:13.454801 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452556    4463 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 19:28:13.454801 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452559    4463 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 19:28:13.454801 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452564    4463 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 19:28:13.454801 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452566    4463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 19:28:13.454801 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452569    4463 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 19:28:13.454801 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452572    4463 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 19:28:13.454801 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452574    4463 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 19:28:13.454801 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452577    4463 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 19:28:13.454801 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452580    4463 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 19:28:13.454801 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452582    4463 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 19:28:13.454801 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452585    4463 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 19:28:13.454801 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452588    4463 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 19:28:13.454801 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452591    4463 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 19:28:13.454801 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452594    4463 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 19:28:13.455431 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452596    4463 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 19:28:13.455431 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452599    4463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 19:28:13.455431 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452602    4463 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 19:28:13.455431 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452604    4463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 19:28:13.455431 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452607    4463 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 19:28:13.455431 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452610    4463 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 19:28:13.455431 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452614    4463 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 19:28:13.455431 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452617    4463 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 19:28:13.455431 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452621    4463 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 19:28:13.455431 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452628    4463 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 19:28:13.455431 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452632    4463 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 19:28:13.455431 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452637    4463 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 19:28:13.455431 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452641    4463 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 19:28:13.455431 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452645    4463 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 19:28:13.455431 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452648    4463 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 19:28:13.455431 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452651    4463 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 19:28:13.455431 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452653    4463 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 19:28:13.455431 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452657    4463 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 19:28:13.455431 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452661    4463 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 19:28:13.455431 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452665    4463 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 19:28:13.456097 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452669    4463 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 19:28:13.456097 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452674    4463 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 19:28:13.456097 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452678    4463 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 19:28:13.456097 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452681    4463 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 19:28:13.456097 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452683    4463 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 19:28:13.456097 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452686    4463 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 19:28:13.456097 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452688    4463 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 19:28:13.456097 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452694    4463 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 19:28:13.456097 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452701    4463 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 19:28:13.456097 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452706    4463 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 19:28:13.456097 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452710    4463 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 19:28:13.456097 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452713    4463 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 19:28:13.456097 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452716    4463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 19:28:13.456097 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452719    4463 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 19:28:13.456097 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452722    4463 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 19:28:13.456097 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452725    4463 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 19:28:13.456097 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452727    4463 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 19:28:13.456097 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452730    4463 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 19:28:13.456097 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452735    4463 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 19:28:13.456722 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452739    4463 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 19:28:13.456722 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452743    4463 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 19:28:13.456722 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452749    4463 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 19:28:13.456722 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452775    4463 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 19:28:13.456722 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452783    4463 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 19:28:13.456722 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452789    4463 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 19:28:13.456722 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452834    4463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 19:28:13.456722 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452841    4463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 19:28:13.456722 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452845    4463 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 19:28:13.456722 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452848    4463 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 19:28:13.456722 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452851    4463 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 19:28:13.456722 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452855    4463 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 19:28:13.456722 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452859    4463 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 19:28:13.456722 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452864    4463 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 19:28:13.456722 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452868    4463 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 19:28:13.456722 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452873    4463 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 19:28:13.456722 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452875    4463 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 19:28:13.456722 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452878    4463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 19:28:13.456722 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452881    4463 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 19:28:13.457340 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452883    4463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 19:28:13.457340 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452886    4463 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 19:28:13.457340 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452889    4463 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 19:28:13.457340 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452894    4463 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 19:28:13.457340 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452898    4463 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 19:28:13.457340 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452901    4463 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 19:28:13.457340 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452905    4463 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 19:28:13.457340 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452909    4463 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 19:28:13.457340 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452913    4463 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 19:28:13.457340 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452917    4463 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 19:28:13.457340 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452920    4463 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 19:28:13.457340 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452924    4463 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 19:28:13.457340 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452931    4463 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 19:28:13.457340 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452936    4463 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 19:28:13.457340 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452940    4463 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 19:28:13.457340 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452947    4463 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 19:28:13.457340 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452950    4463 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 19:28:13.457340 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452953    4463 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 19:28:13.457340 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452955    4463 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 19:28:13.457340 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452958    4463 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 19:28:13.457926 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452962    4463 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 19:28:13.457926 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452967    4463 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 19:28:13.457926 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452971    4463 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 19:28:13.457926 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452975    4463 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 19:28:13.457926 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452980    4463 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 19:28:13.457926 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452984    4463 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 19:28:13.457926 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452988    4463 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 19:28:13.457926 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452992    4463 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 19:28:13.457926 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.452996    4463 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 19:28:13.457926 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453001    4463 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 19:28:13.457926 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453005    4463 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 19:28:13.457926 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453021    4463 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 19:28:13.457926 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453026    4463 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 19:28:13.457926 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453679    4463 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 19:28:13.457926 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453686    4463 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 19:28:13.457926 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453690    4463 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 19:28:13.457926 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453693    4463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 19:28:13.457926 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453696    4463 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 19:28:13.457926 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453700    4463 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 19:28:13.458567 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453705    4463 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 19:28:13.458567 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453708    4463 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 19:28:13.458567 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453711    4463 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 19:28:13.458567 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453714    4463 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 19:28:13.458567 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453717    4463 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 19:28:13.458567 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453720    4463 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 19:28:13.458567 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453723    4463 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 19:28:13.458567 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453726    4463 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 19:28:13.458567 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453729    4463 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 19:28:13.458567 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453733    4463 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 19:28:13.458567 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453738    4463 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 19:28:13.458567 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453742    4463 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 19:28:13.458567 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453746    4463 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 19:28:13.458567 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453751    4463 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 19:28:13.458567 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453756    4463 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 19:28:13.458567 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453761    4463 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 19:28:13.458567 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453764    4463 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 19:28:13.458567 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453767    4463 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 19:28:13.458567 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453770    4463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 19:28:13.458567 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453772    4463 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 19:28:13.459206 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453775    4463 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 19:28:13.459206 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453777    4463 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 19:28:13.459206 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453781    4463 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 19:28:13.459206 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453784    4463 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 19:28:13.459206 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453786    4463 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 19:28:13.459206 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453789    4463 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 19:28:13.459206 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453791    4463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 19:28:13.459206 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453794    4463 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 19:28:13.459206 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453796    4463 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 19:28:13.459206 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453799    4463 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 19:28:13.459206 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453803    4463 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 19:28:13.459206 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453806    4463 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 19:28:13.459206 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453810    4463 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 19:28:13.459206 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453814    4463 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 19:28:13.459206 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453819    4463 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 19:28:13.459206 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453823    4463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 19:28:13.459206 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453827    4463 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 19:28:13.459206 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453830    4463 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 19:28:13.459206 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453833    4463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 19:28:13.459823 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453835    4463 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 19:28:13.459823 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453839    4463 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 19:28:13.459823 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453842    4463 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 19:28:13.459823 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453844    4463 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 19:28:13.459823 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453847    4463 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 19:28:13.459823 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453850    4463 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 19:28:13.459823 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453853    4463 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 19:28:13.459823 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453855    4463 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 19:28:13.459823 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453858    4463 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 19:28:13.459823 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453860    4463 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 19:28:13.459823 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453863    4463 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 19:28:13.459823 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453867    4463 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 19:28:13.459823 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453871    4463 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 19:28:13.459823 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453874    4463 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 19:28:13.459823 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453878    4463 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 19:28:13.459823 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453881    4463 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 19:28:13.459823 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453885    4463 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 19:28:13.459823 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453889    4463 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 19:28:13.459823 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453893    4463 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 19:28:13.459823 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453897    4463 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 19:28:13.460478 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453900    4463 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 19:28:13.460478 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453903    4463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 19:28:13.460478 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453906    4463 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 19:28:13.460478 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453908    4463 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 19:28:13.460478 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453911    4463 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 19:28:13.460478 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453914    4463 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 19:28:13.460478 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453917    4463 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 19:28:13.460478 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453919    4463 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 19:28:13.460478 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453922    4463 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 19:28:13.460478 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453925    4463 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 19:28:13.460478 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453928    4463 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 19:28:13.460478 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453930    4463 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 19:28:13.460478 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453932    4463 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 19:28:13.460478 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453935    4463 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 19:28:13.460478 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453939    4463 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 19:28:13.460478 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453943    4463 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 19:28:13.460478 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453946    4463 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 19:28:13.460478 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453948    4463 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 19:28:13.460478 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453951    4463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 19:28:13.461101 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453954    4463 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 19:28:13.461101 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453956    4463 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 19:28:13.461101 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453960    4463 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 19:28:13.461101 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453964    4463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 19:28:13.461101 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453969    4463 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 19:28:13.461101 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453973    4463 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 19:28:13.461101 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453977    4463 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 19:28:13.461101 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453980    4463 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 19:28:13.461101 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453983    4463 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 19:28:13.461101 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453985    4463 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 19:28:13.461101 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453989    4463 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 19:28:13.461101 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453991    4463 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 19:28:13.461101 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453995    4463 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 19:28:13.461101 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.453998    4463 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 19:28:13.461101 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.454001    4463 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 19:28:13.461101 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.454003    4463 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 19:28:13.461101 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.454006    4463 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 19:28:13.461101 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.454008    4463 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 19:28:13.461101 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.454026    4463 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 19:28:13.461693 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.454030    4463 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 19:28:13.461693 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.454034    4463 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 19:28:13.461693 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.454038    4463 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 19:28:13.461693 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454155    4463 flags.go:64] FLAG: --address="0.0.0.0"
Nov 02 19:28:13.461693 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454164    4463 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 02 19:28:13.461693 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454170    4463 flags.go:64] FLAG: --anonymous-auth="true"
Nov 02 19:28:13.461693 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454175    4463 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 02 19:28:13.461693 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454180    4463 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 02 19:28:13.461693 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454184    4463 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 02 19:28:13.461693 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454200    4463 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 02 19:28:13.461693 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454207    4463 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 02 19:28:13.461693 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454210    4463 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 02 19:28:13.461693 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454213    4463 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 02 19:28:13.461693 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454216    4463 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 02 19:28:13.461693 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454220    4463 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 02 19:28:13.461693 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454223    4463 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 02 19:28:13.461693 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454226    4463 flags.go:64] FLAG: --cgroup-root=""
Nov 02 19:28:13.461693 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454228    4463 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 02 19:28:13.461693 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454231    4463 flags.go:64] FLAG: --client-ca-file=""
Nov 02 19:28:13.461693 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454234    4463 flags.go:64] FLAG: --cloud-provider="external"
Nov 02 19:28:13.461693 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454237    4463 flags.go:64] FLAG: --cluster-dns="[]"
Nov 02 19:28:13.461693 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454242    4463 flags.go:64] FLAG: --cluster-domain=""
Nov 02 19:28:13.461693 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454245    4463 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 02 19:28:13.461693 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454248    4463 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 02 19:28:13.462402 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454252    4463 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 02 19:28:13.462402 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454255    4463 flags.go:64] FLAG: --container-log-max-files="5"
Nov 02 19:28:13.462402 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454261    4463 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 02 19:28:13.462402 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454265    4463 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 02 19:28:13.462402 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454270    4463 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 02 19:28:13.462402 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454275    4463 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 02 19:28:13.462402 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454280    4463 flags.go:64] FLAG: --contention-profiling="false"
Nov 02 19:28:13.462402 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454285    4463 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 02 19:28:13.462402 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454290    4463 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 02 19:28:13.462402 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454296    4463 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 02 19:28:13.462402 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454299    4463 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 02 19:28:13.462402 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454303    4463 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 02 19:28:13.462402 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454306    4463 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 02 19:28:13.462402 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454309    4463 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 02 19:28:13.462402 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454314    4463 flags.go:64] FLAG: --enable-load-reader="false"
Nov 02 19:28:13.462402 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454317    4463 flags.go:64] FLAG: --enable-server="true"
Nov 02 19:28:13.462402 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454320    4463 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 02 19:28:13.462402 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454324    4463 flags.go:64] FLAG: --event-burst="100"
Nov 02 19:28:13.462402 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454327    4463 flags.go:64] FLAG: --event-qps="50"
Nov 02 19:28:13.462402 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454330    4463 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 02 19:28:13.462402 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454333    4463 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 02 19:28:13.462402 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454337    4463 flags.go:64] FLAG: --eviction-hard=""
Nov 02 19:28:13.462402 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454344    4463 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 02 19:28:13.462402 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454349    4463 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 02 19:28:13.463118 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454354    4463 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 02 19:28:13.463118 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454359    4463 flags.go:64] FLAG: --eviction-soft=""
Nov 02 19:28:13.463118 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454365    4463 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 02 19:28:13.463118 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454368    4463 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 02 19:28:13.463118 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454371    4463 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 02 19:28:13.463118 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454374    4463 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 02 19:28:13.463118 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454377    4463 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 02 19:28:13.463118 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454380    4463 flags.go:64] FLAG: --fail-swap-on="true"
Nov 02 19:28:13.463118 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454383    4463 flags.go:64] FLAG: --feature-gates=""
Nov 02 19:28:13.463118 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454388    4463 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 02 19:28:13.463118 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454392    4463 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 02 19:28:13.463118 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454395    4463 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 02 19:28:13.463118 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454398    4463 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 02 19:28:13.463118 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454401    4463 flags.go:64] FLAG: --healthz-port="10248"
Nov 02 19:28:13.463118 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454405    4463 flags.go:64] FLAG: --help="false"
Nov 02 19:28:13.463118 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454407    4463 flags.go:64] FLAG: --hostname-override="ip-10-0-133-9.us-west-2.compute.internal"
Nov 02 19:28:13.463118 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454410    4463 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 02 19:28:13.463118 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454414    4463 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 02 19:28:13.463118 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454417    4463 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 02 19:28:13.463118 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454420    4463 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 02 19:28:13.463118 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454424    4463 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 02 19:28:13.463118 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454426    4463 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 02 19:28:13.463118 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454431    4463 flags.go:64] FLAG: --image-service-endpoint=""
Nov 02 19:28:13.463851 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454434    4463 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 02 19:28:13.463851 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454436    4463 flags.go:64] FLAG: --kube-api-burst="100"
Nov 02 19:28:13.463851 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454439    4463 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 02 19:28:13.463851 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454442    4463 flags.go:64] FLAG: --kube-api-qps="50"
Nov 02 19:28:13.463851 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454446    4463 flags.go:64] FLAG: --kube-reserved=""
Nov 02 19:28:13.463851 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454449    4463 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 02 19:28:13.463851 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454452    4463 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 02 19:28:13.463851 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454455    4463 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 02 19:28:13.463851 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454458    4463 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 02 19:28:13.463851 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454463    4463 flags.go:64] FLAG: --lock-file=""
Nov 02 19:28:13.463851 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454467    4463 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 02 19:28:13.463851 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454472    4463 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 02 19:28:13.463851 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454477    4463 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 02 19:28:13.463851 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454483    4463 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 02 19:28:13.463851 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454486    4463 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 02 19:28:13.463851 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454489    4463 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 02 19:28:13.463851 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454492    4463 flags.go:64] FLAG: --logging-format="text"
Nov 02 19:28:13.463851 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454495    4463 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 02 19:28:13.463851 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454498    4463 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 02 19:28:13.463851 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454505    4463 flags.go:64] FLAG: --manifest-url=""
Nov 02 19:28:13.463851 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454509    4463 flags.go:64] FLAG: --manifest-url-header=""
Nov 02 19:28:13.463851 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454517    4463 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 02 19:28:13.463851 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454522    4463 flags.go:64] FLAG: --max-open-files="1000000"
Nov 02 19:28:13.463851 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454528    4463 flags.go:64] FLAG: --max-pods="110"
Nov 02 19:28:13.463851 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454534    4463 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 02 19:28:13.464875 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454539    4463 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 02 19:28:13.464875 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454543    4463 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 02 19:28:13.464875 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454548    4463 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 02 19:28:13.464875 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454552    4463 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 02 19:28:13.464875 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454555    4463 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 02 19:28:13.464875 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454557    4463 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 02 19:28:13.464875 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454566    4463 flags.go:64] FLAG: --node-status-max-images="50"
Nov 02 19:28:13.464875 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454569    4463 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 02 19:28:13.464875 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454572    4463 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 02 19:28:13.464875 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454577    4463 flags.go:64] FLAG: --pod-cidr=""
Nov 02 19:28:13.464875 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454582    4463 flags.go:64] FLAG: --pod-manifest-path=""
Nov 02 19:28:13.464875 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454586    4463 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 02 19:28:13.464875 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454592    4463 flags.go:64] FLAG: --pods-per-core="0"
Nov 02 19:28:13.464875 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454596    4463 flags.go:64] FLAG: --port="10250"
Nov 02 19:28:13.464875 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454601    4463 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 02 19:28:13.464875 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454607    4463 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0b653f3ce8ab2eeb9"
Nov 02 19:28:13.464875 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454612    4463 flags.go:64] FLAG: --qos-reserved=""
Nov 02 19:28:13.464875 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454617    4463 flags.go:64] FLAG: --read-only-port="10255"
Nov 02 19:28:13.464875 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454622    4463 flags.go:64] FLAG: --register-node="true"
Nov 02 19:28:13.464875 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454626    4463 flags.go:64] FLAG: --register-with-taints=""
Nov 02 19:28:13.464875 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454630    4463 flags.go:64] FLAG: --registry-burst="10"
Nov 02 19:28:13.464875 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454633    4463 flags.go:64] FLAG: --registry-qps="5"
Nov 02 19:28:13.464875 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454636    4463 flags.go:64] FLAG: --reserved-cpus=""
Nov 02 19:28:13.464875 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454638    4463 flags.go:64] FLAG: --reserved-memory=""
Nov 02 19:28:13.464875 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454644    4463 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 02 19:28:13.464875 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454649    4463 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 02 19:28:13.466112 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454656    4463 flags.go:64] FLAG: --rotate-certificates="false"
Nov 02 19:28:13.466112 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454662    4463 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 02 19:28:13.466112 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454665    4463 flags.go:64] FLAG: --runonce="false"
Nov 02 19:28:13.466112 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454668    4463 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 02 19:28:13.466112 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454671    4463 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 02 19:28:13.466112 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454674    4463 flags.go:64] FLAG: --seccomp-default="false"
Nov 02 19:28:13.466112 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454677    4463 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 02 19:28:13.466112 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454680    4463 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 02 19:28:13.466112 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454684    4463 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 02 19:28:13.466112 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454689    4463 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 02 19:28:13.466112 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454694    4463 flags.go:64] FLAG: --storage-driver-password="root"
Nov 02 19:28:13.466112 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454698    4463 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 02 19:28:13.466112 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454703    4463 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 02 19:28:13.466112 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454708    4463 flags.go:64] FLAG: --storage-driver-user="root"
Nov 02 19:28:13.466112 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454713    4463 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 02 19:28:13.466112 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454718    4463 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 02 19:28:13.466112 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454723    4463 flags.go:64] FLAG: --system-cgroups=""
Nov 02 19:28:13.466112 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454728    4463 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 02 19:28:13.466112 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454737    4463 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 02 19:28:13.466112 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454742    4463 flags.go:64] FLAG: --tls-cert-file=""
Nov 02 19:28:13.466112 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454747    4463 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 02 19:28:13.466112 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454754    4463 flags.go:64] FLAG: --tls-min-version=""
Nov 02 19:28:13.466112 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454758    4463 flags.go:64] FLAG: --tls-private-key-file=""
Nov 02 19:28:13.466112 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454763    4463 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 02 19:28:13.466112 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454768    4463 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 02 19:28:13.467088 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454773    4463 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 02 19:28:13.467088 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454778    4463 flags.go:64] FLAG: --v="2"
Nov 02 19:28:13.467088 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454787    4463 flags.go:64] FLAG: --version="false"
Nov 02 19:28:13.467088 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454794    4463 flags.go:64] FLAG: --vmodule=""
Nov 02 19:28:13.467088 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454799    4463 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 02 19:28:13.467088 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.454802    4463 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 02 19:28:13.467088 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.454981    4463 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 19:28:13.467088 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.454991    4463 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 19:28:13.467088 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.454996    4463 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 19:28:13.467088 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455002    4463 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 19:28:13.467088 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455007    4463 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 19:28:13.467088 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455024    4463 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 19:28:13.467088 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455030    4463 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 19:28:13.467088 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455035    4463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 19:28:13.467088 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455040    4463 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 19:28:13.467088 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455045    4463 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 19:28:13.467088 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455050    4463 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 19:28:13.467088 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455056    4463 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 19:28:13.467088 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455061    4463 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 19:28:13.467088 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455069    4463 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 19:28:13.467088 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455074    4463 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 19:28:13.467927 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455078    4463 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 19:28:13.467927 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455083    4463 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 19:28:13.467927 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455086    4463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 19:28:13.467927 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455089    4463 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 19:28:13.467927 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455093    4463 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 19:28:13.467927 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455097    4463 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 19:28:13.467927 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455102    4463 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 19:28:13.467927 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455107    4463 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 19:28:13.467927 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455111    4463 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 19:28:13.467927 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455114    4463 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 19:28:13.467927 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455117    4463 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 19:28:13.467927 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455120    4463 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 19:28:13.467927 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455122    4463 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 19:28:13.467927 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455125    4463 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 19:28:13.467927 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455128    4463 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 19:28:13.467927 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455131    4463 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 19:28:13.467927 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455134    4463 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 19:28:13.467927 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455137    4463 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 19:28:13.467927 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455141    4463 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 19:28:13.468744 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455143    4463 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 19:28:13.468744 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455146    4463 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 19:28:13.468744 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455152    4463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 19:28:13.468744 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455157    4463 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 19:28:13.468744 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455162    4463 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 19:28:13.468744 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455167    4463 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 19:28:13.468744 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455170    4463 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 19:28:13.468744 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455177    4463 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 19:28:13.468744 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455183    4463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 19:28:13.468744 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455188    4463 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 19:28:13.468744 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455192    4463 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 19:28:13.468744 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455202    4463 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 19:28:13.468744 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455206    4463 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 19:28:13.468744 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455210    4463 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 19:28:13.468744 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455214    4463 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 19:28:13.468744 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455219    4463 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 19:28:13.468744 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455223    4463 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 19:28:13.468744 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455227    4463 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 19:28:13.468744 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455231    4463 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 19:28:13.469598 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455235    4463 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 19:28:13.469598 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455238    4463 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 19:28:13.469598 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455240    4463 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 19:28:13.469598 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455243    4463 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 19:28:13.469598 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455245    4463 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 19:28:13.469598 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455248    4463 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 19:28:13.469598 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455251    4463 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 19:28:13.469598 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455253    4463 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 19:28:13.469598 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455256    4463 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 19:28:13.469598 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455258    4463 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 19:28:13.469598 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455261    4463 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 19:28:13.469598 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455265    4463 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 19:28:13.469598 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455271    4463 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 19:28:13.469598 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455275    4463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 19:28:13.469598 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455280    4463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 19:28:13.469598 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455284    4463 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 19:28:13.469598 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455289    4463 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 19:28:13.469598 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455294    4463 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 19:28:13.469598 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455299    4463 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 19:28:13.469598 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455303    4463 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 19:28:13.470335 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455307    4463 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 19:28:13.470335 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455311    4463 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 19:28:13.470335 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455315    4463 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 19:28:13.470335 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455320    4463 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 19:28:13.470335 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455325    4463 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 19:28:13.470335 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455329    4463 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 19:28:13.470335 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455333    4463 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 19:28:13.470335 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455338    4463 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 19:28:13.470335 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455342    4463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 19:28:13.470335 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455345    4463 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 19:28:13.470335 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455348    4463 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 19:28:13.470335 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455351    4463 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 19:28:13.470335 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455354    4463 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 19:28:13.470335 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455356    4463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 19:28:13.470335 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455359    4463 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 19:28:13.470335 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455361    4463 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 19:28:13.470335 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455364    4463 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 19:28:13.470335 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455367    4463 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 19:28:13.470335 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455369    4463 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 19:28:13.470335 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455372    4463 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 19:28:13.471059 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455374    4463 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 19:28:13.471059 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455377    4463 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 19:28:13.471059 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455379    4463 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 19:28:13.471059 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455382    4463 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 19:28:13.471059 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455388    4463 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 19:28:13.471059 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455392    4463 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 19:28:13.471059 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455396    4463 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 19:28:13.471059 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455400    4463 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 19:28:13.471059 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455405    4463 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 19:28:13.471059 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455410    4463 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 19:28:13.471059 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455414    4463 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 19:28:13.471059 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455418    4463 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 19:28:13.471059 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455424    4463 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 19:28:13.471059 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.455460    4463 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 19:28:13.471059 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455501    4463 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 19:28:13.471059 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455508    4463 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 19:28:13.471545 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455516    4463 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 19:28:13.471545 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455521    4463 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 19:28:13.471545 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455527    4463 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 19:28:13.471545 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455532    4463 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 19:28:13.471545 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455537    4463 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 19:28:13.471545 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455542    4463 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 19:28:13.471545 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455546    4463 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 19:28:13.471545 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455551    4463 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 19:28:13.471545 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455554    4463 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 19:28:13.471545 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455557    4463 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 19:28:13.471545 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455560    4463 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 19:28:13.471545 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455562    4463 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 19:28:13.471545 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455565    4463 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 19:28:13.471545 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455568    4463 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 19:28:13.471545 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455571    4463 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 19:28:13.471545 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455573    4463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 19:28:13.471545 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455576    4463 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 19:28:13.471545 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455579    4463 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 19:28:13.471545 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455581    4463 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 19:28:13.472167 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455584    4463 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 19:28:13.472167 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455588    4463 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 19:28:13.472167 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455594    4463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 19:28:13.472167 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455599    4463 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 19:28:13.472167 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455604    4463 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 19:28:13.472167 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455608    4463 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 19:28:13.472167 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455612    4463 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 19:28:13.472167 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455617    4463 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 19:28:13.472167 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455622    4463 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 19:28:13.472167 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455627    4463 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 19:28:13.472167 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455632    4463 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 19:28:13.472167 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455637    4463 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 19:28:13.472167 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455641    4463 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 19:28:13.472167 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455647    4463 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 19:28:13.472167 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455651    4463 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 19:28:13.472167 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455654    4463 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 19:28:13.472167 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455656    4463 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 19:28:13.472167 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455659    4463 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 19:28:13.472167 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455661    4463 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 19:28:13.472725 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455664    4463 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 19:28:13.472725 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455667    4463 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 19:28:13.472725 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455669    4463 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 19:28:13.472725 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455672    4463 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 19:28:13.472725 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455675    4463 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 19:28:13.472725 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455678    4463 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 19:28:13.472725 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455680    4463 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 19:28:13.472725 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455683    4463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 19:28:13.472725 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455686    4463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 19:28:13.472725 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455689    4463 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 19:28:13.472725 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455691    4463 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 19:28:13.472725 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455694    4463 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 19:28:13.472725 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455696    4463 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 19:28:13.472725 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455699    4463 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 19:28:13.472725 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455701    4463 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 19:28:13.472725 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455706    4463 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 19:28:13.472725 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455708    4463 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 19:28:13.472725 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455711    4463 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 19:28:13.472725 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455713    4463 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 19:28:13.473447 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455716    4463 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 19:28:13.473447 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455719    4463 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 19:28:13.473447 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455722    4463 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 19:28:13.473447 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455725    4463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 19:28:13.473447 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455728    4463 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 19:28:13.473447 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455731    4463 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 19:28:13.473447 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455734    4463 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 19:28:13.473447 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455738    4463 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 19:28:13.473447 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455740    4463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 19:28:13.473447 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455743    4463 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 19:28:13.473447 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455745    4463 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 19:28:13.473447 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455748    4463 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 19:28:13.473447 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455751    4463 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 19:28:13.473447 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455753    4463 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 19:28:13.473447 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455756    4463 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 19:28:13.473447 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455758    4463 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 19:28:13.473447 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455761    4463 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 19:28:13.473447 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455764    4463 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 19:28:13.473447 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455766    4463 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 19:28:13.473447 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455769    4463 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 19:28:13.474332 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455773    4463 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 19:28:13.474332 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455777    4463 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 19:28:13.474332 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455781    4463 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 19:28:13.474332 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455785    4463 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 19:28:13.474332 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455789    4463 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 19:28:13.474332 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455792    4463 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 19:28:13.474332 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455794    4463 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 19:28:13.474332 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455797    4463 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 19:28:13.474332 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455802    4463 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 19:28:13.474332 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455805    4463 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 19:28:13.474332 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455807    4463 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 19:28:13.474332 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455810    4463 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 19:28:13.474332 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455813    4463 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 19:28:13.474332 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455817    4463 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 19:28:13.474332 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455822    4463 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 19:28:13.474332 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455827    4463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 19:28:13.474332 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455832    4463 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 19:28:13.474332 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455837    4463 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 19:28:13.474332 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455849    4463 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 19:28:13.474884 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455855    4463 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 19:28:13.474884 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455858    4463 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 19:28:13.474884 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455861    4463 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 19:28:13.474884 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455863    4463 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 19:28:13.474884 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455866    4463 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 19:28:13.474884 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455868    4463 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 19:28:13.474884 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455871    4463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 19:28:13.474884 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.455873    4463 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 19:28:13.474884 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.460121    4463 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 02 19:28:13.474884 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.460135    4463 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 02 19:28:13.474884 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.460154    4463 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 02 19:28:13.474884 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.460162    4463 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 02 19:28:13.474884 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460225    4463 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 19:28:13.474884 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460230    4463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 19:28:13.474884 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460234    4463 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 19:28:13.474884 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460238    4463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 19:28:13.474884 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460240    4463 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 19:28:13.474884 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460243    4463 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 19:28:13.474884 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460246    4463 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 19:28:13.475463 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460249    4463 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 19:28:13.475463 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460252    4463 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 19:28:13.475463 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460254    4463 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 19:28:13.475463 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460265    4463 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 19:28:13.475463 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460270    4463 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 19:28:13.475463 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460276    4463 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 19:28:13.475463 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460280    4463 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 19:28:13.475463 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460286    4463 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 19:28:13.475463 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460290    4463 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 19:28:13.475463 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460295    4463 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 19:28:13.475463 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460298    4463 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 19:28:13.475463 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460301    4463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 19:28:13.475463 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460304    4463 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 19:28:13.475463 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460306    4463 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 19:28:13.475463 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460309    4463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 19:28:13.475463 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460312    4463 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 19:28:13.475463 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460314    4463 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 19:28:13.475463 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460317    4463 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 19:28:13.475463 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460320    4463 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 19:28:13.475463 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460322    4463 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 19:28:13.476094 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460325    4463 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 19:28:13.476094 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460328    4463 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 19:28:13.476094 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460331    4463 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 19:28:13.476094 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460334    4463 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 19:28:13.476094 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460337    4463 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 19:28:13.476094 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460339    4463 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 19:28:13.476094 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460342    4463 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 19:28:13.476094 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460345    4463 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 19:28:13.476094 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460347    4463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 19:28:13.476094 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460350    4463 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 19:28:13.476094 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460352    4463 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 19:28:13.476094 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460355    4463 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 19:28:13.476094 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460358    4463 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 19:28:13.476094 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460360    4463 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 19:28:13.476094 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460363    4463 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 19:28:13.476094 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460365    4463 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 19:28:13.476094 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460368    4463 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 19:28:13.476094 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460371    4463 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 19:28:13.476094 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460375    4463 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 19:28:13.476684 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460379    4463 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 19:28:13.476684 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460384    4463 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 19:28:13.476684 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460390    4463 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 19:28:13.476684 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460395    4463 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 19:28:13.476684 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460400    4463 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 19:28:13.476684 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460405    4463 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 19:28:13.476684 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460408    4463 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 19:28:13.476684 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460411    4463 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 19:28:13.476684 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460413    4463 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 19:28:13.476684 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460416    4463 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 19:28:13.476684 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460419    4463 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 19:28:13.476684 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460422    4463 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 19:28:13.476684 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460425    4463 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 19:28:13.476684 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460427    4463 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 19:28:13.476684 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460430    4463 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 19:28:13.476684 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460433    4463 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 19:28:13.476684 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460436    4463 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 19:28:13.476684 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460439    4463 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 19:28:13.476684 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460442    4463 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 19:28:13.476684 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460446    4463 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 19:28:13.477383 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460450    4463 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 19:28:13.477383 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460454    4463 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 19:28:13.477383 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460459    4463 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 19:28:13.477383 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460467    4463 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 19:28:13.477383 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460474    4463 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 19:28:13.477383 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460478    4463 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 19:28:13.477383 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460480    4463 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 19:28:13.477383 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460484    4463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 19:28:13.477383 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460486    4463 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 19:28:13.477383 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460489    4463 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 19:28:13.477383 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460492    4463 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 19:28:13.477383 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460495    4463 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 19:28:13.477383 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460498    4463 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 19:28:13.477383 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460501    4463 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 19:28:13.477383 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460504    4463 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 19:28:13.477383 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460507    4463 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 19:28:13.477383 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460511    4463 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 19:28:13.477383 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460514    4463 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 19:28:13.477383 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460518    4463 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 19:28:13.477957 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460521    4463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 19:28:13.477957 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460523    4463 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 19:28:13.477957 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460526    4463 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 19:28:13.477957 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460528    4463 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 19:28:13.477957 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460531    4463 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 19:28:13.477957 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460534    4463 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 19:28:13.477957 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460536    4463 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 19:28:13.477957 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460539    4463 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 19:28:13.477957 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460541    4463 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 19:28:13.477957 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460544    4463 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 19:28:13.477957 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460546    4463 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 19:28:13.477957 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460549    4463 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 19:28:13.477957 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460551    4463 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 19:28:13.477957 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460554    4463 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 19:28:13.477957 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460557    4463 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 19:28:13.477957 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460561    4463 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 19:28:13.477957 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460566    4463 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 19:28:13.477957 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460570    4463 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 19:28:13.477957 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460575    4463 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 19:28:13.477957 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460580    4463 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 19:28:13.478536 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460585    4463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 19:28:13.478536 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.460615    4463 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 19:28:13.478536 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460787    4463 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 19:28:13.478536 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460793    4463 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 19:28:13.478536 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460796    4463 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 19:28:13.478536 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460799    4463 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 19:28:13.478536 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460802    4463 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 19:28:13.478536 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460805    4463 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 19:28:13.478536 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460807    4463 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 19:28:13.478536 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460810    4463 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 19:28:13.478536 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460813    4463 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 19:28:13.478536 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460816    4463 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 19:28:13.478536 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460819    4463 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 19:28:13.478536 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460822    4463 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 19:28:13.478536 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460824    4463 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 19:28:13.478536 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460827    4463 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 19:28:13.479090 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460830    4463 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 19:28:13.479090 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460834    4463 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 19:28:13.479090 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460840    4463 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 19:28:13.479090 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460844    4463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 19:28:13.479090 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460849    4463 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 19:28:13.479090 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460852    4463 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 19:28:13.479090 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460856    4463 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 19:28:13.479090 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460861    4463 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 19:28:13.479090 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460866    4463 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 19:28:13.479090 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460870    4463 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 19:28:13.479090 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460874    4463 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 19:28:13.479090 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460879    4463 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 19:28:13.479090 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460883    4463 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 19:28:13.479090 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460888    4463 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 19:28:13.479090 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460892    4463 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 19:28:13.479090 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460896    4463 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 19:28:13.479090 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460900    4463 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 19:28:13.479090 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460905    4463 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 19:28:13.479090 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460909    4463 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 19:28:13.479090 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460913    4463 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 19:28:13.479710 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460918    4463 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 19:28:13.479710 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460922    4463 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 19:28:13.479710 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460925    4463 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 19:28:13.479710 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460927    4463 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 19:28:13.479710 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460930    4463 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 19:28:13.479710 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460932    4463 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 19:28:13.479710 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460935    4463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 19:28:13.479710 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460938    4463 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 19:28:13.479710 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460942    4463 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 19:28:13.479710 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460946    4463 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 19:28:13.479710 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460951    4463 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 19:28:13.479710 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460955    4463 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 19:28:13.479710 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460960    4463 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 19:28:13.479710 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460964    4463 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 19:28:13.479710 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460968    4463 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 19:28:13.479710 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460973    4463 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 19:28:13.479710 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460977    4463 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 19:28:13.479710 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460981    4463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 19:28:13.479710 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460986    4463 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 19:28:13.480286 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460989    4463 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 19:28:13.480286 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460992    4463 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 19:28:13.480286 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.460995    4463 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 19:28:13.480286 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461000    4463 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 19:28:13.480286 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461003    4463 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 19:28:13.480286 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461021    4463 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 19:28:13.480286 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461027    4463 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 19:28:13.480286 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461030    4463 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 19:28:13.480286 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461033    4463 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 19:28:13.480286 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461036    4463 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 19:28:13.480286 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461039    4463 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 19:28:13.480286 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461041    4463 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 19:28:13.480286 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461044    4463 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 19:28:13.480286 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461046    4463 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 19:28:13.480286 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461049    4463 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 19:28:13.480286 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461051    4463 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 19:28:13.480286 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461054    4463 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 19:28:13.480286 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461057    4463 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 19:28:13.480286 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461061    4463 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 19:28:13.480850 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461066    4463 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 19:28:13.480850 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461071    4463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 19:28:13.480850 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461077    4463 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 19:28:13.480850 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461081    4463 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 19:28:13.480850 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461084    4463 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 19:28:13.480850 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461086    4463 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 19:28:13.480850 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461089    4463 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 19:28:13.480850 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461091    4463 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 19:28:13.480850 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461094    4463 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 19:28:13.480850 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461096    4463 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 19:28:13.480850 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461099    4463 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 19:28:13.480850 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461102    4463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 19:28:13.480850 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461104    4463 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 19:28:13.480850 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461107    4463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 19:28:13.480850 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461110    4463 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 19:28:13.480850 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461112    4463 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 19:28:13.480850 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461115    4463 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 19:28:13.480850 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461118    4463 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 19:28:13.480850 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461121    4463 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 19:28:13.481415 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461125    4463 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 19:28:13.481415 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461128    4463 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 19:28:13.481415 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461131    4463 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 19:28:13.481415 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461134    4463 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 19:28:13.481415 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461136    4463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 19:28:13.481415 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461139    4463 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 19:28:13.481415 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461141    4463 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 19:28:13.481415 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461144    4463 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 19:28:13.481415 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461147    4463 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 19:28:13.481415 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461149    4463 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 19:28:13.481415 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461152    4463 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 19:28:13.481415 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461154    4463 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 19:28:13.481415 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461157    4463 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 19:28:13.481415 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461160    4463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 19:28:13.481415 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461162    4463 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 19:28:13.481874 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.461192    4463 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 19:28:13.481874 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461228    4463 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 19:28:13.481874 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461233    4463 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 19:28:13.481874 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461236    4463 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 19:28:13.481874 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461239    4463 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 19:28:13.481874 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461242    4463 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 19:28:13.481874 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461244    4463 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 19:28:13.481874 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461247    4463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 19:28:13.481874 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461250    4463 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 19:28:13.481874 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461253    4463 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 19:28:13.481874 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461255    4463 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 19:28:13.481874 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461258    4463 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 19:28:13.481874 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461260    4463 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 19:28:13.481874 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461263    4463 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 19:28:13.481874 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461266    4463 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 19:28:13.481874 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461269    4463 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 19:28:13.482325 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461272    4463 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 19:28:13.482325 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461275    4463 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 19:28:13.482325 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461277    4463 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 19:28:13.482325 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461280    4463 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 19:28:13.482325 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461282    4463 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 19:28:13.482325 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461285    4463 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 19:28:13.482325 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461287    4463 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 19:28:13.482325 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461290    4463 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 19:28:13.482325 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461293    4463 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 19:28:13.482325 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461295    4463 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 19:28:13.482325 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461298    4463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 19:28:13.482325 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461300    4463 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 19:28:13.482325 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461303    4463 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 19:28:13.482325 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461305    4463 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 19:28:13.482325 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461308    4463 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 19:28:13.482325 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461311    4463 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 19:28:13.482325 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461313    4463 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 19:28:13.482325 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461316    4463 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 19:28:13.482325 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461319    4463 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 19:28:13.482325 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461322    4463 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 19:28:13.482887 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461324    4463 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 19:28:13.482887 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461327    4463 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 19:28:13.482887 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461329    4463 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 19:28:13.482887 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461331    4463 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 19:28:13.482887 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461334    4463 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 19:28:13.482887 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461336    4463 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 19:28:13.482887 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461340    4463 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 19:28:13.482887 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461343    4463 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 19:28:13.482887 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461345    4463 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 19:28:13.482887 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461348    4463 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 19:28:13.482887 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461350    4463 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 19:28:13.482887 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461355    4463 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 19:28:13.482887 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461358    4463 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 19:28:13.482887 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461361    4463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 19:28:13.482887 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461363    4463 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 19:28:13.482887 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461367    4463 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 19:28:13.482887 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461369    4463 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 19:28:13.482887 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461372    4463 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 19:28:13.482887 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461375    4463 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 19:28:13.483461 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461377    4463 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 19:28:13.483461 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461380    4463 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 19:28:13.483461 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461382    4463 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 19:28:13.483461 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461385    4463 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 19:28:13.483461 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461387    4463 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 19:28:13.483461 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461390    4463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 19:28:13.483461 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461393    4463 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 19:28:13.483461 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461395    4463 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 19:28:13.483461 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461398    4463 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 19:28:13.483461 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461400    4463 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 19:28:13.483461 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461403    4463 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 19:28:13.483461 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461405    4463 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 19:28:13.483461 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461409    4463 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 19:28:13.483461 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461413    4463 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 19:28:13.483461 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461416    4463 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 19:28:13.483461 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461419    4463 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 19:28:13.483461 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461422    4463 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 19:28:13.483461 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461424    4463 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 19:28:13.483461 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461427    4463 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 19:28:13.483986 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461429    4463 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 19:28:13.483986 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461432    4463 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 19:28:13.483986 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461434    4463 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 19:28:13.483986 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461437    4463 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 19:28:13.483986 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461439    4463 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 19:28:13.483986 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461443    4463 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 19:28:13.483986 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461446    4463 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 19:28:13.483986 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461449    4463 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 19:28:13.483986 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461451    4463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 19:28:13.483986 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461454    4463 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 19:28:13.483986 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461456    4463 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 19:28:13.483986 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461459    4463 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 19:28:13.483986 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461462    4463 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 19:28:13.483986 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461465    4463 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 19:28:13.483986 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461468    4463 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 19:28:13.483986 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461470    4463 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 19:28:13.483986 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461473    4463 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 19:28:13.483986 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461476    4463 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 19:28:13.483986 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461478    4463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 19:28:13.484561 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461480    4463 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 19:28:13.484561 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461483    4463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 19:28:13.484561 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461486    4463 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 19:28:13.484561 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461489    4463 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 19:28:13.484561 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461491    4463 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 19:28:13.484561 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461494    4463 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 19:28:13.484561 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461496    4463 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 19:28:13.484561 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461501    4463 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 19:28:13.484561 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461505    4463 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 19:28:13.484561 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461508    4463 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 19:28:13.484561 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461511    4463 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 19:28:13.484561 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461513    4463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 19:28:13.484561 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461516    4463 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 19:28:13.484561 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:13.461519    4463 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 19:28:13.484561 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.461573    4463 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 02 19:28:13.484561 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.462565    4463 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 02 19:28:13.484561 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.462625    4463 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 02 19:28:13.484561 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.462822    4463 server.go:1014] "Starting client certificate rotation"
Nov 02 19:28:13.485209 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.462967    4463 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 02 19:28:13.485209 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.463058    4463 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 19:22:34 +0000 UTC" deadline="2028-07-14 07:20:18.545256134 +0000 UTC"
Nov 02 19:28:13.485209 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.463103    4463 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14867h52m5.082155523s"
Nov 02 19:28:13.485209 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.463376    4463 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 19:28:13.485209 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.465628    4463 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 19:28:13.485209 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.467621    4463 log.go:25] "Validated CRI v1 runtime API"
Nov 02 19:28:13.485209 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.469487    4463 log.go:25] "Validated CRI v1 image API"
Nov 02 19:28:13.485209 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.470398    4463 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 02 19:28:13.485209 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.472151    4463 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 96a9f595-0df5-4f03-8f0c-c6da713f0add:/dev/nvme0n1p4 d0dc1a70-3182-4b90-b28d-cb9e1470b581:/dev/nvme0n1p3]
Nov 02 19:28:13.485607 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.472173    4463 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1e383e5f91ee92619c7178649dc6b6d776a9eb3b2e97103bccfdca415dab6887/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1e383e5f91ee92619c7178649dc6b6d776a9eb3b2e97103bccfdca415dab6887/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/205f9a1cdfee951b8ab921391eb6de239e18105e14febb2b60c181b59f644e32/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/205f9a1cdfee951b8ab921391eb6de239e18105e14febb2b60c181b59f644e32/userdata/shm major:0 minor:229 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/29b253e3dff4d86876a235f3c9b0a36d3815e1fb4be3fbdc60497516afaf9b34/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/29b253e3dff4d86876a235f3c9b0a36d3815e1fb4be3fbdc60497516afaf9b34/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2fddafd70439b8e26feb1fff6e97dcb631902ca9cb63bf44a13df77960ad2022/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2fddafd70439b8e26feb1fff6e97dcb631902ca9cb63bf44a13df77960ad2022/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/34ffb39bf1d7fea750bf39bc7b22f4ef70166d88d57a034b4fa561753454caee/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/34ffb39bf1d7fea750bf39bc7b22f4ef70166d88d57a034b4fa561753454caee/userdata/shm major:0 minor:208 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4ce5d46f7311979bfa15d5f696ef882348ffbf80aa8c5acf5c07b8316af4865d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4ce5d46f7311979bfa15d5f696ef882348ffbf80aa8c5acf5c07b8316af4865d/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/625b7eb489e12552e74b6ff8823e0dffd3c14fb16c148ef7f9f401a4327db444/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/625b7eb489e12552e74b6ff8823e0dffd3c14fb16c148ef7f9f401a4327db444/userdata/shm major:0 minor:241 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/655d515e9caa08a107af1d948364e4afa72c74189600ae462600c0de1ee66879/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/655d515e9caa08a107af1d948364e4afa72c74189600ae462600c0de1ee66879/userdata/shm major:0 minor:79 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/667d2758ccf9eaad27f48fb50e5834c02305aa559406fa46b65f01390ffe20bd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/667d2758ccf9eaad27f48fb50e5834c02305aa559406fa46b65f01390ffe20bd/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7655f51aa5471290e3a2ff847995a116fe03885514a5c839f545a75f27a82c0e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7655f51aa5471290e3a2ff847995a116fe03885514a5c839f545a75f27a82c0e/userdata/shm major:0 minor:91 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7fe4319957955c98ed7c6909bd338655b884e2d6cb6f89741f4252df918c536f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7fe4319957955c98ed7c6909bd338655b884e2d6cb6f89741f4252df918c536f/userdata/shm major:0 minor:70 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/abfbe7f65e87c82cfa535f8e1f3445cb20d888ecbb7f8abb697df2085a6a3c73/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/abfbe7f65e87c82cfa535f8e1f3445cb20d888ecbb7f8abb697df2085a6a3c73/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b2cb51f772139f867440341c3eabff8fe9e6429b771fbd5c866d0cfe265a74a2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b2cb51f772139f867440341c3eabff8fe9e6429b771fbd5c866d0cfe265a74a2/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ba7396663395b30e8caee684e3b103368f2aa74a904ffab9058964d5bf446d10/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ba7396663395b30e8caee684e3b103368f2aa74a904ffab9058964d5bf446d10/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c8861ce03831186f570d157c4d607cc643f2909b90a3a4e9d33fcf0b3dec3570/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c8861ce03831186f570d157c4d607cc643f2909b90a3a4e9d33fcf0b3dec3570/userdata/shm major:0 minor:224 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e49208b3830b22cd3c99cbb381515dcd9235cf0a71aff04e049925e4863af5f6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e49208b3830b22cd3c99cbb381515dcd9235cf0a71aff04e049925e4863af5f6/userdata/shm major:0 minor:210 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2/volumes/kubernetes.io~projected/kube-api-access-j2g6f:{mountpoint:/var/lib/kubelet/pods/0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2/volumes/kubernetes.io~projected/kube-api-access-j2g6f major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0b2d7938-0c97-4f07-a7d2-b5c35d2bc8f9/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/0b2d7938-0c97-4f07-a7d2-b5c35d2bc8f9/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/17fe5072-f4c5-4075-8614-4229ed05d4b2/volumes/kubernetes.io~projected/kube-api-access-lbxts:{mountpoint:/var/lib/kubelet/pods/17fe5072-f4c5-4075-8614-4229ed05d4b2/volumes/kubernetes.io~projected/kube-api-access-lbxts major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/28b76ecd-8da7-428f-a825-323670ae5395/volumes/kubernetes.io~projected/kube-api-access-n45ql:{mountpoint:/var/lib/kubelet/pods/28b76ecd-8da7-428f-a825-323670ae5395/volumes/kubernetes.io~projected/kube-api-access-n45ql major:0 minor:240 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/28b76ecd-8da7-428f-a825-323670ae5395/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/28b76ecd-8da7-428f-a825-323670ae5395/volumes/kubernetes.io~secret/signing-key major:0 minor:235 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/31a69a0e-cc9c-47ad-b577-36b50d61d213/volumes/kubernetes.io~projected/kube-api-access-ccf8l:{mountpoint:/var/lib/kubelet/pods/31a69a0e-cc9c-47ad-b577-36b50d61d213/volumes/kubernetes.io~projected/kube-api-access-ccf8l major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3fabcb64-faf2-49f9-813a-ddafde0fd159/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/3fabcb64-faf2-49f9-813a-ddafde0fd159/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4874b345-eac7-4694-8351-d24085bf180a/volumes/kubernetes.io~projected/kube-api-access-jz8qt:{mountpoint:/var/lib/kubelet/pods/4874b345-eac7-4694-8351-d24085bf180a/volumes/kubernetes.io~projected/kube-api-access-jz8qt major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343/volumes/kubernetes.io~projected/kube-api-access-mptpf:{mountpoint:/var/lib/kubelet/pods/4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343/volumes/kubernetes.io~projected/kube-api-access-mptpf major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/54c1de40-ce9d-4d7d-82ca-c5fa41c97e97/volumes/kubernetes.io~projected/kube-api-access-fcgjq:{mountpoint:/var/lib/kubelet/pods/54c1de40-ce9d-4d7d-82ca-c5fa41c97e97/volumes/kubernetes.io~projected/kube-api-access-fcgjq major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5d436341-2000-4c2b-87ac-7b67e4cdb909/volumes/kubernetes.io~projected/kube-api-access-dgbw6:{mountpoint:/var/lib/kubelet/pods/5d436341-2000-4c2b-87ac-7b67e4cdb909/volumes/kubernetes.io~projected/kube-api-access-dgbw6 major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5d436341-2000-4c2b-87ac-7b67e4cdb909/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/5d436341-2000-4c2b-87ac-7b67e4cdb909/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/74106611-359e-4353-b113-70521144bed1/volumes/kubernetes.io~projected/kube-api-access-pgbgt:{mountpoint:/var/lib/kubelet/pods/74106611-359e-4353-b113-70521144bed1/volumes/kubernetes.io~projected/kube-api-access-pgbgt major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8756b800-1b4e-41da-934a-25c6b8ee738c/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/8756b800-1b4e-41da-934a-25c6b8ee738c/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8756b800-1b4e-41da-934a-25c6b8ee738c/volumes/kubernetes.io~projected/kube-api-access-828qc:{mountpoint:/var/lib/kubelet/pods/8756b800-1b4e-41da-934a-25c6b8ee738c/volumes/kubernetes.io~projected/kube-api-access-828qc major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8756b800-1b4e-41da-934a-25c6b8ee738c/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/8756b800-1b4e-41da-934a-25c6b8ee738c/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9b7f05bf-12cd-4aad-b4ee-8b517e901ca6/volumes/kubernetes.io~projected/kube-api-access-2d956:{mountpoint:/var/lib/kubelet/pods/9b7f05bf-12cd-4aad-b4ee-8b517e901ca6/volumes/kubernetes.io~projected/kube-api-access-2d956 major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a8e20267-1b77-4eb3-b940-a29d79b0ec19/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/a8e20267-1b77-4eb3-b940-a29d79b0ec19/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a8e20267-1b77-4eb3-b940-a29d79b0ec19/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/a8e20267-1b77-4eb3-b940-a29d79b0ec19/volumes/kubernetes.io~empty-dir/tmp major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a8e20267-1b77-4eb3-b940-a29d79b0ec19/volumes/kubernetes.io~projected/kube-api-access-7jxxh:{mountpoint:/var/lib/kubelet/pods/a8e20267-1b77-4eb3-b940-a29d79b0ec19/volumes/kubernetes.io~projected/kube-api-access-7jxxh major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b6ad2c98-52b5-496e-9824-614a2e509a8a/volumes/kubernetes.io~projected/kube-api-access-sxwsm:{mountpoint:/var/lib/kubelet/pods/b6ad2c98-52b5-496e-9824-614a2e509a8a/volumes/kubernetes.io~projected/kube-api-access-sxwsm major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b6cb1c47-e00c-484d-bf34-445b9380553c/volumes/kubernetes.io~projected/kube-api-access-l4sz9:{mountpoint:/var/lib/kubelet/pods/b6cb1c47-e00c-484d-bf34-445b9380553c/volumes/kubernetes.io~projected/kube-api-access-l4sz9 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dd804657-39fd-4ae9-899d-cd1f1b62717e/volumes/kubernetes.io~projected/kube-api-access-bcpxd:{mountpoint:/var/lib/kubelet/pods/dd804657-39fd-4ae9-899d-cd1f1b62717e/volumes/kubernetes.io~projected/kube-api-access-bcpxd major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dfb225e8-fabd-49fb-9763-4e41bf6ae7de/volumes/kubernetes.io~projected/kube-api-access-rgmdg:{mountpoint:/var/lib/kubelet/pods/dfb225e8-fabd-49fb-9763-4e41bf6ae7de/volumes/kubernetes.io~projected/kube-api-access-rgmdg major:0 minor:201 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/0b2de393a4584d538ba7e3b6c59ffdfa4631ffe845608ca10cb85a0562c4d0ac/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/253eb510a8a20df8f38dfe4ba3310c1668d52e2d56eb4adf414ba3f7263115fa/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/3a761fb946700db6d733172f4617fc967337a42eda4fe3f58995de22fb5559d7/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-108:{mountpoint:/var/lib/containers/storage/overlay/fca7ae8312c3ec348849b9d7f2fdc7b1750715a264801e8fb3de4fb49f4a3ccd/merged major:0 minor:108 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/0e023d2460b4d1a02a04a2a374ba30cbf687fb8d6e1c0fffde81cced01bbdff9/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/7598ef70438e3e4926a14f0590b8d6698278854bd2a6365ddf4b2abe453ac649/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/2b48c0bcce7c010c605b2ddacee03955aeb003375b8ad102ef2d6f725063e7f7/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/53da365882098e2cf5ecdd75b8fcf9ac107e836a6210678d3daa3d21f3f6ea5a/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/d908c61c0d530b3c0908bbfe52f6c07ce6369a3c6c50f6b3d5d27ef3e459d785/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/9e73cb0d9530efba572932b4327c64b98ec61ba8b083870b051f5ed149a914c8/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/90a41df3871a396df990ed1fd04824e9847ac047da3e12f55efe2615153ac723/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/ec8679bd0bfd6b8d895293bf679ac99ac730b7a6f0dee77eed793b009f5e4ad9/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/fbedfd8513ccb5288a12a18c9633e5f0ed0c6e0219ec94a32a85c9328a2fbc57/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/b290ffefec18405ceba082da7105180ea0f37f9ed537805f70648205fedb7535/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/f8b14a848b60fcee95debe8078b466194d81b94764d85c9ed0d861ee82fae36d/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/b0d298c8aac6a9451a82763d6eb47a47aba45c24a90777b48a762a4d244616e8/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/2460d12ae035825ccc27801c53e5df0c3b85ca6fd7dcc0a462a4610a055d03a1/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/e8bd3730a7faebf0c25a6bb37fc221a67127ac9df07cb2b830a551da531e936b/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/4d1db73fd193c2ea333a69c739a5f5950f53488de08eb0b08ce8ba938417bb9c/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/3b3ffc374fdd44502815d8708d9a41bfe132c3fccf4661451118f42b94a3944a/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/725df9dd69f5ae6de5925f8aba579c951d838b7ff40b5c3f40c6137fd9b0a443/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/16664494e9ca8e1d4d79468f692f479afd222acb5a4cd2d2f350101c3753895a/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-212:{mountpoint:/var/lib/containers/storage/overlay/1e1f72bf0cd589ceb07db5bf12d20edeece677d7257c19c0db091efb88f06cbe/merged major:0 minor:212 fsType:overlay blockSize:0} overlay_0-214:{mountpoint:/var/lib/containers/storage/overlay/36850427ed2dd9e58502143ccb63d91a261ff1af9f44c500e8ac5ddb46dacb4a/merged major:0 minor:214 fsType:overlay blockSize:0} overlay_0-216:{mountpoint:/var/lib/containers/storage/overlay/6b6b297a58272980fbc663a6dfb8ed9978d40e31e1bc2357324efb8df4b91dff/merged major:0 minor:216 fsType:overlay blockSize:0} overlay_0-227:{mountpoint:/var/lib/containers/storage/overlay/45384badf8643191a03c9a8304a0c33c467ee56ecff3f4dc41d9ff4d0e76c0aa/merged major:0 minor:227 fsType:overlay blockSize:0} overlay_0-231:{mountpoint:/var/lib/containers/storage/overlay/d512f32a53f2d6b03d28fec463a1c69d109a6b68aa3382a76462c3ae68c87a56/merged major:0 minor:231 fsType:overlay blockSize:0} overlay_0-233:{mountpoint:/var/lib/containers/storage/overlay/3b86478894e575bebb5c5031cce321b8916060078d179081e03e0bfb1a5725cc/merged major:0 minor:233 fsType:overlay blockSize:0} overlay_0-243:{mountpoint:/var/lib/containers/storage/overlay/a66a051718c780a76dc9bd18ab21c00a793d7631599f47ca4292000d792f3f8b/merged major:0 minor:243 fsType:overlay blockSize:0} overlay_0-245:{mountpoint:/var/lib/containers/storage/overlay/9fe280daf827739a77fb769c820d340b646b4502cc9733813d7f71326f470493/merged major:0 minor:245 fsType:overlay blockSize:0} overlay_0-64:{mountpoint:/var/lib/containers/storage/overlay/977d468a23b054cfd67ef8642dbcbf98597b321422ff8606bcb0a50410fdb451/merged major:0 minor:64 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/bf4957ea03a860b6933e9bdf90cac3b88273f8b2e2bc1abc87bd7382fb1d87ed/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/e8f19b9b1f6b641fdaa38e62e1f0d284d31e5a7d65f0c06595c15a10631e8268/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/651ddf2a8591e651e44d299976d2c0f915b90d59c355187e533f3ce5f85e7ae2/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/3779b002e9779f4003ae1ced3b20699598c7e0e10fecc24f954b19255dd4f8f4/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/ec3499289de5229933fd039c21574aff48f354f11dcf4a33f64d8823a7f30867/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/336e7dacfb9dab5c8a09e46f7c764e32dee7b8a66865b922ee44c3e492a0ed0e/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/8d8b538723146df4e327c02dda7f4079d84ac1934ab73bcd855e7220249d8124/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-93:{mountpoint:/var/lib/containers/storage/overlay/fe4e51d87a2d6a2e7c95c32862b62dd59e6ab5c786dddddd207a886187653d0d/merged major:0 minor:93 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/5dd01477c47088b86e8d7078f2296398898f3f6d51d2f944b412b19ce4354864/merged major:0 minor:96 fsType:overlay blockSize:0}]
Nov 02 19:28:13.485843 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.479719    4463 manager.go:217] Machine: {Timestamp:2026-11-02 19:28:13.479161847 +0000 UTC m=+0.086195607 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099193 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec26c1c90f4d11743645d2d184f7ad79 SystemUUID:ec26c1c9-0f4d-1174-3645-d2d184f7ad79 BootID:8ee7c652-3485-4821-8e0e-6dd701b91d80 Filesystems:[{Device:/var/lib/kubelet/pods/5d436341-2000-4c2b-87ac-7b67e4cdb909/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b6cb1c47-e00c-484d-bf34-445b9380553c/volumes/kubernetes.io~projected/kube-api-access-l4sz9 DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/abfbe7f65e87c82cfa535f8e1f3445cb20d888ecbb7f8abb697df2085a6a3c73/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-214 DeviceMajor:0 DeviceMinor:214 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/b6ad2c98-52b5-496e-9824-614a2e509a8a/volumes/kubernetes.io~projected/kube-api-access-sxwsm DeviceMajor:0 DeviceMinor:52 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-64 DeviceMajor:0 DeviceMinor:64 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7fe4319957955c98ed7c6909bd338655b884e2d6cb6f89741f4252df918c536f/userdata/shm DeviceMajor:0 DeviceMinor:70 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4874b345-eac7-4694-8351-d24085bf180a/volumes/kubernetes.io~projected/kube-api-access-jz8qt DeviceMajor:0 DeviceMinor:223 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/667d2758ccf9eaad27f48fb50e5834c02305aa559406fa46b65f01390ffe20bd/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/34ffb39bf1d7fea750bf39bc7b22f4ef70166d88d57a034b4fa561753454caee/userdata/shm DeviceMajor:0 DeviceMinor:208 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9b7f05bf-12cd-4aad-b4ee-8b517e901ca6/volumes/kubernetes.io~projected/kube-api-access-2d956 DeviceMajor:0 DeviceMinor:48 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8756b800-1b4e-41da-934a-25c6b8ee738c/volumes/kubernetes.io~projected/kube-api-access-828qc DeviceMajor:0 DeviceMinor:50 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4ce5d46f7311979bfa15d5f696ef882348ffbf80aa8c5acf5c07b8316af4865d/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2fddafd70439b8e26feb1fff6e97dcb631902ca9cb63bf44a13df77960ad2022/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8756b800-1b4e-41da-934a-25c6b8ee738c/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/74106611-359e-4353-b113-70521144bed1/volumes/kubernetes.io~projected/kube-api-access-pgbgt DeviceMajor:0 DeviceMinor:49 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3fabcb64-faf2-49f9-813a-ddafde0fd159/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a8e20267-1b77-4eb3-b940-a29d79b0ec19/volumes/kubernetes.io~projected/kube-api-access-7jxxh DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/655d515e9caa08a107af1d948364e4afa72c74189600ae462600c0de1ee66879/userdata/shm DeviceMajor:0 DeviceMinor:79 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dfb225e8-fabd-49fb-9763-4e41bf6ae7de/volumes/kubernetes.io~projected/kube-api-access-rgmdg DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2/volumes/kubernetes.io~projected/kube-api-access-j2g6f DeviceMajor:0 DeviceMinor:218 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-108 DeviceMajor:0 DeviceMinor:108 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/205f9a1cdfee951b8ab921391eb6de239e18105e14febb2b60c181b59f644e32/userdata/shm DeviceMajor:0 DeviceMinor:229 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/17fe5072-f4c5-4075-8614-4229ed05d4b2/volumes/kubernetes.io~projected/kube-api-access-lbxts DeviceMajor:0 DeviceMinor:51 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a8e20267-1b77-4eb3-b940-a29d79b0ec19/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:54 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/dd804657-39fd-4ae9-899d-cd1f1b62717e/volumes/kubernetes.io~projected/kube-api-access-bcpxd DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e49208b3830b22cd3c99cbb381515dcd9235cf0a71aff04e049925e4863af5f6/userdata/shm DeviceMajor:0 DeviceMinor:210 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-231 DeviceMajor:0 DeviceMinor:231 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1e383e5f91ee92619c7178649dc6b6d776a9eb3b2e97103bccfdca415dab6887/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-212 DeviceMajor:0 DeviceMinor:212 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-227 DeviceMajor:0 DeviceMinor:227 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8756b800-1b4e-41da-934a-25c6b8ee738c/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:47 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a8e20267-1b77-4eb3-b940-a29d79b0ec19/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:53 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/29b253e3dff4d86876a235f3c9b0a36d3815e1fb4be3fbdc60497516afaf9b34/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-216 DeviceMajor:0 DeviceMinor:216 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-233 DeviceMajor:0 DeviceMinor:233 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7655f51aa5471290e3a2ff847995a116fe03885514a5c839f545a75f27a82c0e/userdata/shm DeviceMajor:0 DeviceMinor:91 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/31a69a0e-cc9c-47ad-b577-36b50d61d213/volumes/kubernetes.io~projected/kube-api-access-ccf8l DeviceMajor:0 DeviceMinor:207 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/28b76ecd-8da7-428f-a825-323670ae5395/volumes/kubernetes.io~projected/kube-api-access-n45ql DeviceMajor:0 DeviceMinor:240 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-245 DeviceMajor:0 DeviceMinor:245 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343/volumes/kubernetes.io~projected/kube-api-access-mptpf DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ba7396663395b30e8caee684e3b103368f2aa74a904ffab9058964d5bf446d10/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b2cb51f772139f867440341c3eabff8fe9e6429b771fbd5c866d0cfe265a74a2/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5d436341-2000-4c2b-87ac-7b67e4cdb909/volumes/kubernetes.io~projected/kube-api-access-dgbw6 DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c8861ce03831186f570d157c4d607cc643f2909b90a3a4e9d33fcf0b3dec3570/userdata/shm DeviceMajor:0 DeviceMinor:224 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/54c1de40-ce9d-4d7d-82ca-c5fa41c97e97/volumes/kubernetes.io~projected/kube-api-access-fcgjq DeviceMajor:0 DeviceMinor:226 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/625b7eb489e12552e74b6ff8823e0dffd3c14fb16c148ef7f9f401a4327db444/userdata/shm DeviceMajor:0 DeviceMinor:241 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-243 DeviceMajor:0 DeviceMinor:243 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0b2d7938-0c97-4f07-a7d2-b5c35d2bc8f9/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:206 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/28b76ecd-8da7-428f-a825-323670ae5395/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:235 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-93 DeviceMajor:0 DeviceMinor:93 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:205f9a1cdfee951 MacAddress:c2:95:10:f4:e3:c5 Speed:10000 Mtu:8901} {Name:34ffb39bf1d7fea MacAddress:52:69:7d:47:8e:8e Speed:10000 Mtu:8901} {Name:625b7eb489e1255 MacAddress:7e:9b:3c:dd:25:8f Speed:10000 Mtu:8901} {Name:b2cb51f772139f8 MacAddress:2e:69:88:10:8e:20 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:de:70:42:9d Speed:0 Mtu:9001} {Name:br-int MacAddress:f2:2b:59:d8:60:99 Speed:0 Mtu:8901} {Name:c8861ce03831186 MacAddress:72:c6:74:45:10:1d Speed:10000 Mtu:8901} {Name:e49208b3830b22c MacAddress:56:15:66:cd:d5:d1 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:de:70:42:9d Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:86:66:94:35:6a:9f Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:86:47:23:d9:6a:02 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 02 19:28:13.485843 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.480065    4463 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 02 19:28:13.485843 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.480113    4463 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 02 19:28:13.485843 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.480424    4463 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 02 19:28:13.485843 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.480441    4463 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-9.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 02 19:28:13.485843 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.480585    4463 topology_manager.go:143] "Creating topology manager with none policy"
Nov 02 19:28:13.485843 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.480597    4463 container_manager_linux.go:311] "Creating device plugin manager"
Nov 02 19:28:13.485843 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.480615    4463 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 19:28:13.485843 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.480641    4463 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 19:28:13.485843 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.480718    4463 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 02 19:28:13.485843 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.480881    4463 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 02 19:28:13.485843 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.481049    4463 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 02 19:28:13.485843 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.481084    4463 kubelet.go:493] "Attempting to sync node with API server"
Nov 02 19:28:13.485843 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.481097    4463 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 02 19:28:13.485843 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.481111    4463 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 02 19:28:13.485843 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.481125    4463 kubelet.go:405] "Adding apiserver pod source"
Nov 02 19:28:13.485843 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.481136    4463 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 02 19:28:13.485843 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.481852    4463 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 02 19:28:13.485843 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.481864    4463 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 19:28:13.485843 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.483314    4463 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 02 19:28:13.538935 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.538917    4463 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 02 19:28:13.538935 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.538934    4463 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 19:28:13.539125 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.539104    4463 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 02 19:28:13.539361 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.539337    4463 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 02 19:28:13.539449 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.539388    4463 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 02 19:28:13.539541 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.539529    4463 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 02 19:28:13.539592 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.539550    4463 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 02 19:28:13.539592 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.539561    4463 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 02 19:28:13.539592 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.539571    4463 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 02 19:28:13.539592 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.539580    4463 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 02 19:28:13.539592 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.539591    4463 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 02 19:28:13.539829 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.539600    4463 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 02 19:28:13.539829 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.539609    4463 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 02 19:28:13.539829 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.539622    4463 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 02 19:28:13.539829 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.539632    4463 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 02 19:28:13.539829 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.539645    4463 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 02 19:28:13.539829 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.539659    4463 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 02 19:28:13.539829 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.539694    4463 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 02 19:28:13.539829 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.539705    4463 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 02 19:28:13.543852 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.543837    4463 server.go:1263] "Started kubelet"
Nov 02 19:28:13.544222 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.544074    4463 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 02 19:28:13.544222 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.544151    4463 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 02 19:28:13.544222 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.544174    4463 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 02 19:28:13.544440 ip-10-0-133-9 systemd[1]: Started Kubernetes Kubelet.
Nov 02 19:28:13.544630 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.544574    4463 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 02 19:28:13.552359 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.545521    4463 server.go:317] "Adding debug handlers to kubelet server"
Nov 02 19:28:13.552359 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.547915    4463 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 02 19:28:13.552359 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.547944    4463 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 02 19:28:13.552359 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.547988    4463 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 19:22:35 +0000 UTC" deadline="2028-06-28 23:39:34.300476411 +0000 UTC"
Nov 02 19:28:13.552359 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.548066    4463 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14500h11m20.75241675s"
Nov 02 19:28:13.552359 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.548120    4463 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 02 19:28:13.552359 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.548128    4463 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 02 19:28:13.552359 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:28:13.548132    4463 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-9.us-west-2.compute.internal\" not found"
Nov 02 19:28:13.552359 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.548409    4463 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 02 19:28:13.552359 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.548847    4463 factory.go:55] Registering systemd factory
Nov 02 19:28:13.552359 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.548867    4463 factory.go:223] Registration of the systemd container factory successfully
Nov 02 19:28:13.552359 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.550228    4463 factory.go:153] Registering CRI-O factory
Nov 02 19:28:13.552359 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.550246    4463 factory.go:223] Registration of the crio container factory successfully
Nov 02 19:28:13.552359 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.550302    4463 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 02 19:28:13.552359 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.550320    4463 factory.go:103] Registering Raw factory
Nov 02 19:28:13.552359 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.550334    4463 manager.go:1199] Started watching for new ooms in manager
Nov 02 19:28:13.552359 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.550697    4463 manager.go:319] Starting recovery of all containers
Nov 02 19:28:13.552359 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.551121    4463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31a69a0e-cc9c-47ad-b577-36b50d61d213" volumeName="kubernetes.io/projected/31a69a0e-cc9c-47ad-b577-36b50d61d213-kube-api-access-ccf8l" seLinuxMountContext=""
Nov 02 19:28:13.552359 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.551170    4463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a8e20267-1b77-4eb3-b940-a29d79b0ec19" volumeName="kubernetes.io/empty-dir/a8e20267-1b77-4eb3-b940-a29d79b0ec19-etc-tuned" seLinuxMountContext=""
Nov 02 19:28:13.552359 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.551193    4463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dfb225e8-fabd-49fb-9763-4e41bf6ae7de" volumeName="kubernetes.io/empty-dir/dfb225e8-fabd-49fb-9763-4e41bf6ae7de-tmp-dir" seLinuxMountContext=""
Nov 02 19:28:13.552359 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.551215    4463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2" volumeName="kubernetes.io/projected/0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2-kube-api-access-j2g6f" seLinuxMountContext=""
Nov 02 19:28:13.552359 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.551228    4463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b2d7938-0c97-4f07-a7d2-b5c35d2bc8f9" volumeName="kubernetes.io/secret/0b2d7938-0c97-4f07-a7d2-b5c35d2bc8f9-original-pull-secret" seLinuxMountContext=""
Nov 02 19:28:13.552359 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.551241    4463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17fe5072-f4c5-4075-8614-4229ed05d4b2" volumeName="kubernetes.io/projected/17fe5072-f4c5-4075-8614-4229ed05d4b2-kube-api-access-lbxts" seLinuxMountContext=""
Nov 02 19:28:13.552359 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.551279    4463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3fabcb64-faf2-49f9-813a-ddafde0fd159" volumeName="kubernetes.io/configmap/3fabcb64-faf2-49f9-813a-ddafde0fd159-konnectivity-ca" seLinuxMountContext=""
Nov 02 19:28:13.552359 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.551295    4463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dd804657-39fd-4ae9-899d-cd1f1b62717e" volumeName="kubernetes.io/empty-dir/dd804657-39fd-4ae9-899d-cd1f1b62717e-tmp-dir" seLinuxMountContext=""
Nov 02 19:28:13.552359 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.551312    4463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54c1de40-ce9d-4d7d-82ca-c5fa41c97e97" volumeName="kubernetes.io/projected/54c1de40-ce9d-4d7d-82ca-c5fa41c97e97-kube-api-access-fcgjq" seLinuxMountContext=""
Nov 02 19:28:13.552359 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.551336    4463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74106611-359e-4353-b113-70521144bed1" volumeName="kubernetes.io/projected/74106611-359e-4353-b113-70521144bed1-kube-api-access-pgbgt" seLinuxMountContext=""
Nov 02 19:28:13.552359 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.551362    4463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b7f05bf-12cd-4aad-b4ee-8b517e901ca6" volumeName="kubernetes.io/configmap/9b7f05bf-12cd-4aad-b4ee-8b517e901ca6-cni-binary-copy" seLinuxMountContext=""
Nov 02 19:28:13.552359 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.551421    4463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6ad2c98-52b5-496e-9824-614a2e509a8a" volumeName="kubernetes.io/projected/b6ad2c98-52b5-496e-9824-614a2e509a8a-kube-api-access-sxwsm" seLinuxMountContext=""
Nov 02 19:28:13.552359 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.551449    4463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6cb1c47-e00c-484d-bf34-445b9380553c" volumeName="kubernetes.io/projected/b6cb1c47-e00c-484d-bf34-445b9380553c-kube-api-access-l4sz9" seLinuxMountContext=""
Nov 02 19:28:13.552359 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.551463    4463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343" volumeName="kubernetes.io/configmap/4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 02 19:28:13.552359 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.551481    4463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343" volumeName="kubernetes.io/configmap/4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343-cni-binary-copy" seLinuxMountContext=""
Nov 02 19:28:13.552359 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.551505    4463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b7f05bf-12cd-4aad-b4ee-8b517e901ca6" volumeName="kubernetes.io/projected/9b7f05bf-12cd-4aad-b4ee-8b517e901ca6-kube-api-access-2d956" seLinuxMountContext=""
Nov 02 19:28:13.552359 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.551519    4463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a8e20267-1b77-4eb3-b940-a29d79b0ec19" volumeName="kubernetes.io/empty-dir/a8e20267-1b77-4eb3-b940-a29d79b0ec19-tmp" seLinuxMountContext=""
Nov 02 19:28:13.552359 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.551545    4463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="28b76ecd-8da7-428f-a825-323670ae5395" volumeName="kubernetes.io/configmap/28b76ecd-8da7-428f-a825-323670ae5395-signing-cabundle" seLinuxMountContext=""
Nov 02 19:28:13.552359 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.551588    4463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3fabcb64-faf2-49f9-813a-ddafde0fd159" volumeName="kubernetes.io/secret/3fabcb64-faf2-49f9-813a-ddafde0fd159-agent-certs" seLinuxMountContext=""
Nov 02 19:28:13.552359 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.551602    4463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4874b345-eac7-4694-8351-d24085bf180a" volumeName="kubernetes.io/projected/4874b345-eac7-4694-8351-d24085bf180a-kube-api-access-jz8qt" seLinuxMountContext=""
Nov 02 19:28:13.552359 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.551617    4463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343" volumeName="kubernetes.io/projected/4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343-kube-api-access-mptpf" seLinuxMountContext=""
Nov 02 19:28:13.552359 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.551675    4463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5d436341-2000-4c2b-87ac-7b67e4cdb909" volumeName="kubernetes.io/projected/5d436341-2000-4c2b-87ac-7b67e4cdb909-kube-api-access-dgbw6" seLinuxMountContext=""
Nov 02 19:28:13.552359 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.551713    4463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b7f05bf-12cd-4aad-b4ee-8b517e901ca6" volumeName="kubernetes.io/configmap/9b7f05bf-12cd-4aad-b4ee-8b517e901ca6-multus-daemon-config" seLinuxMountContext=""
Nov 02 19:28:13.552359 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.551731    4463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dfb225e8-fabd-49fb-9763-4e41bf6ae7de" volumeName="kubernetes.io/configmap/dfb225e8-fabd-49fb-9763-4e41bf6ae7de-config-volume" seLinuxMountContext=""
Nov 02 19:28:13.552359 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.551761    4463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4874b345-eac7-4694-8351-d24085bf180a" volumeName="kubernetes.io/configmap/4874b345-eac7-4694-8351-d24085bf180a-service-ca-bundle" seLinuxMountContext=""
Nov 02 19:28:13.552359 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.551786    4463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8756b800-1b4e-41da-934a-25c6b8ee738c" volumeName="kubernetes.io/configmap/8756b800-1b4e-41da-934a-25c6b8ee738c-ovnkube-script-lib" seLinuxMountContext=""
Nov 02 19:28:13.552359 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.551797    4463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8756b800-1b4e-41da-934a-25c6b8ee738c" volumeName="kubernetes.io/projected/8756b800-1b4e-41da-934a-25c6b8ee738c-kube-api-access-828qc" seLinuxMountContext=""
Nov 02 19:28:13.552359 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.551824    4463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8756b800-1b4e-41da-934a-25c6b8ee738c" volumeName="kubernetes.io/secret/8756b800-1b4e-41da-934a-25c6b8ee738c-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 02 19:28:13.552359 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.551876    4463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4874b345-eac7-4694-8351-d24085bf180a" volumeName="kubernetes.io/configmap/4874b345-eac7-4694-8351-d24085bf180a-trusted-ca-bundle" seLinuxMountContext=""
Nov 02 19:28:13.552359 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.551902    4463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4874b345-eac7-4694-8351-d24085bf180a" volumeName="kubernetes.io/empty-dir/4874b345-eac7-4694-8351-d24085bf180a-archives-path" seLinuxMountContext=""
Nov 02 19:28:13.552359 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.551925    4463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343" volumeName="kubernetes.io/configmap/4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 02 19:28:13.552359 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.551938    4463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8756b800-1b4e-41da-934a-25c6b8ee738c" volumeName="kubernetes.io/configmap/8756b800-1b4e-41da-934a-25c6b8ee738c-ovnkube-config" seLinuxMountContext=""
Nov 02 19:28:13.552359 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.551976    4463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8756b800-1b4e-41da-934a-25c6b8ee738c" volumeName="kubernetes.io/configmap/8756b800-1b4e-41da-934a-25c6b8ee738c-env-overrides" seLinuxMountContext=""
Nov 02 19:28:13.552359 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.552028    4463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a8e20267-1b77-4eb3-b940-a29d79b0ec19" volumeName="kubernetes.io/projected/a8e20267-1b77-4eb3-b940-a29d79b0ec19-kube-api-access-7jxxh" seLinuxMountContext=""
Nov 02 19:28:13.552359 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.552050    4463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6ad2c98-52b5-496e-9824-614a2e509a8a" volumeName="kubernetes.io/configmap/b6ad2c98-52b5-496e-9824-614a2e509a8a-serviceca" seLinuxMountContext=""
Nov 02 19:28:13.552359 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.552067    4463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6cb1c47-e00c-484d-bf34-445b9380553c" volumeName="kubernetes.io/configmap/b6cb1c47-e00c-484d-bf34-445b9380553c-iptables-alerter-script" seLinuxMountContext=""
Nov 02 19:28:13.552359 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.552092    4463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2" volumeName="kubernetes.io/empty-dir/0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2-data-volume" seLinuxMountContext=""
Nov 02 19:28:13.552359 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.552104    4463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="28b76ecd-8da7-428f-a825-323670ae5395" volumeName="kubernetes.io/projected/28b76ecd-8da7-428f-a825-323670ae5395-kube-api-access-n45ql" seLinuxMountContext=""
Nov 02 19:28:13.552359 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.552118    4463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="28b76ecd-8da7-428f-a825-323670ae5395" volumeName="kubernetes.io/secret/28b76ecd-8da7-428f-a825-323670ae5395-signing-key" seLinuxMountContext=""
Nov 02 19:28:13.552359 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.552145    4463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5d436341-2000-4c2b-87ac-7b67e4cdb909" volumeName="kubernetes.io/secret/5d436341-2000-4c2b-87ac-7b67e4cdb909-cert" seLinuxMountContext=""
Nov 02 19:28:13.552359 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.552156    4463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dd804657-39fd-4ae9-899d-cd1f1b62717e" volumeName="kubernetes.io/projected/dd804657-39fd-4ae9-899d-cd1f1b62717e-kube-api-access-bcpxd" seLinuxMountContext=""
Nov 02 19:28:13.552359 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.552181    4463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dfb225e8-fabd-49fb-9763-4e41bf6ae7de" volumeName="kubernetes.io/projected/dfb225e8-fabd-49fb-9763-4e41bf6ae7de-kube-api-access-rgmdg" seLinuxMountContext=""
Nov 02 19:28:13.552359 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.552189    4463 reconstruct.go:97] "Volume reconstruction finished"
Nov 02 19:28:13.552359 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.552194    4463 reconciler.go:29] "Reconciler: start to sync state"
Nov 02 19:28:13.574598 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.574399    4463 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 02 19:28:13.576405 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.576337    4463 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 02 19:28:13.576405 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.576365    4463 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 02 19:28:13.576405 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.576385    4463 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 02 19:28:13.576648 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:28:13.576434    4463 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 02 19:28:13.607062 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.607003    4463 generic.go:356] "Generic (PLEG): container finished" podID="4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343" containerID="77beba51e1d986850256508f7319d9d84cce1764452332f2a5952d9b32e61435" exitCode=0
Nov 02 19:28:13.607062 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.607056    4463 generic.go:356] "Generic (PLEG): container finished" podID="4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343" containerID="67a7e23ec25a1a59a219af07c8fc672d978f64a76f0065c10597f72db2606392" exitCode=0
Nov 02 19:28:13.607062 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.607069    4463 generic.go:356] "Generic (PLEG): container finished" podID="4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343" containerID="45648625589c2bcc99e47f800b0d4569857fd84049cfa03c5744fc75a23269ba" exitCode=0
Nov 02 19:28:13.607260 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.607075    4463 generic.go:356] "Generic (PLEG): container finished" podID="4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343" containerID="54528f4bec65f2f6c39231332edad67d23fbc44865fa24e2af6fc22662971c17" exitCode=0
Nov 02 19:28:13.607260 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.607080    4463 generic.go:356] "Generic (PLEG): container finished" podID="4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343" containerID="6a6ee7801ec9837ab0021f979afba29017c79437d8dbb1f14d56b4453b2afc46" exitCode=0
Nov 02 19:28:13.607260 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.607086    4463 generic.go:356] "Generic (PLEG): container finished" podID="4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343" containerID="ba119f338d444c26564d72c637569e929c193f3fbd586e6c6820ac8c6b9c2cbe" exitCode=0
Nov 02 19:28:13.618481 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.618451    4463 generic.go:356] "Generic (PLEG): container finished" podID="b70d1060da282fd74b3ebbf01b381326" containerID="99b56a4cd13b0048eb354a2432abd2878b8ad0ec5cc126fcd16fbf66fac0abd8" exitCode=0
Nov 02 19:28:13.632857 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.632830    4463 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 19:28:13.635153 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.633658    4463 manager.go:324] Recovery completed
Nov 02 19:28:13.648232 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:28:13.648208    4463 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-9.us-west-2.compute.internal\" not found"
Nov 02 19:28:13.649695 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.649675    4463 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 19:28:13.650869 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.650852    4463 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-9.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 19:28:13.650942 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.650881    4463 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-9.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 19:28:13.650942 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.650894    4463 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-9.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 19:28:13.652600 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.652586    4463 cpu_manager.go:226] "Starting" policy="none"
Nov 02 19:28:13.652600 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.652598    4463 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 02 19:28:13.652686 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.652602    4463 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 19:28:13.652686 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.652614    4463 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 02 19:28:13.652769 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.652754    4463 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 02 19:28:13.652800 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.652766    4463 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 02 19:28:13.652800 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.652785    4463 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 02 19:28:13.652800 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.652790    4463 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 02 19:28:13.652800 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.652796    4463 policy_none.go:50] "Start"
Nov 02 19:28:13.652972 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.652804    4463 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 02 19:28:13.652972 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.652813    4463 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 02 19:28:13.653506 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.653489    4463 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-9.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 19:28:13.653568 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.653520    4463 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-9.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 19:28:13.653568 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.653534    4463 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-9.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 19:28:13.658243 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.658226    4463 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 02 19:28:13.658243 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.658244    4463 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 02 19:28:13.658353 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.658251    4463 policy_none.go:44] "Start"
Nov 02 19:28:13.662043 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.662009    4463 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 02 19:28:13.666967 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.666952    4463 manager.go:341] "Starting Device Plugin manager"
Nov 02 19:28:13.667068 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:28:13.666976    4463 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 02 19:28:13.667068 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.666985    4463 server.go:87] "Starting device plugin registration server"
Nov 02 19:28:13.667296 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.667283    4463 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 02 19:28:13.667948 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.667297    4463 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 02 19:28:13.668829 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.668053    4463 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 02 19:28:13.668829 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.668152    4463 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 02 19:28:13.668829 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.668189    4463 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 02 19:28:13.668829 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.668196    4463 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 02 19:28:13.668829 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.668244    4463 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T19:28:13.668179947Z","UUID":"57e8e392-6c81-4804-9feb-fd40870bfe17","Handler":null,"Name":"","Endpoint":""}
Nov 02 19:28:13.668829 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:28:13.668594    4463 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 02 19:28:13.668829 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:28:13.668628    4463 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-9.us-west-2.compute.internal\" not found"
Nov 02 19:28:13.669552 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.669533    4463 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 02 19:28:13.669623 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.669559    4463 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 02 19:28:13.676583 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.676554    4463 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-133-9.us-west-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-9.us-west-2.compute.internal"]
Nov 02 19:28:13.676662 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.676652    4463 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 19:28:13.676898 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.676883    4463 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 19:28:13.677360 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.677347    4463 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-9.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 19:28:13.677424 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.677371    4463 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-9.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 19:28:13.677424 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.677380    4463 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-9.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 19:28:13.677598 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.677588    4463 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 19:28:13.677712 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.677700    4463 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 19:28:13.678266 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.678250    4463 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-9.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 19:28:13.678266 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.678259    4463 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-9.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 19:28:13.678359 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.678279    4463 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-9.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 19:28:13.678359 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.678289    4463 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-9.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 19:28:13.678421 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.678280    4463 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-9.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 19:28:13.678421 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.678415    4463 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-9.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 19:28:13.678580 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.678537    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-9.us-west-2.compute.internal" event={"ID":"b70d1060da282fd74b3ebbf01b381326","Type":"ContainerStarted","Data":"562ff5488cc749de466161bb83aad8ad4a17a89c784bb1ed1eda42d7d6e55a34"}
Nov 02 19:28:13.678580 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.678577    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-9.us-west-2.compute.internal" event={"ID":"b70d1060da282fd74b3ebbf01b381326","Type":"ContainerDied","Data":"99b56a4cd13b0048eb354a2432abd2878b8ad0ec5cc126fcd16fbf66fac0abd8"}
Nov 02 19:28:13.678701 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.678591    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-9.us-west-2.compute.internal" event={"ID":"b70d1060da282fd74b3ebbf01b381326","Type":"ContainerStarted","Data":"7fe4319957955c98ed7c6909bd338655b884e2d6cb6f89741f4252df918c536f"}
Nov 02 19:28:13.678701 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.678609    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-9.us-west-2.compute.internal" event={"ID":"d6e1f694aea2cef1694653a075ee3dd5","Type":"ContainerStarted","Data":"530bfca33e295dd1d1a0b0e78873d32784ad7abf446824f3bd8f99ccf04ff7a3"}
Nov 02 19:28:13.678701 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:28:13.678584    4463 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-9.us-west-2.compute.internal\" not found" node="ip-10-0-133-9.us-west-2.compute.internal"
Nov 02 19:28:13.678701 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.678627    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-9.us-west-2.compute.internal" event={"ID":"d6e1f694aea2cef1694653a075ee3dd5","Type":"ContainerStarted","Data":"ba7396663395b30e8caee684e3b103368f2aa74a904ffab9058964d5bf446d10"}
Nov 02 19:28:13.678701 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.678613    4463 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 19:28:13.678701 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.678686    4463 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 19:28:13.679371 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.679354    4463 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-9.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 19:28:13.679451 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.679384    4463 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-9.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 19:28:13.679451 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.679396    4463 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-9.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 19:28:13.679560 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.679525    4463 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-9.us-west-2.compute.internal"
Nov 02 19:28:13.725140 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.725123    4463 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 19:28:13.725237 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.725202    4463 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 02 19:28:13.736675 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.736654    4463 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 19:28:13.736738 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:28:13.736701    4463 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-133-9.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-9.us-west-2.compute.internal"
Nov 02 19:28:13.748872 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.748852    4463 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-9.us-west-2.compute.internal"
Nov 02 19:28:13.769023 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.768995    4463 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 19:28:13.769776 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.769753    4463 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-9.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 19:28:13.769835 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.769779    4463 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-9.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 19:28:13.769835 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.769794    4463 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-9.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 19:28:13.769835 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.769834    4463 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-9.us-west-2.compute.internal"
Nov 02 19:28:13.825891 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.825405    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d6e1f694aea2cef1694653a075ee3dd5-config\") pod \"kube-apiserver-proxy-ip-10-0-133-9.us-west-2.compute.internal\" (UID: \"d6e1f694aea2cef1694653a075ee3dd5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-9.us-west-2.compute.internal"
Nov 02 19:28:13.825891 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.825445    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b70d1060da282fd74b3ebbf01b381326-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-9.us-west-2.compute.internal\" (UID: \"b70d1060da282fd74b3ebbf01b381326\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-9.us-west-2.compute.internal"
Nov 02 19:28:13.825891 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.825481    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b70d1060da282fd74b3ebbf01b381326-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-9.us-west-2.compute.internal\" (UID: \"b70d1060da282fd74b3ebbf01b381326\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-9.us-west-2.compute.internal"
Nov 02 19:28:13.838094 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.838077    4463 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 19:28:13.838170 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:28:13.838111    4463 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-133-9.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-9.us-west-2.compute.internal"
Nov 02 19:28:13.838170 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.838122    4463 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-9.us-west-2.compute.internal"
Nov 02 19:28:13.872575 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.872561    4463 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-133-9.us-west-2.compute.internal"
Nov 02 19:28:13.872658 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.872610    4463 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-133-9.us-west-2.compute.internal"
Nov 02 19:28:13.893827 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.893808    4463 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 19:28:13.893898 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:28:13.893838    4463 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-133-9.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-9.us-west-2.compute.internal"
Nov 02 19:28:13.926246 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.926224    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d6e1f694aea2cef1694653a075ee3dd5-config\") pod \"kube-apiserver-proxy-ip-10-0-133-9.us-west-2.compute.internal\" (UID: \"d6e1f694aea2cef1694653a075ee3dd5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-9.us-west-2.compute.internal"
Nov 02 19:28:13.926309 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.926249    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b70d1060da282fd74b3ebbf01b381326-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-9.us-west-2.compute.internal\" (UID: \"b70d1060da282fd74b3ebbf01b381326\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-9.us-west-2.compute.internal"
Nov 02 19:28:13.926309 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.926265    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b70d1060da282fd74b3ebbf01b381326-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-9.us-west-2.compute.internal\" (UID: \"b70d1060da282fd74b3ebbf01b381326\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-9.us-west-2.compute.internal"
Nov 02 19:28:13.926402 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.926315    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b70d1060da282fd74b3ebbf01b381326-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-9.us-west-2.compute.internal\" (UID: \"b70d1060da282fd74b3ebbf01b381326\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-9.us-west-2.compute.internal"
Nov 02 19:28:13.926402 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.926343    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b70d1060da282fd74b3ebbf01b381326-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-9.us-west-2.compute.internal\" (UID: \"b70d1060da282fd74b3ebbf01b381326\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-9.us-west-2.compute.internal"
Nov 02 19:28:13.926402 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:13.926345    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d6e1f694aea2cef1694653a075ee3dd5-config\") pod \"kube-apiserver-proxy-ip-10-0-133-9.us-west-2.compute.internal\" (UID: \"d6e1f694aea2cef1694653a075ee3dd5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-9.us-west-2.compute.internal"
Nov 02 19:28:14.482539 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.482506    4463 apiserver.go:52] "Watching apiserver"
Nov 02 19:28:14.627816 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.627789    4463 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 02 19:28:14.628303 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.628280    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-hrhpj","openshift-kube-storage-version-migrator/migrator-66df5d496b-zqz7w","openshift-multus/multus-qhswb","openshift-ingress-canary/ingress-canary-sw9mp","openshift-cluster-node-tuning-operator/tuned-nvdtg","kube-system/konnectivity-agent-jgfjj","openshift-multus/network-metrics-daemon-f6r6d","openshift-insights/periodic-gathering-xvzjr-ggg8w","openshift-network-diagnostics/network-check-target-rhdd6","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-9.us-west-2.compute.internal","openshift-multus/multus-additional-cni-plugins-4ddbs","openshift-network-operator/iptables-alerter-cmk66","openshift-dns/dns-default-qj8mr","openshift-ovn-kubernetes/ovnkube-node-vst5r","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5wvz","kube-system/global-pull-secret-syncer-4b2pr","kube-system/kube-apiserver-proxy-ip-10-0-133-9.us-west-2.compute.internal","openshift-service-ca/service-ca-5fffc97bdf-h5fc9","openshift-dns/node-resolver-r9fcz","openshift-insights/insights-runtime-extractor-bh65q"]
Nov 02 19:28:14.630887 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.630850    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f6r6d"
Nov 02 19:28:14.632383 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.632363    4463 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-9.us-west-2.compute.internal"
Nov 02 19:28:14.632555 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.632536    4463 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-9.us-west-2.compute.internal"
Nov 02 19:28:14.633032 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.633003    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qj8mr"
Nov 02 19:28:14.633712 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.633698    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bh65q"
Nov 02 19:28:14.649415 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.649395    4463 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 02 19:28:14.685406 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.685376    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 02 19:28:14.685406 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.685389    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 02 19:28:14.685406 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.685403    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 02 19:28:14.685574 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.685413    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 02 19:28:14.685574 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.685518    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 02 19:28:14.685760 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.685740    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 02 19:28:14.685760 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.685741    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 02 19:28:14.685900 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.685805    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 02 19:28:14.685900 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.685809    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 02 19:28:14.685999 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.685984    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-wxdrr\""
Nov 02 19:28:14.685999 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.685993    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-lnn7q\""
Nov 02 19:28:14.686142 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.686130    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 02 19:28:14.686261 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.686251    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 02 19:28:14.686335 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.686270    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 02 19:28:14.686335 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.686312    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 02 19:28:14.686335 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.686315    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 02 19:28:14.686335 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.686330    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-nnqx8\""
Nov 02 19:28:14.686335 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.686328    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 02 19:28:14.686580 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.686338    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 02 19:28:14.686580 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.686371    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 02 19:28:14.686580 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.686374    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-sp8wk\""
Nov 02 19:28:14.686580 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.686454    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 02 19:28:14.686580 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.686466    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 02 19:28:14.686580 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.686476    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 02 19:28:14.686580 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.686511    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 02 19:28:14.686580 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.686518    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 02 19:28:14.686580 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.686523    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 02 19:28:14.686580 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.686524    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 02 19:28:14.686580 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.686543    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 02 19:28:14.686580 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.686551    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 02 19:28:14.686580 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.686537    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 02 19:28:14.686580 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.686511    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-c8nwr\""
Nov 02 19:28:14.686580 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.686513    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6bb8t\""
Nov 02 19:28:14.686580 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.686567    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 02 19:28:14.686580 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.686585    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9t85s\""
Nov 02 19:28:14.686580 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.686591    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-gsgfx\""
Nov 02 19:28:14.687215 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.686618    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-n2mjf\""
Nov 02 19:28:14.687215 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.686637    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-bg56f\""
Nov 02 19:28:14.687215 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.686721    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-t4nhx\""
Nov 02 19:28:14.687876 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.687865    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 02 19:28:14.724151 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.724134    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 02 19:28:14.724151 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.724147    4463 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 19:28:14.724264 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.724164    4463 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 19:28:14.724264 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:28:14.724188    4463 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-133-9.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-9.us-west-2.compute.internal"
Nov 02 19:28:14.724264 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:28:14.724199    4463 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-133-9.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-9.us-west-2.compute.internal"
Nov 02 19:28:14.730999 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.730984    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8756b800-1b4e-41da-934a-25c6b8ee738c-run-systemd\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:28:14.731066 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.731007    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8756b800-1b4e-41da-934a-25c6b8ee738c-host-run-ovn-kubernetes\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:28:14.731066 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.731038    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8756b800-1b4e-41da-934a-25c6b8ee738c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:28:14.731066 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.731053    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17fe5072-f4c5-4075-8614-4229ed05d4b2-metrics-certs\") pod \"network-metrics-daemon-f6r6d\" (UID: \"17fe5072-f4c5-4075-8614-4229ed05d4b2\") " pod="openshift-multus/network-metrics-daemon-f6r6d"
Nov 02 19:28:14.731165 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.731107    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/28b76ecd-8da7-428f-a825-323670ae5395-signing-key\") pod \"service-ca-5fffc97bdf-h5fc9\" (UID: \"28b76ecd-8da7-428f-a825-323670ae5395\") " pod="openshift-service-ca/service-ca-5fffc97bdf-h5fc9"
Nov 02 19:28:14.731203 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.731159    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8756b800-1b4e-41da-934a-25c6b8ee738c-etc-openvswitch\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:28:14.731203 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.731185    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/74106611-359e-4353-b113-70521144bed1-registration-dir\") pod \"aws-ebs-csi-driver-node-f5wvz\" (UID: \"74106611-359e-4353-b113-70521144bed1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5wvz"
Nov 02 19:28:14.731268 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.731228    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343-system-cni-dir\") pod \"multus-additional-cni-plugins-4ddbs\" (UID: \"4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343\") " pod="openshift-multus/multus-additional-cni-plugins-4ddbs"
Nov 02 19:28:14.731268 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.731250    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mptpf\" (UniqueName: \"kubernetes.io/projected/4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343-kube-api-access-mptpf\") pod \"multus-additional-cni-plugins-4ddbs\" (UID: \"4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343\") " pod="openshift-multus/multus-additional-cni-plugins-4ddbs"
Nov 02 19:28:14.731334 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.731268    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9b7f05bf-12cd-4aad-b4ee-8b517e901ca6-multus-socket-dir-parent\") pod \"multus-qhswb\" (UID: \"9b7f05bf-12cd-4aad-b4ee-8b517e901ca6\") " pod="openshift-multus/multus-qhswb"
Nov 02 19:28:14.731334 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.731291    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9b7f05bf-12cd-4aad-b4ee-8b517e901ca6-hostroot\") pod \"multus-qhswb\" (UID: \"9b7f05bf-12cd-4aad-b4ee-8b517e901ca6\") " pod="openshift-multus/multus-qhswb"
Nov 02 19:28:14.731402 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.731338    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a8e20267-1b77-4eb3-b940-a29d79b0ec19-etc-sysctl-d\") pod \"tuned-nvdtg\" (UID: \"a8e20267-1b77-4eb3-b940-a29d79b0ec19\") " pod="openshift-cluster-node-tuning-operator/tuned-nvdtg"
Nov 02 19:28:14.731402 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.731357    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a8e20267-1b77-4eb3-b940-a29d79b0ec19-sys\") pod \"tuned-nvdtg\" (UID: \"a8e20267-1b77-4eb3-b940-a29d79b0ec19\") " pod="openshift-cluster-node-tuning-operator/tuned-nvdtg"
Nov 02 19:28:14.731464 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.731401    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbxts\" (UniqueName: \"kubernetes.io/projected/17fe5072-f4c5-4075-8614-4229ed05d4b2-kube-api-access-lbxts\") pod \"network-metrics-daemon-f6r6d\" (UID: \"17fe5072-f4c5-4075-8614-4229ed05d4b2\") " pod="openshift-multus/network-metrics-daemon-f6r6d"
Nov 02 19:28:14.731464 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.731445    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9b7f05bf-12cd-4aad-b4ee-8b517e901ca6-host-run-k8s-cni-cncf-io\") pod \"multus-qhswb\" (UID: \"9b7f05bf-12cd-4aad-b4ee-8b517e901ca6\") " pod="openshift-multus/multus-qhswb"
Nov 02 19:28:14.731531 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.731470    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9b7f05bf-12cd-4aad-b4ee-8b517e901ca6-host-var-lib-cni-bin\") pod \"multus-qhswb\" (UID: \"9b7f05bf-12cd-4aad-b4ee-8b517e901ca6\") " pod="openshift-multus/multus-qhswb"
Nov 02 19:28:14.731624 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.731554    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9b7f05bf-12cd-4aad-b4ee-8b517e901ca6-host-var-lib-cni-multus\") pod \"multus-qhswb\" (UID: \"9b7f05bf-12cd-4aad-b4ee-8b517e901ca6\") " pod="openshift-multus/multus-qhswb"
Nov 02 19:28:14.731624 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.731584    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a8e20267-1b77-4eb3-b940-a29d79b0ec19-etc-systemd\") pod \"tuned-nvdtg\" (UID: \"a8e20267-1b77-4eb3-b940-a29d79b0ec19\") " pod="openshift-cluster-node-tuning-operator/tuned-nvdtg"
Nov 02 19:28:14.731624 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.731604    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxwsm\" (UniqueName: \"kubernetes.io/projected/b6ad2c98-52b5-496e-9824-614a2e509a8a-kube-api-access-sxwsm\") pod \"node-ca-hrhpj\" (UID: \"b6ad2c98-52b5-496e-9824-614a2e509a8a\") " pod="openshift-image-registry/node-ca-hrhpj"
Nov 02 19:28:14.731732 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.731645    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4874b345-eac7-4694-8351-d24085bf180a-service-ca-bundle\") pod \"periodic-gathering-xvzjr-ggg8w\" (UID: \"4874b345-eac7-4694-8351-d24085bf180a\") " pod="openshift-insights/periodic-gathering-xvzjr-ggg8w"
Nov 02 19:28:14.731732 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.731719    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8756b800-1b4e-41da-934a-25c6b8ee738c-ovn-node-metrics-cert\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:28:14.731808 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.731745    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9b7f05bf-12cd-4aad-b4ee-8b517e901ca6-cni-binary-copy\") pod \"multus-qhswb\" (UID: \"9b7f05bf-12cd-4aad-b4ee-8b517e901ca6\") " pod="openshift-multus/multus-qhswb"
Nov 02 19:28:14.731808 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.731769    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a8e20267-1b77-4eb3-b940-a29d79b0ec19-lib-modules\") pod \"tuned-nvdtg\" (UID: \"a8e20267-1b77-4eb3-b940-a29d79b0ec19\") " pod="openshift-cluster-node-tuning-operator/tuned-nvdtg"
Nov 02 19:28:14.731808 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.731798    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcgjq\" (UniqueName: \"kubernetes.io/projected/54c1de40-ce9d-4d7d-82ca-c5fa41c97e97-kube-api-access-fcgjq\") pod \"migrator-66df5d496b-zqz7w\" (UID: \"54c1de40-ce9d-4d7d-82ca-c5fa41c97e97\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-zqz7w"
Nov 02 19:28:14.731917 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.731821    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgmdg\" (UniqueName: \"kubernetes.io/projected/dfb225e8-fabd-49fb-9763-4e41bf6ae7de-kube-api-access-rgmdg\") pod \"dns-default-qj8mr\" (UID: \"dfb225e8-fabd-49fb-9763-4e41bf6ae7de\") " pod="openshift-dns/dns-default-qj8mr"
Nov 02 19:28:14.731917 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.731844    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8756b800-1b4e-41da-934a-25c6b8ee738c-systemd-units\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:28:14.731917 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.731868    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/74106611-359e-4353-b113-70521144bed1-etc-selinux\") pod \"aws-ebs-csi-driver-node-f5wvz\" (UID: \"74106611-359e-4353-b113-70521144bed1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5wvz"
Nov 02 19:28:14.731917 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.731913    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a8e20267-1b77-4eb3-b940-a29d79b0ec19-etc-sysconfig\") pod \"tuned-nvdtg\" (UID: \"a8e20267-1b77-4eb3-b940-a29d79b0ec19\") " pod="openshift-cluster-node-tuning-operator/tuned-nvdtg"
Nov 02 19:28:14.732068 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.731933    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a8e20267-1b77-4eb3-b940-a29d79b0ec19-etc-sysctl-conf\") pod \"tuned-nvdtg\" (UID: \"a8e20267-1b77-4eb3-b940-a29d79b0ec19\") " pod="openshift-cluster-node-tuning-operator/tuned-nvdtg"
Nov 02 19:28:14.732068 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.731962    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd804657-39fd-4ae9-899d-cd1f1b62717e-tmp-dir\") pod \"node-resolver-r9fcz\" (UID: \"dd804657-39fd-4ae9-899d-cd1f1b62717e\") " pod="openshift-dns/node-resolver-r9fcz"
Nov 02 19:28:14.732068 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.732052    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd804657-39fd-4ae9-899d-cd1f1b62717e-tmp-dir\") pod \"node-resolver-r9fcz\" (UID: \"dd804657-39fd-4ae9-899d-cd1f1b62717e\") " pod="openshift-dns/node-resolver-r9fcz"
Nov 02 19:28:14.732174 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.732057    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8756b800-1b4e-41da-934a-25c6b8ee738c-ovn-node-metrics-cert\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:28:14.732174 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.732084    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9b7f05bf-12cd-4aad-b4ee-8b517e901ca6-cni-binary-copy\") pod \"multus-qhswb\" (UID: \"9b7f05bf-12cd-4aad-b4ee-8b517e901ca6\") " pod="openshift-multus/multus-qhswb"
Nov 02 19:28:14.732174 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.732108    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dfb225e8-fabd-49fb-9763-4e41bf6ae7de-metrics-tls\") pod \"dns-default-qj8mr\" (UID: \"dfb225e8-fabd-49fb-9763-4e41bf6ae7de\") " pod="openshift-dns/dns-default-qj8mr"
Nov 02 19:28:14.732174 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.732142    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3fabcb64-faf2-49f9-813a-ddafde0fd159-agent-certs\") pod \"konnectivity-agent-jgfjj\" (UID: \"3fabcb64-faf2-49f9-813a-ddafde0fd159\") " pod="kube-system/konnectivity-agent-jgfjj"
Nov 02 19:28:14.732174 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.732164    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2-data-volume\") pod \"insights-runtime-extractor-bh65q\" (UID: \"0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2\") " pod="openshift-insights/insights-runtime-extractor-bh65q"
Nov 02 19:28:14.732331 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.732182    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2g6f\" (UniqueName: \"kubernetes.io/projected/0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2-kube-api-access-j2g6f\") pod \"insights-runtime-extractor-bh65q\" (UID: \"0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2\") " pod="openshift-insights/insights-runtime-extractor-bh65q"
Nov 02 19:28:14.732331 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.732200    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/74106611-359e-4353-b113-70521144bed1-socket-dir\") pod \"aws-ebs-csi-driver-node-f5wvz\" (UID: \"74106611-359e-4353-b113-70521144bed1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5wvz"
Nov 02 19:28:14.732331 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.732215    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4ddbs\" (UID: \"4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343\") " pod="openshift-multus/multus-additional-cni-plugins-4ddbs"
Nov 02 19:28:14.732331 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.732230    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9b7f05bf-12cd-4aad-b4ee-8b517e901ca6-host-run-netns\") pod \"multus-qhswb\" (UID: \"9b7f05bf-12cd-4aad-b4ee-8b517e901ca6\") " pod="openshift-multus/multus-qhswb"
Nov 02 19:28:14.732331 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.732232    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2-data-volume\") pod \"insights-runtime-extractor-bh65q\" (UID: \"0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2\") " pod="openshift-insights/insights-runtime-extractor-bh65q"
Nov 02 19:28:14.732331 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.732245    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/dd804657-39fd-4ae9-899d-cd1f1b62717e-hosts-file\") pod \"node-resolver-r9fcz\" (UID: \"dd804657-39fd-4ae9-899d-cd1f1b62717e\") " pod="openshift-dns/node-resolver-r9fcz"
Nov 02 19:28:14.732522 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.732369    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5d436341-2000-4c2b-87ac-7b67e4cdb909-cert\") pod \"ingress-canary-sw9mp\" (UID: \"5d436341-2000-4c2b-87ac-7b67e4cdb909\") " pod="openshift-ingress-canary/ingress-canary-sw9mp"
Nov 02 19:28:14.732522 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.732401    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4ddbs\" (UID: \"4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343\") " pod="openshift-multus/multus-additional-cni-plugins-4ddbs"
Nov 02 19:28:14.732522 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.732413    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3fabcb64-faf2-49f9-813a-ddafde0fd159-agent-certs\") pod \"konnectivity-agent-jgfjj\" (UID: \"3fabcb64-faf2-49f9-813a-ddafde0fd159\") " pod="kube-system/konnectivity-agent-jgfjj"
Nov 02 19:28:14.732522 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.732459    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3fabcb64-faf2-49f9-813a-ddafde0fd159-konnectivity-ca\") pod \"konnectivity-agent-jgfjj\" (UID: \"3fabcb64-faf2-49f9-813a-ddafde0fd159\") " pod="kube-system/konnectivity-agent-jgfjj"
Nov 02 19:28:14.732670 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.732518    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bh65q\" (UID: \"0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2\") " pod="openshift-insights/insights-runtime-extractor-bh65q"
Nov 02 19:28:14.732670 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.732612    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8756b800-1b4e-41da-934a-25c6b8ee738c-log-socket\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:28:14.732670 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.732636    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8756b800-1b4e-41da-934a-25c6b8ee738c-env-overrides\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:28:14.732670 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.732650    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3fabcb64-faf2-49f9-813a-ddafde0fd159-konnectivity-ca\") pod \"konnectivity-agent-jgfjj\" (UID: \"3fabcb64-faf2-49f9-813a-ddafde0fd159\") " pod="kube-system/konnectivity-agent-jgfjj"
Nov 02 19:28:14.732670 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.732661    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9b7f05bf-12cd-4aad-b4ee-8b517e901ca6-multus-daemon-config\") pod \"multus-qhswb\" (UID: \"9b7f05bf-12cd-4aad-b4ee-8b517e901ca6\") " pod="openshift-multus/multus-qhswb"
Nov 02 19:28:14.732820 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.732766    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8756b800-1b4e-41da-934a-25c6b8ee738c-env-overrides\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:28:14.732820 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.732784    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9b7f05bf-12cd-4aad-b4ee-8b517e901ca6-etc-kubernetes\") pod \"multus-qhswb\" (UID: \"9b7f05bf-12cd-4aad-b4ee-8b517e901ca6\") " pod="openshift-multus/multus-qhswb"
Nov 02 19:28:14.732820 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.732806    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a8e20267-1b77-4eb3-b940-a29d79b0ec19-etc-modprobe-d\") pod \"tuned-nvdtg\" (UID: \"a8e20267-1b77-4eb3-b940-a29d79b0ec19\") " pod="openshift-cluster-node-tuning-operator/tuned-nvdtg"
Nov 02 19:28:14.732905 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.732824    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4874b345-eac7-4694-8351-d24085bf180a-archives-path\") pod \"periodic-gathering-xvzjr-ggg8w\" (UID: \"4874b345-eac7-4694-8351-d24085bf180a\") " pod="openshift-insights/periodic-gathering-xvzjr-ggg8w"
Nov 02 19:28:14.732905 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.732839    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4874b345-eac7-4694-8351-d24085bf180a-trusted-ca-bundle\") pod \"periodic-gathering-xvzjr-ggg8w\" (UID: \"4874b345-eac7-4694-8351-d24085bf180a\") " pod="openshift-insights/periodic-gathering-xvzjr-ggg8w"
Nov 02 19:28:14.732905 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.732865    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jz8qt\" (UniqueName: \"kubernetes.io/projected/4874b345-eac7-4694-8351-d24085bf180a-kube-api-access-jz8qt\") pod \"periodic-gathering-xvzjr-ggg8w\" (UID: \"4874b345-eac7-4694-8351-d24085bf180a\") " pod="openshift-insights/periodic-gathering-xvzjr-ggg8w"
Nov 02 19:28:14.732905 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.732891    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/28b76ecd-8da7-428f-a825-323670ae5395-signing-cabundle\") pod \"service-ca-5fffc97bdf-h5fc9\" (UID: \"28b76ecd-8da7-428f-a825-323670ae5395\") " pod="openshift-service-ca/service-ca-5fffc97bdf-h5fc9"
Nov 02 19:28:14.733082 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.732914    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9b7f05bf-12cd-4aad-b4ee-8b517e901ca6-multus-daemon-config\") pod \"multus-qhswb\" (UID: \"9b7f05bf-12cd-4aad-b4ee-8b517e901ca6\") " pod="openshift-multus/multus-qhswb"
Nov 02 19:28:14.733082 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.732919    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4874b345-eac7-4694-8351-d24085bf180a-archives-path\") pod \"periodic-gathering-xvzjr-ggg8w\" (UID: \"4874b345-eac7-4694-8351-d24085bf180a\") " pod="openshift-insights/periodic-gathering-xvzjr-ggg8w"
Nov 02 19:28:14.733082 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.732914    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9b7f05bf-12cd-4aad-b4ee-8b517e901ca6-system-cni-dir\") pod \"multus-qhswb\" (UID: \"9b7f05bf-12cd-4aad-b4ee-8b517e901ca6\") " pod="openshift-multus/multus-qhswb"
Nov 02 19:28:14.733082 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.733056    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a8e20267-1b77-4eb3-b940-a29d79b0ec19-etc-kubernetes\") pod \"tuned-nvdtg\" (UID: \"a8e20267-1b77-4eb3-b940-a29d79b0ec19\") " pod="openshift-cluster-node-tuning-operator/tuned-nvdtg"
Nov 02 19:28:14.733211 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.733087    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a8e20267-1b77-4eb3-b940-a29d79b0ec19-host\") pod \"tuned-nvdtg\" (UID: \"a8e20267-1b77-4eb3-b940-a29d79b0ec19\") " pod="openshift-cluster-node-tuning-operator/tuned-nvdtg"
Nov 02 19:28:14.733211 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.733115    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/dfb225e8-fabd-49fb-9763-4e41bf6ae7de-config-volume\") pod \"dns-default-qj8mr\" (UID: \"dfb225e8-fabd-49fb-9763-4e41bf6ae7de\") " pod="openshift-dns/dns-default-qj8mr"
Nov 02 19:28:14.733211 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.733145    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8756b800-1b4e-41da-934a-25c6b8ee738c-ovnkube-script-lib\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:28:14.733304 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.733225    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-828qc\" (UniqueName: \"kubernetes.io/projected/8756b800-1b4e-41da-934a-25c6b8ee738c-kube-api-access-828qc\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:28:14.733343 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.733305    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/74106611-359e-4353-b113-70521144bed1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-f5wvz\" (UID: \"74106611-359e-4353-b113-70521144bed1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5wvz"
Nov 02 19:28:14.733343 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.733330    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8756b800-1b4e-41da-934a-25c6b8ee738c-ovnkube-script-lib\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:28:14.733343 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.733336    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343-cni-binary-copy\") pod \"multus-additional-cni-plugins-4ddbs\" (UID: \"4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343\") " pod="openshift-multus/multus-additional-cni-plugins-4ddbs"
Nov 02 19:28:14.733470 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.733368    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4ddbs\" (UID: \"4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343\") " pod="openshift-multus/multus-additional-cni-plugins-4ddbs"
Nov 02 19:28:14.733470 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.733409    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d956\" (UniqueName: \"kubernetes.io/projected/9b7f05bf-12cd-4aad-b4ee-8b517e901ca6-kube-api-access-2d956\") pod \"multus-qhswb\" (UID: \"9b7f05bf-12cd-4aad-b4ee-8b517e901ca6\") " pod="openshift-multus/multus-qhswb"
Nov 02 19:28:14.733527 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.733481    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343-cni-binary-copy\") pod \"multus-additional-cni-plugins-4ddbs\" (UID: \"4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343\") " pod="openshift-multus/multus-additional-cni-plugins-4ddbs"
Nov 02 19:28:14.733527 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.733487    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a8e20267-1b77-4eb3-b940-a29d79b0ec19-tmp\") pod \"tuned-nvdtg\" (UID: \"a8e20267-1b77-4eb3-b940-a29d79b0ec19\") " pod="openshift-cluster-node-tuning-operator/tuned-nvdtg"
Nov 02 19:28:14.733527 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.733519    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0b2d7938-0c97-4f07-a7d2-b5c35d2bc8f9-kubelet-config\") pod \"global-pull-secret-syncer-4b2pr\" (UID: \"0b2d7938-0c97-4f07-a7d2-b5c35d2bc8f9\") " pod="kube-system/global-pull-secret-syncer-4b2pr"
Nov 02 19:28:14.733655 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.733546    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgbw6\" (UniqueName: \"kubernetes.io/projected/5d436341-2000-4c2b-87ac-7b67e4cdb909-kube-api-access-dgbw6\") pod \"ingress-canary-sw9mp\" (UID: \"5d436341-2000-4c2b-87ac-7b67e4cdb909\") " pod="openshift-ingress-canary/ingress-canary-sw9mp"
Nov 02 19:28:14.733655 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.733571    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b6cb1c47-e00c-484d-bf34-445b9380553c-iptables-alerter-script\") pod \"iptables-alerter-cmk66\" (UID: \"b6cb1c47-e00c-484d-bf34-445b9380553c\") " pod="openshift-network-operator/iptables-alerter-cmk66"
Nov 02 19:28:14.733655 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.733573    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4ddbs\" (UID: \"4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343\") " pod="openshift-multus/multus-additional-cni-plugins-4ddbs"
Nov 02 19:28:14.733655 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.733602    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a8e20267-1b77-4eb3-b940-a29d79b0ec19-tmp\") pod \"tuned-nvdtg\" (UID: \"a8e20267-1b77-4eb3-b940-a29d79b0ec19\") " pod="openshift-cluster-node-tuning-operator/tuned-nvdtg"
Nov 02 19:28:14.733655 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.733606    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b6cb1c47-e00c-484d-bf34-445b9380553c-host-slash\") pod \"iptables-alerter-cmk66\" (UID: \"b6cb1c47-e00c-484d-bf34-445b9380553c\") " pod="openshift-network-operator/iptables-alerter-cmk66"
Nov 02 19:28:14.733813 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.733717    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9b7f05bf-12cd-4aad-b4ee-8b517e901ca6-host-run-multus-certs\") pod \"multus-qhswb\" (UID: \"9b7f05bf-12cd-4aad-b4ee-8b517e901ca6\") " pod="openshift-multus/multus-qhswb"
Nov 02 19:28:14.733813 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.733761    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b6cb1c47-e00c-484d-bf34-445b9380553c-iptables-alerter-script\") pod \"iptables-alerter-cmk66\" (UID: \"b6cb1c47-e00c-484d-bf34-445b9380553c\") " pod="openshift-network-operator/iptables-alerter-cmk66"
Nov 02 19:28:14.733813 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.733758    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b2d7938-0c97-4f07-a7d2-b5c35d2bc8f9-original-pull-secret\") pod \"global-pull-secret-syncer-4b2pr\" (UID: \"0b2d7938-0c97-4f07-a7d2-b5c35d2bc8f9\") " pod="kube-system/global-pull-secret-syncer-4b2pr"
Nov 02 19:28:14.733916 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.733846    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4sz9\" (UniqueName: \"kubernetes.io/projected/b6cb1c47-e00c-484d-bf34-445b9380553c-kube-api-access-l4sz9\") pod \"iptables-alerter-cmk66\" (UID: \"b6cb1c47-e00c-484d-bf34-445b9380553c\") " pod="openshift-network-operator/iptables-alerter-cmk66"
Nov 02 19:28:14.733916 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.733867    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8756b800-1b4e-41da-934a-25c6b8ee738c-host-kubelet\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:28:14.733916 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.733894    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8756b800-1b4e-41da-934a-25c6b8ee738c-node-log\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:28:14.734026 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.733989    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b2d7938-0c97-4f07-a7d2-b5c35d2bc8f9-original-pull-secret\") pod \"global-pull-secret-syncer-4b2pr\" (UID: \"0b2d7938-0c97-4f07-a7d2-b5c35d2bc8f9\") " pod="kube-system/global-pull-secret-syncer-4b2pr"
Nov 02 19:28:14.734026 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.733992    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8756b800-1b4e-41da-934a-25c6b8ee738c-host-cni-bin\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:28:14.734094 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.734050    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a8e20267-1b77-4eb3-b940-a29d79b0ec19-var-lib-kubelet\") pod \"tuned-nvdtg\" (UID: \"a8e20267-1b77-4eb3-b940-a29d79b0ec19\") " pod="openshift-cluster-node-tuning-operator/tuned-nvdtg"
Nov 02 19:28:14.734094 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.734076    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0b2d7938-0c97-4f07-a7d2-b5c35d2bc8f9-dbus\") pod \"global-pull-secret-syncer-4b2pr\" (UID: \"0b2d7938-0c97-4f07-a7d2-b5c35d2bc8f9\") " pod="kube-system/global-pull-secret-syncer-4b2pr"
Nov 02 19:28:14.734094 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.734092    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgbgt\" (UniqueName: \"kubernetes.io/projected/74106611-359e-4353-b113-70521144bed1-kube-api-access-pgbgt\") pod \"aws-ebs-csi-driver-node-f5wvz\" (UID: \"74106611-359e-4353-b113-70521144bed1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5wvz"
Nov 02 19:28:14.734234 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.734111    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n45ql\" (UniqueName: \"kubernetes.io/projected/28b76ecd-8da7-428f-a825-323670ae5395-kube-api-access-n45ql\") pod \"service-ca-5fffc97bdf-h5fc9\" (UID: \"28b76ecd-8da7-428f-a825-323670ae5395\") " pod="openshift-service-ca/service-ca-5fffc97bdf-h5fc9"
Nov 02 19:28:14.734234 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.734178    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343-cnibin\") pod \"multus-additional-cni-plugins-4ddbs\" (UID: \"4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343\") " pod="openshift-multus/multus-additional-cni-plugins-4ddbs"
Nov 02 19:28:14.734234 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.734215    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a8e20267-1b77-4eb3-b940-a29d79b0ec19-etc-tuned\") pod \"tuned-nvdtg\" (UID: \"a8e20267-1b77-4eb3-b940-a29d79b0ec19\") " pod="openshift-cluster-node-tuning-operator/tuned-nvdtg"
Nov 02 19:28:14.734383 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.734244    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcpxd\" (UniqueName: \"kubernetes.io/projected/dd804657-39fd-4ae9-899d-cd1f1b62717e-kube-api-access-bcpxd\") pod \"node-resolver-r9fcz\" (UID: \"dd804657-39fd-4ae9-899d-cd1f1b62717e\") " pod="openshift-dns/node-resolver-r9fcz"
Nov 02 19:28:14.734383 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.734271    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccf8l\" (UniqueName: \"kubernetes.io/projected/31a69a0e-cc9c-47ad-b577-36b50d61d213-kube-api-access-ccf8l\") pod \"network-check-target-rhdd6\" (UID: \"31a69a0e-cc9c-47ad-b577-36b50d61d213\") " pod="openshift-network-diagnostics/network-check-target-rhdd6"
Nov 02 19:28:14.734383 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.734299    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2-crio-socket\") pod \"insights-runtime-extractor-bh65q\" (UID: \"0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2\") " pod="openshift-insights/insights-runtime-extractor-bh65q"
Nov 02 19:28:14.734383 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.734324    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8756b800-1b4e-41da-934a-25c6b8ee738c-host-run-netns\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:28:14.734383 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.734330    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a8e20267-1b77-4eb3-b940-a29d79b0ec19-etc-tuned\") pod \"tuned-nvdtg\" (UID: \"a8e20267-1b77-4eb3-b940-a29d79b0ec19\") " pod="openshift-cluster-node-tuning-operator/tuned-nvdtg"
Nov 02 19:28:14.734383 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.734347    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8756b800-1b4e-41da-934a-25c6b8ee738c-run-ovn\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:28:14.734684 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.734387    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4ddbs\" (UID: \"4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343\") " pod="openshift-multus/multus-additional-cni-plugins-4ddbs"
Nov 02 19:28:14.734684 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.734418    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9b7f05bf-12cd-4aad-b4ee-8b517e901ca6-multus-conf-dir\") pod \"multus-qhswb\" (UID: \"9b7f05bf-12cd-4aad-b4ee-8b517e901ca6\") " pod="openshift-multus/multus-qhswb"
Nov 02 19:28:14.734684 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.734458    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jxxh\" (UniqueName: \"kubernetes.io/projected/a8e20267-1b77-4eb3-b940-a29d79b0ec19-kube-api-access-7jxxh\") pod \"tuned-nvdtg\" (UID: \"a8e20267-1b77-4eb3-b940-a29d79b0ec19\") " pod="openshift-cluster-node-tuning-operator/tuned-nvdtg"
Nov 02 19:28:14.734684 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.734502    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8756b800-1b4e-41da-934a-25c6b8ee738c-host-slash\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:28:14.734684 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.734519    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8756b800-1b4e-41da-934a-25c6b8ee738c-var-lib-openvswitch\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:28:14.734684 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.734534    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8756b800-1b4e-41da-934a-25c6b8ee738c-host-cni-netd\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:28:14.734684 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.734548    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/74106611-359e-4353-b113-70521144bed1-sys-fs\") pod \"aws-ebs-csi-driver-node-f5wvz\" (UID: \"74106611-359e-4353-b113-70521144bed1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5wvz"
Nov 02 19:28:14.734684 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.734561    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a8e20267-1b77-4eb3-b940-a29d79b0ec19-run\") pod \"tuned-nvdtg\" (UID: \"a8e20267-1b77-4eb3-b940-a29d79b0ec19\") " pod="openshift-cluster-node-tuning-operator/tuned-nvdtg"
Nov 02 19:28:14.734684 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.734587    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b6ad2c98-52b5-496e-9824-614a2e509a8a-serviceca\") pod \"node-ca-hrhpj\" (UID: \"b6ad2c98-52b5-496e-9824-614a2e509a8a\") " pod="openshift-image-registry/node-ca-hrhpj"
Nov 02 19:28:14.734684 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.734626    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8756b800-1b4e-41da-934a-25c6b8ee738c-run-openvswitch\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:28:14.734684 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.734670    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8756b800-1b4e-41da-934a-25c6b8ee738c-ovnkube-config\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:28:14.735060 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.734696    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9b7f05bf-12cd-4aad-b4ee-8b517e901ca6-multus-cni-dir\") pod \"multus-qhswb\" (UID: \"9b7f05bf-12cd-4aad-b4ee-8b517e901ca6\") " pod="openshift-multus/multus-qhswb"
Nov 02 19:28:14.735060 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.734718    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b6ad2c98-52b5-496e-9824-614a2e509a8a-host\") pod \"node-ca-hrhpj\" (UID: \"b6ad2c98-52b5-496e-9824-614a2e509a8a\") " pod="openshift-image-registry/node-ca-hrhpj"
Nov 02 19:28:14.735060 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.734740    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfb225e8-fabd-49fb-9763-4e41bf6ae7de-tmp-dir\") pod \"dns-default-qj8mr\" (UID: \"dfb225e8-fabd-49fb-9763-4e41bf6ae7de\") " pod="openshift-dns/dns-default-qj8mr"
Nov 02 19:28:14.735060 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.734765    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bh65q\" (UID: \"0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2\") " pod="openshift-insights/insights-runtime-extractor-bh65q"
Nov 02 19:28:14.735060 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.734781    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b6ad2c98-52b5-496e-9824-614a2e509a8a-serviceca\") pod \"node-ca-hrhpj\" (UID: \"b6ad2c98-52b5-496e-9824-614a2e509a8a\") " pod="openshift-image-registry/node-ca-hrhpj"
Nov 02 19:28:14.735060 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.734803    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/74106611-359e-4353-b113-70521144bed1-device-dir\") pod \"aws-ebs-csi-driver-node-f5wvz\" (UID: \"74106611-359e-4353-b113-70521144bed1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5wvz"
Nov 02 19:28:14.735060 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.734818    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8756b800-1b4e-41da-934a-25c6b8ee738c-ovnkube-config\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:28:14.735060 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.734831    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9b7f05bf-12cd-4aad-b4ee-8b517e901ca6-cnibin\") pod \"multus-qhswb\" (UID: \"9b7f05bf-12cd-4aad-b4ee-8b517e901ca6\") " pod="openshift-multus/multus-qhswb"
Nov 02 19:28:14.735060 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.734854    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9b7f05bf-12cd-4aad-b4ee-8b517e901ca6-host-var-lib-kubelet\") pod \"multus-qhswb\" (UID: \"9b7f05bf-12cd-4aad-b4ee-8b517e901ca6\") " pod="openshift-multus/multus-qhswb"
Nov 02 19:28:14.735060 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.734859    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfb225e8-fabd-49fb-9763-4e41bf6ae7de-tmp-dir\") pod \"dns-default-qj8mr\" (UID: \"dfb225e8-fabd-49fb-9763-4e41bf6ae7de\") " pod="openshift-dns/dns-default-qj8mr"
Nov 02 19:28:14.780374 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.780351    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 02 19:28:14.780848 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.780817    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 02 19:28:14.781142 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.781126    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Nov 02 19:28:14.781233 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.781178    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 02 19:28:14.781338 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.781323    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-k9tfs\""
Nov 02 19:28:14.781390 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.781350    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-dc6gd\""
Nov 02 19:28:14.781390 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.781378    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/28b76ecd-8da7-428f-a825-323670ae5395-signing-key\") pod \"service-ca-5fffc97bdf-h5fc9\" (UID: \"28b76ecd-8da7-428f-a825-323670ae5395\") " pod="openshift-service-ca/service-ca-5fffc97bdf-h5fc9"
Nov 02 19:28:14.781465 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.781454    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-g68w8\""
Nov 02 19:28:14.781494 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.781453    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-fbvgr\""
Nov 02 19:28:14.781621 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.781608    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-m9rmj\""
Nov 02 19:28:14.781702 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.781686    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-rnxqc\""
Nov 02 19:28:14.781733 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.781725    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-fzxnx\""
Nov 02 19:28:14.782766 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.782750    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5d436341-2000-4c2b-87ac-7b67e4cdb909-cert\") pod \"ingress-canary-sw9mp\" (UID: \"5d436341-2000-4c2b-87ac-7b67e4cdb909\") " pod="openshift-ingress-canary/ingress-canary-sw9mp"
Nov 02 19:28:14.782806 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.782792    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 02 19:28:14.783127 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.783113    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Nov 02 19:28:14.783208 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.783197    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 02 19:28:14.783248 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.783203    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 02 19:28:14.783248 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.783231    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 02 19:28:14.783248 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.783237    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Nov 02 19:28:14.783433 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.783415    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 02 19:28:14.783514 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.783499    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Nov 02 19:28:14.783575 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.783503    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 02 19:28:14.783575 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.783507    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 02 19:28:14.783670 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.783661    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 02 19:28:14.783727 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.783686    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 02 19:28:14.783782 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.783750    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 02 19:28:14.784698 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.784682    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbxts\" (UniqueName: \"kubernetes.io/projected/17fe5072-f4c5-4075-8614-4229ed05d4b2-kube-api-access-lbxts\") pod \"network-metrics-daemon-f6r6d\" (UID: \"17fe5072-f4c5-4075-8614-4229ed05d4b2\") " pod="openshift-multus/network-metrics-daemon-f6r6d"
Nov 02 19:28:14.784789 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.784712    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 02 19:28:14.785286 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.785267    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2g6f\" (UniqueName: \"kubernetes.io/projected/0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2-kube-api-access-j2g6f\") pod \"insights-runtime-extractor-bh65q\" (UID: \"0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2\") " pod="openshift-insights/insights-runtime-extractor-bh65q"
Nov 02 19:28:14.785365 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.785301    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgmdg\" (UniqueName: \"kubernetes.io/projected/dfb225e8-fabd-49fb-9763-4e41bf6ae7de-kube-api-access-rgmdg\") pod \"dns-default-qj8mr\" (UID: \"dfb225e8-fabd-49fb-9763-4e41bf6ae7de\") " pod="openshift-dns/dns-default-qj8mr"
Nov 02 19:28:14.785400 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.785358    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jz8qt\" (UniqueName: \"kubernetes.io/projected/4874b345-eac7-4694-8351-d24085bf180a-kube-api-access-jz8qt\") pod \"periodic-gathering-xvzjr-ggg8w\" (UID: \"4874b345-eac7-4694-8351-d24085bf180a\") " pod="openshift-insights/periodic-gathering-xvzjr-ggg8w"
Nov 02 19:28:14.785431 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.785404    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-828qc\" (UniqueName: \"kubernetes.io/projected/8756b800-1b4e-41da-934a-25c6b8ee738c-kube-api-access-828qc\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:28:14.785481 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.785463    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcgjq\" (UniqueName: \"kubernetes.io/projected/54c1de40-ce9d-4d7d-82ca-c5fa41c97e97-kube-api-access-fcgjq\") pod \"migrator-66df5d496b-zqz7w\" (UID: \"54c1de40-ce9d-4d7d-82ca-c5fa41c97e97\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-zqz7w"
Nov 02 19:28:14.785512 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.785470    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxwsm\" (UniqueName: \"kubernetes.io/projected/b6ad2c98-52b5-496e-9824-614a2e509a8a-kube-api-access-sxwsm\") pod \"node-ca-hrhpj\" (UID: \"b6ad2c98-52b5-496e-9824-614a2e509a8a\") " pod="openshift-image-registry/node-ca-hrhpj"
Nov 02 19:28:14.785512 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.785497    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mptpf\" (UniqueName: \"kubernetes.io/projected/4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343-kube-api-access-mptpf\") pod \"multus-additional-cni-plugins-4ddbs\" (UID: \"4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343\") " pod="openshift-multus/multus-additional-cni-plugins-4ddbs"
Nov 02 19:28:14.787095 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.787077    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4sz9\" (UniqueName: \"kubernetes.io/projected/b6cb1c47-e00c-484d-bf34-445b9380553c-kube-api-access-l4sz9\") pod \"iptables-alerter-cmk66\" (UID: \"b6cb1c47-e00c-484d-bf34-445b9380553c\") " pod="openshift-network-operator/iptables-alerter-cmk66"
Nov 02 19:28:14.787095 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.787085    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2d956\" (UniqueName: \"kubernetes.io/projected/9b7f05bf-12cd-4aad-b4ee-8b517e901ca6-kube-api-access-2d956\") pod \"multus-qhswb\" (UID: \"9b7f05bf-12cd-4aad-b4ee-8b517e901ca6\") " pod="openshift-multus/multus-qhswb"
Nov 02 19:28:14.787204 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.787112    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n45ql\" (UniqueName: \"kubernetes.io/projected/28b76ecd-8da7-428f-a825-323670ae5395-kube-api-access-n45ql\") pod \"service-ca-5fffc97bdf-h5fc9\" (UID: \"28b76ecd-8da7-428f-a825-323670ae5395\") " pod="openshift-service-ca/service-ca-5fffc97bdf-h5fc9"
Nov 02 19:28:14.787742 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.787678    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgbw6\" (UniqueName: \"kubernetes.io/projected/5d436341-2000-4c2b-87ac-7b67e4cdb909-kube-api-access-dgbw6\") pod \"ingress-canary-sw9mp\" (UID: \"5d436341-2000-4c2b-87ac-7b67e4cdb909\") " pod="openshift-ingress-canary/ingress-canary-sw9mp"
Nov 02 19:28:14.787868 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.787851    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcpxd\" (UniqueName: \"kubernetes.io/projected/dd804657-39fd-4ae9-899d-cd1f1b62717e-kube-api-access-bcpxd\") pod \"node-resolver-r9fcz\" (UID: \"dd804657-39fd-4ae9-899d-cd1f1b62717e\") " pod="openshift-dns/node-resolver-r9fcz"
Nov 02 19:28:14.787917 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.787859    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccf8l\" (UniqueName: \"kubernetes.io/projected/31a69a0e-cc9c-47ad-b577-36b50d61d213-kube-api-access-ccf8l\") pod \"network-check-target-rhdd6\" (UID: \"31a69a0e-cc9c-47ad-b577-36b50d61d213\") " pod="openshift-network-diagnostics/network-check-target-rhdd6"
Nov 02 19:28:14.787955 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.787921    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgbgt\" (UniqueName: \"kubernetes.io/projected/74106611-359e-4353-b113-70521144bed1-kube-api-access-pgbgt\") pod \"aws-ebs-csi-driver-node-f5wvz\" (UID: \"74106611-359e-4353-b113-70521144bed1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5wvz"
Nov 02 19:28:14.788249 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.788234    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jxxh\" (UniqueName: \"kubernetes.io/projected/a8e20267-1b77-4eb3-b940-a29d79b0ec19-kube-api-access-7jxxh\") pod \"tuned-nvdtg\" (UID: \"a8e20267-1b77-4eb3-b940-a29d79b0ec19\") " pod="openshift-cluster-node-tuning-operator/tuned-nvdtg"
Nov 02 19:28:14.792074 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.792059    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4874b345-eac7-4694-8351-d24085bf180a-service-ca-bundle\") pod \"periodic-gathering-xvzjr-ggg8w\" (UID: \"4874b345-eac7-4694-8351-d24085bf180a\") " pod="openshift-insights/periodic-gathering-xvzjr-ggg8w"
Nov 02 19:28:14.793282 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.793262    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/28b76ecd-8da7-428f-a825-323670ae5395-signing-cabundle\") pod \"service-ca-5fffc97bdf-h5fc9\" (UID: \"28b76ecd-8da7-428f-a825-323670ae5395\") " pod="openshift-service-ca/service-ca-5fffc97bdf-h5fc9"
Nov 02 19:28:14.793321 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.793308    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/dfb225e8-fabd-49fb-9763-4e41bf6ae7de-config-volume\") pod \"dns-default-qj8mr\" (UID: \"dfb225e8-fabd-49fb-9763-4e41bf6ae7de\") " pod="openshift-dns/dns-default-qj8mr"
Nov 02 19:28:14.793350 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.793317    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4874b345-eac7-4694-8351-d24085bf180a-trusted-ca-bundle\") pod \"periodic-gathering-xvzjr-ggg8w\" (UID: \"4874b345-eac7-4694-8351-d24085bf180a\") " pod="openshift-insights/periodic-gathering-xvzjr-ggg8w"
Nov 02 19:28:14.835075 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.835053    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9b7f05bf-12cd-4aad-b4ee-8b517e901ca6-multus-socket-dir-parent\") pod \"multus-qhswb\" (UID: \"9b7f05bf-12cd-4aad-b4ee-8b517e901ca6\") " pod="openshift-multus/multus-qhswb"
Nov 02 19:28:14.835145 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.835076    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9b7f05bf-12cd-4aad-b4ee-8b517e901ca6-hostroot\") pod \"multus-qhswb\" (UID: \"9b7f05bf-12cd-4aad-b4ee-8b517e901ca6\") " pod="openshift-multus/multus-qhswb"
Nov 02 19:28:14.835145 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.835091    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a8e20267-1b77-4eb3-b940-a29d79b0ec19-etc-sysctl-d\") pod \"tuned-nvdtg\" (UID: \"a8e20267-1b77-4eb3-b940-a29d79b0ec19\") " pod="openshift-cluster-node-tuning-operator/tuned-nvdtg"
Nov 02 19:28:14.835145 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.835106    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a8e20267-1b77-4eb3-b940-a29d79b0ec19-sys\") pod \"tuned-nvdtg\" (UID: \"a8e20267-1b77-4eb3-b940-a29d79b0ec19\") " pod="openshift-cluster-node-tuning-operator/tuned-nvdtg"
Nov 02 19:28:14.835145 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.835121    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9b7f05bf-12cd-4aad-b4ee-8b517e901ca6-host-run-k8s-cni-cncf-io\") pod \"multus-qhswb\" (UID: \"9b7f05bf-12cd-4aad-b4ee-8b517e901ca6\") " pod="openshift-multus/multus-qhswb"
Nov 02 19:28:14.835317 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.835155    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9b7f05bf-12cd-4aad-b4ee-8b517e901ca6-host-var-lib-cni-bin\") pod \"multus-qhswb\" (UID: \"9b7f05bf-12cd-4aad-b4ee-8b517e901ca6\") " pod="openshift-multus/multus-qhswb"
Nov 02 19:28:14.835317 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.835157    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9b7f05bf-12cd-4aad-b4ee-8b517e901ca6-hostroot\") pod \"multus-qhswb\" (UID: \"9b7f05bf-12cd-4aad-b4ee-8b517e901ca6\") " pod="openshift-multus/multus-qhswb"
Nov 02 19:28:14.835317 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.835167    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9b7f05bf-12cd-4aad-b4ee-8b517e901ca6-host-run-k8s-cni-cncf-io\") pod \"multus-qhswb\" (UID: \"9b7f05bf-12cd-4aad-b4ee-8b517e901ca6\") " pod="openshift-multus/multus-qhswb"
Nov 02 19:28:14.835317 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.835187    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a8e20267-1b77-4eb3-b940-a29d79b0ec19-sys\") pod \"tuned-nvdtg\" (UID: \"a8e20267-1b77-4eb3-b940-a29d79b0ec19\") " pod="openshift-cluster-node-tuning-operator/tuned-nvdtg"
Nov 02 19:28:14.835317 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.835208    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9b7f05bf-12cd-4aad-b4ee-8b517e901ca6-host-var-lib-cni-multus\") pod \"multus-qhswb\" (UID: \"9b7f05bf-12cd-4aad-b4ee-8b517e901ca6\") " pod="openshift-multus/multus-qhswb"
Nov 02 19:28:14.835317 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.835181    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9b7f05bf-12cd-4aad-b4ee-8b517e901ca6-host-var-lib-cni-multus\") pod \"multus-qhswb\" (UID: \"9b7f05bf-12cd-4aad-b4ee-8b517e901ca6\") " pod="openshift-multus/multus-qhswb"
Nov 02 19:28:14.835317 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.835223    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9b7f05bf-12cd-4aad-b4ee-8b517e901ca6-multus-socket-dir-parent\") pod \"multus-qhswb\" (UID: \"9b7f05bf-12cd-4aad-b4ee-8b517e901ca6\") " pod="openshift-multus/multus-qhswb"
Nov 02 19:28:14.835317 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.835240    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a8e20267-1b77-4eb3-b940-a29d79b0ec19-etc-systemd\") pod \"tuned-nvdtg\" (UID: \"a8e20267-1b77-4eb3-b940-a29d79b0ec19\") " pod="openshift-cluster-node-tuning-operator/tuned-nvdtg"
Nov 02 19:28:14.835317 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.835221    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9b7f05bf-12cd-4aad-b4ee-8b517e901ca6-host-var-lib-cni-bin\") pod \"multus-qhswb\" (UID: \"9b7f05bf-12cd-4aad-b4ee-8b517e901ca6\") " pod="openshift-multus/multus-qhswb"
Nov 02 19:28:14.835317 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.835284    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a8e20267-1b77-4eb3-b940-a29d79b0ec19-etc-sysctl-d\") pod \"tuned-nvdtg\" (UID: \"a8e20267-1b77-4eb3-b940-a29d79b0ec19\") " pod="openshift-cluster-node-tuning-operator/tuned-nvdtg"
Nov 02 19:28:14.835714 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.835322    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a8e20267-1b77-4eb3-b940-a29d79b0ec19-lib-modules\") pod \"tuned-nvdtg\" (UID: \"a8e20267-1b77-4eb3-b940-a29d79b0ec19\") " pod="openshift-cluster-node-tuning-operator/tuned-nvdtg"
Nov 02 19:28:14.835714 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.835360    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a8e20267-1b77-4eb3-b940-a29d79b0ec19-etc-systemd\") pod \"tuned-nvdtg\" (UID: \"a8e20267-1b77-4eb3-b940-a29d79b0ec19\") " pod="openshift-cluster-node-tuning-operator/tuned-nvdtg"
Nov 02 19:28:14.835714 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.835371    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8756b800-1b4e-41da-934a-25c6b8ee738c-systemd-units\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:28:14.835714 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.835395    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/74106611-359e-4353-b113-70521144bed1-etc-selinux\") pod \"aws-ebs-csi-driver-node-f5wvz\" (UID: \"74106611-359e-4353-b113-70521144bed1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5wvz"
Nov 02 19:28:14.835714 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.835418    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8756b800-1b4e-41da-934a-25c6b8ee738c-systemd-units\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:28:14.835714 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.835420    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a8e20267-1b77-4eb3-b940-a29d79b0ec19-lib-modules\") pod \"tuned-nvdtg\" (UID: \"a8e20267-1b77-4eb3-b940-a29d79b0ec19\") " pod="openshift-cluster-node-tuning-operator/tuned-nvdtg"
Nov 02 19:28:14.835714 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.835428    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/74106611-359e-4353-b113-70521144bed1-etc-selinux\") pod \"aws-ebs-csi-driver-node-f5wvz\" (UID: \"74106611-359e-4353-b113-70521144bed1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5wvz"
Nov 02 19:28:14.835714 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.835435    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a8e20267-1b77-4eb3-b940-a29d79b0ec19-etc-sysconfig\") pod \"tuned-nvdtg\" (UID: \"a8e20267-1b77-4eb3-b940-a29d79b0ec19\") " pod="openshift-cluster-node-tuning-operator/tuned-nvdtg"
Nov 02 19:28:14.835714 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.835456    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a8e20267-1b77-4eb3-b940-a29d79b0ec19-etc-sysctl-conf\") pod \"tuned-nvdtg\" (UID: \"a8e20267-1b77-4eb3-b940-a29d79b0ec19\") " pod="openshift-cluster-node-tuning-operator/tuned-nvdtg"
Nov 02 19:28:14.835714 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.835475    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a8e20267-1b77-4eb3-b940-a29d79b0ec19-etc-sysconfig\") pod \"tuned-nvdtg\" (UID: \"a8e20267-1b77-4eb3-b940-a29d79b0ec19\") " pod="openshift-cluster-node-tuning-operator/tuned-nvdtg"
Nov 02 19:28:14.835714 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.835494    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dfb225e8-fabd-49fb-9763-4e41bf6ae7de-metrics-tls\") pod \"dns-default-qj8mr\" (UID: \"dfb225e8-fabd-49fb-9763-4e41bf6ae7de\") " pod="openshift-dns/dns-default-qj8mr"
Nov 02 19:28:14.835714 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.835510    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/74106611-359e-4353-b113-70521144bed1-socket-dir\") pod \"aws-ebs-csi-driver-node-f5wvz\" (UID: \"74106611-359e-4353-b113-70521144bed1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5wvz"
Nov 02 19:28:14.835714 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.835524    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9b7f05bf-12cd-4aad-b4ee-8b517e901ca6-host-run-netns\") pod \"multus-qhswb\" (UID: \"9b7f05bf-12cd-4aad-b4ee-8b517e901ca6\") " pod="openshift-multus/multus-qhswb"
Nov 02 19:28:14.835714 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.835553    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/dd804657-39fd-4ae9-899d-cd1f1b62717e-hosts-file\") pod \"node-resolver-r9fcz\" (UID: \"dd804657-39fd-4ae9-899d-cd1f1b62717e\") " pod="openshift-dns/node-resolver-r9fcz"
Nov 02 19:28:14.835714 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.835577    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/74106611-359e-4353-b113-70521144bed1-socket-dir\") pod \"aws-ebs-csi-driver-node-f5wvz\" (UID: \"74106611-359e-4353-b113-70521144bed1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5wvz"
Nov 02 19:28:14.835714 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.835580    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9b7f05bf-12cd-4aad-b4ee-8b517e901ca6-host-run-netns\") pod \"multus-qhswb\" (UID: \"9b7f05bf-12cd-4aad-b4ee-8b517e901ca6\") " pod="openshift-multus/multus-qhswb"
Nov 02 19:28:14.835714 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.835577    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bh65q\" (UID: \"0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2\") " pod="openshift-insights/insights-runtime-extractor-bh65q"
Nov 02 19:28:14.835714 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:28:14.835611    4463 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 19:28:14.835714 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.835596    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a8e20267-1b77-4eb3-b940-a29d79b0ec19-etc-sysctl-conf\") pod \"tuned-nvdtg\" (UID: \"a8e20267-1b77-4eb3-b940-a29d79b0ec19\") " pod="openshift-cluster-node-tuning-operator/tuned-nvdtg"
Nov 02 19:28:14.835714 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.835623    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/dd804657-39fd-4ae9-899d-cd1f1b62717e-hosts-file\") pod \"node-resolver-r9fcz\" (UID: \"dd804657-39fd-4ae9-899d-cd1f1b62717e\") " pod="openshift-dns/node-resolver-r9fcz"
Nov 02 19:28:14.835714 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.835635    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8756b800-1b4e-41da-934a-25c6b8ee738c-log-socket\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:28:14.835714 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:28:14.835660    4463 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 19:28:14.835714 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.835662    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9b7f05bf-12cd-4aad-b4ee-8b517e901ca6-etc-kubernetes\") pod \"multus-qhswb\" (UID: \"9b7f05bf-12cd-4aad-b4ee-8b517e901ca6\") " pod="openshift-multus/multus-qhswb"
Nov 02 19:28:14.835714 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:28:14.835691    4463 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfb225e8-fabd-49fb-9763-4e41bf6ae7de-metrics-tls podName:dfb225e8-fabd-49fb-9763-4e41bf6ae7de nodeName:}" failed. No retries permitted until 2026-11-02 19:28:15.33566967 +0000 UTC m=+1.942703442 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dfb225e8-fabd-49fb-9763-4e41bf6ae7de-metrics-tls") pod "dns-default-qj8mr" (UID: "dfb225e8-fabd-49fb-9763-4e41bf6ae7de") : secret "dns-default-metrics-tls" not found
Nov 02 19:28:14.835714 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.835697    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9b7f05bf-12cd-4aad-b4ee-8b517e901ca6-etc-kubernetes\") pod \"multus-qhswb\" (UID: \"9b7f05bf-12cd-4aad-b4ee-8b517e901ca6\") " pod="openshift-multus/multus-qhswb"
Nov 02 19:28:14.835714 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:28:14.835719    4463 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2-insights-runtime-extractor-tls podName:0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2 nodeName:}" failed. No retries permitted until 2026-11-02 19:28:15.33570983 +0000 UTC m=+1.942743583 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bh65q" (UID: "0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2") : secret "insights-runtime-extractor-tls" not found
Nov 02 19:28:14.835714 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.835719    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8756b800-1b4e-41da-934a-25c6b8ee738c-log-socket\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:28:14.836766 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.835734    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a8e20267-1b77-4eb3-b940-a29d79b0ec19-etc-modprobe-d\") pod \"tuned-nvdtg\" (UID: \"a8e20267-1b77-4eb3-b940-a29d79b0ec19\") " pod="openshift-cluster-node-tuning-operator/tuned-nvdtg"
Nov 02 19:28:14.836766 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.835752    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9b7f05bf-12cd-4aad-b4ee-8b517e901ca6-system-cni-dir\") pod \"multus-qhswb\" (UID: \"9b7f05bf-12cd-4aad-b4ee-8b517e901ca6\") " pod="openshift-multus/multus-qhswb"
Nov 02 19:28:14.836766 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.835766    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a8e20267-1b77-4eb3-b940-a29d79b0ec19-etc-kubernetes\") pod \"tuned-nvdtg\" (UID: \"a8e20267-1b77-4eb3-b940-a29d79b0ec19\") " pod="openshift-cluster-node-tuning-operator/tuned-nvdtg"
Nov 02 19:28:14.836766 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.835780    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a8e20267-1b77-4eb3-b940-a29d79b0ec19-host\") pod \"tuned-nvdtg\" (UID: \"a8e20267-1b77-4eb3-b940-a29d79b0ec19\") " pod="openshift-cluster-node-tuning-operator/tuned-nvdtg"
Nov 02 19:28:14.836766 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.835806    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/74106611-359e-4353-b113-70521144bed1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-f5wvz\" (UID: \"74106611-359e-4353-b113-70521144bed1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5wvz"
Nov 02 19:28:14.836766 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.835814    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9b7f05bf-12cd-4aad-b4ee-8b517e901ca6-system-cni-dir\") pod \"multus-qhswb\" (UID: \"9b7f05bf-12cd-4aad-b4ee-8b517e901ca6\") " pod="openshift-multus/multus-qhswb"
Nov 02 19:28:14.836766 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.835823    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0b2d7938-0c97-4f07-a7d2-b5c35d2bc8f9-kubelet-config\") pod \"global-pull-secret-syncer-4b2pr\" (UID: \"0b2d7938-0c97-4f07-a7d2-b5c35d2bc8f9\") " pod="kube-system/global-pull-secret-syncer-4b2pr"
Nov 02 19:28:14.836766 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.835845    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a8e20267-1b77-4eb3-b940-a29d79b0ec19-etc-modprobe-d\") pod \"tuned-nvdtg\" (UID: \"a8e20267-1b77-4eb3-b940-a29d79b0ec19\") " pod="openshift-cluster-node-tuning-operator/tuned-nvdtg"
Nov 02 19:28:14.836766 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.835857    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b6cb1c47-e00c-484d-bf34-445b9380553c-host-slash\") pod \"iptables-alerter-cmk66\" (UID: \"b6cb1c47-e00c-484d-bf34-445b9380553c\") " pod="openshift-network-operator/iptables-alerter-cmk66"
Nov 02 19:28:14.836766 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.835859    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0b2d7938-0c97-4f07-a7d2-b5c35d2bc8f9-kubelet-config\") pod \"global-pull-secret-syncer-4b2pr\" (UID: \"0b2d7938-0c97-4f07-a7d2-b5c35d2bc8f9\") " pod="kube-system/global-pull-secret-syncer-4b2pr"
Nov 02 19:28:14.836766 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.835881    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a8e20267-1b77-4eb3-b940-a29d79b0ec19-host\") pod \"tuned-nvdtg\" (UID: \"a8e20267-1b77-4eb3-b940-a29d79b0ec19\") " pod="openshift-cluster-node-tuning-operator/tuned-nvdtg"
Nov 02 19:28:14.836766 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.835884    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9b7f05bf-12cd-4aad-b4ee-8b517e901ca6-host-run-multus-certs\") pod \"multus-qhswb\" (UID: \"9b7f05bf-12cd-4aad-b4ee-8b517e901ca6\") " pod="openshift-multus/multus-qhswb"
Nov 02 19:28:14.836766 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.835894    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/74106611-359e-4353-b113-70521144bed1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-f5wvz\" (UID: \"74106611-359e-4353-b113-70521144bed1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5wvz"
Nov 02 19:28:14.836766 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.835897    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b6cb1c47-e00c-484d-bf34-445b9380553c-host-slash\") pod \"iptables-alerter-cmk66\" (UID: \"b6cb1c47-e00c-484d-bf34-445b9380553c\") " pod="openshift-network-operator/iptables-alerter-cmk66"
Nov 02 19:28:14.836766 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.835918    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9b7f05bf-12cd-4aad-b4ee-8b517e901ca6-host-run-multus-certs\") pod \"multus-qhswb\" (UID: \"9b7f05bf-12cd-4aad-b4ee-8b517e901ca6\") " pod="openshift-multus/multus-qhswb"
Nov 02 19:28:14.836766 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.835934    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8756b800-1b4e-41da-934a-25c6b8ee738c-host-kubelet\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:28:14.836766 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.835949    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8756b800-1b4e-41da-934a-25c6b8ee738c-node-log\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:28:14.836766 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.835890    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a8e20267-1b77-4eb3-b940-a29d79b0ec19-etc-kubernetes\") pod \"tuned-nvdtg\" (UID: \"a8e20267-1b77-4eb3-b940-a29d79b0ec19\") " pod="openshift-cluster-node-tuning-operator/tuned-nvdtg"
Nov 02 19:28:14.836766 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.835966    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8756b800-1b4e-41da-934a-25c6b8ee738c-host-cni-bin\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:28:14.836766 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.835991    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a8e20267-1b77-4eb3-b940-a29d79b0ec19-var-lib-kubelet\") pod \"tuned-nvdtg\" (UID: \"a8e20267-1b77-4eb3-b940-a29d79b0ec19\") " pod="openshift-cluster-node-tuning-operator/tuned-nvdtg"
Nov 02 19:28:14.836766 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.835997    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8756b800-1b4e-41da-934a-25c6b8ee738c-host-kubelet\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:28:14.836766 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.836003    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8756b800-1b4e-41da-934a-25c6b8ee738c-node-log\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:28:14.836766 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.836026    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0b2d7938-0c97-4f07-a7d2-b5c35d2bc8f9-dbus\") pod \"global-pull-secret-syncer-4b2pr\" (UID: \"0b2d7938-0c97-4f07-a7d2-b5c35d2bc8f9\") " pod="kube-system/global-pull-secret-syncer-4b2pr"
Nov 02 19:28:14.836766 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.836032    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8756b800-1b4e-41da-934a-25c6b8ee738c-host-cni-bin\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:28:14.836766 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.836050    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343-cnibin\") pod \"multus-additional-cni-plugins-4ddbs\" (UID: \"4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343\") " pod="openshift-multus/multus-additional-cni-plugins-4ddbs"
Nov 02 19:28:14.836766 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.836067    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2-crio-socket\") pod \"insights-runtime-extractor-bh65q\" (UID: \"0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2\") " pod="openshift-insights/insights-runtime-extractor-bh65q"
Nov 02 19:28:14.836766 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.836069    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a8e20267-1b77-4eb3-b940-a29d79b0ec19-var-lib-kubelet\") pod \"tuned-nvdtg\" (UID: \"a8e20267-1b77-4eb3-b940-a29d79b0ec19\") " pod="openshift-cluster-node-tuning-operator/tuned-nvdtg"
Nov 02 19:28:14.836766 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.836081    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8756b800-1b4e-41da-934a-25c6b8ee738c-host-run-netns\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:28:14.836766 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.836105    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8756b800-1b4e-41da-934a-25c6b8ee738c-run-ovn\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:28:14.836766 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.836116    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8756b800-1b4e-41da-934a-25c6b8ee738c-host-run-netns\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:28:14.836766 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.836123    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4ddbs\" (UID: \"4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343\") " pod="openshift-multus/multus-additional-cni-plugins-4ddbs"
Nov 02 19:28:14.836766 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.836134    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0b2d7938-0c97-4f07-a7d2-b5c35d2bc8f9-dbus\") pod \"global-pull-secret-syncer-4b2pr\" (UID: \"0b2d7938-0c97-4f07-a7d2-b5c35d2bc8f9\") " pod="kube-system/global-pull-secret-syncer-4b2pr"
Nov 02 19:28:14.836766 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.836137    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9b7f05bf-12cd-4aad-b4ee-8b517e901ca6-multus-conf-dir\") pod \"multus-qhswb\" (UID: \"9b7f05bf-12cd-4aad-b4ee-8b517e901ca6\") " pod="openshift-multus/multus-qhswb"
Nov 02 19:28:14.836766 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.836142    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8756b800-1b4e-41da-934a-25c6b8ee738c-run-ovn\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:28:14.836766 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.836155    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2-crio-socket\") pod \"insights-runtime-extractor-bh65q\" (UID: \"0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2\") " pod="openshift-insights/insights-runtime-extractor-bh65q"
Nov 02 19:28:14.836766 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.836121    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343-cnibin\") pod \"multus-additional-cni-plugins-4ddbs\" (UID: \"4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343\") " pod="openshift-multus/multus-additional-cni-plugins-4ddbs"
Nov 02 19:28:14.836766 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.836160    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8756b800-1b4e-41da-934a-25c6b8ee738c-host-slash\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:28:14.836766 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.836183    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8756b800-1b4e-41da-934a-25c6b8ee738c-host-slash\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:28:14.836766 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.836189    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4ddbs\" (UID: \"4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343\") " pod="openshift-multus/multus-additional-cni-plugins-4ddbs"
Nov 02 19:28:14.836766 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.836200    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8756b800-1b4e-41da-934a-25c6b8ee738c-var-lib-openvswitch\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:28:14.836766 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.836222    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8756b800-1b4e-41da-934a-25c6b8ee738c-host-cni-netd\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:28:14.836766 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.836166    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9b7f05bf-12cd-4aad-b4ee-8b517e901ca6-multus-conf-dir\") pod \"multus-qhswb\" (UID: \"9b7f05bf-12cd-4aad-b4ee-8b517e901ca6\") " pod="openshift-multus/multus-qhswb"
Nov 02 19:28:14.836766 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.836237    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/74106611-359e-4353-b113-70521144bed1-sys-fs\") pod \"aws-ebs-csi-driver-node-f5wvz\" (UID: \"74106611-359e-4353-b113-70521144bed1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5wvz"
Nov 02 19:28:14.836766 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.836256    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a8e20267-1b77-4eb3-b940-a29d79b0ec19-run\") pod \"tuned-nvdtg\" (UID: \"a8e20267-1b77-4eb3-b940-a29d79b0ec19\") " pod="openshift-cluster-node-tuning-operator/tuned-nvdtg"
Nov 02 19:28:14.836766 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.836270    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8756b800-1b4e-41da-934a-25c6b8ee738c-var-lib-openvswitch\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:28:14.836766 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.836279    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8756b800-1b4e-41da-934a-25c6b8ee738c-run-openvswitch\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:28:14.836766 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.836291    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/74106611-359e-4353-b113-70521144bed1-sys-fs\") pod \"aws-ebs-csi-driver-node-f5wvz\" (UID: \"74106611-359e-4353-b113-70521144bed1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5wvz"
Nov 02 19:28:14.836766 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.836300    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8756b800-1b4e-41da-934a-25c6b8ee738c-run-openvswitch\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:28:14.836766 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.836264    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8756b800-1b4e-41da-934a-25c6b8ee738c-host-cni-netd\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:28:14.836766 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.836320    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a8e20267-1b77-4eb3-b940-a29d79b0ec19-run\") pod \"tuned-nvdtg\" (UID: \"a8e20267-1b77-4eb3-b940-a29d79b0ec19\") " pod="openshift-cluster-node-tuning-operator/tuned-nvdtg"
Nov 02 19:28:14.836766 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.836308    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9b7f05bf-12cd-4aad-b4ee-8b517e901ca6-multus-cni-dir\") pod \"multus-qhswb\" (UID: \"9b7f05bf-12cd-4aad-b4ee-8b517e901ca6\") " pod="openshift-multus/multus-qhswb"
Nov 02 19:28:14.836766 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.836348    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b6ad2c98-52b5-496e-9824-614a2e509a8a-host\") pod \"node-ca-hrhpj\" (UID: \"b6ad2c98-52b5-496e-9824-614a2e509a8a\") " pod="openshift-image-registry/node-ca-hrhpj"
Nov 02 19:28:14.836766 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.836372    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bh65q\" (UID: \"0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2\") " pod="openshift-insights/insights-runtime-extractor-bh65q"
Nov 02 19:28:14.836766 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.836403    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b6ad2c98-52b5-496e-9824-614a2e509a8a-host\") pod \"node-ca-hrhpj\" (UID: \"b6ad2c98-52b5-496e-9824-614a2e509a8a\") " pod="openshift-image-registry/node-ca-hrhpj"
Nov 02 19:28:14.836766 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:28:14.836419    4463 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 02 19:28:14.836766 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.836422    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9b7f05bf-12cd-4aad-b4ee-8b517e901ca6-multus-cni-dir\") pod \"multus-qhswb\" (UID: \"9b7f05bf-12cd-4aad-b4ee-8b517e901ca6\") " pod="openshift-multus/multus-qhswb"
Nov 02 19:28:14.836766 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.836396    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/74106611-359e-4353-b113-70521144bed1-device-dir\") pod \"aws-ebs-csi-driver-node-f5wvz\" (UID: \"74106611-359e-4353-b113-70521144bed1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5wvz"
Nov 02 19:28:14.836766 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:28:14.836461    4463 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2-kube-rbac-proxy-cm podName:0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2 nodeName:}" failed. No retries permitted until 2026-11-02 19:28:15.33644969 +0000 UTC m=+1.943483458 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bh65q" (UID: "0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2") : configmap "kube-rbac-proxy" not found
Nov 02 19:28:14.836766 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.836485    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9b7f05bf-12cd-4aad-b4ee-8b517e901ca6-cnibin\") pod \"multus-qhswb\" (UID: \"9b7f05bf-12cd-4aad-b4ee-8b517e901ca6\") " pod="openshift-multus/multus-qhswb"
Nov 02 19:28:14.836766 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.836449    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/74106611-359e-4353-b113-70521144bed1-device-dir\") pod \"aws-ebs-csi-driver-node-f5wvz\" (UID: \"74106611-359e-4353-b113-70521144bed1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5wvz"
Nov 02 19:28:14.836766 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.836501    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9b7f05bf-12cd-4aad-b4ee-8b517e901ca6-host-var-lib-kubelet\") pod \"multus-qhswb\" (UID: \"9b7f05bf-12cd-4aad-b4ee-8b517e901ca6\") " pod="openshift-multus/multus-qhswb"
Nov 02 19:28:14.836766 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.836515    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8756b800-1b4e-41da-934a-25c6b8ee738c-run-systemd\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:28:14.836766 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.836530    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8756b800-1b4e-41da-934a-25c6b8ee738c-host-run-ovn-kubernetes\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:28:14.836766 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.836541    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9b7f05bf-12cd-4aad-b4ee-8b517e901ca6-cnibin\") pod \"multus-qhswb\" (UID: \"9b7f05bf-12cd-4aad-b4ee-8b517e901ca6\") " pod="openshift-multus/multus-qhswb"
Nov 02 19:28:14.836766 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.836546    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8756b800-1b4e-41da-934a-25c6b8ee738c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:28:14.836766 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.836561    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9b7f05bf-12cd-4aad-b4ee-8b517e901ca6-host-var-lib-kubelet\") pod \"multus-qhswb\" (UID: \"9b7f05bf-12cd-4aad-b4ee-8b517e901ca6\") " pod="openshift-multus/multus-qhswb"
Nov 02 19:28:14.836766 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.836583    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17fe5072-f4c5-4075-8614-4229ed05d4b2-metrics-certs\") pod \"network-metrics-daemon-f6r6d\" (UID: \"17fe5072-f4c5-4075-8614-4229ed05d4b2\") " pod="openshift-multus/network-metrics-daemon-f6r6d"
Nov 02 19:28:14.836766 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.836599    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8756b800-1b4e-41da-934a-25c6b8ee738c-run-systemd\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:28:14.836766 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.836623    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8756b800-1b4e-41da-934a-25c6b8ee738c-etc-openvswitch\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:28:14.836766 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.836600    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8756b800-1b4e-41da-934a-25c6b8ee738c-etc-openvswitch\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:28:14.836766 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:28:14.836646    4463 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 19:28:14.836766 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.836650    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8756b800-1b4e-41da-934a-25c6b8ee738c-host-run-ovn-kubernetes\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:28:14.836766 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.836678    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8756b800-1b4e-41da-934a-25c6b8ee738c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vst5r\" (UID: \"8756b800-1b4e-41da-934a-25c6b8ee738c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:28:14.836766 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.836695    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/74106611-359e-4353-b113-70521144bed1-registration-dir\") pod \"aws-ebs-csi-driver-node-f5wvz\" (UID: \"74106611-359e-4353-b113-70521144bed1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5wvz"
Nov 02 19:28:14.836766 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.836718    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/74106611-359e-4353-b113-70521144bed1-registration-dir\") pod \"aws-ebs-csi-driver-node-f5wvz\" (UID: \"74106611-359e-4353-b113-70521144bed1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5wvz"
Nov 02 19:28:14.836766 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.836736    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343-system-cni-dir\") pod \"multus-additional-cni-plugins-4ddbs\" (UID: \"4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343\") " pod="openshift-multus/multus-additional-cni-plugins-4ddbs"
Nov 02 19:28:14.836766 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:28:14.836763    4463 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17fe5072-f4c5-4075-8614-4229ed05d4b2-metrics-certs podName:17fe5072-f4c5-4075-8614-4229ed05d4b2 nodeName:}" failed. No retries permitted until 2026-11-02 19:28:15.336752491 +0000 UTC m=+1.943786238 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/17fe5072-f4c5-4075-8614-4229ed05d4b2-metrics-certs") pod "network-metrics-daemon-f6r6d" (UID: "17fe5072-f4c5-4075-8614-4229ed05d4b2") : secret "metrics-daemon-secret" not found
Nov 02 19:28:14.836766 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.836789    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343-system-cni-dir\") pod \"multus-additional-cni-plugins-4ddbs\" (UID: \"4c8ad35a-7977-4e7a-8eb0-a4c8d1ce2343\") " pod="openshift-multus/multus-additional-cni-plugins-4ddbs"
Nov 02 19:28:14.935742 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:14.935726    4463 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 19:28:15.339800 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:15.339769    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bh65q\" (UID: \"0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2\") " pod="openshift-insights/insights-runtime-extractor-bh65q"
Nov 02 19:28:15.339800 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:15.339804    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17fe5072-f4c5-4075-8614-4229ed05d4b2-metrics-certs\") pod \"network-metrics-daemon-f6r6d\" (UID: \"17fe5072-f4c5-4075-8614-4229ed05d4b2\") " pod="openshift-multus/network-metrics-daemon-f6r6d"
Nov 02 19:28:15.339994 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:15.339828    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dfb225e8-fabd-49fb-9763-4e41bf6ae7de-metrics-tls\") pod \"dns-default-qj8mr\" (UID: \"dfb225e8-fabd-49fb-9763-4e41bf6ae7de\") " pod="openshift-dns/dns-default-qj8mr"
Nov 02 19:28:15.339994 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:15.339848    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bh65q\" (UID: \"0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2\") " pod="openshift-insights/insights-runtime-extractor-bh65q"
Nov 02 19:28:15.339994 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:28:15.339925    4463 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 19:28:15.340137 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:28:15.339931    4463 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 19:28:15.340137 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:28:15.339931    4463 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 19:28:15.340137 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:28:15.340060    4463 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17fe5072-f4c5-4075-8614-4229ed05d4b2-metrics-certs podName:17fe5072-f4c5-4075-8614-4229ed05d4b2 nodeName:}" failed. No retries permitted until 2026-11-02 19:28:16.340032085 +0000 UTC m=+2.947065846 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/17fe5072-f4c5-4075-8614-4229ed05d4b2-metrics-certs") pod "network-metrics-daemon-f6r6d" (UID: "17fe5072-f4c5-4075-8614-4229ed05d4b2") : secret "metrics-daemon-secret" not found
Nov 02 19:28:15.340137 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:28:15.340091    4463 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfb225e8-fabd-49fb-9763-4e41bf6ae7de-metrics-tls podName:dfb225e8-fabd-49fb-9763-4e41bf6ae7de nodeName:}" failed. No retries permitted until 2026-11-02 19:28:16.340080317 +0000 UTC m=+2.947114067 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dfb225e8-fabd-49fb-9763-4e41bf6ae7de-metrics-tls") pod "dns-default-qj8mr" (UID: "dfb225e8-fabd-49fb-9763-4e41bf6ae7de") : secret "dns-default-metrics-tls" not found
Nov 02 19:28:15.340137 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:28:15.340106    4463 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2-insights-runtime-extractor-tls podName:0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2 nodeName:}" failed. No retries permitted until 2026-11-02 19:28:16.34009673 +0000 UTC m=+2.947130488 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bh65q" (UID: "0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2") : secret "insights-runtime-extractor-tls" not found
Nov 02 19:28:15.340327 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:15.340308    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bh65q\" (UID: \"0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2\") " pod="openshift-insights/insights-runtime-extractor-bh65q"
Nov 02 19:28:15.376949 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:15.376933    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-rhdd6"
Nov 02 19:28:16.147989 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:16.147964    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:28:16.181793 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:16.181767    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:28:16.347575 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:16.347542    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17fe5072-f4c5-4075-8614-4229ed05d4b2-metrics-certs\") pod \"network-metrics-daemon-f6r6d\" (UID: \"17fe5072-f4c5-4075-8614-4229ed05d4b2\") " pod="openshift-multus/network-metrics-daemon-f6r6d"
Nov 02 19:28:16.347738 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:16.347590    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dfb225e8-fabd-49fb-9763-4e41bf6ae7de-metrics-tls\") pod \"dns-default-qj8mr\" (UID: \"dfb225e8-fabd-49fb-9763-4e41bf6ae7de\") " pod="openshift-dns/dns-default-qj8mr"
Nov 02 19:28:16.347738 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:16.347620    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bh65q\" (UID: \"0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2\") " pod="openshift-insights/insights-runtime-extractor-bh65q"
Nov 02 19:28:16.347738 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:28:16.347698    4463 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 19:28:16.347738 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:28:16.347726    4463 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 19:28:16.347946 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:28:16.347765    4463 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17fe5072-f4c5-4075-8614-4229ed05d4b2-metrics-certs podName:17fe5072-f4c5-4075-8614-4229ed05d4b2 nodeName:}" failed. No retries permitted until 2026-11-02 19:28:18.347747965 +0000 UTC m=+4.954781719 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/17fe5072-f4c5-4075-8614-4229ed05d4b2-metrics-certs") pod "network-metrics-daemon-f6r6d" (UID: "17fe5072-f4c5-4075-8614-4229ed05d4b2") : secret "metrics-daemon-secret" not found
Nov 02 19:28:16.347946 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:28:16.347779    4463 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2-insights-runtime-extractor-tls podName:0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2 nodeName:}" failed. No retries permitted until 2026-11-02 19:28:18.347772836 +0000 UTC m=+4.954806582 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bh65q" (UID: "0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2") : secret "insights-runtime-extractor-tls" not found
Nov 02 19:28:16.347946 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:28:16.347778    4463 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 19:28:16.347946 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:28:16.347827    4463 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfb225e8-fabd-49fb-9763-4e41bf6ae7de-metrics-tls podName:dfb225e8-fabd-49fb-9763-4e41bf6ae7de nodeName:}" failed. No retries permitted until 2026-11-02 19:28:18.347814687 +0000 UTC m=+4.954848435 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dfb225e8-fabd-49fb-9763-4e41bf6ae7de-metrics-tls") pod "dns-default-qj8mr" (UID: "dfb225e8-fabd-49fb-9763-4e41bf6ae7de") : secret "dns-default-metrics-tls" not found
Nov 02 19:28:16.639386 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:16.639051    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-xvzjr-ggg8w" event={"ID":"4874b345-eac7-4694-8351-d24085bf180a","Type":"ContainerStarted","Data":"8ec058e8f2189f7936cd79d0710a93b07e6d8a58abeec9756522d41c1f3f117d"}
Nov 02 19:28:16.640725 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:16.640699    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-zqz7w" event={"ID":"54c1de40-ce9d-4d7d-82ca-c5fa41c97e97","Type":"ContainerStarted","Data":"cf29f1fcadd0a43282bd9a12f6c70992f42a114a934dc2745547c22a8f9e7432"}
Nov 02 19:28:16.640853 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:16.640777    4463 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 19:28:18.361534 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:18.361504    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17fe5072-f4c5-4075-8614-4229ed05d4b2-metrics-certs\") pod \"network-metrics-daemon-f6r6d\" (UID: \"17fe5072-f4c5-4075-8614-4229ed05d4b2\") " pod="openshift-multus/network-metrics-daemon-f6r6d"
Nov 02 19:28:18.361925 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:18.361545    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dfb225e8-fabd-49fb-9763-4e41bf6ae7de-metrics-tls\") pod \"dns-default-qj8mr\" (UID: \"dfb225e8-fabd-49fb-9763-4e41bf6ae7de\") " pod="openshift-dns/dns-default-qj8mr"
Nov 02 19:28:18.361925 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:18.361565    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bh65q\" (UID: \"0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2\") " pod="openshift-insights/insights-runtime-extractor-bh65q"
Nov 02 19:28:18.361925 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:28:18.361659    4463 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 19:28:18.361925 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:28:18.361661    4463 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 19:28:18.361925 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:28:18.361701    4463 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2-insights-runtime-extractor-tls podName:0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2 nodeName:}" failed. No retries permitted until 2026-11-02 19:28:22.361688826 +0000 UTC m=+8.968722574 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bh65q" (UID: "0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2") : secret "insights-runtime-extractor-tls" not found
Nov 02 19:28:18.361925 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:28:18.361719    4463 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17fe5072-f4c5-4075-8614-4229ed05d4b2-metrics-certs podName:17fe5072-f4c5-4075-8614-4229ed05d4b2 nodeName:}" failed. No retries permitted until 2026-11-02 19:28:22.361708148 +0000 UTC m=+8.968741895 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/17fe5072-f4c5-4075-8614-4229ed05d4b2-metrics-certs") pod "network-metrics-daemon-f6r6d" (UID: "17fe5072-f4c5-4075-8614-4229ed05d4b2") : secret "metrics-daemon-secret" not found
Nov 02 19:28:18.361925 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:28:18.361662    4463 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 19:28:18.361925 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:28:18.361744    4463 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfb225e8-fabd-49fb-9763-4e41bf6ae7de-metrics-tls podName:dfb225e8-fabd-49fb-9763-4e41bf6ae7de nodeName:}" failed. No retries permitted until 2026-11-02 19:28:22.361738084 +0000 UTC m=+8.968771830 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dfb225e8-fabd-49fb-9763-4e41bf6ae7de-metrics-tls") pod "dns-default-qj8mr" (UID: "dfb225e8-fabd-49fb-9763-4e41bf6ae7de") : secret "dns-default-metrics-tls" not found
Nov 02 19:28:18.648848 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:18.648782    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-h5fc9" event={"ID":"28b76ecd-8da7-428f-a825-323670ae5395","Type":"ContainerStarted","Data":"9ae0a0d70264f4f9e93a7980779342981972b15f80b2a7443cb319caa657259c"}
Nov 02 19:28:19.501814 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:19.501778    4463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-r9fcz_dd804657-39fd-4ae9-899d-cd1f1b62717e/dns-node-resolver/0.log"
Nov 02 19:28:20.695129 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:20.695099    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-jgfjj"
Nov 02 19:28:20.701962 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:20.701940    4463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hrhpj_b6ad2c98-52b5-496e-9824-614a2e509a8a/node-ca/0.log"
Nov 02 19:28:21.302220 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:21.302191    4463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-sw9mp_5d436341-2000-4c2b-87ac-7b67e4cdb909/serve-healthcheck-canary/0.log"
Nov 02 19:28:21.902146 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:21.902120    4463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-zqz7w_54c1de40-ce9d-4d7d-82ca-c5fa41c97e97/migrator/0.log"
Nov 02 19:28:22.385979 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:22.385942    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17fe5072-f4c5-4075-8614-4229ed05d4b2-metrics-certs\") pod \"network-metrics-daemon-f6r6d\" (UID: \"17fe5072-f4c5-4075-8614-4229ed05d4b2\") " pod="openshift-multus/network-metrics-daemon-f6r6d"
Nov 02 19:28:22.386145 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:22.385987    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dfb225e8-fabd-49fb-9763-4e41bf6ae7de-metrics-tls\") pod \"dns-default-qj8mr\" (UID: \"dfb225e8-fabd-49fb-9763-4e41bf6ae7de\") " pod="openshift-dns/dns-default-qj8mr"
Nov 02 19:28:22.386145 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:22.386008    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bh65q\" (UID: \"0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2\") " pod="openshift-insights/insights-runtime-extractor-bh65q"
Nov 02 19:28:22.386145 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:28:22.386115    4463 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 19:28:22.386273 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:28:22.386172    4463 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2-insights-runtime-extractor-tls podName:0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2 nodeName:}" failed. No retries permitted until 2026-11-02 19:28:30.386159794 +0000 UTC m=+16.993193554 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bh65q" (UID: "0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2") : secret "insights-runtime-extractor-tls" not found
Nov 02 19:28:22.386273 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:28:22.386115    4463 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 19:28:22.386273 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:28:22.386252    4463 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17fe5072-f4c5-4075-8614-4229ed05d4b2-metrics-certs podName:17fe5072-f4c5-4075-8614-4229ed05d4b2 nodeName:}" failed. No retries permitted until 2026-11-02 19:28:30.386235342 +0000 UTC m=+16.993269099 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/17fe5072-f4c5-4075-8614-4229ed05d4b2-metrics-certs") pod "network-metrics-daemon-f6r6d" (UID: "17fe5072-f4c5-4075-8614-4229ed05d4b2") : secret "metrics-daemon-secret" not found
Nov 02 19:28:22.386428 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:22.386396    4463 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 02 19:28:22.388321 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:22.388298    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dfb225e8-fabd-49fb-9763-4e41bf6ae7de-metrics-tls\") pod \"dns-default-qj8mr\" (UID: \"dfb225e8-fabd-49fb-9763-4e41bf6ae7de\") " pod="openshift-dns/dns-default-qj8mr"
Nov 02 19:28:22.433353 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:22.433325    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qj8mr"
Nov 02 19:28:22.792069 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:22.792038    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qj8mr"]
Nov 02 19:28:22.795319 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:22.795288    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfb225e8_fabd_49fb_9763_4e41bf6ae7de.slice/crio-052476dd75bd9c6bd4bb56d530c6ac29cd127cf2ffbdfa6c57011e6cf197468c WatchSource:0}: Error finding container 052476dd75bd9c6bd4bb56d530c6ac29cd127cf2ffbdfa6c57011e6cf197468c: Status 404 returned error can't find the container with id 052476dd75bd9c6bd4bb56d530c6ac29cd127cf2ffbdfa6c57011e6cf197468c
Nov 02 19:28:23.663109 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:23.663072    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qj8mr" event={"ID":"dfb225e8-fabd-49fb-9763-4e41bf6ae7de","Type":"ContainerStarted","Data":"052476dd75bd9c6bd4bb56d530c6ac29cd127cf2ffbdfa6c57011e6cf197468c"}
Nov 02 19:28:25.670492 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:25.670406    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qj8mr" event={"ID":"dfb225e8-fabd-49fb-9763-4e41bf6ae7de","Type":"ContainerStarted","Data":"2e91945761580e13060cefc69b3f8a18f2b96a76ad56337569b885e9156e2efe"}
Nov 02 19:28:25.670492 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:25.670467    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qj8mr" event={"ID":"dfb225e8-fabd-49fb-9763-4e41bf6ae7de","Type":"ContainerStarted","Data":"ec7f956065a0fa92be1ffa1f7f1f3c2b41255f4a829b1a4f5059bcc4f8e6e295"}
Nov 02 19:28:25.670884 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:25.670507    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-qj8mr"
Nov 02 19:28:30.443678 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:30.443627    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17fe5072-f4c5-4075-8614-4229ed05d4b2-metrics-certs\") pod \"network-metrics-daemon-f6r6d\" (UID: \"17fe5072-f4c5-4075-8614-4229ed05d4b2\") " pod="openshift-multus/network-metrics-daemon-f6r6d"
Nov 02 19:28:30.444203 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:30.443701    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bh65q\" (UID: \"0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2\") " pod="openshift-insights/insights-runtime-extractor-bh65q"
Nov 02 19:28:30.446154 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:30.446130    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17fe5072-f4c5-4075-8614-4229ed05d4b2-metrics-certs\") pod \"network-metrics-daemon-f6r6d\" (UID: \"17fe5072-f4c5-4075-8614-4229ed05d4b2\") " pod="openshift-multus/network-metrics-daemon-f6r6d"
Nov 02 19:28:30.446154 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:30.446144    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bh65q\" (UID: \"0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2\") " pod="openshift-insights/insights-runtime-extractor-bh65q"
Nov 02 19:28:30.531420 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:30.531385    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f6r6d"
Nov 02 19:28:30.534340 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:30.534318    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bh65q"
Nov 02 19:28:30.598166 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:30.596749    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vst5r"
Nov 02 19:28:30.954677 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:30.954647    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-f6r6d"]
Nov 02 19:28:30.957452 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:30.957387    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bh65q"]
Nov 02 19:28:30.958273 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:30.958248    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17fe5072_f4c5_4075_8614_4229ed05d4b2.slice/crio-fec3f087b6c36abacf75ac8775d680fca9900b380bf0d7df6bf75716d90b1912 WatchSource:0}: Error finding container fec3f087b6c36abacf75ac8775d680fca9900b380bf0d7df6bf75716d90b1912: Status 404 returned error can't find the container with id fec3f087b6c36abacf75ac8775d680fca9900b380bf0d7df6bf75716d90b1912
Nov 02 19:28:30.961113 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:30.961093    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a5cb87d_e24d_4e8c_b7c1_6fb4a7e86dc2.slice/crio-aefec8d91f8903a6e8ee9f93df26a4de781fee38a626a6f60e70b40490082fdf WatchSource:0}: Error finding container aefec8d91f8903a6e8ee9f93df26a4de781fee38a626a6f60e70b40490082fdf: Status 404 returned error can't find the container with id aefec8d91f8903a6e8ee9f93df26a4de781fee38a626a6f60e70b40490082fdf
Nov 02 19:28:31.689943 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:31.689903    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-f6r6d" event={"ID":"17fe5072-f4c5-4075-8614-4229ed05d4b2","Type":"ContainerStarted","Data":"fec3f087b6c36abacf75ac8775d680fca9900b380bf0d7df6bf75716d90b1912"}
Nov 02 19:28:31.691308 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:31.691286    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bh65q" event={"ID":"0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2","Type":"ContainerStarted","Data":"df75823a38c4f80b4799dad440bab85fc5fe8ab6ae6789337cc4458406669b2f"}
Nov 02 19:28:31.691393 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:31.691333    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bh65q" event={"ID":"0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2","Type":"ContainerStarted","Data":"aefec8d91f8903a6e8ee9f93df26a4de781fee38a626a6f60e70b40490082fdf"}
Nov 02 19:28:32.695847 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:32.695807    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-f6r6d" event={"ID":"17fe5072-f4c5-4075-8614-4229ed05d4b2","Type":"ContainerStarted","Data":"1591660693bf54b83a17d619ec3d856c4111ac1b93d17c618a8a1be01b346437"}
Nov 02 19:28:32.696209 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:32.695905    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-f6r6d" event={"ID":"17fe5072-f4c5-4075-8614-4229ed05d4b2","Type":"ContainerStarted","Data":"e05ffb910a842789a331aa3765deb0ac70101bc0b3f5c354815ef7ea95c82f5b"}
Nov 02 19:28:32.699930 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:32.699906    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bh65q" event={"ID":"0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2","Type":"ContainerStarted","Data":"ab94815ae9ffe5d55fe175cfe0ec8f8399bfee9dd1eff1c948c01a71f68e2ae3"}
Nov 02 19:28:34.708150 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:34.708115    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bh65q" event={"ID":"0a5cb87d-e24d-4e8c-b7c1-6fb4a7e86dc2","Type":"ContainerStarted","Data":"0273949274d2397a3147ac69367ddf6355560cc4741589eb8fd0d30ffd940397"}
Nov 02 19:28:34.872242 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:34.871832    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7c555f8bcd-cq7qt"]
Nov 02 19:28:34.875753 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:34.875724    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c555f8bcd-cq7qt"
Nov 02 19:28:34.972104 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:34.972073    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Nov 02 19:28:34.972806 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:34.972786    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7c555f8bcd-cq7qt"]
Nov 02 19:28:34.976529 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:34.976509    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/52bbcac3-92a5-4c6f-90d7-5237247a86b6-bound-sa-token\") pod \"image-registry-7c555f8bcd-cq7qt\" (UID: \"52bbcac3-92a5-4c6f-90d7-5237247a86b6\") " pod="openshift-image-registry/image-registry-7c555f8bcd-cq7qt"
Nov 02 19:28:34.976621 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:34.976555    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjj62\" (UniqueName: \"kubernetes.io/projected/52bbcac3-92a5-4c6f-90d7-5237247a86b6-kube-api-access-gjj62\") pod \"image-registry-7c555f8bcd-cq7qt\" (UID: \"52bbcac3-92a5-4c6f-90d7-5237247a86b6\") " pod="openshift-image-registry/image-registry-7c555f8bcd-cq7qt"
Nov 02 19:28:34.976689 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:34.976623    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/52bbcac3-92a5-4c6f-90d7-5237247a86b6-ca-trust-extracted\") pod \"image-registry-7c555f8bcd-cq7qt\" (UID: \"52bbcac3-92a5-4c6f-90d7-5237247a86b6\") " pod="openshift-image-registry/image-registry-7c555f8bcd-cq7qt"
Nov 02 19:28:34.976689 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:34.976649    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/52bbcac3-92a5-4c6f-90d7-5237247a86b6-registry-certificates\") pod \"image-registry-7c555f8bcd-cq7qt\" (UID: \"52bbcac3-92a5-4c6f-90d7-5237247a86b6\") " pod="openshift-image-registry/image-registry-7c555f8bcd-cq7qt"
Nov 02 19:28:34.976779 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:34.976686    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/52bbcac3-92a5-4c6f-90d7-5237247a86b6-installation-pull-secrets\") pod \"image-registry-7c555f8bcd-cq7qt\" (UID: \"52bbcac3-92a5-4c6f-90d7-5237247a86b6\") " pod="openshift-image-registry/image-registry-7c555f8bcd-cq7qt"
Nov 02 19:28:34.976779 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:34.976761    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/52bbcac3-92a5-4c6f-90d7-5237247a86b6-image-registry-private-configuration\") pod \"image-registry-7c555f8bcd-cq7qt\" (UID: \"52bbcac3-92a5-4c6f-90d7-5237247a86b6\") " pod="openshift-image-registry/image-registry-7c555f8bcd-cq7qt"
Nov 02 19:28:34.976857 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:34.976789    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/52bbcac3-92a5-4c6f-90d7-5237247a86b6-registry-tls\") pod \"image-registry-7c555f8bcd-cq7qt\" (UID: \"52bbcac3-92a5-4c6f-90d7-5237247a86b6\") " pod="openshift-image-registry/image-registry-7c555f8bcd-cq7qt"
Nov 02 19:28:34.976857 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:34.976806    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/52bbcac3-92a5-4c6f-90d7-5237247a86b6-trusted-ca\") pod \"image-registry-7c555f8bcd-cq7qt\" (UID: \"52bbcac3-92a5-4c6f-90d7-5237247a86b6\") " pod="openshift-image-registry/image-registry-7c555f8bcd-cq7qt"
Nov 02 19:28:34.991750 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:34.991726    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Nov 02 19:28:34.991750 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:34.991727    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Nov 02 19:28:35.017153 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:35.017129    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Nov 02 19:28:35.017153 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:35.017134    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-ktd87\""
Nov 02 19:28:35.077544 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:35.077523    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjj62\" (UniqueName: \"kubernetes.io/projected/52bbcac3-92a5-4c6f-90d7-5237247a86b6-kube-api-access-gjj62\") pod \"image-registry-7c555f8bcd-cq7qt\" (UID: \"52bbcac3-92a5-4c6f-90d7-5237247a86b6\") " pod="openshift-image-registry/image-registry-7c555f8bcd-cq7qt"
Nov 02 19:28:35.077651 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:35.077551    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/52bbcac3-92a5-4c6f-90d7-5237247a86b6-ca-trust-extracted\") pod \"image-registry-7c555f8bcd-cq7qt\" (UID: \"52bbcac3-92a5-4c6f-90d7-5237247a86b6\") " pod="openshift-image-registry/image-registry-7c555f8bcd-cq7qt"
Nov 02 19:28:35.077651 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:35.077587    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/52bbcac3-92a5-4c6f-90d7-5237247a86b6-registry-certificates\") pod \"image-registry-7c555f8bcd-cq7qt\" (UID: \"52bbcac3-92a5-4c6f-90d7-5237247a86b6\") " pod="openshift-image-registry/image-registry-7c555f8bcd-cq7qt"
Nov 02 19:28:35.077651 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:35.077622    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/52bbcac3-92a5-4c6f-90d7-5237247a86b6-installation-pull-secrets\") pod \"image-registry-7c555f8bcd-cq7qt\" (UID: \"52bbcac3-92a5-4c6f-90d7-5237247a86b6\") " pod="openshift-image-registry/image-registry-7c555f8bcd-cq7qt"
Nov 02 19:28:35.077793 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:35.077660    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/52bbcac3-92a5-4c6f-90d7-5237247a86b6-image-registry-private-configuration\") pod \"image-registry-7c555f8bcd-cq7qt\" (UID: \"52bbcac3-92a5-4c6f-90d7-5237247a86b6\") " pod="openshift-image-registry/image-registry-7c555f8bcd-cq7qt"
Nov 02 19:28:35.077793 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:35.077686    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/52bbcac3-92a5-4c6f-90d7-5237247a86b6-registry-tls\") pod \"image-registry-7c555f8bcd-cq7qt\" (UID: \"52bbcac3-92a5-4c6f-90d7-5237247a86b6\") " pod="openshift-image-registry/image-registry-7c555f8bcd-cq7qt"
Nov 02 19:28:35.077793 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:35.077706    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/52bbcac3-92a5-4c6f-90d7-5237247a86b6-trusted-ca\") pod \"image-registry-7c555f8bcd-cq7qt\" (UID: \"52bbcac3-92a5-4c6f-90d7-5237247a86b6\") " pod="openshift-image-registry/image-registry-7c555f8bcd-cq7qt"
Nov 02 19:28:35.077793 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:35.077740    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/52bbcac3-92a5-4c6f-90d7-5237247a86b6-bound-sa-token\") pod \"image-registry-7c555f8bcd-cq7qt\" (UID: \"52bbcac3-92a5-4c6f-90d7-5237247a86b6\") " pod="openshift-image-registry/image-registry-7c555f8bcd-cq7qt"
Nov 02 19:28:35.078039 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:35.078002    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/52bbcac3-92a5-4c6f-90d7-5237247a86b6-ca-trust-extracted\") pod \"image-registry-7c555f8bcd-cq7qt\" (UID: \"52bbcac3-92a5-4c6f-90d7-5237247a86b6\") " pod="openshift-image-registry/image-registry-7c555f8bcd-cq7qt"
Nov 02 19:28:35.078619 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:35.078590    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/52bbcac3-92a5-4c6f-90d7-5237247a86b6-registry-certificates\") pod \"image-registry-7c555f8bcd-cq7qt\" (UID: \"52bbcac3-92a5-4c6f-90d7-5237247a86b6\") " pod="openshift-image-registry/image-registry-7c555f8bcd-cq7qt"
Nov 02 19:28:35.078794 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:35.078773    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/52bbcac3-92a5-4c6f-90d7-5237247a86b6-trusted-ca\") pod \"image-registry-7c555f8bcd-cq7qt\" (UID: \"52bbcac3-92a5-4c6f-90d7-5237247a86b6\") " pod="openshift-image-registry/image-registry-7c555f8bcd-cq7qt"
Nov 02 19:28:35.080237 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:35.080216    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/52bbcac3-92a5-4c6f-90d7-5237247a86b6-image-registry-private-configuration\") pod \"image-registry-7c555f8bcd-cq7qt\" (UID: \"52bbcac3-92a5-4c6f-90d7-5237247a86b6\") " pod="openshift-image-registry/image-registry-7c555f8bcd-cq7qt"
Nov 02 19:28:35.080470 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:35.080452    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/52bbcac3-92a5-4c6f-90d7-5237247a86b6-installation-pull-secrets\") pod \"image-registry-7c555f8bcd-cq7qt\" (UID: \"52bbcac3-92a5-4c6f-90d7-5237247a86b6\") " pod="openshift-image-registry/image-registry-7c555f8bcd-cq7qt"
Nov 02 19:28:35.080660 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:35.080638    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/52bbcac3-92a5-4c6f-90d7-5237247a86b6-registry-tls\") pod \"image-registry-7c555f8bcd-cq7qt\" (UID: \"52bbcac3-92a5-4c6f-90d7-5237247a86b6\") " pod="openshift-image-registry/image-registry-7c555f8bcd-cq7qt"
Nov 02 19:28:35.132266 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:35.132215    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/52bbcac3-92a5-4c6f-90d7-5237247a86b6-bound-sa-token\") pod \"image-registry-7c555f8bcd-cq7qt\" (UID: \"52bbcac3-92a5-4c6f-90d7-5237247a86b6\") " pod="openshift-image-registry/image-registry-7c555f8bcd-cq7qt"
Nov 02 19:28:35.132342 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:35.132318    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjj62\" (UniqueName: \"kubernetes.io/projected/52bbcac3-92a5-4c6f-90d7-5237247a86b6-kube-api-access-gjj62\") pod \"image-registry-7c555f8bcd-cq7qt\" (UID: \"52bbcac3-92a5-4c6f-90d7-5237247a86b6\") " pod="openshift-image-registry/image-registry-7c555f8bcd-cq7qt"
Nov 02 19:28:35.186284 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:35.186263    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c555f8bcd-cq7qt"
Nov 02 19:28:35.595065 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:35.595036    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7c555f8bcd-cq7qt"]
Nov 02 19:28:35.598833 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:35.598802    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52bbcac3_92a5_4c6f_90d7_5237247a86b6.slice/crio-cd8bcd99e2a6f273ad08ef57ee6e1243b23c815567a517acb018916f42642232 WatchSource:0}: Error finding container cd8bcd99e2a6f273ad08ef57ee6e1243b23c815567a517acb018916f42642232: Status 404 returned error can't find the container with id cd8bcd99e2a6f273ad08ef57ee6e1243b23c815567a517acb018916f42642232
Nov 02 19:28:35.675552 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:35.675530    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-qj8mr"
Nov 02 19:28:35.712376 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:35.712346    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c555f8bcd-cq7qt" event={"ID":"52bbcac3-92a5-4c6f-90d7-5237247a86b6","Type":"ContainerStarted","Data":"255c12eee43b0b1c46aa37319863e3a855c46eb042a1d91467cb14fe3377774a"}
Nov 02 19:28:35.712726 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:35.712396    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c555f8bcd-cq7qt" event={"ID":"52bbcac3-92a5-4c6f-90d7-5237247a86b6","Type":"ContainerStarted","Data":"cd8bcd99e2a6f273ad08ef57ee6e1243b23c815567a517acb018916f42642232"}
Nov 02 19:28:35.891568 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:35.891323    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7c555f8bcd-cq7qt" podStartSLOduration=1.891303416 podStartE2EDuration="1.891303416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:28:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:28:35.890780378 +0000 UTC m=+22.497814172" watchObservedRunningTime="2026-11-02 19:28:35.891303416 +0000 UTC m=+22.498337186"
Nov 02 19:28:36.715142 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:36.715101    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7c555f8bcd-cq7qt"
Nov 02 19:28:45.637079 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:45.637050    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-rhdd6"
Nov 02 19:28:47.407404 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:47.407364    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-ckhs2"]
Nov 02 19:28:47.447210 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:47.447186    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ckhs2"
Nov 02 19:28:47.453321 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:47.452953    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtqg4\" (UniqueName: \"kubernetes.io/projected/bef314aa-0c49-4545-b518-cd0015b5b159-kube-api-access-vtqg4\") pod \"openshift-state-metrics-7bd7489957-ckhs2\" (UID: \"bef314aa-0c49-4545-b518-cd0015b5b159\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ckhs2"
Nov 02 19:28:47.453321 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:47.452995    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bef314aa-0c49-4545-b518-cd0015b5b159-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-ckhs2\" (UID: \"bef314aa-0c49-4545-b518-cd0015b5b159\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ckhs2"
Nov 02 19:28:47.453321 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:47.453117    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bef314aa-0c49-4545-b518-cd0015b5b159-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-ckhs2\" (UID: \"bef314aa-0c49-4545-b518-cd0015b5b159\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ckhs2"
Nov 02 19:28:47.453321 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:47.453168    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bef314aa-0c49-4545-b518-cd0015b5b159-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-ckhs2\" (UID: \"bef314aa-0c49-4545-b518-cd0015b5b159\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ckhs2"
Nov 02 19:28:47.465472 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:47.465449    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-ckhs2"]
Nov 02 19:28:47.510991 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:47.510965    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 02 19:28:47.510991 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:47.510976    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Nov 02 19:28:47.510991 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:47.510965    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 02 19:28:47.511168 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:47.511048    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-rwkp6\""
Nov 02 19:28:47.542181 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:47.542160    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Nov 02 19:28:47.542343 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:47.542329    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 02 19:28:47.553729 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:47.553709    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bef314aa-0c49-4545-b518-cd0015b5b159-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-ckhs2\" (UID: \"bef314aa-0c49-4545-b518-cd0015b5b159\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ckhs2"
Nov 02 19:28:47.553787 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:47.553738    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bef314aa-0c49-4545-b518-cd0015b5b159-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-ckhs2\" (UID: \"bef314aa-0c49-4545-b518-cd0015b5b159\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ckhs2"
Nov 02 19:28:47.553787 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:47.553764    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bef314aa-0c49-4545-b518-cd0015b5b159-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-ckhs2\" (UID: \"bef314aa-0c49-4545-b518-cd0015b5b159\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ckhs2"
Nov 02 19:28:47.553849 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:47.553810    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtqg4\" (UniqueName: \"kubernetes.io/projected/bef314aa-0c49-4545-b518-cd0015b5b159-kube-api-access-vtqg4\") pod \"openshift-state-metrics-7bd7489957-ckhs2\" (UID: \"bef314aa-0c49-4545-b518-cd0015b5b159\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ckhs2"
Nov 02 19:28:47.554454 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:47.554432    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bef314aa-0c49-4545-b518-cd0015b5b159-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-ckhs2\" (UID: \"bef314aa-0c49-4545-b518-cd0015b5b159\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ckhs2"
Nov 02 19:28:47.556089 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:47.556066    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bef314aa-0c49-4545-b518-cd0015b5b159-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-ckhs2\" (UID: \"bef314aa-0c49-4545-b518-cd0015b5b159\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ckhs2"
Nov 02 19:28:47.556160 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:47.556151    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bef314aa-0c49-4545-b518-cd0015b5b159-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-ckhs2\" (UID: \"bef314aa-0c49-4545-b518-cd0015b5b159\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ckhs2"
Nov 02 19:28:47.557909 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:47.557892    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-gfk8j"]
Nov 02 19:28:47.575719 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:47.575698    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gfk8j"
Nov 02 19:28:47.606024 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:47.605989    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtqg4\" (UniqueName: \"kubernetes.io/projected/bef314aa-0c49-4545-b518-cd0015b5b159-kube-api-access-vtqg4\") pod \"openshift-state-metrics-7bd7489957-ckhs2\" (UID: \"bef314aa-0c49-4545-b518-cd0015b5b159\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ckhs2"
Nov 02 19:28:47.624206 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:47.624190    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 02 19:28:47.639165 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:47.639143    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 02 19:28:47.639250 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:47.639181    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 02 19:28:47.639250 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:47.639202    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-njtg7\""
Nov 02 19:28:47.654815 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:47.654791    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7ffd9b32-1bf0-4c92-959e-237289d6f609-metrics-client-ca\") pod \"node-exporter-gfk8j\" (UID: \"7ffd9b32-1bf0-4c92-959e-237289d6f609\") " pod="openshift-monitoring/node-exporter-gfk8j"
Nov 02 19:28:47.654912 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:47.654820    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7ffd9b32-1bf0-4c92-959e-237289d6f609-sys\") pod \"node-exporter-gfk8j\" (UID: \"7ffd9b32-1bf0-4c92-959e-237289d6f609\") " pod="openshift-monitoring/node-exporter-gfk8j"
Nov 02 19:28:47.654912 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:47.654835    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzk8j\" (UniqueName: \"kubernetes.io/projected/7ffd9b32-1bf0-4c92-959e-237289d6f609-kube-api-access-wzk8j\") pod \"node-exporter-gfk8j\" (UID: \"7ffd9b32-1bf0-4c92-959e-237289d6f609\") " pod="openshift-monitoring/node-exporter-gfk8j"
Nov 02 19:28:47.654912 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:47.654851    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7ffd9b32-1bf0-4c92-959e-237289d6f609-root\") pod \"node-exporter-gfk8j\" (UID: \"7ffd9b32-1bf0-4c92-959e-237289d6f609\") " pod="openshift-monitoring/node-exporter-gfk8j"
Nov 02 19:28:47.654912 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:47.654868    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7ffd9b32-1bf0-4c92-959e-237289d6f609-node-exporter-wtmp\") pod \"node-exporter-gfk8j\" (UID: \"7ffd9b32-1bf0-4c92-959e-237289d6f609\") " pod="openshift-monitoring/node-exporter-gfk8j"
Nov 02 19:28:47.655100 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:47.654946    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7ffd9b32-1bf0-4c92-959e-237289d6f609-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gfk8j\" (UID: \"7ffd9b32-1bf0-4c92-959e-237289d6f609\") " pod="openshift-monitoring/node-exporter-gfk8j"
Nov 02 19:28:47.655100 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:47.655056    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7ffd9b32-1bf0-4c92-959e-237289d6f609-node-exporter-accelerators-collector-config\") pod \"node-exporter-gfk8j\" (UID: \"7ffd9b32-1bf0-4c92-959e-237289d6f609\") " pod="openshift-monitoring/node-exporter-gfk8j"
Nov 02 19:28:47.655100 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:47.655079    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7ffd9b32-1bf0-4c92-959e-237289d6f609-node-exporter-textfile\") pod \"node-exporter-gfk8j\" (UID: \"7ffd9b32-1bf0-4c92-959e-237289d6f609\") " pod="openshift-monitoring/node-exporter-gfk8j"
Nov 02 19:28:47.655100 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:47.655098    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7ffd9b32-1bf0-4c92-959e-237289d6f609-node-exporter-tls\") pod \"node-exporter-gfk8j\" (UID: \"7ffd9b32-1bf0-4c92-959e-237289d6f609\") " pod="openshift-monitoring/node-exporter-gfk8j"
Nov 02 19:28:47.756323 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:47.756295    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7ffd9b32-1bf0-4c92-959e-237289d6f609-node-exporter-accelerators-collector-config\") pod \"node-exporter-gfk8j\" (UID: \"7ffd9b32-1bf0-4c92-959e-237289d6f609\") " pod="openshift-monitoring/node-exporter-gfk8j"
Nov 02 19:28:47.756422 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:47.756331    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7ffd9b32-1bf0-4c92-959e-237289d6f609-node-exporter-textfile\") pod \"node-exporter-gfk8j\" (UID: \"7ffd9b32-1bf0-4c92-959e-237289d6f609\") " pod="openshift-monitoring/node-exporter-gfk8j"
Nov 02 19:28:47.756422 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:47.756362    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7ffd9b32-1bf0-4c92-959e-237289d6f609-node-exporter-tls\") pod \"node-exporter-gfk8j\" (UID: \"7ffd9b32-1bf0-4c92-959e-237289d6f609\") " pod="openshift-monitoring/node-exporter-gfk8j"
Nov 02 19:28:47.756422 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:47.756416    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7ffd9b32-1bf0-4c92-959e-237289d6f609-metrics-client-ca\") pod \"node-exporter-gfk8j\" (UID: \"7ffd9b32-1bf0-4c92-959e-237289d6f609\") " pod="openshift-monitoring/node-exporter-gfk8j"
Nov 02 19:28:47.756565 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:47.756442    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7ffd9b32-1bf0-4c92-959e-237289d6f609-sys\") pod \"node-exporter-gfk8j\" (UID: \"7ffd9b32-1bf0-4c92-959e-237289d6f609\") " pod="openshift-monitoring/node-exporter-gfk8j"
Nov 02 19:28:47.756565 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:47.756491    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7ffd9b32-1bf0-4c92-959e-237289d6f609-sys\") pod \"node-exporter-gfk8j\" (UID: \"7ffd9b32-1bf0-4c92-959e-237289d6f609\") " pod="openshift-monitoring/node-exporter-gfk8j"
Nov 02 19:28:47.756565 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:47.756549    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzk8j\" (UniqueName: \"kubernetes.io/projected/7ffd9b32-1bf0-4c92-959e-237289d6f609-kube-api-access-wzk8j\") pod \"node-exporter-gfk8j\" (UID: \"7ffd9b32-1bf0-4c92-959e-237289d6f609\") " pod="openshift-monitoring/node-exporter-gfk8j"
Nov 02 19:28:47.756747 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:47.756592    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7ffd9b32-1bf0-4c92-959e-237289d6f609-root\") pod \"node-exporter-gfk8j\" (UID: \"7ffd9b32-1bf0-4c92-959e-237289d6f609\") " pod="openshift-monitoring/node-exporter-gfk8j"
Nov 02 19:28:47.756747 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:47.756629    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7ffd9b32-1bf0-4c92-959e-237289d6f609-node-exporter-wtmp\") pod \"node-exporter-gfk8j\" (UID: \"7ffd9b32-1bf0-4c92-959e-237289d6f609\") " pod="openshift-monitoring/node-exporter-gfk8j"
Nov 02 19:28:47.756747 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:47.756658    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7ffd9b32-1bf0-4c92-959e-237289d6f609-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gfk8j\" (UID: \"7ffd9b32-1bf0-4c92-959e-237289d6f609\") " pod="openshift-monitoring/node-exporter-gfk8j"
Nov 02 19:28:47.756747 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:47.756672    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7ffd9b32-1bf0-4c92-959e-237289d6f609-root\") pod \"node-exporter-gfk8j\" (UID: \"7ffd9b32-1bf0-4c92-959e-237289d6f609\") " pod="openshift-monitoring/node-exporter-gfk8j"
Nov 02 19:28:47.756970 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:47.756793    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7ffd9b32-1bf0-4c92-959e-237289d6f609-node-exporter-textfile\") pod \"node-exporter-gfk8j\" (UID: \"7ffd9b32-1bf0-4c92-959e-237289d6f609\") " pod="openshift-monitoring/node-exporter-gfk8j"
Nov 02 19:28:47.756970 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:47.756808    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7ffd9b32-1bf0-4c92-959e-237289d6f609-node-exporter-wtmp\") pod \"node-exporter-gfk8j\" (UID: \"7ffd9b32-1bf0-4c92-959e-237289d6f609\") " pod="openshift-monitoring/node-exporter-gfk8j"
Nov 02 19:28:47.756970 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:47.756928    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7ffd9b32-1bf0-4c92-959e-237289d6f609-node-exporter-accelerators-collector-config\") pod \"node-exporter-gfk8j\" (UID: \"7ffd9b32-1bf0-4c92-959e-237289d6f609\") " pod="openshift-monitoring/node-exporter-gfk8j"
Nov 02 19:28:47.756970 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:47.756962    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7ffd9b32-1bf0-4c92-959e-237289d6f609-metrics-client-ca\") pod \"node-exporter-gfk8j\" (UID: \"7ffd9b32-1bf0-4c92-959e-237289d6f609\") " pod="openshift-monitoring/node-exporter-gfk8j"
Nov 02 19:28:47.758644 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:47.758626    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7ffd9b32-1bf0-4c92-959e-237289d6f609-node-exporter-tls\") pod \"node-exporter-gfk8j\" (UID: \"7ffd9b32-1bf0-4c92-959e-237289d6f609\") " pod="openshift-monitoring/node-exporter-gfk8j"
Nov 02 19:28:47.758730 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:47.758691    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7ffd9b32-1bf0-4c92-959e-237289d6f609-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gfk8j\" (UID: \"7ffd9b32-1bf0-4c92-959e-237289d6f609\") " pod="openshift-monitoring/node-exporter-gfk8j"
Nov 02 19:28:47.796390 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:47.796363    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ckhs2"
Nov 02 19:28:47.809183 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:47.809161    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzk8j\" (UniqueName: \"kubernetes.io/projected/7ffd9b32-1bf0-4c92-959e-237289d6f609-kube-api-access-wzk8j\") pod \"node-exporter-gfk8j\" (UID: \"7ffd9b32-1bf0-4c92-959e-237289d6f609\") " pod="openshift-monitoring/node-exporter-gfk8j"
Nov 02 19:28:47.939055 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:47.939007    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gfk8j"
Nov 02 19:28:47.969238 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:47.969213    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ffd9b32_1bf0_4c92_959e_237289d6f609.slice/crio-3da959a606153ab0344db771b93aaad409e219411b1ec561c6dbe5cde57705b2 WatchSource:0}: Error finding container 3da959a606153ab0344db771b93aaad409e219411b1ec561c6dbe5cde57705b2: Status 404 returned error can't find the container with id 3da959a606153ab0344db771b93aaad409e219411b1ec561c6dbe5cde57705b2
Nov 02 19:28:48.241870 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:48.241840    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-ckhs2"]
Nov 02 19:28:48.245041 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:48.244997    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbef314aa_0c49_4545_b518_cd0015b5b159.slice/crio-74be0abb2c44656bd30cf533cfa4929f14c466499306c2ea75e8d6f41931706a WatchSource:0}: Error finding container 74be0abb2c44656bd30cf533cfa4929f14c466499306c2ea75e8d6f41931706a: Status 404 returned error can't find the container with id 74be0abb2c44656bd30cf533cfa4929f14c466499306c2ea75e8d6f41931706a
Nov 02 19:28:48.750953 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:48.750920    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ckhs2" event={"ID":"bef314aa-0c49-4545-b518-cd0015b5b159","Type":"ContainerStarted","Data":"c1c31a72661050b706e59d99722f2e5edacc6393df212cbba99893e17e772202"}
Nov 02 19:28:48.751387 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:48.751020    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ckhs2" event={"ID":"bef314aa-0c49-4545-b518-cd0015b5b159","Type":"ContainerStarted","Data":"654216be1321d9a0af9514e2a80385a2b26e3c9445d67b3a050f7502312adf3e"}
Nov 02 19:28:48.751387 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:48.751043    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ckhs2" event={"ID":"bef314aa-0c49-4545-b518-cd0015b5b159","Type":"ContainerStarted","Data":"74be0abb2c44656bd30cf533cfa4929f14c466499306c2ea75e8d6f41931706a"}
Nov 02 19:28:48.752042 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:48.752006    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gfk8j" event={"ID":"7ffd9b32-1bf0-4c92-959e-237289d6f609","Type":"ContainerStarted","Data":"3da959a606153ab0344db771b93aaad409e219411b1ec561c6dbe5cde57705b2"}
Nov 02 19:28:50.759737 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:50.759700    4463 generic.go:356] "Generic (PLEG): container finished" podID="7ffd9b32-1bf0-4c92-959e-237289d6f609" containerID="3ad7114cd10e2ae0a90db08ff0c68499547d8bf301deba9b8763ce777355b15c" exitCode=0
Nov 02 19:28:50.760229 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:50.759788    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gfk8j" event={"ID":"7ffd9b32-1bf0-4c92-959e-237289d6f609","Type":"ContainerDied","Data":"3ad7114cd10e2ae0a90db08ff0c68499547d8bf301deba9b8763ce777355b15c"}
Nov 02 19:28:50.761914 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:50.761892    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ckhs2" event={"ID":"bef314aa-0c49-4545-b518-cd0015b5b159","Type":"ContainerStarted","Data":"bcc9998b7a588ff79c809ece550d8f9687d7a0948286fa5d3c9f4fa540b7898d"}
Nov 02 19:28:51.032462 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:51.032143    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ckhs2" podStartSLOduration=2.172367072 podStartE2EDuration="4.032126141s" totalImagesPullingTime="1.859759069s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:28:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:28:51.031434157 +0000 UTC m=+37.638467926" watchObservedRunningTime="2026-11-02 19:28:51.032126141 +0000 UTC m=+37.639159911"
Nov 02 19:28:51.767362 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:51.767323    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gfk8j" event={"ID":"7ffd9b32-1bf0-4c92-959e-237289d6f609","Type":"ContainerStarted","Data":"588eb7c539daf7ad7f85f55f0dca12c65bab9de707bd8609e0331617a5e4837c"}
Nov 02 19:28:51.767796 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:51.767395    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gfk8j" event={"ID":"7ffd9b32-1bf0-4c92-959e-237289d6f609","Type":"ContainerStarted","Data":"fb7207848f7557be8a2bdd113303ea99d6704244dee94ecf216d63310451705c"}
Nov 02 19:28:51.877421 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:51.877372    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-gfk8j" podStartSLOduration=2.78797333 podStartE2EDuration="4.877358373s" totalImagesPullingTime="2.013382107s" totalInitContainerRuntime="76.002936ms" isStatefulPod=true podCreationTimestamp="2026-11-02 19:28:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:28:51.876742476 +0000 UTC m=+38.483776258" watchObservedRunningTime="2026-11-02 19:28:51.877358373 +0000 UTC m=+38.484392140"
Nov 02 19:28:56.628831 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:56.628753    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 02 19:28:56.631266 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:56.631246    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:28:56.680311 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:56.680290    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Nov 02 19:28:56.680428 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:56.680365    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Nov 02 19:28:56.720276 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:56.720248    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:28:56.720276 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:56.720279    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-config-out\") pod \"prometheus-k8s-0\" (UID: \"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:28:56.720428 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:56.720294    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsvdc\" (UniqueName: \"kubernetes.io/projected/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-kube-api-access-tsvdc\") pod \"prometheus-k8s-0\" (UID: \"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:28:56.720428 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:56.720315    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:28:56.720428 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:56.720376    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:28:56.720523 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:56.720417    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:28:56.720523 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:56.720469    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:28:56.720523 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:56.720489    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:28:56.720682 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:56.720531    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:28:56.720682 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:56.720558    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:28:56.720682 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:56.720579    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:28:56.720682 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:56.720619    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:28:56.720682 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:56.720650    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:28:56.720826 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:56.720713    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:28:56.720826 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:56.720736    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-web-config\") pod \"prometheus-k8s-0\" (UID: \"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:28:56.720826 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:56.720784    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-config\") pod \"prometheus-k8s-0\" (UID: \"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:28:56.720826 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:56.720816    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:28:56.720941 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:56.720842    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:28:56.720941 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:56.720892    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:28:56.720941 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:56.720911    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:28:56.727304 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:56.727280    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Nov 02 19:28:56.727462 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:56.727436    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Nov 02 19:28:56.727462 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:56.727353    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Nov 02 19:28:56.727641 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:56.727400    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Nov 02 19:28:56.727641 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:56.727481    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Nov 02 19:28:56.727641 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:56.727289    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Nov 02 19:28:56.727641 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:56.727514    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Nov 02 19:28:56.727641 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:56.727526    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Nov 02 19:28:56.727641 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:56.727565    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-t4rdb\""
Nov 02 19:28:56.727641 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:56.727630    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Nov 02 19:28:56.727933 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:56.727714    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-9cdm8fh30lr9b\""
Nov 02 19:28:56.728413 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:56.728396    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 02 19:28:56.780348 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:56.780323    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Nov 02 19:28:56.780438 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:56.780364    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Nov 02 19:28:56.822128 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:56.822107    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:28:56.822217 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:56.822135    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:28:56.822217 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:56.822155    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:28:56.822217 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:56.822173    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:28:56.822217 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:56.822197    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:28:56.822405 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:56.822223    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:28:56.822405 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:56.822240    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:28:56.822405 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:56.822254    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:28:56.822405 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:56.822281    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:28:56.822405 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:56.822306    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-web-config\") pod \"prometheus-k8s-0\" (UID: \"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:28:56.822405 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:56.822341    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-config\") pod \"prometheus-k8s-0\" (UID: \"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:28:56.822405 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:56.822366    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:28:56.822405 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:56.822395    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:28:56.822681 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:56.822416    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:28:56.822681 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:56.822437    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:28:56.822681 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:56.822464    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:28:56.822681 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:56.822483    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-config-out\") pod \"prometheus-k8s-0\" (UID: \"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:28:56.822681 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:56.822498    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsvdc\" (UniqueName: \"kubernetes.io/projected/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-kube-api-access-tsvdc\") pod \"prometheus-k8s-0\" (UID: \"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:28:56.822681 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:56.822517    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:28:56.822681 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:56.822531    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:28:56.822969 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:56.822828    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:28:56.822969 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:56.822947    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:28:56.823216 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:56.823192    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:28:56.823646 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:56.823620    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:28:56.824405 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:56.823823    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:28:56.824405 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:56.824294    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:28:56.825602 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:56.825579    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-web-config\") pod \"prometheus-k8s-0\" (UID: \"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:28:56.826362 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:56.826341    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:28:56.826615 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:56.826593    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:28:56.826782 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:56.826759    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:28:56.826867 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:56.826796    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:28:56.827487 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:56.827310    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:28:56.827641 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:56.827490    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-config-out\") pod \"prometheus-k8s-0\" (UID: \"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:28:56.827641 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:56.827544    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:28:56.827991 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:56.827969    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Nov 02 19:28:56.828080 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:56.828001    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Nov 02 19:28:56.828235 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:56.828211    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:28:56.828457 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:56.828437    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:28:56.829065 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:56.829008    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-config\") pod \"prometheus-k8s-0\" (UID: \"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:28:56.833653 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:56.833632    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:28:56.835093 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:56.835076    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:28:56.876806 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:56.876786    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsvdc\" (UniqueName: \"kubernetes.io/projected/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-kube-api-access-tsvdc\") pod \"prometheus-k8s-0\" (UID: \"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:28:56.946951 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:56.946928    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:28:57.412632 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:57.412561    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 02 19:28:57.463235 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:28:57.463211    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f9e31fc_0bbd_42de_8afb_1d62bb4b3ef6.slice/crio-ae01abf175d8b84bd7e55310d5e23b6bd88da3c4f1a53d851065df87880aa38c WatchSource:0}: Error finding container ae01abf175d8b84bd7e55310d5e23b6bd88da3c4f1a53d851065df87880aa38c: Status 404 returned error can't find the container with id ae01abf175d8b84bd7e55310d5e23b6bd88da3c4f1a53d851065df87880aa38c
Nov 02 19:28:57.721647 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:57.721623    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7c555f8bcd-cq7qt"
Nov 02 19:28:57.787937 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:57.787903    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6","Type":"ContainerStarted","Data":"ae01abf175d8b84bd7e55310d5e23b6bd88da3c4f1a53d851065df87880aa38c"}
Nov 02 19:28:59.795642 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:59.795601    4463 generic.go:356] "Generic (PLEG): container finished" podID="3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6" containerID="6a4f1f869b329698cf706634ed7e7e9346ba2ca7fb58a6ea4b7dcb2871d2b91b" exitCode=0
Nov 02 19:28:59.796129 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:28:59.795642    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6","Type":"ContainerDied","Data":"6a4f1f869b329698cf706634ed7e7e9346ba2ca7fb58a6ea4b7dcb2871d2b91b"}
Nov 02 19:29:02.698779 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:29:02.698747    4463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jgfjj_3fabcb64-faf2-49f9-813a-ddafde0fd159/konnectivity-agent/0.log" limit=1024
Nov 02 19:29:03.818698 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:29:03.818616    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6","Type":"ContainerStarted","Data":"0f086ed8657f6bfba49edbd31efefccad0546d49366036a18306d4c9e7d447db"}
Nov 02 19:29:03.819070 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:29:03.818744    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6","Type":"ContainerStarted","Data":"70c9116606bd5bc31b3c8d441aef43a47d50794c451250fe8bd072c4e6e6063a"}
Nov 02 19:29:04.293634 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:29:04.293591    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 19:29:06.832555 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:29:06.832522    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6","Type":"ContainerStarted","Data":"1191899c144edc026e3faa7a91bfa808848360317d529f243954165c9236c54a"}
Nov 02 19:29:06.832953 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:29:06.832584    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6","Type":"ContainerStarted","Data":"e805a31089b420a0bb031d7bd8d139f97e36b5fa80c3f33d02ff138ae9812f52"}
Nov 02 19:29:06.832953 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:29:06.832608    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6","Type":"ContainerStarted","Data":"a19b6396e6bb883ee21fb5bb101ac44eef51a42f57640f232d81d99db225e4ff"}
Nov 02 19:29:06.832953 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:29:06.832633    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6","Type":"ContainerStarted","Data":"a44073d3e07dc0bb7b8a1ffe71825d6ab45ca0edd6b85d661c37d7faf1bfc6db"}
Nov 02 19:29:06.947944 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:29:06.947907    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:29:07.091477 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:29:07.091379    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.528486446 podStartE2EDuration="11.091365369s" totalImagesPullingTime="7.547887835s" totalInitContainerRuntime="14.991088ms" isStatefulPod=false podCreationTimestamp="2026-11-02 19:28:56 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:29:07.067385967 +0000 UTC m=+53.674419747" watchObservedRunningTime="2026-11-02 19:29:07.091365369 +0000 UTC m=+53.698399138"
Nov 02 19:29:34.292955 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:29:34.292920    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 19:29:40.563220 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:29:40.563175    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-xvzjr-ggg8w"]
Nov 02 19:29:40.563851 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:29:40.563433    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-xvzjr-ggg8w" podUID="4874b345-eac7-4694-8351-d24085bf180a" containerName="insights-gathering" containerID="cri-o://8ec058e8f2189f7936cd79d0710a93b07e6d8a58abeec9756522d41c1f3f117d" gracePeriod=30
Nov 02 19:29:40.820775 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:29:40.820722    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-xvzjr-ggg8w"
Nov 02 19:29:40.944954 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:29:40.944910    4463 generic.go:356] "Generic (PLEG): container finished" podID="4874b345-eac7-4694-8351-d24085bf180a" containerID="8ec058e8f2189f7936cd79d0710a93b07e6d8a58abeec9756522d41c1f3f117d" exitCode=2
Nov 02 19:29:40.944954 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:29:40.944962    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-xvzjr-ggg8w" event={"ID":"4874b345-eac7-4694-8351-d24085bf180a","Type":"ContainerDied","Data":"8ec058e8f2189f7936cd79d0710a93b07e6d8a58abeec9756522d41c1f3f117d"}
Nov 02 19:29:40.945166 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:29:40.944970    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-xvzjr-ggg8w"
Nov 02 19:29:40.945166 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:29:40.945024    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-xvzjr-ggg8w" event={"ID":"4874b345-eac7-4694-8351-d24085bf180a","Type":"ContainerDied","Data":"c8861ce03831186f570d157c4d607cc643f2909b90a3a4e9d33fcf0b3dec3570"}
Nov 02 19:29:40.945166 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:29:40.945058    4463 scope.go:122] "RemoveContainer" containerID="8ec058e8f2189f7936cd79d0710a93b07e6d8a58abeec9756522d41c1f3f117d"
Nov 02 19:29:40.953139 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:29:40.953120    4463 scope.go:122] "RemoveContainer" containerID="8ec058e8f2189f7936cd79d0710a93b07e6d8a58abeec9756522d41c1f3f117d"
Nov 02 19:29:40.953413 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:29:40.953393    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ec058e8f2189f7936cd79d0710a93b07e6d8a58abeec9756522d41c1f3f117d\": container with ID starting with 8ec058e8f2189f7936cd79d0710a93b07e6d8a58abeec9756522d41c1f3f117d not found: ID does not exist" containerID="8ec058e8f2189f7936cd79d0710a93b07e6d8a58abeec9756522d41c1f3f117d"
Nov 02 19:29:40.953467 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:29:40.953421    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ec058e8f2189f7936cd79d0710a93b07e6d8a58abeec9756522d41c1f3f117d"} err="failed to get container status \"8ec058e8f2189f7936cd79d0710a93b07e6d8a58abeec9756522d41c1f3f117d\": rpc error: code = NotFound desc = could not find container \"8ec058e8f2189f7936cd79d0710a93b07e6d8a58abeec9756522d41c1f3f117d\": container with ID starting with 8ec058e8f2189f7936cd79d0710a93b07e6d8a58abeec9756522d41c1f3f117d not found: ID does not exist"
Nov 02 19:29:40.956995 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:29:40.956980    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4874b345-eac7-4694-8351-d24085bf180a-kube-api-access-jz8qt\" (UniqueName: \"kubernetes.io/projected/4874b345-eac7-4694-8351-d24085bf180a-kube-api-access-jz8qt\") pod \"4874b345-eac7-4694-8351-d24085bf180a\" (UID: \"4874b345-eac7-4694-8351-d24085bf180a\") "
Nov 02 19:29:40.957056 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:29:40.957023    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4874b345-eac7-4694-8351-d24085bf180a-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4874b345-eac7-4694-8351-d24085bf180a-service-ca-bundle\") pod \"4874b345-eac7-4694-8351-d24085bf180a\" (UID: \"4874b345-eac7-4694-8351-d24085bf180a\") "
Nov 02 19:29:40.957090 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:29:40.957059    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4874b345-eac7-4694-8351-d24085bf180a-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4874b345-eac7-4694-8351-d24085bf180a-trusted-ca-bundle\") pod \"4874b345-eac7-4694-8351-d24085bf180a\" (UID: \"4874b345-eac7-4694-8351-d24085bf180a\") "
Nov 02 19:29:40.957160 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:29:40.957138    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4874b345-eac7-4694-8351-d24085bf180a-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4874b345-eac7-4694-8351-d24085bf180a-archives-path\") pod \"4874b345-eac7-4694-8351-d24085bf180a\" (UID: \"4874b345-eac7-4694-8351-d24085bf180a\") "
Nov 02 19:29:40.957429 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:29:40.957410    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4874b345-eac7-4694-8351-d24085bf180a-service-ca-bundle" pod "4874b345-eac7-4694-8351-d24085bf180a" (UID: "4874b345-eac7-4694-8351-d24085bf180a"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 19:29:40.957505 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:29:40.957434    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4874b345-eac7-4694-8351-d24085bf180a-trusted-ca-bundle" pod "4874b345-eac7-4694-8351-d24085bf180a" (UID: "4874b345-eac7-4694-8351-d24085bf180a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 19:29:40.957505 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:29:40.957461    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4874b345-eac7-4694-8351-d24085bf180a-archives-path" pod "4874b345-eac7-4694-8351-d24085bf180a" (UID: "4874b345-eac7-4694-8351-d24085bf180a"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:29:40.959057 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:29:40.959036    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4874b345-eac7-4694-8351-d24085bf180a-kube-api-access-jz8qt" pod "4874b345-eac7-4694-8351-d24085bf180a" (UID: "4874b345-eac7-4694-8351-d24085bf180a"). InnerVolumeSpecName "kube-api-access-jz8qt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:29:41.058568 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:29:41.058533    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jz8qt\" (UniqueName: \"kubernetes.io/projected/4874b345-eac7-4694-8351-d24085bf180a-kube-api-access-jz8qt\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:29:41.058568 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:29:41.058564    4463 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4874b345-eac7-4694-8351-d24085bf180a-service-ca-bundle\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:29:41.058568 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:29:41.058576    4463 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4874b345-eac7-4694-8351-d24085bf180a-trusted-ca-bundle\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:29:41.058785 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:29:41.058587    4463 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4874b345-eac7-4694-8351-d24085bf180a-archives-path\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:29:41.355329 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:29:41.355296    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-xvzjr-ggg8w"]
Nov 02 19:29:41.361178 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:29:41.361148    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-xvzjr-ggg8w"]
Nov 02 19:29:41.580878 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:29:41.580814    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4874b345-eac7-4694-8351-d24085bf180a" path="/var/lib/kubelet/pods/4874b345-eac7-4694-8351-d24085bf180a/volumes"
Nov 02 19:29:47.425884 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:29:47.425835    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:29:47.443832 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:29:47.443807    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:29:47.983043 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:29:47.983001    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:30:04.292565 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:30:04.292529    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 19:30:09.064716 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:09.064679    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 02 19:30:09.065132 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:09.065068    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6" containerName="prometheus" containerID="cri-o://70c9116606bd5bc31b3c8d441aef43a47d50794c451250fe8bd072c4e6e6063a" gracePeriod=600
Nov 02 19:30:09.065132 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:09.065090    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6" containerName="thanos-sidecar" containerID="cri-o://a44073d3e07dc0bb7b8a1ffe71825d6ab45ca0edd6b85d661c37d7faf1bfc6db" gracePeriod=600
Nov 02 19:30:09.065252 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:09.065129    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6" containerName="config-reloader" containerID="cri-o://0f086ed8657f6bfba49edbd31efefccad0546d49366036a18306d4c9e7d447db" gracePeriod=600
Nov 02 19:30:09.065252 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:09.065138    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6" containerName="kube-rbac-proxy-thanos" containerID="cri-o://1191899c144edc026e3faa7a91bfa808848360317d529f243954165c9236c54a" gracePeriod=600
Nov 02 19:30:09.065252 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:09.065151    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6" containerName="kube-rbac-proxy-web" containerID="cri-o://a19b6396e6bb883ee21fb5bb101ac44eef51a42f57640f232d81d99db225e4ff" gracePeriod=600
Nov 02 19:30:09.065252 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:09.065086    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6" containerName="kube-rbac-proxy" containerID="cri-o://e805a31089b420a0bb031d7bd8d139f97e36b5fa80c3f33d02ff138ae9812f52" gracePeriod=600
Nov 02 19:30:09.302697 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:09.302673    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:30:09.351278 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:09.351240    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6\" (UID: \"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6\") "
Nov 02 19:30:09.351278 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:09.351285    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-configmap-serving-certs-ca-bundle\") pod \"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6\" (UID: \"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6\") "
Nov 02 19:30:09.351522 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:09.351313    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-prometheus-k8s-rulefiles-2\") pod \"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6\" (UID: \"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6\") "
Nov 02 19:30:09.351522 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:09.351405    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-prometheus-k8s-rulefiles-1\") pod \"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6\" (UID: \"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6\") "
Nov 02 19:30:09.351522 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:09.351489    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6\" (UID: \"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6\") "
Nov 02 19:30:09.351522 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:09.351521    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-secret-prometheus-k8s-tls\") pod \"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6\" (UID: \"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6\") "
Nov 02 19:30:09.351731 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:09.351552    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-kube-api-access-tsvdc\" (UniqueName: \"kubernetes.io/projected/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-kube-api-access-tsvdc\") pod \"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6\" (UID: \"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6\") "
Nov 02 19:30:09.351731 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:09.351588    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-prometheus-k8s-db\") pod \"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6\" (UID: \"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6\") "
Nov 02 19:30:09.351731 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:09.351621    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-prometheus-k8s-rulefiles-2" pod "3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6" (UID: "3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 19:30:09.351731 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:09.351629    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-config-out\" (UniqueName: \"kubernetes.io/empty-dir/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-config-out\") pod \"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6\" (UID: \"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6\") "
Nov 02 19:30:09.351731 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:09.351726    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-configmap-kubelet-serving-ca-bundle\") pod \"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6\" (UID: \"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6\") "
Nov 02 19:30:09.351917 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:09.351753    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-prometheus-trusted-ca-bundle\") pod \"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6\" (UID: \"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6\") "
Nov 02 19:30:09.351917 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:09.351748    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-configmap-serving-certs-ca-bundle" pod "3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6" (UID: "3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 19:30:09.351917 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:09.351763    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-prometheus-k8s-rulefiles-1" pod "3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6" (UID: "3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 19:30:09.351917 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:09.351772    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-web-config\" (UniqueName: \"kubernetes.io/secret/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-web-config\") pod \"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6\" (UID: \"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6\") "
Nov 02 19:30:09.351917 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:09.351836    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-secret-kube-rbac-proxy\") pod \"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6\" (UID: \"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6\") "
Nov 02 19:30:09.351917 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:09.351870    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-secret-metrics-client-certs\") pod \"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6\" (UID: \"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6\") "
Nov 02 19:30:09.351917 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:09.351901    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-config\" (UniqueName: \"kubernetes.io/secret/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-config\") pod \"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6\" (UID: \"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6\") "
Nov 02 19:30:09.352160 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:09.351929    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-tls-assets\" (UniqueName: \"kubernetes.io/projected/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-tls-assets\") pod \"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6\" (UID: \"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6\") "
Nov 02 19:30:09.352160 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:09.351959    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-secret-grpc-tls\") pod \"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6\" (UID: \"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6\") "
Nov 02 19:30:09.352160 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:09.351984    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-prometheus-k8s-rulefiles-0\") pod \"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6\" (UID: \"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6\") "
Nov 02 19:30:09.352160 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:09.352042    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-configmap-metrics-client-ca\") pod \"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6\" (UID: \"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6\") "
Nov 02 19:30:09.352160 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:09.352084    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-thanos-prometheus-http-client-file\") pod \"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6\" (UID: \"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6\") "
Nov 02 19:30:09.352408 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:09.352305    4463 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:30:09.352408 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:09.352322    4463 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:30:09.352408 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:09.352337    4463 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:30:09.354255 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:09.354001    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-configmap-kubelet-serving-ca-bundle" pod "3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6" (UID: "3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 19:30:09.354452 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:09.354425    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-prometheus-k8s-db" pod "3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6" (UID: "3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:30:09.355529 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:09.355439    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-secret-metrics-client-certs" pod "3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6" (UID: "3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 19:30:09.355529 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:09.355454    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-secret-prometheus-k8s-kube-rbac-proxy-web" pod "3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6" (UID: "3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 19:30:09.355685 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:09.355539    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-prometheus-trusted-ca-bundle" pod "3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6" (UID: "3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 19:30:09.355851 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:09.355817    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-secret-prometheus-k8s-thanos-sidecar-tls" pod "3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6" (UID: "3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 19:30:09.355919 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:09.355845    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-config-out" pod "3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6" (UID: "3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:30:09.356096 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:09.356032    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-configmap-metrics-client-ca" pod "3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6" (UID: "3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 19:30:09.356096 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:09.356071    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-thanos-prometheus-http-client-file" pod "3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6" (UID: "3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 19:30:09.356325 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:09.356258    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-prometheus-k8s-rulefiles-0" pod "3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6" (UID: "3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 19:30:09.356412 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:09.356390    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-secret-prometheus-k8s-tls" pod "3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6" (UID: "3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 19:30:09.356690 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:09.356659    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-kube-api-access-tsvdc" pod "3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6" (UID: "3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6"). InnerVolumeSpecName "kube-api-access-tsvdc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:30:09.356913 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:09.356895    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-secret-grpc-tls" pod "3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6" (UID: "3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 19:30:09.357404 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:09.357379    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-config" pod "3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6" (UID: "3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 19:30:09.357494 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:09.357427    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-secret-kube-rbac-proxy" pod "3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6" (UID: "3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 19:30:09.357732 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:09.357710    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-tls-assets" pod "3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6" (UID: "3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:30:09.364527 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:09.364509    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-web-config" pod "3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6" (UID: "3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 19:30:09.453281 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:09.453208    4463 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-configmap-metrics-client-ca\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:30:09.453281 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:09.453239    4463 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-thanos-prometheus-http-client-file\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:30:09.453281 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:09.453254    4463 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:30:09.453281 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:09.453268    4463 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:30:09.453281 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:09.453282    4463 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-secret-prometheus-k8s-tls\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:30:09.453555 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:09.453295    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsvdc\" (UniqueName: \"kubernetes.io/projected/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-kube-api-access-tsvdc\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:30:09.453555 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:09.453308    4463 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-prometheus-k8s-db\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:30:09.453555 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:09.453320    4463 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-config-out\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:30:09.453555 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:09.453332    4463 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:30:09.453555 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:09.453345    4463 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-prometheus-trusted-ca-bundle\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:30:09.453555 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:09.453357    4463 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-web-config\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:30:09.453555 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:09.453369    4463 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-secret-kube-rbac-proxy\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:30:09.453555 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:09.453381    4463 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-secret-metrics-client-certs\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:30:09.453555 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:09.453395    4463 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-config\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:30:09.453555 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:09.453406    4463 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-tls-assets\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:30:09.453555 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:09.453419    4463 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-secret-grpc-tls\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:30:09.453555 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:09.453432    4463 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:30:10.044473 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.044437    4463 generic.go:356] "Generic (PLEG): container finished" podID="3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6" containerID="1191899c144edc026e3faa7a91bfa808848360317d529f243954165c9236c54a" exitCode=0
Nov 02 19:30:10.044473 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.044463    4463 generic.go:356] "Generic (PLEG): container finished" podID="3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6" containerID="e805a31089b420a0bb031d7bd8d139f97e36b5fa80c3f33d02ff138ae9812f52" exitCode=0
Nov 02 19:30:10.044473 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.044470    4463 generic.go:356] "Generic (PLEG): container finished" podID="3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6" containerID="a19b6396e6bb883ee21fb5bb101ac44eef51a42f57640f232d81d99db225e4ff" exitCode=0
Nov 02 19:30:10.044473 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.044476    4463 generic.go:356] "Generic (PLEG): container finished" podID="3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6" containerID="a44073d3e07dc0bb7b8a1ffe71825d6ab45ca0edd6b85d661c37d7faf1bfc6db" exitCode=0
Nov 02 19:30:10.044473 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.044481    4463 generic.go:356] "Generic (PLEG): container finished" podID="3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6" containerID="0f086ed8657f6bfba49edbd31efefccad0546d49366036a18306d4c9e7d447db" exitCode=0
Nov 02 19:30:10.044747 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.044486    4463 generic.go:356] "Generic (PLEG): container finished" podID="3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6" containerID="70c9116606bd5bc31b3c8d441aef43a47d50794c451250fe8bd072c4e6e6063a" exitCode=0
Nov 02 19:30:10.044747 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.044508    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6","Type":"ContainerDied","Data":"1191899c144edc026e3faa7a91bfa808848360317d529f243954165c9236c54a"}
Nov 02 19:30:10.044747 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.044520    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:30:10.044747 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.044590    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6","Type":"ContainerDied","Data":"e805a31089b420a0bb031d7bd8d139f97e36b5fa80c3f33d02ff138ae9812f52"}
Nov 02 19:30:10.044747 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.044616    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6","Type":"ContainerDied","Data":"a19b6396e6bb883ee21fb5bb101ac44eef51a42f57640f232d81d99db225e4ff"}
Nov 02 19:30:10.044747 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.044641    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6","Type":"ContainerDied","Data":"a44073d3e07dc0bb7b8a1ffe71825d6ab45ca0edd6b85d661c37d7faf1bfc6db"}
Nov 02 19:30:10.044747 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.044662    4463 scope.go:122] "RemoveContainer" containerID="1191899c144edc026e3faa7a91bfa808848360317d529f243954165c9236c54a"
Nov 02 19:30:10.044747 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.044663    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6","Type":"ContainerDied","Data":"0f086ed8657f6bfba49edbd31efefccad0546d49366036a18306d4c9e7d447db"}
Nov 02 19:30:10.045066 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.044779    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6","Type":"ContainerDied","Data":"70c9116606bd5bc31b3c8d441aef43a47d50794c451250fe8bd072c4e6e6063a"}
Nov 02 19:30:10.045066 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.044807    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6","Type":"ContainerDied","Data":"ae01abf175d8b84bd7e55310d5e23b6bd88da3c4f1a53d851065df87880aa38c"}
Nov 02 19:30:10.051911 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.051895    4463 scope.go:122] "RemoveContainer" containerID="e805a31089b420a0bb031d7bd8d139f97e36b5fa80c3f33d02ff138ae9812f52"
Nov 02 19:30:10.058128 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.058112    4463 scope.go:122] "RemoveContainer" containerID="a19b6396e6bb883ee21fb5bb101ac44eef51a42f57640f232d81d99db225e4ff"
Nov 02 19:30:10.063997 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.063975    4463 scope.go:122] "RemoveContainer" containerID="a44073d3e07dc0bb7b8a1ffe71825d6ab45ca0edd6b85d661c37d7faf1bfc6db"
Nov 02 19:30:10.069980 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.069836    4463 scope.go:122] "RemoveContainer" containerID="0f086ed8657f6bfba49edbd31efefccad0546d49366036a18306d4c9e7d447db"
Nov 02 19:30:10.088177 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.088151    4463 scope.go:122] "RemoveContainer" containerID="70c9116606bd5bc31b3c8d441aef43a47d50794c451250fe8bd072c4e6e6063a"
Nov 02 19:30:10.094513 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.094499    4463 scope.go:122] "RemoveContainer" containerID="6a4f1f869b329698cf706634ed7e7e9346ba2ca7fb58a6ea4b7dcb2871d2b91b"
Nov 02 19:30:10.100496 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.100477    4463 scope.go:122] "RemoveContainer" containerID="1191899c144edc026e3faa7a91bfa808848360317d529f243954165c9236c54a"
Nov 02 19:30:10.100730 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:30:10.100711    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1191899c144edc026e3faa7a91bfa808848360317d529f243954165c9236c54a\": container with ID starting with 1191899c144edc026e3faa7a91bfa808848360317d529f243954165c9236c54a not found: ID does not exist" containerID="1191899c144edc026e3faa7a91bfa808848360317d529f243954165c9236c54a"
Nov 02 19:30:10.100787 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.100737    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1191899c144edc026e3faa7a91bfa808848360317d529f243954165c9236c54a"} err="failed to get container status \"1191899c144edc026e3faa7a91bfa808848360317d529f243954165c9236c54a\": rpc error: code = NotFound desc = could not find container \"1191899c144edc026e3faa7a91bfa808848360317d529f243954165c9236c54a\": container with ID starting with 1191899c144edc026e3faa7a91bfa808848360317d529f243954165c9236c54a not found: ID does not exist"
Nov 02 19:30:10.100787 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.100753    4463 scope.go:122] "RemoveContainer" containerID="e805a31089b420a0bb031d7bd8d139f97e36b5fa80c3f33d02ff138ae9812f52"
Nov 02 19:30:10.100959 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:30:10.100943    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e805a31089b420a0bb031d7bd8d139f97e36b5fa80c3f33d02ff138ae9812f52\": container with ID starting with e805a31089b420a0bb031d7bd8d139f97e36b5fa80c3f33d02ff138ae9812f52 not found: ID does not exist" containerID="e805a31089b420a0bb031d7bd8d139f97e36b5fa80c3f33d02ff138ae9812f52"
Nov 02 19:30:10.101001 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.100964    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e805a31089b420a0bb031d7bd8d139f97e36b5fa80c3f33d02ff138ae9812f52"} err="failed to get container status \"e805a31089b420a0bb031d7bd8d139f97e36b5fa80c3f33d02ff138ae9812f52\": rpc error: code = NotFound desc = could not find container \"e805a31089b420a0bb031d7bd8d139f97e36b5fa80c3f33d02ff138ae9812f52\": container with ID starting with e805a31089b420a0bb031d7bd8d139f97e36b5fa80c3f33d02ff138ae9812f52 not found: ID does not exist"
Nov 02 19:30:10.101001 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.100979    4463 scope.go:122] "RemoveContainer" containerID="a19b6396e6bb883ee21fb5bb101ac44eef51a42f57640f232d81d99db225e4ff"
Nov 02 19:30:10.101227 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:30:10.101211    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a19b6396e6bb883ee21fb5bb101ac44eef51a42f57640f232d81d99db225e4ff\": container with ID starting with a19b6396e6bb883ee21fb5bb101ac44eef51a42f57640f232d81d99db225e4ff not found: ID does not exist" containerID="a19b6396e6bb883ee21fb5bb101ac44eef51a42f57640f232d81d99db225e4ff"
Nov 02 19:30:10.101280 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.101230    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a19b6396e6bb883ee21fb5bb101ac44eef51a42f57640f232d81d99db225e4ff"} err="failed to get container status \"a19b6396e6bb883ee21fb5bb101ac44eef51a42f57640f232d81d99db225e4ff\": rpc error: code = NotFound desc = could not find container \"a19b6396e6bb883ee21fb5bb101ac44eef51a42f57640f232d81d99db225e4ff\": container with ID starting with a19b6396e6bb883ee21fb5bb101ac44eef51a42f57640f232d81d99db225e4ff not found: ID does not exist"
Nov 02 19:30:10.101280 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.101241    4463 scope.go:122] "RemoveContainer" containerID="a44073d3e07dc0bb7b8a1ffe71825d6ab45ca0edd6b85d661c37d7faf1bfc6db"
Nov 02 19:30:10.101472 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:30:10.101458    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a44073d3e07dc0bb7b8a1ffe71825d6ab45ca0edd6b85d661c37d7faf1bfc6db\": container with ID starting with a44073d3e07dc0bb7b8a1ffe71825d6ab45ca0edd6b85d661c37d7faf1bfc6db not found: ID does not exist" containerID="a44073d3e07dc0bb7b8a1ffe71825d6ab45ca0edd6b85d661c37d7faf1bfc6db"
Nov 02 19:30:10.101512 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.101474    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a44073d3e07dc0bb7b8a1ffe71825d6ab45ca0edd6b85d661c37d7faf1bfc6db"} err="failed to get container status \"a44073d3e07dc0bb7b8a1ffe71825d6ab45ca0edd6b85d661c37d7faf1bfc6db\": rpc error: code = NotFound desc = could not find container \"a44073d3e07dc0bb7b8a1ffe71825d6ab45ca0edd6b85d661c37d7faf1bfc6db\": container with ID starting with a44073d3e07dc0bb7b8a1ffe71825d6ab45ca0edd6b85d661c37d7faf1bfc6db not found: ID does not exist"
Nov 02 19:30:10.101512 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.101485    4463 scope.go:122] "RemoveContainer" containerID="0f086ed8657f6bfba49edbd31efefccad0546d49366036a18306d4c9e7d447db"
Nov 02 19:30:10.101682 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:30:10.101666    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f086ed8657f6bfba49edbd31efefccad0546d49366036a18306d4c9e7d447db\": container with ID starting with 0f086ed8657f6bfba49edbd31efefccad0546d49366036a18306d4c9e7d447db not found: ID does not exist" containerID="0f086ed8657f6bfba49edbd31efefccad0546d49366036a18306d4c9e7d447db"
Nov 02 19:30:10.101725 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.101685    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f086ed8657f6bfba49edbd31efefccad0546d49366036a18306d4c9e7d447db"} err="failed to get container status \"0f086ed8657f6bfba49edbd31efefccad0546d49366036a18306d4c9e7d447db\": rpc error: code = NotFound desc = could not find container \"0f086ed8657f6bfba49edbd31efefccad0546d49366036a18306d4c9e7d447db\": container with ID starting with 0f086ed8657f6bfba49edbd31efefccad0546d49366036a18306d4c9e7d447db not found: ID does not exist"
Nov 02 19:30:10.101725 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.101696    4463 scope.go:122] "RemoveContainer" containerID="70c9116606bd5bc31b3c8d441aef43a47d50794c451250fe8bd072c4e6e6063a"
Nov 02 19:30:10.101895 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:30:10.101878    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70c9116606bd5bc31b3c8d441aef43a47d50794c451250fe8bd072c4e6e6063a\": container with ID starting with 70c9116606bd5bc31b3c8d441aef43a47d50794c451250fe8bd072c4e6e6063a not found: ID does not exist" containerID="70c9116606bd5bc31b3c8d441aef43a47d50794c451250fe8bd072c4e6e6063a"
Nov 02 19:30:10.101944 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.101898    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70c9116606bd5bc31b3c8d441aef43a47d50794c451250fe8bd072c4e6e6063a"} err="failed to get container status \"70c9116606bd5bc31b3c8d441aef43a47d50794c451250fe8bd072c4e6e6063a\": rpc error: code = NotFound desc = could not find container \"70c9116606bd5bc31b3c8d441aef43a47d50794c451250fe8bd072c4e6e6063a\": container with ID starting with 70c9116606bd5bc31b3c8d441aef43a47d50794c451250fe8bd072c4e6e6063a not found: ID does not exist"
Nov 02 19:30:10.101944 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.101909    4463 scope.go:122] "RemoveContainer" containerID="6a4f1f869b329698cf706634ed7e7e9346ba2ca7fb58a6ea4b7dcb2871d2b91b"
Nov 02 19:30:10.102403 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:30:10.102146    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a4f1f869b329698cf706634ed7e7e9346ba2ca7fb58a6ea4b7dcb2871d2b91b\": container with ID starting with 6a4f1f869b329698cf706634ed7e7e9346ba2ca7fb58a6ea4b7dcb2871d2b91b not found: ID does not exist" containerID="6a4f1f869b329698cf706634ed7e7e9346ba2ca7fb58a6ea4b7dcb2871d2b91b"
Nov 02 19:30:10.102403 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.102162    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a4f1f869b329698cf706634ed7e7e9346ba2ca7fb58a6ea4b7dcb2871d2b91b"} err="failed to get container status \"6a4f1f869b329698cf706634ed7e7e9346ba2ca7fb58a6ea4b7dcb2871d2b91b\": rpc error: code = NotFound desc = could not find container \"6a4f1f869b329698cf706634ed7e7e9346ba2ca7fb58a6ea4b7dcb2871d2b91b\": container with ID starting with 6a4f1f869b329698cf706634ed7e7e9346ba2ca7fb58a6ea4b7dcb2871d2b91b not found: ID does not exist"
Nov 02 19:30:10.102403 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.102173    4463 scope.go:122] "RemoveContainer" containerID="1191899c144edc026e3faa7a91bfa808848360317d529f243954165c9236c54a"
Nov 02 19:30:10.102403 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.102378    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1191899c144edc026e3faa7a91bfa808848360317d529f243954165c9236c54a"} err="failed to get container status \"1191899c144edc026e3faa7a91bfa808848360317d529f243954165c9236c54a\": rpc error: code = NotFound desc = could not find container \"1191899c144edc026e3faa7a91bfa808848360317d529f243954165c9236c54a\": container with ID starting with 1191899c144edc026e3faa7a91bfa808848360317d529f243954165c9236c54a not found: ID does not exist"
Nov 02 19:30:10.102403 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.102396    4463 scope.go:122] "RemoveContainer" containerID="e805a31089b420a0bb031d7bd8d139f97e36b5fa80c3f33d02ff138ae9812f52"
Nov 02 19:30:10.102638 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.102604    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e805a31089b420a0bb031d7bd8d139f97e36b5fa80c3f33d02ff138ae9812f52"} err="failed to get container status \"e805a31089b420a0bb031d7bd8d139f97e36b5fa80c3f33d02ff138ae9812f52\": rpc error: code = NotFound desc = could not find container \"e805a31089b420a0bb031d7bd8d139f97e36b5fa80c3f33d02ff138ae9812f52\": container with ID starting with e805a31089b420a0bb031d7bd8d139f97e36b5fa80c3f33d02ff138ae9812f52 not found: ID does not exist"
Nov 02 19:30:10.102638 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.102617    4463 scope.go:122] "RemoveContainer" containerID="a19b6396e6bb883ee21fb5bb101ac44eef51a42f57640f232d81d99db225e4ff"
Nov 02 19:30:10.102833 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.102815    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a19b6396e6bb883ee21fb5bb101ac44eef51a42f57640f232d81d99db225e4ff"} err="failed to get container status \"a19b6396e6bb883ee21fb5bb101ac44eef51a42f57640f232d81d99db225e4ff\": rpc error: code = NotFound desc = could not find container \"a19b6396e6bb883ee21fb5bb101ac44eef51a42f57640f232d81d99db225e4ff\": container with ID starting with a19b6396e6bb883ee21fb5bb101ac44eef51a42f57640f232d81d99db225e4ff not found: ID does not exist"
Nov 02 19:30:10.102875 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.102833    4463 scope.go:122] "RemoveContainer" containerID="a44073d3e07dc0bb7b8a1ffe71825d6ab45ca0edd6b85d661c37d7faf1bfc6db"
Nov 02 19:30:10.103049 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.103027    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a44073d3e07dc0bb7b8a1ffe71825d6ab45ca0edd6b85d661c37d7faf1bfc6db"} err="failed to get container status \"a44073d3e07dc0bb7b8a1ffe71825d6ab45ca0edd6b85d661c37d7faf1bfc6db\": rpc error: code = NotFound desc = could not find container \"a44073d3e07dc0bb7b8a1ffe71825d6ab45ca0edd6b85d661c37d7faf1bfc6db\": container with ID starting with a44073d3e07dc0bb7b8a1ffe71825d6ab45ca0edd6b85d661c37d7faf1bfc6db not found: ID does not exist"
Nov 02 19:30:10.103049 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.103043    4463 scope.go:122] "RemoveContainer" containerID="0f086ed8657f6bfba49edbd31efefccad0546d49366036a18306d4c9e7d447db"
Nov 02 19:30:10.103256 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.103242    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f086ed8657f6bfba49edbd31efefccad0546d49366036a18306d4c9e7d447db"} err="failed to get container status \"0f086ed8657f6bfba49edbd31efefccad0546d49366036a18306d4c9e7d447db\": rpc error: code = NotFound desc = could not find container \"0f086ed8657f6bfba49edbd31efefccad0546d49366036a18306d4c9e7d447db\": container with ID starting with 0f086ed8657f6bfba49edbd31efefccad0546d49366036a18306d4c9e7d447db not found: ID does not exist"
Nov 02 19:30:10.103256 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.103255    4463 scope.go:122] "RemoveContainer" containerID="70c9116606bd5bc31b3c8d441aef43a47d50794c451250fe8bd072c4e6e6063a"
Nov 02 19:30:10.103479 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.103459    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70c9116606bd5bc31b3c8d441aef43a47d50794c451250fe8bd072c4e6e6063a"} err="failed to get container status \"70c9116606bd5bc31b3c8d441aef43a47d50794c451250fe8bd072c4e6e6063a\": rpc error: code = NotFound desc = could not find container \"70c9116606bd5bc31b3c8d441aef43a47d50794c451250fe8bd072c4e6e6063a\": container with ID starting with 70c9116606bd5bc31b3c8d441aef43a47d50794c451250fe8bd072c4e6e6063a not found: ID does not exist"
Nov 02 19:30:10.103527 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.103479    4463 scope.go:122] "RemoveContainer" containerID="6a4f1f869b329698cf706634ed7e7e9346ba2ca7fb58a6ea4b7dcb2871d2b91b"
Nov 02 19:30:10.103690 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.103672    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a4f1f869b329698cf706634ed7e7e9346ba2ca7fb58a6ea4b7dcb2871d2b91b"} err="failed to get container status \"6a4f1f869b329698cf706634ed7e7e9346ba2ca7fb58a6ea4b7dcb2871d2b91b\": rpc error: code = NotFound desc = could not find container \"6a4f1f869b329698cf706634ed7e7e9346ba2ca7fb58a6ea4b7dcb2871d2b91b\": container with ID starting with 6a4f1f869b329698cf706634ed7e7e9346ba2ca7fb58a6ea4b7dcb2871d2b91b not found: ID does not exist"
Nov 02 19:30:10.103730 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.103689    4463 scope.go:122] "RemoveContainer" containerID="1191899c144edc026e3faa7a91bfa808848360317d529f243954165c9236c54a"
Nov 02 19:30:10.103881 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.103865    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1191899c144edc026e3faa7a91bfa808848360317d529f243954165c9236c54a"} err="failed to get container status \"1191899c144edc026e3faa7a91bfa808848360317d529f243954165c9236c54a\": rpc error: code = NotFound desc = could not find container \"1191899c144edc026e3faa7a91bfa808848360317d529f243954165c9236c54a\": container with ID starting with 1191899c144edc026e3faa7a91bfa808848360317d529f243954165c9236c54a not found: ID does not exist"
Nov 02 19:30:10.103924 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.103881    4463 scope.go:122] "RemoveContainer" containerID="e805a31089b420a0bb031d7bd8d139f97e36b5fa80c3f33d02ff138ae9812f52"
Nov 02 19:30:10.104190 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.104156    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e805a31089b420a0bb031d7bd8d139f97e36b5fa80c3f33d02ff138ae9812f52"} err="failed to get container status \"e805a31089b420a0bb031d7bd8d139f97e36b5fa80c3f33d02ff138ae9812f52\": rpc error: code = NotFound desc = could not find container \"e805a31089b420a0bb031d7bd8d139f97e36b5fa80c3f33d02ff138ae9812f52\": container with ID starting with e805a31089b420a0bb031d7bd8d139f97e36b5fa80c3f33d02ff138ae9812f52 not found: ID does not exist"
Nov 02 19:30:10.104238 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.104191    4463 scope.go:122] "RemoveContainer" containerID="a19b6396e6bb883ee21fb5bb101ac44eef51a42f57640f232d81d99db225e4ff"
Nov 02 19:30:10.104418 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.104402    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a19b6396e6bb883ee21fb5bb101ac44eef51a42f57640f232d81d99db225e4ff"} err="failed to get container status \"a19b6396e6bb883ee21fb5bb101ac44eef51a42f57640f232d81d99db225e4ff\": rpc error: code = NotFound desc = could not find container \"a19b6396e6bb883ee21fb5bb101ac44eef51a42f57640f232d81d99db225e4ff\": container with ID starting with a19b6396e6bb883ee21fb5bb101ac44eef51a42f57640f232d81d99db225e4ff not found: ID does not exist"
Nov 02 19:30:10.104458 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.104418    4463 scope.go:122] "RemoveContainer" containerID="a44073d3e07dc0bb7b8a1ffe71825d6ab45ca0edd6b85d661c37d7faf1bfc6db"
Nov 02 19:30:10.104613 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.104595    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a44073d3e07dc0bb7b8a1ffe71825d6ab45ca0edd6b85d661c37d7faf1bfc6db"} err="failed to get container status \"a44073d3e07dc0bb7b8a1ffe71825d6ab45ca0edd6b85d661c37d7faf1bfc6db\": rpc error: code = NotFound desc = could not find container \"a44073d3e07dc0bb7b8a1ffe71825d6ab45ca0edd6b85d661c37d7faf1bfc6db\": container with ID starting with a44073d3e07dc0bb7b8a1ffe71825d6ab45ca0edd6b85d661c37d7faf1bfc6db not found: ID does not exist"
Nov 02 19:30:10.104656 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.104630    4463 scope.go:122] "RemoveContainer" containerID="0f086ed8657f6bfba49edbd31efefccad0546d49366036a18306d4c9e7d447db"
Nov 02 19:30:10.104851 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.104833    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f086ed8657f6bfba49edbd31efefccad0546d49366036a18306d4c9e7d447db"} err="failed to get container status \"0f086ed8657f6bfba49edbd31efefccad0546d49366036a18306d4c9e7d447db\": rpc error: code = NotFound desc = could not find container \"0f086ed8657f6bfba49edbd31efefccad0546d49366036a18306d4c9e7d447db\": container with ID starting with 0f086ed8657f6bfba49edbd31efefccad0546d49366036a18306d4c9e7d447db not found: ID does not exist"
Nov 02 19:30:10.104913 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.104850    4463 scope.go:122] "RemoveContainer" containerID="70c9116606bd5bc31b3c8d441aef43a47d50794c451250fe8bd072c4e6e6063a"
Nov 02 19:30:10.105078 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.105061    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70c9116606bd5bc31b3c8d441aef43a47d50794c451250fe8bd072c4e6e6063a"} err="failed to get container status \"70c9116606bd5bc31b3c8d441aef43a47d50794c451250fe8bd072c4e6e6063a\": rpc error: code = NotFound desc = could not find container \"70c9116606bd5bc31b3c8d441aef43a47d50794c451250fe8bd072c4e6e6063a\": container with ID starting with 70c9116606bd5bc31b3c8d441aef43a47d50794c451250fe8bd072c4e6e6063a not found: ID does not exist"
Nov 02 19:30:10.105125 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.105078    4463 scope.go:122] "RemoveContainer" containerID="6a4f1f869b329698cf706634ed7e7e9346ba2ca7fb58a6ea4b7dcb2871d2b91b"
Nov 02 19:30:10.105324 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.105307    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a4f1f869b329698cf706634ed7e7e9346ba2ca7fb58a6ea4b7dcb2871d2b91b"} err="failed to get container status \"6a4f1f869b329698cf706634ed7e7e9346ba2ca7fb58a6ea4b7dcb2871d2b91b\": rpc error: code = NotFound desc = could not find container \"6a4f1f869b329698cf706634ed7e7e9346ba2ca7fb58a6ea4b7dcb2871d2b91b\": container with ID starting with 6a4f1f869b329698cf706634ed7e7e9346ba2ca7fb58a6ea4b7dcb2871d2b91b not found: ID does not exist"
Nov 02 19:30:10.105364 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.105324    4463 scope.go:122] "RemoveContainer" containerID="1191899c144edc026e3faa7a91bfa808848360317d529f243954165c9236c54a"
Nov 02 19:30:10.105564 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.105547    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1191899c144edc026e3faa7a91bfa808848360317d529f243954165c9236c54a"} err="failed to get container status \"1191899c144edc026e3faa7a91bfa808848360317d529f243954165c9236c54a\": rpc error: code = NotFound desc = could not find container \"1191899c144edc026e3faa7a91bfa808848360317d529f243954165c9236c54a\": container with ID starting with 1191899c144edc026e3faa7a91bfa808848360317d529f243954165c9236c54a not found: ID does not exist"
Nov 02 19:30:10.105611 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.105564    4463 scope.go:122] "RemoveContainer" containerID="e805a31089b420a0bb031d7bd8d139f97e36b5fa80c3f33d02ff138ae9812f52"
Nov 02 19:30:10.105783 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.105764    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e805a31089b420a0bb031d7bd8d139f97e36b5fa80c3f33d02ff138ae9812f52"} err="failed to get container status \"e805a31089b420a0bb031d7bd8d139f97e36b5fa80c3f33d02ff138ae9812f52\": rpc error: code = NotFound desc = could not find container \"e805a31089b420a0bb031d7bd8d139f97e36b5fa80c3f33d02ff138ae9812f52\": container with ID starting with e805a31089b420a0bb031d7bd8d139f97e36b5fa80c3f33d02ff138ae9812f52 not found: ID does not exist"
Nov 02 19:30:10.105783 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.105782    4463 scope.go:122] "RemoveContainer" containerID="a19b6396e6bb883ee21fb5bb101ac44eef51a42f57640f232d81d99db225e4ff"
Nov 02 19:30:10.106313 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.106296    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a19b6396e6bb883ee21fb5bb101ac44eef51a42f57640f232d81d99db225e4ff"} err="failed to get container status \"a19b6396e6bb883ee21fb5bb101ac44eef51a42f57640f232d81d99db225e4ff\": rpc error: code = NotFound desc = could not find container \"a19b6396e6bb883ee21fb5bb101ac44eef51a42f57640f232d81d99db225e4ff\": container with ID starting with a19b6396e6bb883ee21fb5bb101ac44eef51a42f57640f232d81d99db225e4ff not found: ID does not exist"
Nov 02 19:30:10.106313 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.106312    4463 scope.go:122] "RemoveContainer" containerID="a44073d3e07dc0bb7b8a1ffe71825d6ab45ca0edd6b85d661c37d7faf1bfc6db"
Nov 02 19:30:10.106513 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.106496    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a44073d3e07dc0bb7b8a1ffe71825d6ab45ca0edd6b85d661c37d7faf1bfc6db"} err="failed to get container status \"a44073d3e07dc0bb7b8a1ffe71825d6ab45ca0edd6b85d661c37d7faf1bfc6db\": rpc error: code = NotFound desc = could not find container \"a44073d3e07dc0bb7b8a1ffe71825d6ab45ca0edd6b85d661c37d7faf1bfc6db\": container with ID starting with a44073d3e07dc0bb7b8a1ffe71825d6ab45ca0edd6b85d661c37d7faf1bfc6db not found: ID does not exist"
Nov 02 19:30:10.106513 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.106511    4463 scope.go:122] "RemoveContainer" containerID="0f086ed8657f6bfba49edbd31efefccad0546d49366036a18306d4c9e7d447db"
Nov 02 19:30:10.106753 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.106715    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f086ed8657f6bfba49edbd31efefccad0546d49366036a18306d4c9e7d447db"} err="failed to get container status \"0f086ed8657f6bfba49edbd31efefccad0546d49366036a18306d4c9e7d447db\": rpc error: code = NotFound desc = could not find container \"0f086ed8657f6bfba49edbd31efefccad0546d49366036a18306d4c9e7d447db\": container with ID starting with 0f086ed8657f6bfba49edbd31efefccad0546d49366036a18306d4c9e7d447db not found: ID does not exist"
Nov 02 19:30:10.106811 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.106752    4463 scope.go:122] "RemoveContainer" containerID="70c9116606bd5bc31b3c8d441aef43a47d50794c451250fe8bd072c4e6e6063a"
Nov 02 19:30:10.106976 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.106958    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70c9116606bd5bc31b3c8d441aef43a47d50794c451250fe8bd072c4e6e6063a"} err="failed to get container status \"70c9116606bd5bc31b3c8d441aef43a47d50794c451250fe8bd072c4e6e6063a\": rpc error: code = NotFound desc = could not find container \"70c9116606bd5bc31b3c8d441aef43a47d50794c451250fe8bd072c4e6e6063a\": container with ID starting with 70c9116606bd5bc31b3c8d441aef43a47d50794c451250fe8bd072c4e6e6063a not found: ID does not exist"
Nov 02 19:30:10.107058 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.106977    4463 scope.go:122] "RemoveContainer" containerID="6a4f1f869b329698cf706634ed7e7e9346ba2ca7fb58a6ea4b7dcb2871d2b91b"
Nov 02 19:30:10.107204 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.107188    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a4f1f869b329698cf706634ed7e7e9346ba2ca7fb58a6ea4b7dcb2871d2b91b"} err="failed to get container status \"6a4f1f869b329698cf706634ed7e7e9346ba2ca7fb58a6ea4b7dcb2871d2b91b\": rpc error: code = NotFound desc = could not find container \"6a4f1f869b329698cf706634ed7e7e9346ba2ca7fb58a6ea4b7dcb2871d2b91b\": container with ID starting with 6a4f1f869b329698cf706634ed7e7e9346ba2ca7fb58a6ea4b7dcb2871d2b91b not found: ID does not exist"
Nov 02 19:30:10.107204 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.107202    4463 scope.go:122] "RemoveContainer" containerID="1191899c144edc026e3faa7a91bfa808848360317d529f243954165c9236c54a"
Nov 02 19:30:10.107403 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.107382    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1191899c144edc026e3faa7a91bfa808848360317d529f243954165c9236c54a"} err="failed to get container status \"1191899c144edc026e3faa7a91bfa808848360317d529f243954165c9236c54a\": rpc error: code = NotFound desc = could not find container \"1191899c144edc026e3faa7a91bfa808848360317d529f243954165c9236c54a\": container with ID starting with 1191899c144edc026e3faa7a91bfa808848360317d529f243954165c9236c54a not found: ID does not exist"
Nov 02 19:30:10.107455 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.107403    4463 scope.go:122] "RemoveContainer" containerID="e805a31089b420a0bb031d7bd8d139f97e36b5fa80c3f33d02ff138ae9812f52"
Nov 02 19:30:10.107603 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.107579    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e805a31089b420a0bb031d7bd8d139f97e36b5fa80c3f33d02ff138ae9812f52"} err="failed to get container status \"e805a31089b420a0bb031d7bd8d139f97e36b5fa80c3f33d02ff138ae9812f52\": rpc error: code = NotFound desc = could not find container \"e805a31089b420a0bb031d7bd8d139f97e36b5fa80c3f33d02ff138ae9812f52\": container with ID starting with e805a31089b420a0bb031d7bd8d139f97e36b5fa80c3f33d02ff138ae9812f52 not found: ID does not exist"
Nov 02 19:30:10.107662 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.107604    4463 scope.go:122] "RemoveContainer" containerID="a19b6396e6bb883ee21fb5bb101ac44eef51a42f57640f232d81d99db225e4ff"
Nov 02 19:30:10.107809 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.107788    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a19b6396e6bb883ee21fb5bb101ac44eef51a42f57640f232d81d99db225e4ff"} err="failed to get container status \"a19b6396e6bb883ee21fb5bb101ac44eef51a42f57640f232d81d99db225e4ff\": rpc error: code = NotFound desc = could not find container \"a19b6396e6bb883ee21fb5bb101ac44eef51a42f57640f232d81d99db225e4ff\": container with ID starting with a19b6396e6bb883ee21fb5bb101ac44eef51a42f57640f232d81d99db225e4ff not found: ID does not exist"
Nov 02 19:30:10.107863 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.107809    4463 scope.go:122] "RemoveContainer" containerID="a44073d3e07dc0bb7b8a1ffe71825d6ab45ca0edd6b85d661c37d7faf1bfc6db"
Nov 02 19:30:10.108026 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.107993    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a44073d3e07dc0bb7b8a1ffe71825d6ab45ca0edd6b85d661c37d7faf1bfc6db"} err="failed to get container status \"a44073d3e07dc0bb7b8a1ffe71825d6ab45ca0edd6b85d661c37d7faf1bfc6db\": rpc error: code = NotFound desc = could not find container \"a44073d3e07dc0bb7b8a1ffe71825d6ab45ca0edd6b85d661c37d7faf1bfc6db\": container with ID starting with a44073d3e07dc0bb7b8a1ffe71825d6ab45ca0edd6b85d661c37d7faf1bfc6db not found: ID does not exist"
Nov 02 19:30:10.108070 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.108007    4463 scope.go:122] "RemoveContainer" containerID="0f086ed8657f6bfba49edbd31efefccad0546d49366036a18306d4c9e7d447db"
Nov 02 19:30:10.108243 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.108224    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f086ed8657f6bfba49edbd31efefccad0546d49366036a18306d4c9e7d447db"} err="failed to get container status \"0f086ed8657f6bfba49edbd31efefccad0546d49366036a18306d4c9e7d447db\": rpc error: code = NotFound desc = could not find container \"0f086ed8657f6bfba49edbd31efefccad0546d49366036a18306d4c9e7d447db\": container with ID starting with 0f086ed8657f6bfba49edbd31efefccad0546d49366036a18306d4c9e7d447db not found: ID does not exist"
Nov 02 19:30:10.108280 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.108242    4463 scope.go:122] "RemoveContainer" containerID="70c9116606bd5bc31b3c8d441aef43a47d50794c451250fe8bd072c4e6e6063a"
Nov 02 19:30:10.108445 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.108428    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70c9116606bd5bc31b3c8d441aef43a47d50794c451250fe8bd072c4e6e6063a"} err="failed to get container status \"70c9116606bd5bc31b3c8d441aef43a47d50794c451250fe8bd072c4e6e6063a\": rpc error: code = NotFound desc = could not find container \"70c9116606bd5bc31b3c8d441aef43a47d50794c451250fe8bd072c4e6e6063a\": container with ID starting with 70c9116606bd5bc31b3c8d441aef43a47d50794c451250fe8bd072c4e6e6063a not found: ID does not exist"
Nov 02 19:30:10.108486 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.108446    4463 scope.go:122] "RemoveContainer" containerID="6a4f1f869b329698cf706634ed7e7e9346ba2ca7fb58a6ea4b7dcb2871d2b91b"
Nov 02 19:30:10.108654 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.108635    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a4f1f869b329698cf706634ed7e7e9346ba2ca7fb58a6ea4b7dcb2871d2b91b"} err="failed to get container status \"6a4f1f869b329698cf706634ed7e7e9346ba2ca7fb58a6ea4b7dcb2871d2b91b\": rpc error: code = NotFound desc = could not find container \"6a4f1f869b329698cf706634ed7e7e9346ba2ca7fb58a6ea4b7dcb2871d2b91b\": container with ID starting with 6a4f1f869b329698cf706634ed7e7e9346ba2ca7fb58a6ea4b7dcb2871d2b91b not found: ID does not exist"
Nov 02 19:30:10.108694 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.108654    4463 scope.go:122] "RemoveContainer" containerID="1191899c144edc026e3faa7a91bfa808848360317d529f243954165c9236c54a"
Nov 02 19:30:10.108872 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.108847    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1191899c144edc026e3faa7a91bfa808848360317d529f243954165c9236c54a"} err="failed to get container status \"1191899c144edc026e3faa7a91bfa808848360317d529f243954165c9236c54a\": rpc error: code = NotFound desc = could not find container \"1191899c144edc026e3faa7a91bfa808848360317d529f243954165c9236c54a\": container with ID starting with 1191899c144edc026e3faa7a91bfa808848360317d529f243954165c9236c54a not found: ID does not exist"
Nov 02 19:30:10.108908 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.108872    4463 scope.go:122] "RemoveContainer" containerID="e805a31089b420a0bb031d7bd8d139f97e36b5fa80c3f33d02ff138ae9812f52"
Nov 02 19:30:10.109090 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.109073    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e805a31089b420a0bb031d7bd8d139f97e36b5fa80c3f33d02ff138ae9812f52"} err="failed to get container status \"e805a31089b420a0bb031d7bd8d139f97e36b5fa80c3f33d02ff138ae9812f52\": rpc error: code = NotFound desc = could not find container \"e805a31089b420a0bb031d7bd8d139f97e36b5fa80c3f33d02ff138ae9812f52\": container with ID starting with e805a31089b420a0bb031d7bd8d139f97e36b5fa80c3f33d02ff138ae9812f52 not found: ID does not exist"
Nov 02 19:30:10.109140 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.109091    4463 scope.go:122] "RemoveContainer" containerID="a19b6396e6bb883ee21fb5bb101ac44eef51a42f57640f232d81d99db225e4ff"
Nov 02 19:30:10.109300 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.109283    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a19b6396e6bb883ee21fb5bb101ac44eef51a42f57640f232d81d99db225e4ff"} err="failed to get container status \"a19b6396e6bb883ee21fb5bb101ac44eef51a42f57640f232d81d99db225e4ff\": rpc error: code = NotFound desc = could not find container \"a19b6396e6bb883ee21fb5bb101ac44eef51a42f57640f232d81d99db225e4ff\": container with ID starting with a19b6396e6bb883ee21fb5bb101ac44eef51a42f57640f232d81d99db225e4ff not found: ID does not exist"
Nov 02 19:30:10.109300 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.109298    4463 scope.go:122] "RemoveContainer" containerID="a44073d3e07dc0bb7b8a1ffe71825d6ab45ca0edd6b85d661c37d7faf1bfc6db"
Nov 02 19:30:10.109499 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.109479    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a44073d3e07dc0bb7b8a1ffe71825d6ab45ca0edd6b85d661c37d7faf1bfc6db"} err="failed to get container status \"a44073d3e07dc0bb7b8a1ffe71825d6ab45ca0edd6b85d661c37d7faf1bfc6db\": rpc error: code = NotFound desc = could not find container \"a44073d3e07dc0bb7b8a1ffe71825d6ab45ca0edd6b85d661c37d7faf1bfc6db\": container with ID starting with a44073d3e07dc0bb7b8a1ffe71825d6ab45ca0edd6b85d661c37d7faf1bfc6db not found: ID does not exist"
Nov 02 19:30:10.109499 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.109499    4463 scope.go:122] "RemoveContainer" containerID="0f086ed8657f6bfba49edbd31efefccad0546d49366036a18306d4c9e7d447db"
Nov 02 19:30:10.109686 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.109667    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f086ed8657f6bfba49edbd31efefccad0546d49366036a18306d4c9e7d447db"} err="failed to get container status \"0f086ed8657f6bfba49edbd31efefccad0546d49366036a18306d4c9e7d447db\": rpc error: code = NotFound desc = could not find container \"0f086ed8657f6bfba49edbd31efefccad0546d49366036a18306d4c9e7d447db\": container with ID starting with 0f086ed8657f6bfba49edbd31efefccad0546d49366036a18306d4c9e7d447db not found: ID does not exist"
Nov 02 19:30:10.109748 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.109685    4463 scope.go:122] "RemoveContainer" containerID="70c9116606bd5bc31b3c8d441aef43a47d50794c451250fe8bd072c4e6e6063a"
Nov 02 19:30:10.109886 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.109866    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70c9116606bd5bc31b3c8d441aef43a47d50794c451250fe8bd072c4e6e6063a"} err="failed to get container status \"70c9116606bd5bc31b3c8d441aef43a47d50794c451250fe8bd072c4e6e6063a\": rpc error: code = NotFound desc = could not find container \"70c9116606bd5bc31b3c8d441aef43a47d50794c451250fe8bd072c4e6e6063a\": container with ID starting with 70c9116606bd5bc31b3c8d441aef43a47d50794c451250fe8bd072c4e6e6063a not found: ID does not exist"
Nov 02 19:30:10.109936 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.109887    4463 scope.go:122] "RemoveContainer" containerID="6a4f1f869b329698cf706634ed7e7e9346ba2ca7fb58a6ea4b7dcb2871d2b91b"
Nov 02 19:30:10.110100 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.110081    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a4f1f869b329698cf706634ed7e7e9346ba2ca7fb58a6ea4b7dcb2871d2b91b"} err="failed to get container status \"6a4f1f869b329698cf706634ed7e7e9346ba2ca7fb58a6ea4b7dcb2871d2b91b\": rpc error: code = NotFound desc = could not find container \"6a4f1f869b329698cf706634ed7e7e9346ba2ca7fb58a6ea4b7dcb2871d2b91b\": container with ID starting with 6a4f1f869b329698cf706634ed7e7e9346ba2ca7fb58a6ea4b7dcb2871d2b91b not found: ID does not exist"
Nov 02 19:30:10.202500 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.202476    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 02 19:30:10.248709 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.248691    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 02 19:30:10.249368 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.249354    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 02 19:30:10.249653 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.249639    4463 cpu_manager.go:414] "containerMap: removing container" podUID="3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6" containerName="kube-rbac-proxy"
Nov 02 19:30:10.249700 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.249656    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6" containerName="kube-rbac-proxy"
Nov 02 19:30:10.249700 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.249687    4463 cpu_manager.go:414] "containerMap: removing container" podUID="4874b345-eac7-4694-8351-d24085bf180a" containerName="insights-gathering"
Nov 02 19:30:10.249700 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.249693    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4874b345-eac7-4694-8351-d24085bf180a" containerName="insights-gathering"
Nov 02 19:30:10.249793 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.249701    4463 cpu_manager.go:414] "containerMap: removing container" podUID="3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6" containerName="config-reloader"
Nov 02 19:30:10.249793 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.249707    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6" containerName="config-reloader"
Nov 02 19:30:10.249793 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.249713    4463 cpu_manager.go:414] "containerMap: removing container" podUID="3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6" containerName="kube-rbac-proxy-thanos"
Nov 02 19:30:10.249793 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.249718    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6" containerName="kube-rbac-proxy-thanos"
Nov 02 19:30:10.249793 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.249728    4463 cpu_manager.go:414] "containerMap: removing container" podUID="3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6" containerName="thanos-sidecar"
Nov 02 19:30:10.249793 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.249736    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6" containerName="thanos-sidecar"
Nov 02 19:30:10.249793 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.249751    4463 cpu_manager.go:414] "containerMap: removing container" podUID="3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6" containerName="prometheus"
Nov 02 19:30:10.249793 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.249758    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6" containerName="prometheus"
Nov 02 19:30:10.249793 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.249771    4463 cpu_manager.go:414] "containerMap: removing container" podUID="3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6" containerName="init-config-reloader"
Nov 02 19:30:10.249793 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.249777    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6" containerName="init-config-reloader"
Nov 02 19:30:10.249793 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.249784    4463 cpu_manager.go:414] "containerMap: removing container" podUID="3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6" containerName="kube-rbac-proxy-web"
Nov 02 19:30:10.249793 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.249789    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6" containerName="kube-rbac-proxy-web"
Nov 02 19:30:10.250182 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.249840    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="4874b345-eac7-4694-8351-d24085bf180a" containerName="insights-gathering"
Nov 02 19:30:10.250182 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.249852    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6" containerName="config-reloader"
Nov 02 19:30:10.250182 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.249860    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6" containerName="kube-rbac-proxy-thanos"
Nov 02 19:30:10.250182 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.249868    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6" containerName="thanos-sidecar"
Nov 02 19:30:10.250182 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.249876    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6" containerName="prometheus"
Nov 02 19:30:10.250182 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.249882    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6" containerName="kube-rbac-proxy-web"
Nov 02 19:30:10.250182 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.249888    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6" containerName="kube-rbac-proxy"
Nov 02 19:30:10.252283 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.252266    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:30:10.304856 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.304729    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Nov 02 19:30:10.304856 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.304727    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Nov 02 19:30:10.304856 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.304755    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Nov 02 19:30:10.304856 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.304803    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-t4rdb\""
Nov 02 19:30:10.304856 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.304814    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-9cdm8fh30lr9b\""
Nov 02 19:30:10.304856 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.304810    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Nov 02 19:30:10.304856 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.304835    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Nov 02 19:30:10.305218 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.304812    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Nov 02 19:30:10.305218 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.304996    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Nov 02 19:30:10.305218 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.305002    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Nov 02 19:30:10.305218 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.305053    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Nov 02 19:30:10.305218 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.305062    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Nov 02 19:30:10.305218 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.305071    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Nov 02 19:30:10.305218 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.305099    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Nov 02 19:30:10.351900 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.351874    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Nov 02 19:30:10.351900 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.351881    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Nov 02 19:30:10.352061 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.351994    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Nov 02 19:30:10.359746 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.359727    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d91b3696-0787-4913-9bbc-4db403c0139b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d91b3696-0787-4913-9bbc-4db403c0139b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:30:10.359803 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.359756    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d91b3696-0787-4913-9bbc-4db403c0139b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d91b3696-0787-4913-9bbc-4db403c0139b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:30:10.359803 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.359774    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d91b3696-0787-4913-9bbc-4db403c0139b-config-out\") pod \"prometheus-k8s-0\" (UID: \"d91b3696-0787-4913-9bbc-4db403c0139b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:30:10.359803 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.359795    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d91b3696-0787-4913-9bbc-4db403c0139b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d91b3696-0787-4913-9bbc-4db403c0139b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:30:10.359894 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.359834    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d91b3696-0787-4913-9bbc-4db403c0139b-config\") pod \"prometheus-k8s-0\" (UID: \"d91b3696-0787-4913-9bbc-4db403c0139b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:30:10.359935 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.359922    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d91b3696-0787-4913-9bbc-4db403c0139b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d91b3696-0787-4913-9bbc-4db403c0139b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:30:10.359969 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.359949    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d91b3696-0787-4913-9bbc-4db403c0139b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d91b3696-0787-4913-9bbc-4db403c0139b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:30:10.360000 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.359966    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d91b3696-0787-4913-9bbc-4db403c0139b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d91b3696-0787-4913-9bbc-4db403c0139b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:30:10.360000 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.359982    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6bzl\" (UniqueName: \"kubernetes.io/projected/d91b3696-0787-4913-9bbc-4db403c0139b-kube-api-access-l6bzl\") pod \"prometheus-k8s-0\" (UID: \"d91b3696-0787-4913-9bbc-4db403c0139b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:30:10.360072 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.360041    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d91b3696-0787-4913-9bbc-4db403c0139b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d91b3696-0787-4913-9bbc-4db403c0139b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:30:10.360127 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.360114    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d91b3696-0787-4913-9bbc-4db403c0139b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d91b3696-0787-4913-9bbc-4db403c0139b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:30:10.360159 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.360136    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d91b3696-0787-4913-9bbc-4db403c0139b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d91b3696-0787-4913-9bbc-4db403c0139b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:30:10.360159 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.360152    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d91b3696-0787-4913-9bbc-4db403c0139b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d91b3696-0787-4913-9bbc-4db403c0139b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:30:10.360221 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.360174    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d91b3696-0787-4913-9bbc-4db403c0139b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d91b3696-0787-4913-9bbc-4db403c0139b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:30:10.360253 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.360230    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d91b3696-0787-4913-9bbc-4db403c0139b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d91b3696-0787-4913-9bbc-4db403c0139b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:30:10.360284 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.360273    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d91b3696-0787-4913-9bbc-4db403c0139b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d91b3696-0787-4913-9bbc-4db403c0139b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:30:10.360316 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.360296    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d91b3696-0787-4913-9bbc-4db403c0139b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d91b3696-0787-4913-9bbc-4db403c0139b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:30:10.360347 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.360334    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d91b3696-0787-4913-9bbc-4db403c0139b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d91b3696-0787-4913-9bbc-4db403c0139b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:30:10.360379 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.360354    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d91b3696-0787-4913-9bbc-4db403c0139b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d91b3696-0787-4913-9bbc-4db403c0139b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:30:10.360407 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.360382    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d91b3696-0787-4913-9bbc-4db403c0139b-web-config\") pod \"prometheus-k8s-0\" (UID: \"d91b3696-0787-4913-9bbc-4db403c0139b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:30:10.461227 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.461204    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d91b3696-0787-4913-9bbc-4db403c0139b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d91b3696-0787-4913-9bbc-4db403c0139b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:30:10.461310 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.461232    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d91b3696-0787-4913-9bbc-4db403c0139b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d91b3696-0787-4913-9bbc-4db403c0139b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:30:10.461310 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.461252    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d91b3696-0787-4913-9bbc-4db403c0139b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d91b3696-0787-4913-9bbc-4db403c0139b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:30:10.461310 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.461267    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d91b3696-0787-4913-9bbc-4db403c0139b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d91b3696-0787-4913-9bbc-4db403c0139b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:30:10.461310 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.461290    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d91b3696-0787-4913-9bbc-4db403c0139b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d91b3696-0787-4913-9bbc-4db403c0139b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:30:10.461485 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.461316    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d91b3696-0787-4913-9bbc-4db403c0139b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d91b3696-0787-4913-9bbc-4db403c0139b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:30:10.461485 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.461343    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d91b3696-0787-4913-9bbc-4db403c0139b-web-config\") pod \"prometheus-k8s-0\" (UID: \"d91b3696-0787-4913-9bbc-4db403c0139b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:30:10.461485 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.461381    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d91b3696-0787-4913-9bbc-4db403c0139b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d91b3696-0787-4913-9bbc-4db403c0139b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:30:10.461485 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.461412    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d91b3696-0787-4913-9bbc-4db403c0139b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d91b3696-0787-4913-9bbc-4db403c0139b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:30:10.462076 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.462050    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d91b3696-0787-4913-9bbc-4db403c0139b-config-out\") pod \"prometheus-k8s-0\" (UID: \"d91b3696-0787-4913-9bbc-4db403c0139b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:30:10.462159 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.462100    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d91b3696-0787-4913-9bbc-4db403c0139b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d91b3696-0787-4913-9bbc-4db403c0139b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:30:10.462159 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.462129    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d91b3696-0787-4913-9bbc-4db403c0139b-config\") pod \"prometheus-k8s-0\" (UID: \"d91b3696-0787-4913-9bbc-4db403c0139b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:30:10.462272 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.462166    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d91b3696-0787-4913-9bbc-4db403c0139b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d91b3696-0787-4913-9bbc-4db403c0139b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:30:10.462272 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.462192    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d91b3696-0787-4913-9bbc-4db403c0139b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d91b3696-0787-4913-9bbc-4db403c0139b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:30:10.462272 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.462215    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d91b3696-0787-4913-9bbc-4db403c0139b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d91b3696-0787-4913-9bbc-4db403c0139b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:30:10.462272 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.462239    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6bzl\" (UniqueName: \"kubernetes.io/projected/d91b3696-0787-4913-9bbc-4db403c0139b-kube-api-access-l6bzl\") pod \"prometheus-k8s-0\" (UID: \"d91b3696-0787-4913-9bbc-4db403c0139b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:30:10.462272 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.462266    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d91b3696-0787-4913-9bbc-4db403c0139b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d91b3696-0787-4913-9bbc-4db403c0139b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:30:10.462533 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.462301    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d91b3696-0787-4913-9bbc-4db403c0139b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d91b3696-0787-4913-9bbc-4db403c0139b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:30:10.462533 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.462332    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d91b3696-0787-4913-9bbc-4db403c0139b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d91b3696-0787-4913-9bbc-4db403c0139b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:30:10.462533 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.462355    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d91b3696-0787-4913-9bbc-4db403c0139b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d91b3696-0787-4913-9bbc-4db403c0139b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:30:10.462684 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.462578    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d91b3696-0787-4913-9bbc-4db403c0139b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d91b3696-0787-4913-9bbc-4db403c0139b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:30:10.463006 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.462981    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d91b3696-0787-4913-9bbc-4db403c0139b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d91b3696-0787-4913-9bbc-4db403c0139b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:30:10.463440 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.463414    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d91b3696-0787-4913-9bbc-4db403c0139b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d91b3696-0787-4913-9bbc-4db403c0139b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:30:10.463625 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.463600    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d91b3696-0787-4913-9bbc-4db403c0139b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d91b3696-0787-4913-9bbc-4db403c0139b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:30:10.463696 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.462060    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d91b3696-0787-4913-9bbc-4db403c0139b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d91b3696-0787-4913-9bbc-4db403c0139b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:30:10.465434 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.463902    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d91b3696-0787-4913-9bbc-4db403c0139b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d91b3696-0787-4913-9bbc-4db403c0139b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:30:10.465434 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.464363    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d91b3696-0787-4913-9bbc-4db403c0139b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d91b3696-0787-4913-9bbc-4db403c0139b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:30:10.465434 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.464673    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d91b3696-0787-4913-9bbc-4db403c0139b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d91b3696-0787-4913-9bbc-4db403c0139b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:30:10.465434 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.464739    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d91b3696-0787-4913-9bbc-4db403c0139b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d91b3696-0787-4913-9bbc-4db403c0139b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:30:10.465434 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.465134    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d91b3696-0787-4913-9bbc-4db403c0139b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d91b3696-0787-4913-9bbc-4db403c0139b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:30:10.465434 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.465420    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d91b3696-0787-4913-9bbc-4db403c0139b-web-config\") pod \"prometheus-k8s-0\" (UID: \"d91b3696-0787-4913-9bbc-4db403c0139b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:30:10.465792 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.465652    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d91b3696-0787-4913-9bbc-4db403c0139b-config-out\") pod \"prometheus-k8s-0\" (UID: \"d91b3696-0787-4913-9bbc-4db403c0139b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:30:10.466035 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.465965    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d91b3696-0787-4913-9bbc-4db403c0139b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d91b3696-0787-4913-9bbc-4db403c0139b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:30:10.466104 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.466061    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d91b3696-0787-4913-9bbc-4db403c0139b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d91b3696-0787-4913-9bbc-4db403c0139b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:30:10.466320 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.466302    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/d91b3696-0787-4913-9bbc-4db403c0139b-config\") pod \"prometheus-k8s-0\" (UID: \"d91b3696-0787-4913-9bbc-4db403c0139b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:30:10.467145 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.467120    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d91b3696-0787-4913-9bbc-4db403c0139b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d91b3696-0787-4913-9bbc-4db403c0139b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:30:10.467305 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.467289    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d91b3696-0787-4913-9bbc-4db403c0139b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d91b3696-0787-4913-9bbc-4db403c0139b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:30:10.467548 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.467528    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d91b3696-0787-4913-9bbc-4db403c0139b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d91b3696-0787-4913-9bbc-4db403c0139b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:30:10.467812 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.467795    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d91b3696-0787-4913-9bbc-4db403c0139b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d91b3696-0787-4913-9bbc-4db403c0139b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:30:10.470843 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.470822    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 02 19:30:10.517840 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.517815    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6bzl\" (UniqueName: \"kubernetes.io/projected/d91b3696-0787-4913-9bbc-4db403c0139b-kube-api-access-l6bzl\") pod \"prometheus-k8s-0\" (UID: \"d91b3696-0787-4913-9bbc-4db403c0139b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:30:10.568481 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.568431    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:30:10.980326 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:10.980226    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 02 19:30:10.984811 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:30:10.984784    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd91b3696_0787_4913_9bbc_4db403c0139b.slice/crio-96aec91f39b7726f15f62296760cf67080986d91262eeeea0f3092df6692fe7d WatchSource:0}: Error finding container 96aec91f39b7726f15f62296760cf67080986d91262eeeea0f3092df6692fe7d: Status 404 returned error can't find the container with id 96aec91f39b7726f15f62296760cf67080986d91262eeeea0f3092df6692fe7d
Nov 02 19:30:11.049681 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:11.049653    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d91b3696-0787-4913-9bbc-4db403c0139b","Type":"ContainerStarted","Data":"49151e84073e8acef2d3b1bcee9ea4de45e38b148f810c755468a5dca08d489a"}
Nov 02 19:30:11.049783 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:11.049730    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d91b3696-0787-4913-9bbc-4db403c0139b","Type":"ContainerStarted","Data":"96aec91f39b7726f15f62296760cf67080986d91262eeeea0f3092df6692fe7d"}
Nov 02 19:30:11.580885 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:11.580852    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6" path="/var/lib/kubelet/pods/3f9e31fc-0bbd-42de-8afb-1d62bb4b3ef6/volumes"
Nov 02 19:30:12.057602 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:12.057564    4463 generic.go:356] "Generic (PLEG): container finished" podID="d91b3696-0787-4913-9bbc-4db403c0139b" containerID="49151e84073e8acef2d3b1bcee9ea4de45e38b148f810c755468a5dca08d489a" exitCode=0
Nov 02 19:30:12.057891 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:12.057617    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d91b3696-0787-4913-9bbc-4db403c0139b","Type":"ContainerDied","Data":"49151e84073e8acef2d3b1bcee9ea4de45e38b148f810c755468a5dca08d489a"}
Nov 02 19:30:13.064781 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:13.064742    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d91b3696-0787-4913-9bbc-4db403c0139b","Type":"ContainerStarted","Data":"37c2c9fce62b938e03c92c2442758b31fdf35393cb5a15e014cac262f23e6a36"}
Nov 02 19:30:13.065189 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:13.064801    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d91b3696-0787-4913-9bbc-4db403c0139b","Type":"ContainerStarted","Data":"9f3c8d7c0e3f4e6c23d9eb27369c728b81aa3eb4cc6bd4f7c13c672dec09aa4c"}
Nov 02 19:30:13.065189 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:13.064826    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d91b3696-0787-4913-9bbc-4db403c0139b","Type":"ContainerStarted","Data":"7adf49ee2616d0c9012e418ed7807120bdaa8d76efb308e0dc5f35d3d79a4084"}
Nov 02 19:30:13.065189 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:13.064852    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d91b3696-0787-4913-9bbc-4db403c0139b","Type":"ContainerStarted","Data":"68f1f0f84ff5eec765f49c976cb81c961bf952e88da12e4626fc19ed9f3ad817"}
Nov 02 19:30:13.065189 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:13.064874    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d91b3696-0787-4913-9bbc-4db403c0139b","Type":"ContainerStarted","Data":"1661a35edc6d5b7e5ee3760b3e6d7c40249ddedad686793d168a863574e66699"}
Nov 02 19:30:13.065189 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:13.064896    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d91b3696-0787-4913-9bbc-4db403c0139b","Type":"ContainerStarted","Data":"82bad9cceeb3d38fd7edecc2b6032658ec30e46d611808967f4a7fe0bfef8833"}
Nov 02 19:30:13.182973 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:13.182925    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.166668363 podStartE2EDuration="3.18291317s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.244807ms" isStatefulPod=false podCreationTimestamp="2026-11-02 19:30:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:30:13.180473942 +0000 UTC m=+119.787507711" watchObservedRunningTime="2026-11-02 19:30:13.18291317 +0000 UTC m=+119.789946939"
Nov 02 19:30:15.568851 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:30:15.568802    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:30:34.292930 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:30:34.292850    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 19:31:04.292543 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:31:04.292505    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 19:31:10.569503 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:31:10.569467    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:31:10.584410 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:31:10.584386    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:31:11.258342 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:31:11.258307    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:31:34.292052 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:31:34.291998    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 19:32:04.292580 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:32:04.292546    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 19:32:26.628165 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:32:26.628128    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5985f94fb7-7ct2h"]
Nov 02 19:32:26.630246 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:32:26.630229    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5985f94fb7-7ct2h"
Nov 02 19:32:26.679076 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:32:26.679057    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 02 19:32:26.679191 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:32:26.679174    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 02 19:32:26.679233 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:32:26.679211    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 02 19:32:26.679318 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:32:26.679305    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 02 19:32:26.679415 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:32:26.679397    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-lqjs8\""
Nov 02 19:32:26.679576 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:32:26.679564    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 02 19:32:26.696536 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:32:26.696512    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a3017984-7d80-4b3f-8072-2ed75d87cf96-metrics-cert\") pod \"opendatahub-operator-controller-manager-5985f94fb7-7ct2h\" (UID: \"a3017984-7d80-4b3f-8072-2ed75d87cf96\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5985f94fb7-7ct2h"
Nov 02 19:32:26.696628 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:32:26.696551    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppnxk\" (UniqueName: \"kubernetes.io/projected/a3017984-7d80-4b3f-8072-2ed75d87cf96-kube-api-access-ppnxk\") pod \"opendatahub-operator-controller-manager-5985f94fb7-7ct2h\" (UID: \"a3017984-7d80-4b3f-8072-2ed75d87cf96\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5985f94fb7-7ct2h"
Nov 02 19:32:26.696628 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:32:26.696571    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a3017984-7d80-4b3f-8072-2ed75d87cf96-manifests\") pod \"opendatahub-operator-controller-manager-5985f94fb7-7ct2h\" (UID: \"a3017984-7d80-4b3f-8072-2ed75d87cf96\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5985f94fb7-7ct2h"
Nov 02 19:32:26.696628 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:32:26.696588    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a3017984-7d80-4b3f-8072-2ed75d87cf96-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5985f94fb7-7ct2h\" (UID: \"a3017984-7d80-4b3f-8072-2ed75d87cf96\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5985f94fb7-7ct2h"
Nov 02 19:32:26.696756 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:32:26.696740    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a3017984-7d80-4b3f-8072-2ed75d87cf96-tmp\") pod \"opendatahub-operator-controller-manager-5985f94fb7-7ct2h\" (UID: \"a3017984-7d80-4b3f-8072-2ed75d87cf96\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5985f94fb7-7ct2h"
Nov 02 19:32:26.696790 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:32:26.696776    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a3017984-7d80-4b3f-8072-2ed75d87cf96-webhook-cert\") pod \"opendatahub-operator-controller-manager-5985f94fb7-7ct2h\" (UID: \"a3017984-7d80-4b3f-8072-2ed75d87cf96\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5985f94fb7-7ct2h"
Nov 02 19:32:26.771846 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:32:26.771825    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5985f94fb7-7ct2h"]
Nov 02 19:32:26.797894 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:32:26.797873    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a3017984-7d80-4b3f-8072-2ed75d87cf96-manifests\") pod \"opendatahub-operator-controller-manager-5985f94fb7-7ct2h\" (UID: \"a3017984-7d80-4b3f-8072-2ed75d87cf96\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5985f94fb7-7ct2h"
Nov 02 19:32:26.798037 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:32:26.797905    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a3017984-7d80-4b3f-8072-2ed75d87cf96-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5985f94fb7-7ct2h\" (UID: \"a3017984-7d80-4b3f-8072-2ed75d87cf96\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5985f94fb7-7ct2h"
Nov 02 19:32:26.798037 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:32:26.797966    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a3017984-7d80-4b3f-8072-2ed75d87cf96-tmp\") pod \"opendatahub-operator-controller-manager-5985f94fb7-7ct2h\" (UID: \"a3017984-7d80-4b3f-8072-2ed75d87cf96\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5985f94fb7-7ct2h"
Nov 02 19:32:26.798037 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:32:26.798009    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a3017984-7d80-4b3f-8072-2ed75d87cf96-webhook-cert\") pod \"opendatahub-operator-controller-manager-5985f94fb7-7ct2h\" (UID: \"a3017984-7d80-4b3f-8072-2ed75d87cf96\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5985f94fb7-7ct2h"
Nov 02 19:32:26.798214 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:32:26.798067    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a3017984-7d80-4b3f-8072-2ed75d87cf96-metrics-cert\") pod \"opendatahub-operator-controller-manager-5985f94fb7-7ct2h\" (UID: \"a3017984-7d80-4b3f-8072-2ed75d87cf96\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5985f94fb7-7ct2h"
Nov 02 19:32:26.798214 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:32:26.798107    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppnxk\" (UniqueName: \"kubernetes.io/projected/a3017984-7d80-4b3f-8072-2ed75d87cf96-kube-api-access-ppnxk\") pod \"opendatahub-operator-controller-manager-5985f94fb7-7ct2h\" (UID: \"a3017984-7d80-4b3f-8072-2ed75d87cf96\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5985f94fb7-7ct2h"
Nov 02 19:32:26.798378 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:32:26.798355    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a3017984-7d80-4b3f-8072-2ed75d87cf96-manifests\") pod \"opendatahub-operator-controller-manager-5985f94fb7-7ct2h\" (UID: \"a3017984-7d80-4b3f-8072-2ed75d87cf96\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5985f94fb7-7ct2h"
Nov 02 19:32:26.800518 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:32:26.800496    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a3017984-7d80-4b3f-8072-2ed75d87cf96-tmp\") pod \"opendatahub-operator-controller-manager-5985f94fb7-7ct2h\" (UID: \"a3017984-7d80-4b3f-8072-2ed75d87cf96\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5985f94fb7-7ct2h"
Nov 02 19:32:26.800614 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:32:26.800529    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a3017984-7d80-4b3f-8072-2ed75d87cf96-metrics-cert\") pod \"opendatahub-operator-controller-manager-5985f94fb7-7ct2h\" (UID: \"a3017984-7d80-4b3f-8072-2ed75d87cf96\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5985f94fb7-7ct2h"
Nov 02 19:32:26.800736 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:32:26.800716    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a3017984-7d80-4b3f-8072-2ed75d87cf96-webhook-cert\") pod \"opendatahub-operator-controller-manager-5985f94fb7-7ct2h\" (UID: \"a3017984-7d80-4b3f-8072-2ed75d87cf96\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5985f94fb7-7ct2h"
Nov 02 19:32:26.800794 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:32:26.800716    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a3017984-7d80-4b3f-8072-2ed75d87cf96-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5985f94fb7-7ct2h\" (UID: \"a3017984-7d80-4b3f-8072-2ed75d87cf96\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5985f94fb7-7ct2h"
Nov 02 19:32:26.854851 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:32:26.854825    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppnxk\" (UniqueName: \"kubernetes.io/projected/a3017984-7d80-4b3f-8072-2ed75d87cf96-kube-api-access-ppnxk\") pod \"opendatahub-operator-controller-manager-5985f94fb7-7ct2h\" (UID: \"a3017984-7d80-4b3f-8072-2ed75d87cf96\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5985f94fb7-7ct2h"
Nov 02 19:32:26.942450 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:32:26.942427    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5985f94fb7-7ct2h"
Nov 02 19:32:27.417171 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:32:27.417043    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5985f94fb7-7ct2h"]
Nov 02 19:32:27.419813 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:32:27.419785    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3017984_7d80_4b3f_8072_2ed75d87cf96.slice/crio-d88ab473e38abcfec6c6b794cbeb2f7a076312fa50827d046642e19f4816cbf5 WatchSource:0}: Error finding container d88ab473e38abcfec6c6b794cbeb2f7a076312fa50827d046642e19f4816cbf5: Status 404 returned error can't find the container with id d88ab473e38abcfec6c6b794cbeb2f7a076312fa50827d046642e19f4816cbf5
Nov 02 19:32:27.482362 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:32:27.482334    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5985f94fb7-7ct2h" event={"ID":"a3017984-7d80-4b3f-8072-2ed75d87cf96","Type":"ContainerStarted","Data":"d88ab473e38abcfec6c6b794cbeb2f7a076312fa50827d046642e19f4816cbf5"}
Nov 02 19:32:31.496743 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:32:31.496705    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5985f94fb7-7ct2h" event={"ID":"a3017984-7d80-4b3f-8072-2ed75d87cf96","Type":"ContainerStarted","Data":"94afb3a71ad9eec5bcf593ad94ba49cd049c1d5fccf845c86a95a4969605a6d8"}
Nov 02 19:32:32.500955 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:32:32.500916    4463 generic.go:356] "Generic (PLEG): container finished" podID="a3017984-7d80-4b3f-8072-2ed75d87cf96" containerID="94afb3a71ad9eec5bcf593ad94ba49cd049c1d5fccf845c86a95a4969605a6d8" exitCode=0
Nov 02 19:32:32.501340 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:32:32.500966    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5985f94fb7-7ct2h" event={"ID":"a3017984-7d80-4b3f-8072-2ed75d87cf96","Type":"ContainerDied","Data":"94afb3a71ad9eec5bcf593ad94ba49cd049c1d5fccf845c86a95a4969605a6d8"}
Nov 02 19:32:33.506004 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:32:33.505972    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5985f94fb7-7ct2h" event={"ID":"a3017984-7d80-4b3f-8072-2ed75d87cf96","Type":"ContainerStarted","Data":"2754e2d33b03e99786821be370fcb4bbd0603ab4a7227a4b7337fa1f739d0d7b"}
Nov 02 19:32:33.506386 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:32:33.506116    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5985f94fb7-7ct2h"
Nov 02 19:32:33.720462 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:32:33.720411    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5985f94fb7-7ct2h" podStartSLOduration=2.695270782 podStartE2EDuration="7.72039847s" totalImagesPullingTime="4.586338359s" totalInitContainerRuntime="438.789329ms" isStatefulPod=false podCreationTimestamp="2026-11-02 19:32:26 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:32:33.72014379 +0000 UTC m=+260.327177558" watchObservedRunningTime="2026-11-02 19:32:33.72039847 +0000 UTC m=+260.327432238"
Nov 02 19:32:34.292743 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:32:34.292707    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 19:32:37.144629 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:32:37.144595    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-sjx6s"]
Nov 02 19:32:37.151106 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:32:37.151087    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-sjx6s"
Nov 02 19:32:37.246780 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:32:37.246753    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Nov 02 19:32:37.246780 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:32:37.246761    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-5n584\""
Nov 02 19:32:37.246780 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:32:37.246787    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Nov 02 19:32:37.247714 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:32:37.247691    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-sjx6s"]
Nov 02 19:32:37.277006 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:32:37.276981    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hstzj\" (UniqueName: \"kubernetes.io/projected/228a6205-ce35-4594-8bc8-aebe182ffb1c-kube-api-access-hstzj\") pod \"jobset-operator-674cd59dc5-sjx6s\" (UID: \"228a6205-ce35-4594-8bc8-aebe182ffb1c\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-sjx6s"
Nov 02 19:32:37.277143 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:32:37.277042    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/228a6205-ce35-4594-8bc8-aebe182ffb1c-tmp\") pod \"jobset-operator-674cd59dc5-sjx6s\" (UID: \"228a6205-ce35-4594-8bc8-aebe182ffb1c\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-sjx6s"
Nov 02 19:32:37.378007 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:32:37.377976    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hstzj\" (UniqueName: \"kubernetes.io/projected/228a6205-ce35-4594-8bc8-aebe182ffb1c-kube-api-access-hstzj\") pod \"jobset-operator-674cd59dc5-sjx6s\" (UID: \"228a6205-ce35-4594-8bc8-aebe182ffb1c\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-sjx6s"
Nov 02 19:32:37.378007 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:32:37.378023    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/228a6205-ce35-4594-8bc8-aebe182ffb1c-tmp\") pod \"jobset-operator-674cd59dc5-sjx6s\" (UID: \"228a6205-ce35-4594-8bc8-aebe182ffb1c\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-sjx6s"
Nov 02 19:32:37.378332 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:32:37.378317    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/228a6205-ce35-4594-8bc8-aebe182ffb1c-tmp\") pod \"jobset-operator-674cd59dc5-sjx6s\" (UID: \"228a6205-ce35-4594-8bc8-aebe182ffb1c\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-sjx6s"
Nov 02 19:32:37.431787 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:32:37.431761    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hstzj\" (UniqueName: \"kubernetes.io/projected/228a6205-ce35-4594-8bc8-aebe182ffb1c-kube-api-access-hstzj\") pod \"jobset-operator-674cd59dc5-sjx6s\" (UID: \"228a6205-ce35-4594-8bc8-aebe182ffb1c\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-sjx6s"
Nov 02 19:32:37.478080 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:32:37.478056    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-sjx6s"
Nov 02 19:32:37.836927 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:32:37.836852    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-sjx6s"]
Nov 02 19:32:37.839873 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:32:37.839836    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod228a6205_ce35_4594_8bc8_aebe182ffb1c.slice/crio-faf00e2991a31aca6161a8296618dae71a9ba2c6be5b21e4b48685923b70128a WatchSource:0}: Error finding container faf00e2991a31aca6161a8296618dae71a9ba2c6be5b21e4b48685923b70128a: Status 404 returned error can't find the container with id faf00e2991a31aca6161a8296618dae71a9ba2c6be5b21e4b48685923b70128a
Nov 02 19:32:38.523837 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:32:38.523798    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-sjx6s" event={"ID":"228a6205-ce35-4594-8bc8-aebe182ffb1c","Type":"ContainerStarted","Data":"faf00e2991a31aca6161a8296618dae71a9ba2c6be5b21e4b48685923b70128a"}
Nov 02 19:32:42.540285 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:32:42.540243    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-sjx6s" event={"ID":"228a6205-ce35-4594-8bc8-aebe182ffb1c","Type":"ContainerStarted","Data":"a04dfd5212adb9e40b74d01cb9d4a579ac447f0a774b18617df980a55b60cc7e"}
Nov 02 19:32:42.656965 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:32:42.656915    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-sjx6s" podStartSLOduration=1.466086014 podStartE2EDuration="5.656899769s" totalImagesPullingTime="4.190813755s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:32:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:32:42.655393467 +0000 UTC m=+269.262427255" watchObservedRunningTime="2026-11-02 19:32:42.656899769 +0000 UTC m=+269.263933538"
Nov 02 19:32:44.511895 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:32:44.511868    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5985f94fb7-7ct2h"
Nov 02 19:32:50.980429 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:32:50.980394    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-25nw5"]
Nov 02 19:32:50.985106 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:32:50.985089    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-25nw5"
Nov 02 19:32:51.039244 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:32:51.039220    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 02 19:32:51.081523 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:32:51.081496    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6z2b\" (UniqueName: \"kubernetes.io/projected/01019899-b2a2-4b23-ac2f-33dd143c6476-kube-api-access-f6z2b\") pod \"authorino-operator-7b7bb4bc77-25nw5\" (UID: \"01019899-b2a2-4b23-ac2f-33dd143c6476\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-25nw5"
Nov 02 19:32:51.085335 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:32:51.085315    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 02 19:32:51.085513 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:32:51.085497    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-7zx7v\""
Nov 02 19:32:51.086456 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:32:51.086440    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-25nw5"]
Nov 02 19:32:51.182126 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:32:51.182102    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6z2b\" (UniqueName: \"kubernetes.io/projected/01019899-b2a2-4b23-ac2f-33dd143c6476-kube-api-access-f6z2b\") pod \"authorino-operator-7b7bb4bc77-25nw5\" (UID: \"01019899-b2a2-4b23-ac2f-33dd143c6476\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-25nw5"
Nov 02 19:32:51.236288 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:32:51.236216    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6z2b\" (UniqueName: \"kubernetes.io/projected/01019899-b2a2-4b23-ac2f-33dd143c6476-kube-api-access-f6z2b\") pod \"authorino-operator-7b7bb4bc77-25nw5\" (UID: \"01019899-b2a2-4b23-ac2f-33dd143c6476\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-25nw5"
Nov 02 19:32:51.295437 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:32:51.295411    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-25nw5"
Nov 02 19:32:51.657086 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:32:51.657005    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-25nw5"]
Nov 02 19:32:51.660647 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:32:51.660618    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01019899_b2a2_4b23_ac2f_33dd143c6476.slice/crio-643b822b8124c2555715e83bf259030d2f5fdb9e3678611b57496c4e47f2d844 WatchSource:0}: Error finding container 643b822b8124c2555715e83bf259030d2f5fdb9e3678611b57496c4e47f2d844: Status 404 returned error can't find the container with id 643b822b8124c2555715e83bf259030d2f5fdb9e3678611b57496c4e47f2d844
Nov 02 19:32:52.575733 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:32:52.575696    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-25nw5" event={"ID":"01019899-b2a2-4b23-ac2f-33dd143c6476","Type":"ContainerStarted","Data":"643b822b8124c2555715e83bf259030d2f5fdb9e3678611b57496c4e47f2d844"}
Nov 02 19:32:54.450405 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:32:54.450365    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-jjs9d"]
Nov 02 19:32:54.455905 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:32:54.455881    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jjs9d"
Nov 02 19:32:54.504919 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:32:54.504878    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Nov 02 19:32:54.551386 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:32:54.551359    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Nov 02 19:32:54.551524 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:32:54.551500    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-f9djg\""
Nov 02 19:32:54.552651 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:32:54.552620    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-jjs9d"]
Nov 02 19:32:54.612076 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:32:54.612042    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tx7lk\" (UniqueName: \"kubernetes.io/projected/760eeed1-48ec-4029-b3c1-837009e77fbf-kube-api-access-tx7lk\") pod \"kuadrant-console-plugin-66c4fb575d-jjs9d\" (UID: \"760eeed1-48ec-4029-b3c1-837009e77fbf\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jjs9d"
Nov 02 19:32:54.612273 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:32:54.612152    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/760eeed1-48ec-4029-b3c1-837009e77fbf-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-jjs9d\" (UID: \"760eeed1-48ec-4029-b3c1-837009e77fbf\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jjs9d"
Nov 02 19:32:54.612338 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:32:54.612278    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/760eeed1-48ec-4029-b3c1-837009e77fbf-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-jjs9d\" (UID: \"760eeed1-48ec-4029-b3c1-837009e77fbf\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jjs9d"
Nov 02 19:32:54.713603 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:32:54.713537    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tx7lk\" (UniqueName: \"kubernetes.io/projected/760eeed1-48ec-4029-b3c1-837009e77fbf-kube-api-access-tx7lk\") pod \"kuadrant-console-plugin-66c4fb575d-jjs9d\" (UID: \"760eeed1-48ec-4029-b3c1-837009e77fbf\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jjs9d"
Nov 02 19:32:54.713769 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:32:54.713608    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/760eeed1-48ec-4029-b3c1-837009e77fbf-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-jjs9d\" (UID: \"760eeed1-48ec-4029-b3c1-837009e77fbf\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jjs9d"
Nov 02 19:32:54.713769 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:32:54.713658    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/760eeed1-48ec-4029-b3c1-837009e77fbf-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-jjs9d\" (UID: \"760eeed1-48ec-4029-b3c1-837009e77fbf\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jjs9d"
Nov 02 19:32:54.714403 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:32:54.714373    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/760eeed1-48ec-4029-b3c1-837009e77fbf-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-jjs9d\" (UID: \"760eeed1-48ec-4029-b3c1-837009e77fbf\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jjs9d"
Nov 02 19:32:54.715921 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:32:54.715900    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/760eeed1-48ec-4029-b3c1-837009e77fbf-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-jjs9d\" (UID: \"760eeed1-48ec-4029-b3c1-837009e77fbf\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jjs9d"
Nov 02 19:32:54.767299 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:32:54.767272    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tx7lk\" (UniqueName: \"kubernetes.io/projected/760eeed1-48ec-4029-b3c1-837009e77fbf-kube-api-access-tx7lk\") pod \"kuadrant-console-plugin-66c4fb575d-jjs9d\" (UID: \"760eeed1-48ec-4029-b3c1-837009e77fbf\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jjs9d"
Nov 02 19:32:54.768988 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:32:54.768964    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jjs9d"
Nov 02 19:32:55.130686 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:32:55.130661    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-jjs9d"]
Nov 02 19:32:55.133590 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:32:55.133566    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod760eeed1_48ec_4029_b3c1_837009e77fbf.slice/crio-eeb56a6539b5b69df672af6b596595afd067d58f5196bd0c2c708ed6c66ebc46 WatchSource:0}: Error finding container eeb56a6539b5b69df672af6b596595afd067d58f5196bd0c2c708ed6c66ebc46: Status 404 returned error can't find the container with id eeb56a6539b5b69df672af6b596595afd067d58f5196bd0c2c708ed6c66ebc46
Nov 02 19:32:55.592450 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:32:55.592418    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jjs9d" event={"ID":"760eeed1-48ec-4029-b3c1-837009e77fbf","Type":"ContainerStarted","Data":"eeb56a6539b5b69df672af6b596595afd067d58f5196bd0c2c708ed6c66ebc46"}
Nov 02 19:32:55.593815 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:32:55.593787    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-25nw5" event={"ID":"01019899-b2a2-4b23-ac2f-33dd143c6476","Type":"ContainerStarted","Data":"c645f8cd03d6c43011eb9136d22f8c7eec68a07f6a4ca77a9508a00a8ad42d35"}
Nov 02 19:32:55.593953 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:32:55.593939    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-25nw5"
Nov 02 19:32:55.701822 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:32:55.701760    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-25nw5" podStartSLOduration=2.372427308 podStartE2EDuration="5.701736301s" totalImagesPullingTime="3.329308993s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:32:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:32:55.700774811 +0000 UTC m=+282.307808600" watchObservedRunningTime="2026-11-02 19:32:55.701736301 +0000 UTC m=+282.308770070"
Nov 02 19:33:01.010948 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:33:01.010911    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-wmp44"]
Nov 02 19:33:01.015579 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:33:01.015554    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-wmp44"
Nov 02 19:33:01.115280 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:33:01.115252    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Nov 02 19:33:01.115280 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:33:01.115279    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Nov 02 19:33:01.116487 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:33:01.116468    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-wmp44"]
Nov 02 19:33:01.159945 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:33:01.159919    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5qpx\" (UniqueName: \"kubernetes.io/projected/c5fcfd2d-ed14-45ea-a811-dbe2237490eb-kube-api-access-m5qpx\") pod \"cert-manager-webhook-5dbbbf75cc-wmp44\" (UID: \"c5fcfd2d-ed14-45ea-a811-dbe2237490eb\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-wmp44"
Nov 02 19:33:01.160071 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:33:01.159952    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c5fcfd2d-ed14-45ea-a811-dbe2237490eb-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-wmp44\" (UID: \"c5fcfd2d-ed14-45ea-a811-dbe2237490eb\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-wmp44"
Nov 02 19:33:01.261156 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:33:01.261077    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5qpx\" (UniqueName: \"kubernetes.io/projected/c5fcfd2d-ed14-45ea-a811-dbe2237490eb-kube-api-access-m5qpx\") pod \"cert-manager-webhook-5dbbbf75cc-wmp44\" (UID: \"c5fcfd2d-ed14-45ea-a811-dbe2237490eb\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-wmp44"
Nov 02 19:33:01.261156 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:33:01.261126    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c5fcfd2d-ed14-45ea-a811-dbe2237490eb-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-wmp44\" (UID: \"c5fcfd2d-ed14-45ea-a811-dbe2237490eb\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-wmp44"
Nov 02 19:33:01.314987 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:33:01.314962    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c5fcfd2d-ed14-45ea-a811-dbe2237490eb-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-wmp44\" (UID: \"c5fcfd2d-ed14-45ea-a811-dbe2237490eb\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-wmp44"
Nov 02 19:33:01.315246 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:33:01.315227    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5qpx\" (UniqueName: \"kubernetes.io/projected/c5fcfd2d-ed14-45ea-a811-dbe2237490eb-kube-api-access-m5qpx\") pod \"cert-manager-webhook-5dbbbf75cc-wmp44\" (UID: \"c5fcfd2d-ed14-45ea-a811-dbe2237490eb\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-wmp44"
Nov 02 19:33:01.326984 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:33:01.326966    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-wmp44"
Nov 02 19:33:01.690265 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:33:01.690237    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-wmp44"]
Nov 02 19:33:01.693394 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:33:01.693370    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc5fcfd2d_ed14_45ea_a811_dbe2237490eb.slice/crio-87ad37b46bdda1af4acb0b98dee5277cc708b5822a5295d1703ed5b555fc0c52 WatchSource:0}: Error finding container 87ad37b46bdda1af4acb0b98dee5277cc708b5822a5295d1703ed5b555fc0c52: Status 404 returned error can't find the container with id 87ad37b46bdda1af4acb0b98dee5277cc708b5822a5295d1703ed5b555fc0c52
Nov 02 19:33:02.622539 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:33:02.622501    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-wmp44" event={"ID":"c5fcfd2d-ed14-45ea-a811-dbe2237490eb","Type":"ContainerStarted","Data":"87ad37b46bdda1af4acb0b98dee5277cc708b5822a5295d1703ed5b555fc0c52"}
Nov 02 19:33:04.297419 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:33:04.297363    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 19:33:04.660318 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:33:04.660222    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-65657466b-9hgsh"]
Nov 02 19:33:04.671913 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:33:04.671886    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-65657466b-9hgsh"
Nov 02 19:33:04.690881 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:33:04.690851    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pqc2\" (UniqueName: \"kubernetes.io/projected/a394c32e-69b7-4a96-8e08-e0080be53c80-kube-api-access-2pqc2\") pod \"kube-auth-proxy-65657466b-9hgsh\" (UID: \"a394c32e-69b7-4a96-8e08-e0080be53c80\") " pod="openshift-ingress/kube-auth-proxy-65657466b-9hgsh"
Nov 02 19:33:04.691040 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:33:04.690914    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a394c32e-69b7-4a96-8e08-e0080be53c80-tls-certs\") pod \"kube-auth-proxy-65657466b-9hgsh\" (UID: \"a394c32e-69b7-4a96-8e08-e0080be53c80\") " pod="openshift-ingress/kube-auth-proxy-65657466b-9hgsh"
Nov 02 19:33:04.691040 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:33:04.690953    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a394c32e-69b7-4a96-8e08-e0080be53c80-tmp\") pod \"kube-auth-proxy-65657466b-9hgsh\" (UID: \"a394c32e-69b7-4a96-8e08-e0080be53c80\") " pod="openshift-ingress/kube-auth-proxy-65657466b-9hgsh"
Nov 02 19:33:04.720770 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:33:04.720698    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-65657466b-9hgsh"]
Nov 02 19:33:04.757248 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:33:04.757211    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 02 19:33:04.764382 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:33:04.764059    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 02 19:33:04.764382 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:33:04.764312    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Nov 02 19:33:04.764382 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:33:04.764350    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Nov 02 19:33:04.777170 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:33:04.777134    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-7db5j\""
Nov 02 19:33:04.791687 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:33:04.791657    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pqc2\" (UniqueName: \"kubernetes.io/projected/a394c32e-69b7-4a96-8e08-e0080be53c80-kube-api-access-2pqc2\") pod \"kube-auth-proxy-65657466b-9hgsh\" (UID: \"a394c32e-69b7-4a96-8e08-e0080be53c80\") " pod="openshift-ingress/kube-auth-proxy-65657466b-9hgsh"
Nov 02 19:33:04.791819 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:33:04.791709    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a394c32e-69b7-4a96-8e08-e0080be53c80-tls-certs\") pod \"kube-auth-proxy-65657466b-9hgsh\" (UID: \"a394c32e-69b7-4a96-8e08-e0080be53c80\") " pod="openshift-ingress/kube-auth-proxy-65657466b-9hgsh"
Nov 02 19:33:04.791819 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:33:04.791749    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a394c32e-69b7-4a96-8e08-e0080be53c80-tmp\") pod \"kube-auth-proxy-65657466b-9hgsh\" (UID: \"a394c32e-69b7-4a96-8e08-e0080be53c80\") " pod="openshift-ingress/kube-auth-proxy-65657466b-9hgsh"
Nov 02 19:33:04.794595 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:33:04.794567    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a394c32e-69b7-4a96-8e08-e0080be53c80-tmp\") pod \"kube-auth-proxy-65657466b-9hgsh\" (UID: \"a394c32e-69b7-4a96-8e08-e0080be53c80\") " pod="openshift-ingress/kube-auth-proxy-65657466b-9hgsh"
Nov 02 19:33:04.795124 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:33:04.795101    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a394c32e-69b7-4a96-8e08-e0080be53c80-tls-certs\") pod \"kube-auth-proxy-65657466b-9hgsh\" (UID: \"a394c32e-69b7-4a96-8e08-e0080be53c80\") " pod="openshift-ingress/kube-auth-proxy-65657466b-9hgsh"
Nov 02 19:33:04.853006 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:33:04.852941    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pqc2\" (UniqueName: \"kubernetes.io/projected/a394c32e-69b7-4a96-8e08-e0080be53c80-kube-api-access-2pqc2\") pod \"kube-auth-proxy-65657466b-9hgsh\" (UID: \"a394c32e-69b7-4a96-8e08-e0080be53c80\") " pod="openshift-ingress/kube-auth-proxy-65657466b-9hgsh"
Nov 02 19:33:04.988989 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:33:04.988781    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-65657466b-9hgsh"
Nov 02 19:33:05.379320 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:33:05.379250    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-65657466b-9hgsh"]
Nov 02 19:33:05.385760 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:33:05.385728    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda394c32e_69b7_4a96_8e08_e0080be53c80.slice/crio-7a6138d6ce3d7d8aa41cc53cfe44285baa9256bb996e359202e31971ce9c477b WatchSource:0}: Error finding container 7a6138d6ce3d7d8aa41cc53cfe44285baa9256bb996e359202e31971ce9c477b: Status 404 returned error can't find the container with id 7a6138d6ce3d7d8aa41cc53cfe44285baa9256bb996e359202e31971ce9c477b
Nov 02 19:33:05.655309 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:33:05.655233    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-65657466b-9hgsh" event={"ID":"a394c32e-69b7-4a96-8e08-e0080be53c80","Type":"ContainerStarted","Data":"7a6138d6ce3d7d8aa41cc53cfe44285baa9256bb996e359202e31971ce9c477b"}
Nov 02 19:33:06.600381 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:33:06.600350    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-25nw5"
Nov 02 19:33:06.661625 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:33:06.661588    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-wmp44" event={"ID":"c5fcfd2d-ed14-45ea-a811-dbe2237490eb","Type":"ContainerStarted","Data":"b59c66358eb36aa1612a5c9995ca705e29bf69176dd3b795ca61db99696c29c4"}
Nov 02 19:33:06.661790 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:33:06.661729    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-wmp44"
Nov 02 19:33:06.824987 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:33:06.824883    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-wmp44" podStartSLOduration=2.026409017 podStartE2EDuration="6.824861994s" totalImagesPullingTime="4.798452977s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:33:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:33:06.824439423 +0000 UTC m=+293.431473189" watchObservedRunningTime="2026-11-02 19:33:06.824861994 +0000 UTC m=+293.431895763"
Nov 02 19:33:07.666961 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:33:07.666858    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jjs9d" event={"ID":"760eeed1-48ec-4029-b3c1-837009e77fbf","Type":"ContainerStarted","Data":"79f08ea25d0c0678f6e452d142e5337968b501c290f994cf57f404fa78975a1c"}
Nov 02 19:33:07.773124 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:33:07.773045    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jjs9d" podStartSLOduration=1.511654404 podStartE2EDuration="13.773026717s" totalImagesPullingTime="12.261372313s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:32:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:33:07.77259127 +0000 UTC m=+294.379625038" watchObservedRunningTime="2026-11-02 19:33:07.773026717 +0000 UTC m=+294.380060480"
Nov 02 19:33:08.053692 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:33:08.053656    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 19:33:08.063477 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:33:08.063449    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"
Nov 02 19:33:08.115715 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:33:08.115679    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Nov 02 19:33:08.117295 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:33:08.117262    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 19:33:08.118519 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:33:08.118487    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7wvj\" (UniqueName: \"kubernetes.io/projected/ae3ddaea-c756-486f-bd85-aef7fc1f66b3-kube-api-access-b7wvj\") pod \"limitador-limitador-7c9b87d574-p4mx5\" (UID: \"ae3ddaea-c756-486f-bd85-aef7fc1f66b3\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"
Nov 02 19:33:08.118617 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:33:08.118563    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ae3ddaea-c756-486f-bd85-aef7fc1f66b3-config-file\") pod \"limitador-limitador-7c9b87d574-p4mx5\" (UID: \"ae3ddaea-c756-486f-bd85-aef7fc1f66b3\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"
Nov 02 19:33:08.219824 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:33:08.219796    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7wvj\" (UniqueName: \"kubernetes.io/projected/ae3ddaea-c756-486f-bd85-aef7fc1f66b3-kube-api-access-b7wvj\") pod \"limitador-limitador-7c9b87d574-p4mx5\" (UID: \"ae3ddaea-c756-486f-bd85-aef7fc1f66b3\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"
Nov 02 19:33:08.219974 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:33:08.219843    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ae3ddaea-c756-486f-bd85-aef7fc1f66b3-config-file\") pod \"limitador-limitador-7c9b87d574-p4mx5\" (UID: \"ae3ddaea-c756-486f-bd85-aef7fc1f66b3\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"
Nov 02 19:33:08.220512 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:33:08.220492    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ae3ddaea-c756-486f-bd85-aef7fc1f66b3-config-file\") pod \"limitador-limitador-7c9b87d574-p4mx5\" (UID: \"ae3ddaea-c756-486f-bd85-aef7fc1f66b3\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"
Nov 02 19:33:08.273388 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:33:08.273358    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7wvj\" (UniqueName: \"kubernetes.io/projected/ae3ddaea-c756-486f-bd85-aef7fc1f66b3-kube-api-access-b7wvj\") pod \"limitador-limitador-7c9b87d574-p4mx5\" (UID: \"ae3ddaea-c756-486f-bd85-aef7fc1f66b3\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"
Nov 02 19:33:08.330667 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:33:08.330615    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 19:33:08.376973 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:33:08.376947    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"
Nov 02 19:33:08.749967 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:33:08.749939    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 19:33:08.753180 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:33:08.753157    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae3ddaea_c756_486f_bd85_aef7fc1f66b3.slice/crio-a864ea64646ac053a3219df4717281eae32b465be9f85a18fbf7b64abd7e5e46 WatchSource:0}: Error finding container a864ea64646ac053a3219df4717281eae32b465be9f85a18fbf7b64abd7e5e46: Status 404 returned error can't find the container with id a864ea64646ac053a3219df4717281eae32b465be9f85a18fbf7b64abd7e5e46
Nov 02 19:33:09.676028 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:33:09.675973    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-65657466b-9hgsh" event={"ID":"a394c32e-69b7-4a96-8e08-e0080be53c80","Type":"ContainerStarted","Data":"fc2b42c6ab87907b557d634d713645092aba8df4ddee23b565bdf9d7f2c241b8"}
Nov 02 19:33:09.677200 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:33:09.677175    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-p4mx5" event={"ID":"ae3ddaea-c756-486f-bd85-aef7fc1f66b3","Type":"ContainerStarted","Data":"a864ea64646ac053a3219df4717281eae32b465be9f85a18fbf7b64abd7e5e46"}
Nov 02 19:33:09.784525 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:33:09.784471    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-65657466b-9hgsh" podStartSLOduration=2.419430437 podStartE2EDuration="5.784454474s" totalImagesPullingTime="3.365024037s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:33:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:33:09.782652394 +0000 UTC m=+296.389686160" watchObservedRunningTime="2026-11-02 19:33:09.784454474 +0000 UTC m=+296.391488255"
Nov 02 19:33:11.686969 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:33:11.686933    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-p4mx5" event={"ID":"ae3ddaea-c756-486f-bd85-aef7fc1f66b3","Type":"ContainerStarted","Data":"5e64ae44735936edcc5a0b6ed25f93e1f48b82259a6a9f41af011f6a5d50a1a3"}
Nov 02 19:33:11.687404 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:33:11.687066    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"
Nov 02 19:33:11.795991 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:33:11.795941    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-p4mx5" podStartSLOduration=1.877438196 podStartE2EDuration="3.795927735s" totalImagesPullingTime="1.918489539s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:33:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:33:11.79449116 +0000 UTC m=+298.401524927" watchObservedRunningTime="2026-11-02 19:33:11.795927735 +0000 UTC m=+298.402961502"
Nov 02 19:33:12.669096 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:33:12.669066    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-wmp44"
Nov 02 19:33:22.692099 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:33:22.692069    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"
Nov 02 19:33:34.292654 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:33:34.292568    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 19:34:04.293805 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:34:04.293768    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 19:34:22.627865 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:34:22.627828    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-phk29"]
Nov 02 19:34:22.630190 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:34:22.630173    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-phk29"
Nov 02 19:34:22.678712 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:34:22.678691    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Nov 02 19:34:22.678816 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:34:22.678762    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-t6nb8\""
Nov 02 19:34:22.678944 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:34:22.678929    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 02 19:34:22.679030 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:34:22.678939    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 02 19:34:22.679377 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:34:22.679359    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qc455\" (UniqueName: \"kubernetes.io/projected/00b5dca8-d734-47c4-99d6-00824c07793b-kube-api-access-qc455\") pod \"model-serving-api-75bd89fdfc-phk29\" (UID: \"00b5dca8-d734-47c4-99d6-00824c07793b\") " pod="opendatahub/model-serving-api-75bd89fdfc-phk29"
Nov 02 19:34:22.679412 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:34:22.679392    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00b5dca8-d734-47c4-99d6-00824c07793b-tls-certs\") pod \"model-serving-api-75bd89fdfc-phk29\" (UID: \"00b5dca8-d734-47c4-99d6-00824c07793b\") " pod="opendatahub/model-serving-api-75bd89fdfc-phk29"
Nov 02 19:34:22.684193 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:34:22.684162    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-phk29"]
Nov 02 19:34:22.779747 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:34:22.779719    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc455\" (UniqueName: \"kubernetes.io/projected/00b5dca8-d734-47c4-99d6-00824c07793b-kube-api-access-qc455\") pod \"model-serving-api-75bd89fdfc-phk29\" (UID: \"00b5dca8-d734-47c4-99d6-00824c07793b\") " pod="opendatahub/model-serving-api-75bd89fdfc-phk29"
Nov 02 19:34:22.779892 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:34:22.779757    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00b5dca8-d734-47c4-99d6-00824c07793b-tls-certs\") pod \"model-serving-api-75bd89fdfc-phk29\" (UID: \"00b5dca8-d734-47c4-99d6-00824c07793b\") " pod="opendatahub/model-serving-api-75bd89fdfc-phk29"
Nov 02 19:34:22.782082 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:34:22.782062    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00b5dca8-d734-47c4-99d6-00824c07793b-tls-certs\") pod \"model-serving-api-75bd89fdfc-phk29\" (UID: \"00b5dca8-d734-47c4-99d6-00824c07793b\") " pod="opendatahub/model-serving-api-75bd89fdfc-phk29"
Nov 02 19:34:22.834279 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:34:22.834257    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qc455\" (UniqueName: \"kubernetes.io/projected/00b5dca8-d734-47c4-99d6-00824c07793b-kube-api-access-qc455\") pod \"model-serving-api-75bd89fdfc-phk29\" (UID: \"00b5dca8-d734-47c4-99d6-00824c07793b\") " pod="opendatahub/model-serving-api-75bd89fdfc-phk29"
Nov 02 19:34:22.941030 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:34:22.940981    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-phk29"
Nov 02 19:34:23.299864 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:34:23.299796    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-phk29"]
Nov 02 19:34:23.305748 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:34:23.305716    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00b5dca8_d734_47c4_99d6_00824c07793b.slice/crio-31c851aac375924b54433b913e54a3053965c658452dbcfe03f44c705c0d9cd8 WatchSource:0}: Error finding container 31c851aac375924b54433b913e54a3053965c658452dbcfe03f44c705c0d9cd8: Status 404 returned error can't find the container with id 31c851aac375924b54433b913e54a3053965c658452dbcfe03f44c705c0d9cd8
Nov 02 19:34:23.307912 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:34:23.307896    4463 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 19:34:23.955047 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:34:23.954995    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-phk29" event={"ID":"00b5dca8-d734-47c4-99d6-00824c07793b","Type":"ContainerStarted","Data":"31c851aac375924b54433b913e54a3053965c658452dbcfe03f44c705c0d9cd8"}
Nov 02 19:34:26.968225 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:34:26.968178    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-phk29" event={"ID":"00b5dca8-d734-47c4-99d6-00824c07793b","Type":"ContainerStarted","Data":"d2b1e68272331155c0c2ec270d5a552fa51deb55a74e417e0a78a9f676aec1be"}
Nov 02 19:34:26.968607 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:34:26.968318    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-phk29"
Nov 02 19:34:27.075956 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:34:27.075913    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-phk29" podStartSLOduration=2.46503121 podStartE2EDuration="5.075901733s" totalImagesPullingTime="2.610870523s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:34:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:34:27.074273107 +0000 UTC m=+373.681306875" watchObservedRunningTime="2026-11-02 19:34:27.075901733 +0000 UTC m=+373.682935500"
Nov 02 19:34:34.292934 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:34:34.292898    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 19:34:37.976098 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:34:37.976071    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-phk29"
Nov 02 19:34:46.330394 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:34:46.330365    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 19:34:51.353845 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:34:51.353813    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-769845dd8d-nthbb"]
Nov 02 19:34:51.355949 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:34:51.355934    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-769845dd8d-nthbb"
Nov 02 19:34:51.393114 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:34:51.393086    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fvsw\" (UniqueName: \"kubernetes.io/projected/48058634-fd1d-4c53-a367-d43833c583b4-kube-api-access-4fvsw\") pod \"maas-api-769845dd8d-nthbb\" (UID: \"48058634-fd1d-4c53-a367-d43833c583b4\") " pod="odh-ai-gateway-infra/maas-api-769845dd8d-nthbb"
Nov 02 19:34:51.393228 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:34:51.393138    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48058634-fd1d-4c53-a367-d43833c583b4-openshift-service-ca\") pod \"maas-api-769845dd8d-nthbb\" (UID: \"48058634-fd1d-4c53-a367-d43833c583b4\") " pod="odh-ai-gateway-infra/maas-api-769845dd8d-nthbb"
Nov 02 19:34:51.393228 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:34:51.393194    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48058634-fd1d-4c53-a367-d43833c583b4-maas-api-tls\") pod \"maas-api-769845dd8d-nthbb\" (UID: \"48058634-fd1d-4c53-a367-d43833c583b4\") " pod="odh-ai-gateway-infra/maas-api-769845dd8d-nthbb"
Nov 02 19:34:51.393307 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:34:51.393292    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48058634-fd1d-4c53-a367-d43833c583b4-metrics-certs\") pod \"maas-api-769845dd8d-nthbb\" (UID: \"48058634-fd1d-4c53-a367-d43833c583b4\") " pod="odh-ai-gateway-infra/maas-api-769845dd8d-nthbb"
Nov 02 19:34:51.450898 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:34:51.450875    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 19:34:51.451047 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:34:51.451008    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 19:34:51.451114 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:34:51.451095    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-sg6rf\""
Nov 02 19:34:51.451167 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:34:51.451149    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Nov 02 19:34:51.451205 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:34:51.451189    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 02 19:34:51.452339 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:34:51.452323    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-769845dd8d-nthbb"]
Nov 02 19:34:51.493874 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:34:51.493853    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48058634-fd1d-4c53-a367-d43833c583b4-openshift-service-ca\") pod \"maas-api-769845dd8d-nthbb\" (UID: \"48058634-fd1d-4c53-a367-d43833c583b4\") " pod="odh-ai-gateway-infra/maas-api-769845dd8d-nthbb"
Nov 02 19:34:51.493966 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:34:51.493880    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48058634-fd1d-4c53-a367-d43833c583b4-maas-api-tls\") pod \"maas-api-769845dd8d-nthbb\" (UID: \"48058634-fd1d-4c53-a367-d43833c583b4\") " pod="odh-ai-gateway-infra/maas-api-769845dd8d-nthbb"
Nov 02 19:34:51.493966 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:34:51.493902    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48058634-fd1d-4c53-a367-d43833c583b4-metrics-certs\") pod \"maas-api-769845dd8d-nthbb\" (UID: \"48058634-fd1d-4c53-a367-d43833c583b4\") " pod="odh-ai-gateway-infra/maas-api-769845dd8d-nthbb"
Nov 02 19:34:51.493966 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:34:51.493936    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fvsw\" (UniqueName: \"kubernetes.io/projected/48058634-fd1d-4c53-a367-d43833c583b4-kube-api-access-4fvsw\") pod \"maas-api-769845dd8d-nthbb\" (UID: \"48058634-fd1d-4c53-a367-d43833c583b4\") " pod="odh-ai-gateway-infra/maas-api-769845dd8d-nthbb"
Nov 02 19:34:51.494579 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:34:51.494560    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48058634-fd1d-4c53-a367-d43833c583b4-openshift-service-ca\") pod \"maas-api-769845dd8d-nthbb\" (UID: \"48058634-fd1d-4c53-a367-d43833c583b4\") " pod="odh-ai-gateway-infra/maas-api-769845dd8d-nthbb"
Nov 02 19:34:51.496316 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:34:51.496299    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48058634-fd1d-4c53-a367-d43833c583b4-maas-api-tls\") pod \"maas-api-769845dd8d-nthbb\" (UID: \"48058634-fd1d-4c53-a367-d43833c583b4\") " pod="odh-ai-gateway-infra/maas-api-769845dd8d-nthbb"
Nov 02 19:34:51.496397 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:34:51.496359    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48058634-fd1d-4c53-a367-d43833c583b4-metrics-certs\") pod \"maas-api-769845dd8d-nthbb\" (UID: \"48058634-fd1d-4c53-a367-d43833c583b4\") " pod="odh-ai-gateway-infra/maas-api-769845dd8d-nthbb"
Nov 02 19:34:51.547982 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:34:51.547951    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fvsw\" (UniqueName: \"kubernetes.io/projected/48058634-fd1d-4c53-a367-d43833c583b4-kube-api-access-4fvsw\") pod \"maas-api-769845dd8d-nthbb\" (UID: \"48058634-fd1d-4c53-a367-d43833c583b4\") " pod="odh-ai-gateway-infra/maas-api-769845dd8d-nthbb"
Nov 02 19:34:51.667039 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:34:51.666953    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-769845dd8d-nthbb"
Nov 02 19:34:52.026437 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:34:52.026286    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-769845dd8d-nthbb"]
Nov 02 19:34:52.028985 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:34:52.028957    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48058634_fd1d_4c53_a367_d43833c583b4.slice/crio-0ec8f029adf057f3d66a992a33035a7958f1245520e4aa34a9c36f24aa3db3e9 WatchSource:0}: Error finding container 0ec8f029adf057f3d66a992a33035a7958f1245520e4aa34a9c36f24aa3db3e9: Status 404 returned error can't find the container with id 0ec8f029adf057f3d66a992a33035a7958f1245520e4aa34a9c36f24aa3db3e9
Nov 02 19:34:52.066195 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:34:52.066172    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-769845dd8d-nthbb" event={"ID":"48058634-fd1d-4c53-a367-d43833c583b4","Type":"ContainerStarted","Data":"0ec8f029adf057f3d66a992a33035a7958f1245520e4aa34a9c36f24aa3db3e9"}
Nov 02 19:34:55.081346 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:34:55.081267    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-769845dd8d-nthbb" event={"ID":"48058634-fd1d-4c53-a367-d43833c583b4","Type":"ContainerStarted","Data":"03093b0ae89cc9037e21046ef214c1f4d422e1c3c738a2cca50cc63ddccc6083"}
Nov 02 19:34:55.081788 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:34:55.081415    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-769845dd8d-nthbb"
Nov 02 19:34:55.188707 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:34:55.188657    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-769845dd8d-nthbb" podStartSLOduration=1.495209147 podStartE2EDuration="4.188644955s" totalImagesPullingTime="2.693435808s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:34:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:34:55.1877019 +0000 UTC m=+401.794735666" watchObservedRunningTime="2026-11-02 19:34:55.188644955 +0000 UTC m=+401.795678723"
Nov 02 19:35:01.090931 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:35:01.090899    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-769845dd8d-nthbb"
Nov 02 19:35:04.292296 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:35:04.292233    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 19:35:34.292550 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:35:34.292509    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 19:36:04.293227 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:36:04.293185    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 19:36:30.615691 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:30.615604    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9b2dc"]
Nov 02 19:36:30.618948 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:30.618931    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9b2dc"
Nov 02 19:36:30.667973 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:30.667949    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Nov 02 19:36:30.710306 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:30.710280    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/365a63f7-ce01-4701-95b3-fd3862930ca6-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9b2dc\" (UID: \"365a63f7-ce01-4701-95b3-fd3862930ca6\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9b2dc"
Nov 02 19:36:30.710413 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:30.710319    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/365a63f7-ce01-4701-95b3-fd3862930ca6-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9b2dc\" (UID: \"365a63f7-ce01-4701-95b3-fd3862930ca6\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9b2dc"
Nov 02 19:36:30.710413 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:30.710382    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/365a63f7-ce01-4701-95b3-fd3862930ca6-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9b2dc\" (UID: \"365a63f7-ce01-4701-95b3-fd3862930ca6\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9b2dc"
Nov 02 19:36:30.710413 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:30.710400    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/365a63f7-ce01-4701-95b3-fd3862930ca6-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9b2dc\" (UID: \"365a63f7-ce01-4701-95b3-fd3862930ca6\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9b2dc"
Nov 02 19:36:30.710513 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:30.710468    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxb2g\" (UniqueName: \"kubernetes.io/projected/365a63f7-ce01-4701-95b3-fd3862930ca6-kube-api-access-hxb2g\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9b2dc\" (UID: \"365a63f7-ce01-4701-95b3-fd3862930ca6\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9b2dc"
Nov 02 19:36:30.710546 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:30.710518    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/365a63f7-ce01-4701-95b3-fd3862930ca6-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9b2dc\" (UID: \"365a63f7-ce01-4701-95b3-fd3862930ca6\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9b2dc"
Nov 02 19:36:30.714064 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:30.714045    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Nov 02 19:36:30.714152 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:30.714060    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Nov 02 19:36:30.714152 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:30.714073    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-xwhqg\""
Nov 02 19:36:30.715476 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:30.715460    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9b2dc"]
Nov 02 19:36:30.811570 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:30.811551    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/365a63f7-ce01-4701-95b3-fd3862930ca6-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9b2dc\" (UID: \"365a63f7-ce01-4701-95b3-fd3862930ca6\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9b2dc"
Nov 02 19:36:30.811679 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:30.811581    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/365a63f7-ce01-4701-95b3-fd3862930ca6-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9b2dc\" (UID: \"365a63f7-ce01-4701-95b3-fd3862930ca6\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9b2dc"
Nov 02 19:36:30.811679 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:30.811607    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/365a63f7-ce01-4701-95b3-fd3862930ca6-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9b2dc\" (UID: \"365a63f7-ce01-4701-95b3-fd3862930ca6\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9b2dc"
Nov 02 19:36:30.811679 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:30.811634    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/365a63f7-ce01-4701-95b3-fd3862930ca6-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9b2dc\" (UID: \"365a63f7-ce01-4701-95b3-fd3862930ca6\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9b2dc"
Nov 02 19:36:30.811679 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:30.811657    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/365a63f7-ce01-4701-95b3-fd3862930ca6-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9b2dc\" (UID: \"365a63f7-ce01-4701-95b3-fd3862930ca6\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9b2dc"
Nov 02 19:36:30.811849 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:30.811687    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxb2g\" (UniqueName: \"kubernetes.io/projected/365a63f7-ce01-4701-95b3-fd3862930ca6-kube-api-access-hxb2g\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9b2dc\" (UID: \"365a63f7-ce01-4701-95b3-fd3862930ca6\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9b2dc"
Nov 02 19:36:30.812065 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:30.812048    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/365a63f7-ce01-4701-95b3-fd3862930ca6-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9b2dc\" (UID: \"365a63f7-ce01-4701-95b3-fd3862930ca6\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9b2dc"
Nov 02 19:36:30.812140 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:30.812101    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/365a63f7-ce01-4701-95b3-fd3862930ca6-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9b2dc\" (UID: \"365a63f7-ce01-4701-95b3-fd3862930ca6\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9b2dc"
Nov 02 19:36:30.812206 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:30.812173    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/365a63f7-ce01-4701-95b3-fd3862930ca6-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9b2dc\" (UID: \"365a63f7-ce01-4701-95b3-fd3862930ca6\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9b2dc"
Nov 02 19:36:30.813805 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:30.813788    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/365a63f7-ce01-4701-95b3-fd3862930ca6-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9b2dc\" (UID: \"365a63f7-ce01-4701-95b3-fd3862930ca6\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9b2dc"
Nov 02 19:36:30.813949 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:30.813936    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/365a63f7-ce01-4701-95b3-fd3862930ca6-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9b2dc\" (UID: \"365a63f7-ce01-4701-95b3-fd3862930ca6\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9b2dc"
Nov 02 19:36:30.864794 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:30.864776    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxb2g\" (UniqueName: \"kubernetes.io/projected/365a63f7-ce01-4701-95b3-fd3862930ca6-kube-api-access-hxb2g\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9b2dc\" (UID: \"365a63f7-ce01-4701-95b3-fd3862930ca6\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9b2dc"
Nov 02 19:36:30.931396 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:30.931367    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9b2dc"
Nov 02 19:36:31.291988 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:31.291919    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9b2dc"]
Nov 02 19:36:31.294626 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:36:31.294594    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod365a63f7_ce01_4701_95b3_fd3862930ca6.slice/crio-b45b6fff63e9cf7906687c81e68cff2c887012e0604c594421b6b6133bc4f29b WatchSource:0}: Error finding container b45b6fff63e9cf7906687c81e68cff2c887012e0604c594421b6b6133bc4f29b: Status 404 returned error can't find the container with id b45b6fff63e9cf7906687c81e68cff2c887012e0604c594421b6b6133bc4f29b
Nov 02 19:36:31.445872 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:31.445839    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9b2dc" event={"ID":"365a63f7-ce01-4701-95b3-fd3862930ca6","Type":"ContainerStarted","Data":"b45b6fff63e9cf7906687c81e68cff2c887012e0604c594421b6b6133bc4f29b"}
Nov 02 19:36:33.990461 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:33.987427    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 19:36:34.214617 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:34.214580    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-ffswj"]
Nov 02 19:36:34.217918 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:34.217901    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ffswj"
Nov 02 19:36:34.297439 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:36:34.297352    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 19:36:34.313289 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:34.313264    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Nov 02 19:36:34.314660 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:34.314640    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-ffswj"]
Nov 02 19:36:34.343126 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:34.343102    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf00bfe7-3062-4f29-a243-367a3e6c13b7-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-ffswj\" (UID: \"bf00bfe7-3062-4f29-a243-367a3e6c13b7\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ffswj"
Nov 02 19:36:34.343229 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:34.343129    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf00bfe7-3062-4f29-a243-367a3e6c13b7-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-ffswj\" (UID: \"bf00bfe7-3062-4f29-a243-367a3e6c13b7\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ffswj"
Nov 02 19:36:34.343229 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:34.343153    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf00bfe7-3062-4f29-a243-367a3e6c13b7-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-ffswj\" (UID: \"bf00bfe7-3062-4f29-a243-367a3e6c13b7\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ffswj"
Nov 02 19:36:34.343301 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:34.343234    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf00bfe7-3062-4f29-a243-367a3e6c13b7-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-ffswj\" (UID: \"bf00bfe7-3062-4f29-a243-367a3e6c13b7\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ffswj"
Nov 02 19:36:34.343301 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:34.343257    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzx9x\" (UniqueName: \"kubernetes.io/projected/bf00bfe7-3062-4f29-a243-367a3e6c13b7-kube-api-access-pzx9x\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-ffswj\" (UID: \"bf00bfe7-3062-4f29-a243-367a3e6c13b7\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ffswj"
Nov 02 19:36:34.343397 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:34.343303    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf00bfe7-3062-4f29-a243-367a3e6c13b7-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-ffswj\" (UID: \"bf00bfe7-3062-4f29-a243-367a3e6c13b7\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ffswj"
Nov 02 19:36:34.444393 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:34.444365    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf00bfe7-3062-4f29-a243-367a3e6c13b7-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-ffswj\" (UID: \"bf00bfe7-3062-4f29-a243-367a3e6c13b7\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ffswj"
Nov 02 19:36:34.444485 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:34.444409    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf00bfe7-3062-4f29-a243-367a3e6c13b7-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-ffswj\" (UID: \"bf00bfe7-3062-4f29-a243-367a3e6c13b7\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ffswj"
Nov 02 19:36:34.444485 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:34.444429    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf00bfe7-3062-4f29-a243-367a3e6c13b7-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-ffswj\" (UID: \"bf00bfe7-3062-4f29-a243-367a3e6c13b7\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ffswj"
Nov 02 19:36:34.444485 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:34.444451    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf00bfe7-3062-4f29-a243-367a3e6c13b7-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-ffswj\" (UID: \"bf00bfe7-3062-4f29-a243-367a3e6c13b7\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ffswj"
Nov 02 19:36:34.444485 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:34.444478    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf00bfe7-3062-4f29-a243-367a3e6c13b7-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-ffswj\" (UID: \"bf00bfe7-3062-4f29-a243-367a3e6c13b7\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ffswj"
Nov 02 19:36:34.444674 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:34.444493    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzx9x\" (UniqueName: \"kubernetes.io/projected/bf00bfe7-3062-4f29-a243-367a3e6c13b7-kube-api-access-pzx9x\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-ffswj\" (UID: \"bf00bfe7-3062-4f29-a243-367a3e6c13b7\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ffswj"
Nov 02 19:36:34.444884 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:34.444856    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf00bfe7-3062-4f29-a243-367a3e6c13b7-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-ffswj\" (UID: \"bf00bfe7-3062-4f29-a243-367a3e6c13b7\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ffswj"
Nov 02 19:36:34.444998 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:34.444887    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf00bfe7-3062-4f29-a243-367a3e6c13b7-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-ffswj\" (UID: \"bf00bfe7-3062-4f29-a243-367a3e6c13b7\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ffswj"
Nov 02 19:36:34.444998 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:34.444952    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf00bfe7-3062-4f29-a243-367a3e6c13b7-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-ffswj\" (UID: \"bf00bfe7-3062-4f29-a243-367a3e6c13b7\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ffswj"
Nov 02 19:36:34.446560 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:34.446543    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf00bfe7-3062-4f29-a243-367a3e6c13b7-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-ffswj\" (UID: \"bf00bfe7-3062-4f29-a243-367a3e6c13b7\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ffswj"
Nov 02 19:36:34.446729 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:34.446712    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf00bfe7-3062-4f29-a243-367a3e6c13b7-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-ffswj\" (UID: \"bf00bfe7-3062-4f29-a243-367a3e6c13b7\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ffswj"
Nov 02 19:36:34.458327 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:34.458301    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9b2dc" event={"ID":"365a63f7-ce01-4701-95b3-fd3862930ca6","Type":"ContainerStarted","Data":"ecca8684b529aa513da96ef66726d0c5696bc3be8b792342e077de4474cb9ec7"}
Nov 02 19:36:34.497686 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:34.497660    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzx9x\" (UniqueName: \"kubernetes.io/projected/bf00bfe7-3062-4f29-a243-367a3e6c13b7-kube-api-access-pzx9x\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-ffswj\" (UID: \"bf00bfe7-3062-4f29-a243-367a3e6c13b7\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ffswj"
Nov 02 19:36:34.529577 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:34.529553    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ffswj"
Nov 02 19:36:34.564213 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:34.564118    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9b2dc" podStartSLOduration=2.18161103 podStartE2EDuration="4.564104972s" totalImagesPullingTime="2.382493942s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:36:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:36:34.564054077 +0000 UTC m=+501.171087977" watchObservedRunningTime="2026-11-02 19:36:34.564104972 +0000 UTC m=+501.171138743"
Nov 02 19:36:34.888781 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:34.888714    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-ffswj"]
Nov 02 19:36:34.891823 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:36:34.891788    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf00bfe7_3062_4f29_a243_367a3e6c13b7.slice/crio-ae53e3319f479bc0b990ea065af3f8d616b2cbb0b9625ddff4fb13cf925ddba0 WatchSource:0}: Error finding container ae53e3319f479bc0b990ea065af3f8d616b2cbb0b9625ddff4fb13cf925ddba0: Status 404 returned error can't find the container with id ae53e3319f479bc0b990ea065af3f8d616b2cbb0b9625ddff4fb13cf925ddba0
Nov 02 19:36:34.931668 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:34.931644    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9b2dc"
Nov 02 19:36:35.464067 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:35.464035    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ffswj" event={"ID":"bf00bfe7-3062-4f29-a243-367a3e6c13b7","Type":"ContainerStarted","Data":"ae53e3319f479bc0b990ea065af3f8d616b2cbb0b9625ddff4fb13cf925ddba0"}
Nov 02 19:36:36.469024 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:36.468984    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ffswj" event={"ID":"bf00bfe7-3062-4f29-a243-367a3e6c13b7","Type":"ContainerStarted","Data":"300ff87dfa1b923a928493055726c65fded109e95b49334821e604b76fdc571a"}
Nov 02 19:36:36.530238 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:36.530203    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ffswj"
Nov 02 19:36:36.576284 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:36.576227    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ffswj" podStartSLOduration=2.027552719 podStartE2EDuration="2.576212779s" totalImagesPullingTime="548.66006ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:36:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:36:36.575591469 +0000 UTC m=+503.182625238" watchObservedRunningTime="2026-11-02 19:36:36.576212779 +0000 UTC m=+503.183246544"
Nov 02 19:36:40.932580 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:40.932542    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9b2dc"
Nov 02 19:36:40.944870 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:40.944840    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9b2dc"
Nov 02 19:36:41.052694 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:41.052663    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-kjt8s"]
Nov 02 19:36:41.055961 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:41.055946    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kjt8s"
Nov 02 19:36:41.106257 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:41.106221    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6536ec7e-0d3f-42f2-9e97-180f1e0a35a7-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-kjt8s\" (UID: \"6536ec7e-0d3f-42f2-9e97-180f1e0a35a7\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kjt8s"
Nov 02 19:36:41.106422 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:41.106264    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6536ec7e-0d3f-42f2-9e97-180f1e0a35a7-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-kjt8s\" (UID: \"6536ec7e-0d3f-42f2-9e97-180f1e0a35a7\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kjt8s"
Nov 02 19:36:41.106422 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:41.106292    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Nov 02 19:36:41.106422 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:41.106320    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqmcc\" (UniqueName: \"kubernetes.io/projected/6536ec7e-0d3f-42f2-9e97-180f1e0a35a7-kube-api-access-kqmcc\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-kjt8s\" (UID: \"6536ec7e-0d3f-42f2-9e97-180f1e0a35a7\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kjt8s"
Nov 02 19:36:41.106650 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:41.106426    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6536ec7e-0d3f-42f2-9e97-180f1e0a35a7-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-kjt8s\" (UID: \"6536ec7e-0d3f-42f2-9e97-180f1e0a35a7\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kjt8s"
Nov 02 19:36:41.106650 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:41.106550    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6536ec7e-0d3f-42f2-9e97-180f1e0a35a7-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-kjt8s\" (UID: \"6536ec7e-0d3f-42f2-9e97-180f1e0a35a7\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kjt8s"
Nov 02 19:36:41.106650 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:41.106600    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6536ec7e-0d3f-42f2-9e97-180f1e0a35a7-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-kjt8s\" (UID: \"6536ec7e-0d3f-42f2-9e97-180f1e0a35a7\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kjt8s"
Nov 02 19:36:41.149461 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:41.149424    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-kjt8s"]
Nov 02 19:36:41.207741 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:41.207668    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqmcc\" (UniqueName: \"kubernetes.io/projected/6536ec7e-0d3f-42f2-9e97-180f1e0a35a7-kube-api-access-kqmcc\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-kjt8s\" (UID: \"6536ec7e-0d3f-42f2-9e97-180f1e0a35a7\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kjt8s"
Nov 02 19:36:41.207741 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:41.207708    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6536ec7e-0d3f-42f2-9e97-180f1e0a35a7-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-kjt8s\" (UID: \"6536ec7e-0d3f-42f2-9e97-180f1e0a35a7\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kjt8s"
Nov 02 19:36:41.207937 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:41.207768    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6536ec7e-0d3f-42f2-9e97-180f1e0a35a7-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-kjt8s\" (UID: \"6536ec7e-0d3f-42f2-9e97-180f1e0a35a7\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kjt8s"
Nov 02 19:36:41.207937 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:41.207791    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6536ec7e-0d3f-42f2-9e97-180f1e0a35a7-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-kjt8s\" (UID: \"6536ec7e-0d3f-42f2-9e97-180f1e0a35a7\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kjt8s"
Nov 02 19:36:41.207937 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:41.207818    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6536ec7e-0d3f-42f2-9e97-180f1e0a35a7-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-kjt8s\" (UID: \"6536ec7e-0d3f-42f2-9e97-180f1e0a35a7\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kjt8s"
Nov 02 19:36:41.207937 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:41.207836    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6536ec7e-0d3f-42f2-9e97-180f1e0a35a7-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-kjt8s\" (UID: \"6536ec7e-0d3f-42f2-9e97-180f1e0a35a7\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kjt8s"
Nov 02 19:36:41.208303 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:41.208278    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6536ec7e-0d3f-42f2-9e97-180f1e0a35a7-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-kjt8s\" (UID: \"6536ec7e-0d3f-42f2-9e97-180f1e0a35a7\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kjt8s"
Nov 02 19:36:41.208414 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:41.208317    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6536ec7e-0d3f-42f2-9e97-180f1e0a35a7-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-kjt8s\" (UID: \"6536ec7e-0d3f-42f2-9e97-180f1e0a35a7\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kjt8s"
Nov 02 19:36:41.208414 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:41.208335    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6536ec7e-0d3f-42f2-9e97-180f1e0a35a7-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-kjt8s\" (UID: \"6536ec7e-0d3f-42f2-9e97-180f1e0a35a7\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kjt8s"
Nov 02 19:36:41.210143 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:41.210119    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6536ec7e-0d3f-42f2-9e97-180f1e0a35a7-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-kjt8s\" (UID: \"6536ec7e-0d3f-42f2-9e97-180f1e0a35a7\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kjt8s"
Nov 02 19:36:41.210310 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:41.210294    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6536ec7e-0d3f-42f2-9e97-180f1e0a35a7-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-kjt8s\" (UID: \"6536ec7e-0d3f-42f2-9e97-180f1e0a35a7\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kjt8s"
Nov 02 19:36:41.260466 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:41.260441    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqmcc\" (UniqueName: \"kubernetes.io/projected/6536ec7e-0d3f-42f2-9e97-180f1e0a35a7-kube-api-access-kqmcc\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-kjt8s\" (UID: \"6536ec7e-0d3f-42f2-9e97-180f1e0a35a7\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kjt8s"
Nov 02 19:36:41.367578 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:41.367551    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kjt8s"
Nov 02 19:36:41.500107 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:41.500031    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9b2dc"
Nov 02 19:36:41.730353 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:41.730322    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-kjt8s"]
Nov 02 19:36:41.733492 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:36:41.733466    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6536ec7e_0d3f_42f2_9e97_180f1e0a35a7.slice/crio-f303735f15058395620ba3c943ef1272ea782ca7741ce1789d88a1c629cf27bb WatchSource:0}: Error finding container f303735f15058395620ba3c943ef1272ea782ca7741ce1789d88a1c629cf27bb: Status 404 returned error can't find the container with id f303735f15058395620ba3c943ef1272ea782ca7741ce1789d88a1c629cf27bb
Nov 02 19:36:42.495051 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:42.495000    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kjt8s" event={"ID":"6536ec7e-0d3f-42f2-9e97-180f1e0a35a7","Type":"ContainerStarted","Data":"4505d48ea5d47237a7c2f325afad6de1d7d3306a57abb4199902e132506f1c91"}
Nov 02 19:36:42.495497 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:42.495062    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kjt8s" event={"ID":"6536ec7e-0d3f-42f2-9e97-180f1e0a35a7","Type":"ContainerStarted","Data":"f303735f15058395620ba3c943ef1272ea782ca7741ce1789d88a1c629cf27bb"}
Nov 02 19:36:42.603207 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:42.603160    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kjt8s" podStartSLOduration=0.970767586 podStartE2EDuration="1.603145947s" totalImagesPullingTime="632.378361ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:36:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:36:42.60181512 +0000 UTC m=+509.208848929" watchObservedRunningTime="2026-11-02 19:36:42.603145947 +0000 UTC m=+509.210179716"
Nov 02 19:36:43.368003 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:43.367970    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kjt8s"
Nov 02 19:36:44.530070 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:44.530037    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ffswj"
Nov 02 19:36:44.542520 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:44.542495    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ffswj"
Nov 02 19:36:45.521867 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:45.521834    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ffswj"
Nov 02 19:36:47.459634 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:47.459586    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 19:36:48.374203 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:48.374168    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-cthc4"]
Nov 02 19:36:48.377667 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:48.377648    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-cthc4"
Nov 02 19:36:48.469055 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:48.469000    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27b07c53-9c7a-4cc1-a0cd-8a9cf982d73c-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-cthc4\" (UID: \"27b07c53-9c7a-4cc1-a0cd-8a9cf982d73c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-cthc4"
Nov 02 19:36:48.469479 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:48.469076    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27b07c53-9c7a-4cc1-a0cd-8a9cf982d73c-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-cthc4\" (UID: \"27b07c53-9c7a-4cc1-a0cd-8a9cf982d73c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-cthc4"
Nov 02 19:36:48.469479 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:48.469168    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27b07c53-9c7a-4cc1-a0cd-8a9cf982d73c-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-cthc4\" (UID: \"27b07c53-9c7a-4cc1-a0cd-8a9cf982d73c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-cthc4"
Nov 02 19:36:48.469479 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:48.469190    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krtb8\" (UniqueName: \"kubernetes.io/projected/27b07c53-9c7a-4cc1-a0cd-8a9cf982d73c-kube-api-access-krtb8\") pod \"echo-simulator-kserve-64b6f8f96c-cthc4\" (UID: \"27b07c53-9c7a-4cc1-a0cd-8a9cf982d73c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-cthc4"
Nov 02 19:36:48.469479 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:48.469236    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27b07c53-9c7a-4cc1-a0cd-8a9cf982d73c-home\") pod \"echo-simulator-kserve-64b6f8f96c-cthc4\" (UID: \"27b07c53-9c7a-4cc1-a0cd-8a9cf982d73c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-cthc4"
Nov 02 19:36:48.469479 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:48.469309    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27b07c53-9c7a-4cc1-a0cd-8a9cf982d73c-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-cthc4\" (UID: \"27b07c53-9c7a-4cc1-a0cd-8a9cf982d73c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-cthc4"
Nov 02 19:36:48.472194 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:48.472177    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\""
Nov 02 19:36:48.473604 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:48.473582    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-cthc4"]
Nov 02 19:36:48.570211 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:48.570181    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27b07c53-9c7a-4cc1-a0cd-8a9cf982d73c-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-cthc4\" (UID: \"27b07c53-9c7a-4cc1-a0cd-8a9cf982d73c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-cthc4"
Nov 02 19:36:48.570211 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:48.570217    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27b07c53-9c7a-4cc1-a0cd-8a9cf982d73c-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-cthc4\" (UID: \"27b07c53-9c7a-4cc1-a0cd-8a9cf982d73c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-cthc4"
Nov 02 19:36:48.570403 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:48.570251    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27b07c53-9c7a-4cc1-a0cd-8a9cf982d73c-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-cthc4\" (UID: \"27b07c53-9c7a-4cc1-a0cd-8a9cf982d73c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-cthc4"
Nov 02 19:36:48.570403 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:48.570278    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krtb8\" (UniqueName: \"kubernetes.io/projected/27b07c53-9c7a-4cc1-a0cd-8a9cf982d73c-kube-api-access-krtb8\") pod \"echo-simulator-kserve-64b6f8f96c-cthc4\" (UID: \"27b07c53-9c7a-4cc1-a0cd-8a9cf982d73c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-cthc4"
Nov 02 19:36:48.570403 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:48.570336    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27b07c53-9c7a-4cc1-a0cd-8a9cf982d73c-home\") pod \"echo-simulator-kserve-64b6f8f96c-cthc4\" (UID: \"27b07c53-9c7a-4cc1-a0cd-8a9cf982d73c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-cthc4"
Nov 02 19:36:48.570403 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:48.570359    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27b07c53-9c7a-4cc1-a0cd-8a9cf982d73c-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-cthc4\" (UID: \"27b07c53-9c7a-4cc1-a0cd-8a9cf982d73c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-cthc4"
Nov 02 19:36:48.570766 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:48.570627    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27b07c53-9c7a-4cc1-a0cd-8a9cf982d73c-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-cthc4\" (UID: \"27b07c53-9c7a-4cc1-a0cd-8a9cf982d73c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-cthc4"
Nov 02 19:36:48.570766 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:48.570694    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27b07c53-9c7a-4cc1-a0cd-8a9cf982d73c-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-cthc4\" (UID: \"27b07c53-9c7a-4cc1-a0cd-8a9cf982d73c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-cthc4"
Nov 02 19:36:48.570766 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:48.570732    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27b07c53-9c7a-4cc1-a0cd-8a9cf982d73c-home\") pod \"echo-simulator-kserve-64b6f8f96c-cthc4\" (UID: \"27b07c53-9c7a-4cc1-a0cd-8a9cf982d73c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-cthc4"
Nov 02 19:36:48.572476 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:48.572455    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27b07c53-9c7a-4cc1-a0cd-8a9cf982d73c-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-cthc4\" (UID: \"27b07c53-9c7a-4cc1-a0cd-8a9cf982d73c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-cthc4"
Nov 02 19:36:48.572768 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:48.572749    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27b07c53-9c7a-4cc1-a0cd-8a9cf982d73c-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-cthc4\" (UID: \"27b07c53-9c7a-4cc1-a0cd-8a9cf982d73c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-cthc4"
Nov 02 19:36:48.627091 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:48.627026    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krtb8\" (UniqueName: \"kubernetes.io/projected/27b07c53-9c7a-4cc1-a0cd-8a9cf982d73c-kube-api-access-krtb8\") pod \"echo-simulator-kserve-64b6f8f96c-cthc4\" (UID: \"27b07c53-9c7a-4cc1-a0cd-8a9cf982d73c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-cthc4"
Nov 02 19:36:48.689701 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:48.689669    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-cthc4"
Nov 02 19:36:49.050668 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:49.050636    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-cthc4"]
Nov 02 19:36:49.053723 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:36:49.053695    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27b07c53_9c7a_4cc1_a0cd_8a9cf982d73c.slice/crio-c1da7918ef789273d01ca01a0428ae375b3806def3a4e6000bed718169e4df7b WatchSource:0}: Error finding container c1da7918ef789273d01ca01a0428ae375b3806def3a4e6000bed718169e4df7b: Status 404 returned error can't find the container with id c1da7918ef789273d01ca01a0428ae375b3806def3a4e6000bed718169e4df7b
Nov 02 19:36:49.528175 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:49.528141    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-cthc4" event={"ID":"27b07c53-9c7a-4cc1-a0cd-8a9cf982d73c","Type":"ContainerStarted","Data":"c1da7918ef789273d01ca01a0428ae375b3806def3a4e6000bed718169e4df7b"}
Nov 02 19:36:51.368523 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:51.368495    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kjt8s"
Nov 02 19:36:51.383074 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:51.383048    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kjt8s"
Nov 02 19:36:51.547107 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:51.547073    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kjt8s"
Nov 02 19:36:52.114820 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:52.114794    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-f8bw8"]
Nov 02 19:36:52.118094 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:52.118075    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-f8bw8"
Nov 02 19:36:52.167127 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:52.167099    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Nov 02 19:36:52.192789 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:52.192761    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-f8bw8"]
Nov 02 19:36:52.201863 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:52.201838    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98da7e93-4099-4f42-bcfb-8d820ee124db-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-f8bw8\" (UID: \"98da7e93-4099-4f42-bcfb-8d820ee124db\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-f8bw8"
Nov 02 19:36:52.201992 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:52.201878    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98da7e93-4099-4f42-bcfb-8d820ee124db-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-f8bw8\" (UID: \"98da7e93-4099-4f42-bcfb-8d820ee124db\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-f8bw8"
Nov 02 19:36:52.202069 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:52.201986    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsv8m\" (UniqueName: \"kubernetes.io/projected/98da7e93-4099-4f42-bcfb-8d820ee124db-kube-api-access-lsv8m\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-f8bw8\" (UID: \"98da7e93-4099-4f42-bcfb-8d820ee124db\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-f8bw8"
Nov 02 19:36:52.202115 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:52.202062    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98da7e93-4099-4f42-bcfb-8d820ee124db-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-f8bw8\" (UID: \"98da7e93-4099-4f42-bcfb-8d820ee124db\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-f8bw8"
Nov 02 19:36:52.202115 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:52.202089    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98da7e93-4099-4f42-bcfb-8d820ee124db-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-f8bw8\" (UID: \"98da7e93-4099-4f42-bcfb-8d820ee124db\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-f8bw8"
Nov 02 19:36:52.202115 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:52.202108    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98da7e93-4099-4f42-bcfb-8d820ee124db-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-f8bw8\" (UID: \"98da7e93-4099-4f42-bcfb-8d820ee124db\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-f8bw8"
Nov 02 19:36:52.303561 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:52.303476    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98da7e93-4099-4f42-bcfb-8d820ee124db-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-f8bw8\" (UID: \"98da7e93-4099-4f42-bcfb-8d820ee124db\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-f8bw8"
Nov 02 19:36:52.303561 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:52.303540    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsv8m\" (UniqueName: \"kubernetes.io/projected/98da7e93-4099-4f42-bcfb-8d820ee124db-kube-api-access-lsv8m\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-f8bw8\" (UID: \"98da7e93-4099-4f42-bcfb-8d820ee124db\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-f8bw8"
Nov 02 19:36:52.303757 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:52.303573    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98da7e93-4099-4f42-bcfb-8d820ee124db-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-f8bw8\" (UID: \"98da7e93-4099-4f42-bcfb-8d820ee124db\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-f8bw8"
Nov 02 19:36:52.303757 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:52.303603    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98da7e93-4099-4f42-bcfb-8d820ee124db-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-f8bw8\" (UID: \"98da7e93-4099-4f42-bcfb-8d820ee124db\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-f8bw8"
Nov 02 19:36:52.303757 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:52.303630    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98da7e93-4099-4f42-bcfb-8d820ee124db-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-f8bw8\" (UID: \"98da7e93-4099-4f42-bcfb-8d820ee124db\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-f8bw8"
Nov 02 19:36:52.303757 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:52.303689    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98da7e93-4099-4f42-bcfb-8d820ee124db-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-f8bw8\" (UID: \"98da7e93-4099-4f42-bcfb-8d820ee124db\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-f8bw8"
Nov 02 19:36:52.304090 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:52.304064    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98da7e93-4099-4f42-bcfb-8d820ee124db-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-f8bw8\" (UID: \"98da7e93-4099-4f42-bcfb-8d820ee124db\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-f8bw8"
Nov 02 19:36:52.304218 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:52.304068    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98da7e93-4099-4f42-bcfb-8d820ee124db-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-f8bw8\" (UID: \"98da7e93-4099-4f42-bcfb-8d820ee124db\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-f8bw8"
Nov 02 19:36:52.304218 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:52.304144    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98da7e93-4099-4f42-bcfb-8d820ee124db-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-f8bw8\" (UID: \"98da7e93-4099-4f42-bcfb-8d820ee124db\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-f8bw8"
Nov 02 19:36:52.305803 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:52.305781    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98da7e93-4099-4f42-bcfb-8d820ee124db-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-f8bw8\" (UID: \"98da7e93-4099-4f42-bcfb-8d820ee124db\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-f8bw8"
Nov 02 19:36:52.305963 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:52.305945    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98da7e93-4099-4f42-bcfb-8d820ee124db-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-f8bw8\" (UID: \"98da7e93-4099-4f42-bcfb-8d820ee124db\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-f8bw8"
Nov 02 19:36:52.358392 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:52.358363    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsv8m\" (UniqueName: \"kubernetes.io/projected/98da7e93-4099-4f42-bcfb-8d820ee124db-kube-api-access-lsv8m\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-f8bw8\" (UID: \"98da7e93-4099-4f42-bcfb-8d820ee124db\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-f8bw8"
Nov 02 19:36:52.431417 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:52.431385    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-f8bw8"
Nov 02 19:36:52.542386 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:52.542350    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-cthc4" event={"ID":"27b07c53-9c7a-4cc1-a0cd-8a9cf982d73c","Type":"ContainerStarted","Data":"937b0465974ac30c4ec7c99bfbd18b0aac95e68943b28a23767184295f30c1f4"}
Nov 02 19:36:52.649805 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:52.649706    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-cthc4" podStartSLOduration=1.6609558610000001 podStartE2EDuration="4.649690662s" totalImagesPullingTime="2.988734801s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:36:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:36:52.649205065 +0000 UTC m=+519.256238852" watchObservedRunningTime="2026-11-02 19:36:52.649690662 +0000 UTC m=+519.256724478"
Nov 02 19:36:52.689925 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:52.689898    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-cthc4"
Nov 02 19:36:52.791460 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:52.791440    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-f8bw8"]
Nov 02 19:36:52.793934 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:36:52.793913    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98da7e93_4099_4f42_bcfb_8d820ee124db.slice/crio-3ae388988ced816a9d90e70a17dc76bf23068d52ddbbea56a6d9973a6523dd1b WatchSource:0}: Error finding container 3ae388988ced816a9d90e70a17dc76bf23068d52ddbbea56a6d9973a6523dd1b: Status 404 returned error can't find the container with id 3ae388988ced816a9d90e70a17dc76bf23068d52ddbbea56a6d9973a6523dd1b
Nov 02 19:36:53.548413 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:53.548372    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-f8bw8" event={"ID":"98da7e93-4099-4f42-bcfb-8d820ee124db","Type":"ContainerStarted","Data":"84787a6413764f50d804d715ffacbfb2cc4434a64c01fbbae0c1e36123aa2402"}
Nov 02 19:36:53.548801 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:53.548425    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-f8bw8" event={"ID":"98da7e93-4099-4f42-bcfb-8d820ee124db","Type":"ContainerStarted","Data":"3ae388988ced816a9d90e70a17dc76bf23068d52ddbbea56a6d9973a6523dd1b"}
Nov 02 19:36:53.655312 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:53.655201    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-f8bw8" podStartSLOduration=1.083483354 podStartE2EDuration="1.655185402s" totalImagesPullingTime="571.702048ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:36:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:36:53.654530604 +0000 UTC m=+520.261564377" watchObservedRunningTime="2026-11-02 19:36:53.655185402 +0000 UTC m=+520.262219174"
Nov 02 19:36:54.432057 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:54.432006    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-f8bw8"
Nov 02 19:36:58.690587 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:58.690554    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-cthc4"
Nov 02 19:36:58.702710 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:58.702680    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-cthc4"
Nov 02 19:36:59.583938 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:36:59.583909    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-cthc4"
Nov 02 19:37:01.110157 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:01.110124    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6v2d5"]
Nov 02 19:37:01.113546 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:01.113522    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6v2d5"
Nov 02 19:37:01.162327 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:01.162298    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-qmwl4\""
Nov 02 19:37:01.162459 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:01.162425    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 02 19:37:01.211315 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:01.211283    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6v2d5"]
Nov 02 19:37:01.280121 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:01.280093    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14de05d8-f8d6-4ab0-8671-faa1158ca274-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6v2d5\" (UID: \"14de05d8-f8d6-4ab0-8671-faa1158ca274\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6v2d5"
Nov 02 19:37:01.280283 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:01.280139    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14de05d8-f8d6-4ab0-8671-faa1158ca274-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6v2d5\" (UID: \"14de05d8-f8d6-4ab0-8671-faa1158ca274\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6v2d5"
Nov 02 19:37:01.280283 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:01.280159    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14de05d8-f8d6-4ab0-8671-faa1158ca274-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6v2d5\" (UID: \"14de05d8-f8d6-4ab0-8671-faa1158ca274\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6v2d5"
Nov 02 19:37:01.280283 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:01.280275    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14de05d8-f8d6-4ab0-8671-faa1158ca274-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6v2d5\" (UID: \"14de05d8-f8d6-4ab0-8671-faa1158ca274\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6v2d5"
Nov 02 19:37:01.280425 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:01.280298    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14de05d8-f8d6-4ab0-8671-faa1158ca274-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6v2d5\" (UID: \"14de05d8-f8d6-4ab0-8671-faa1158ca274\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6v2d5"
Nov 02 19:37:01.280425 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:01.280323    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zpsc\" (UniqueName: \"kubernetes.io/projected/14de05d8-f8d6-4ab0-8671-faa1158ca274-kube-api-access-7zpsc\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6v2d5\" (UID: \"14de05d8-f8d6-4ab0-8671-faa1158ca274\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6v2d5"
Nov 02 19:37:01.280523 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:01.280450    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14de05d8-f8d6-4ab0-8671-faa1158ca274-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6v2d5\" (UID: \"14de05d8-f8d6-4ab0-8671-faa1158ca274\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6v2d5"
Nov 02 19:37:01.280523 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:01.280503    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14de05d8-f8d6-4ab0-8671-faa1158ca274-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6v2d5\" (UID: \"14de05d8-f8d6-4ab0-8671-faa1158ca274\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6v2d5"
Nov 02 19:37:01.280594 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:01.280561    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14de05d8-f8d6-4ab0-8671-faa1158ca274-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6v2d5\" (UID: \"14de05d8-f8d6-4ab0-8671-faa1158ca274\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6v2d5"
Nov 02 19:37:01.381147 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:01.381072    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zpsc\" (UniqueName: \"kubernetes.io/projected/14de05d8-f8d6-4ab0-8671-faa1158ca274-kube-api-access-7zpsc\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6v2d5\" (UID: \"14de05d8-f8d6-4ab0-8671-faa1158ca274\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6v2d5"
Nov 02 19:37:01.381147 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:01.381107    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14de05d8-f8d6-4ab0-8671-faa1158ca274-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6v2d5\" (UID: \"14de05d8-f8d6-4ab0-8671-faa1158ca274\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6v2d5"
Nov 02 19:37:01.381147 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:01.381137    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14de05d8-f8d6-4ab0-8671-faa1158ca274-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6v2d5\" (UID: \"14de05d8-f8d6-4ab0-8671-faa1158ca274\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6v2d5"
Nov 02 19:37:01.381404 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:01.381166    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14de05d8-f8d6-4ab0-8671-faa1158ca274-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6v2d5\" (UID: \"14de05d8-f8d6-4ab0-8671-faa1158ca274\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6v2d5"
Nov 02 19:37:01.381404 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:01.381200    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14de05d8-f8d6-4ab0-8671-faa1158ca274-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6v2d5\" (UID: \"14de05d8-f8d6-4ab0-8671-faa1158ca274\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6v2d5"
Nov 02 19:37:01.381404 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:01.381256    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14de05d8-f8d6-4ab0-8671-faa1158ca274-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6v2d5\" (UID: \"14de05d8-f8d6-4ab0-8671-faa1158ca274\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6v2d5"
Nov 02 19:37:01.381404 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:01.381283    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14de05d8-f8d6-4ab0-8671-faa1158ca274-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6v2d5\" (UID: \"14de05d8-f8d6-4ab0-8671-faa1158ca274\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6v2d5"
Nov 02 19:37:01.381404 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:01.381313    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14de05d8-f8d6-4ab0-8671-faa1158ca274-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6v2d5\" (UID: \"14de05d8-f8d6-4ab0-8671-faa1158ca274\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6v2d5"
Nov 02 19:37:01.381404 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:01.381342    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14de05d8-f8d6-4ab0-8671-faa1158ca274-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6v2d5\" (UID: \"14de05d8-f8d6-4ab0-8671-faa1158ca274\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6v2d5"
Nov 02 19:37:01.381699 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:01.381615    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14de05d8-f8d6-4ab0-8671-faa1158ca274-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6v2d5\" (UID: \"14de05d8-f8d6-4ab0-8671-faa1158ca274\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6v2d5"
Nov 02 19:37:01.381699 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:01.381663    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14de05d8-f8d6-4ab0-8671-faa1158ca274-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6v2d5\" (UID: \"14de05d8-f8d6-4ab0-8671-faa1158ca274\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6v2d5"
Nov 02 19:37:01.381768 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:01.381741    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14de05d8-f8d6-4ab0-8671-faa1158ca274-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6v2d5\" (UID: \"14de05d8-f8d6-4ab0-8671-faa1158ca274\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6v2d5"
Nov 02 19:37:01.381891 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:01.381871    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14de05d8-f8d6-4ab0-8671-faa1158ca274-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6v2d5\" (UID: \"14de05d8-f8d6-4ab0-8671-faa1158ca274\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6v2d5"
Nov 02 19:37:01.381984 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:01.381963    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14de05d8-f8d6-4ab0-8671-faa1158ca274-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6v2d5\" (UID: \"14de05d8-f8d6-4ab0-8671-faa1158ca274\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6v2d5"
Nov 02 19:37:01.383702 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:01.383675    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14de05d8-f8d6-4ab0-8671-faa1158ca274-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6v2d5\" (UID: \"14de05d8-f8d6-4ab0-8671-faa1158ca274\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6v2d5"
Nov 02 19:37:01.383802 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:01.383788    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14de05d8-f8d6-4ab0-8671-faa1158ca274-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6v2d5\" (UID: \"14de05d8-f8d6-4ab0-8671-faa1158ca274\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6v2d5"
Nov 02 19:37:01.434447 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:01.434410    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14de05d8-f8d6-4ab0-8671-faa1158ca274-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6v2d5\" (UID: \"14de05d8-f8d6-4ab0-8671-faa1158ca274\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6v2d5"
Nov 02 19:37:01.434612 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:01.434592    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zpsc\" (UniqueName: \"kubernetes.io/projected/14de05d8-f8d6-4ab0-8671-faa1158ca274-kube-api-access-7zpsc\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6v2d5\" (UID: \"14de05d8-f8d6-4ab0-8671-faa1158ca274\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6v2d5"
Nov 02 19:37:01.725336 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:01.725308    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6v2d5"
Nov 02 19:37:02.136493 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:02.136449    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6v2d5"]
Nov 02 19:37:02.139100 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:37:02.139070    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14de05d8_f8d6_4ab0_8671_faa1158ca274.slice/crio-70ad73cfe5853ea47cf1f6b01f7b09a8f9915981a1382b9be4c902d6c64c82c5 WatchSource:0}: Error finding container 70ad73cfe5853ea47cf1f6b01f7b09a8f9915981a1382b9be4c902d6c64c82c5: Status 404 returned error can't find the container with id 70ad73cfe5853ea47cf1f6b01f7b09a8f9915981a1382b9be4c902d6c64c82c5
Nov 02 19:37:02.432271 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:02.432243    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-f8bw8"
Nov 02 19:37:02.444483 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:02.444458    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-f8bw8"
Nov 02 19:37:02.587455 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:02.587422    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6v2d5" event={"ID":"14de05d8-f8d6-4ab0-8671-faa1158ca274","Type":"ContainerStarted","Data":"70ad73cfe5853ea47cf1f6b01f7b09a8f9915981a1382b9be4c902d6c64c82c5"}
Nov 02 19:37:02.598762 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:02.598739    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-f8bw8"
Nov 02 19:37:03.530790 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:03.530750    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 19:37:04.304215 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:37:04.304167    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 19:37:06.118829 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:06.118794    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:37:06.119233 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:06.118873    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:37:06.119233 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:06.118918    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:37:06.607521 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:06.607481    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6v2d5" event={"ID":"14de05d8-f8d6-4ab0-8671-faa1158ca274","Type":"ContainerStarted","Data":"f0786da9edfe1c2555abfd654fbb2f3e30c7c610764dc3063b8fc212f4d6cecd"}
Nov 02 19:37:06.726259 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:06.726227    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6v2d5"
Nov 02 19:37:06.774040 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:06.765054    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6v2d5" podStartSLOduration=1.788041589 podStartE2EDuration="5.765033987s" totalImagesPullingTime="3.976992398s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:37:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:37:06.716526293 +0000 UTC m=+533.323560061" watchObservedRunningTime="2026-11-02 19:37:06.765033987 +0000 UTC m=+533.372067754"
Nov 02 19:37:07.727665 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:07.727626    4463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-6v2d5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Nov 02 19:37:07.728073 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:07.727679    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6v2d5" podUID="14de05d8-f8d6-4ab0-8671-faa1158ca274" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Nov 02 19:37:08.726587 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:08.726550    4463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-6v2d5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Nov 02 19:37:08.726818 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:08.726610    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6v2d5" podUID="14de05d8-f8d6-4ab0-8671-faa1158ca274" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Nov 02 19:37:09.726714 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:09.726679    4463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-6v2d5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Nov 02 19:37:09.727134 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:09.726732    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6v2d5" podUID="14de05d8-f8d6-4ab0-8671-faa1158ca274" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Nov 02 19:37:10.726371 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:10.726334    4463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-6v2d5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Nov 02 19:37:10.726551 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:10.726393    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6v2d5" podUID="14de05d8-f8d6-4ab0-8671-faa1158ca274" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Nov 02 19:37:11.726114 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:11.726079    4463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-6v2d5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Nov 02 19:37:11.726497 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:11.726129    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6v2d5" podUID="14de05d8-f8d6-4ab0-8671-faa1158ca274" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Nov 02 19:37:12.726519 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:12.726474    4463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-6v2d5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Nov 02 19:37:12.726912 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:12.726548    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6v2d5" podUID="14de05d8-f8d6-4ab0-8671-faa1158ca274" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Nov 02 19:37:14.395283 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:14.395251    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6v2d5"
Nov 02 19:37:14.395777 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:14.395582    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6v2d5"
Nov 02 19:37:14.396720 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:14.396693    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6v2d5"
Nov 02 19:37:16.207552 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:16.207522    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z74mv"]
Nov 02 19:37:16.211409 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:16.211388    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z74mv"
Nov 02 19:37:16.229157 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:16.229130    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/658d1f56-a03e-4a38-8911-229de81a3d29-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-z74mv\" (UID: \"658d1f56-a03e-4a38-8911-229de81a3d29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z74mv"
Nov 02 19:37:16.229284 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:16.229178    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/658d1f56-a03e-4a38-8911-229de81a3d29-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-z74mv\" (UID: \"658d1f56-a03e-4a38-8911-229de81a3d29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z74mv"
Nov 02 19:37:16.229284 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:16.229199    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/658d1f56-a03e-4a38-8911-229de81a3d29-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-z74mv\" (UID: \"658d1f56-a03e-4a38-8911-229de81a3d29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z74mv"
Nov 02 19:37:16.229355 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:16.229308    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/658d1f56-a03e-4a38-8911-229de81a3d29-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-z74mv\" (UID: \"658d1f56-a03e-4a38-8911-229de81a3d29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z74mv"
Nov 02 19:37:16.229355 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:16.229346    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/658d1f56-a03e-4a38-8911-229de81a3d29-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-z74mv\" (UID: \"658d1f56-a03e-4a38-8911-229de81a3d29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z74mv"
Nov 02 19:37:16.229491 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:16.229469    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/658d1f56-a03e-4a38-8911-229de81a3d29-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-z74mv\" (UID: \"658d1f56-a03e-4a38-8911-229de81a3d29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z74mv"
Nov 02 19:37:16.229549 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:16.229506    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/658d1f56-a03e-4a38-8911-229de81a3d29-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-z74mv\" (UID: \"658d1f56-a03e-4a38-8911-229de81a3d29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z74mv"
Nov 02 19:37:16.229549 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:16.229540    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/658d1f56-a03e-4a38-8911-229de81a3d29-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-z74mv\" (UID: \"658d1f56-a03e-4a38-8911-229de81a3d29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z74mv"
Nov 02 19:37:16.229628 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:16.229600    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cptpz\" (UniqueName: \"kubernetes.io/projected/658d1f56-a03e-4a38-8911-229de81a3d29-kube-api-access-cptpz\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-z74mv\" (UID: \"658d1f56-a03e-4a38-8911-229de81a3d29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z74mv"
Nov 02 19:37:16.308705 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:16.308666    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z74mv"]
Nov 02 19:37:16.331163 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:16.331126    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cptpz\" (UniqueName: \"kubernetes.io/projected/658d1f56-a03e-4a38-8911-229de81a3d29-kube-api-access-cptpz\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-z74mv\" (UID: \"658d1f56-a03e-4a38-8911-229de81a3d29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z74mv"
Nov 02 19:37:16.331339 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:16.331181    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/658d1f56-a03e-4a38-8911-229de81a3d29-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-z74mv\" (UID: \"658d1f56-a03e-4a38-8911-229de81a3d29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z74mv"
Nov 02 19:37:16.331339 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:16.331237    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/658d1f56-a03e-4a38-8911-229de81a3d29-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-z74mv\" (UID: \"658d1f56-a03e-4a38-8911-229de81a3d29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z74mv"
Nov 02 19:37:16.331465 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:16.331386    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/658d1f56-a03e-4a38-8911-229de81a3d29-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-z74mv\" (UID: \"658d1f56-a03e-4a38-8911-229de81a3d29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z74mv"
Nov 02 19:37:16.331465 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:16.331432    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/658d1f56-a03e-4a38-8911-229de81a3d29-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-z74mv\" (UID: \"658d1f56-a03e-4a38-8911-229de81a3d29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z74mv"
Nov 02 19:37:16.331568 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:16.331471    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/658d1f56-a03e-4a38-8911-229de81a3d29-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-z74mv\" (UID: \"658d1f56-a03e-4a38-8911-229de81a3d29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z74mv"
Nov 02 19:37:16.331568 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:16.331523    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/658d1f56-a03e-4a38-8911-229de81a3d29-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-z74mv\" (UID: \"658d1f56-a03e-4a38-8911-229de81a3d29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z74mv"
Nov 02 19:37:16.331568 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:16.331540    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/658d1f56-a03e-4a38-8911-229de81a3d29-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-z74mv\" (UID: \"658d1f56-a03e-4a38-8911-229de81a3d29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z74mv"
Nov 02 19:37:16.331568 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:16.331561    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/658d1f56-a03e-4a38-8911-229de81a3d29-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-z74mv\" (UID: \"658d1f56-a03e-4a38-8911-229de81a3d29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z74mv"
Nov 02 19:37:16.331772 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:16.331698    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/658d1f56-a03e-4a38-8911-229de81a3d29-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-z74mv\" (UID: \"658d1f56-a03e-4a38-8911-229de81a3d29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z74mv"
Nov 02 19:37:16.332171 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:16.332136    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/658d1f56-a03e-4a38-8911-229de81a3d29-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-z74mv\" (UID: \"658d1f56-a03e-4a38-8911-229de81a3d29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z74mv"
Nov 02 19:37:16.332328 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:16.332279    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/658d1f56-a03e-4a38-8911-229de81a3d29-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-z74mv\" (UID: \"658d1f56-a03e-4a38-8911-229de81a3d29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z74mv"
Nov 02 19:37:16.332459 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:16.332342    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/658d1f56-a03e-4a38-8911-229de81a3d29-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-z74mv\" (UID: \"658d1f56-a03e-4a38-8911-229de81a3d29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z74mv"
Nov 02 19:37:16.332459 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:16.332421    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/658d1f56-a03e-4a38-8911-229de81a3d29-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-z74mv\" (UID: \"658d1f56-a03e-4a38-8911-229de81a3d29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z74mv"
Nov 02 19:37:16.333727 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:16.333710    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/658d1f56-a03e-4a38-8911-229de81a3d29-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-z74mv\" (UID: \"658d1f56-a03e-4a38-8911-229de81a3d29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z74mv"
Nov 02 19:37:16.333859 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:16.333841    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/658d1f56-a03e-4a38-8911-229de81a3d29-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-z74mv\" (UID: \"658d1f56-a03e-4a38-8911-229de81a3d29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z74mv"
Nov 02 19:37:16.385001 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:16.384967    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/658d1f56-a03e-4a38-8911-229de81a3d29-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-z74mv\" (UID: \"658d1f56-a03e-4a38-8911-229de81a3d29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z74mv"
Nov 02 19:37:16.385198 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:16.385180    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cptpz\" (UniqueName: \"kubernetes.io/projected/658d1f56-a03e-4a38-8911-229de81a3d29-kube-api-access-cptpz\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-z74mv\" (UID: \"658d1f56-a03e-4a38-8911-229de81a3d29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z74mv"
Nov 02 19:37:16.522320 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:16.522234    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z74mv"
Nov 02 19:37:16.936525 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:16.936487    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z74mv"]
Nov 02 19:37:16.938425 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:37:16.938390    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod658d1f56_a03e_4a38_8911_229de81a3d29.slice/crio-a1b5df7718e9e6274ed8f95c9086794e8c5693c83fe81d1357b3ab414d8dfb81 WatchSource:0}: Error finding container a1b5df7718e9e6274ed8f95c9086794e8c5693c83fe81d1357b3ab414d8dfb81: Status 404 returned error can't find the container with id a1b5df7718e9e6274ed8f95c9086794e8c5693c83fe81d1357b3ab414d8dfb81
Nov 02 19:37:16.940449 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:16.940419    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:37:16.940510 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:16.940477    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:37:16.940510 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:16.940504    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:37:17.673037 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:17.672930    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z74mv" event={"ID":"658d1f56-a03e-4a38-8911-229de81a3d29","Type":"ContainerStarted","Data":"70d39895a3906c23298558bb65c6c6c121df351bd839a8e20943d293db6d8e2b"}
Nov 02 19:37:17.673610 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:17.673074    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z74mv" event={"ID":"658d1f56-a03e-4a38-8911-229de81a3d29","Type":"ContainerStarted","Data":"a1b5df7718e9e6274ed8f95c9086794e8c5693c83fe81d1357b3ab414d8dfb81"}
Nov 02 19:37:17.783078 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:17.783006    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z74mv" podStartSLOduration=1.782990613 podStartE2EDuration="1.782990613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:37:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:37:17.781461004 +0000 UTC m=+544.388494774" watchObservedRunningTime="2026-11-02 19:37:17.782990613 +0000 UTC m=+544.390024384"
Nov 02 19:37:18.523074 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:18.523044    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z74mv"
Nov 02 19:37:18.524362 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:18.524330    4463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-z74mv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Nov 02 19:37:18.524483 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:18.524374    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z74mv" podUID="658d1f56-a03e-4a38-8911-229de81a3d29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Nov 02 19:37:19.523242 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:19.523208    4463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-z74mv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Nov 02 19:37:19.523629 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:19.523260    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z74mv" podUID="658d1f56-a03e-4a38-8911-229de81a3d29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Nov 02 19:37:20.522864 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:20.522834    4463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-z74mv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Nov 02 19:37:20.523061 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:20.522884    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z74mv" podUID="658d1f56-a03e-4a38-8911-229de81a3d29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Nov 02 19:37:21.523106 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:21.523069    4463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-z74mv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Nov 02 19:37:21.523511 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:21.523122    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z74mv" podUID="658d1f56-a03e-4a38-8911-229de81a3d29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Nov 02 19:37:22.522708 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:22.522674    4463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-z74mv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Nov 02 19:37:22.522875 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:22.522728    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z74mv" podUID="658d1f56-a03e-4a38-8911-229de81a3d29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Nov 02 19:37:23.523101 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:23.523067    4463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-z74mv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Nov 02 19:37:23.523586 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:23.523118    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z74mv" podUID="658d1f56-a03e-4a38-8911-229de81a3d29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Nov 02 19:37:25.198672 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:25.198613    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z74mv"
Nov 02 19:37:25.199174 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:25.198944    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z74mv"
Nov 02 19:37:25.199680 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:25.199655    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z74mv"
Nov 02 19:37:33.706523 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:33.706479    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-slt99"]
Nov 02 19:37:33.709526 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:33.709502    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-slt99"
Nov 02 19:37:33.803935 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:33.803892    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b33fed1-e127-4f1f-8b7f-d56427da0155-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-slt99\" (UID: \"5b33fed1-e127-4f1f-8b7f-d56427da0155\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-slt99"
Nov 02 19:37:33.804130 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:33.804051    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sj75h\" (UniqueName: \"kubernetes.io/projected/5b33fed1-e127-4f1f-8b7f-d56427da0155-kube-api-access-sj75h\") pod \"payload-processing-55fbbbf9dc-slt99\" (UID: \"5b33fed1-e127-4f1f-8b7f-d56427da0155\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-slt99"
Nov 02 19:37:33.806983 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:33.806963    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Nov 02 19:37:33.807191 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:33.807175    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-kq5qx\""
Nov 02 19:37:33.809672 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:33.809648    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-slt99"]
Nov 02 19:37:33.904954 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:33.904911    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b33fed1-e127-4f1f-8b7f-d56427da0155-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-slt99\" (UID: \"5b33fed1-e127-4f1f-8b7f-d56427da0155\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-slt99"
Nov 02 19:37:33.905179 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:33.905005    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj75h\" (UniqueName: \"kubernetes.io/projected/5b33fed1-e127-4f1f-8b7f-d56427da0155-kube-api-access-sj75h\") pod \"payload-processing-55fbbbf9dc-slt99\" (UID: \"5b33fed1-e127-4f1f-8b7f-d56427da0155\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-slt99"
Nov 02 19:37:33.905677 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:33.905651    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b33fed1-e127-4f1f-8b7f-d56427da0155-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-slt99\" (UID: \"5b33fed1-e127-4f1f-8b7f-d56427da0155\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-slt99"
Nov 02 19:37:33.958538 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:33.958471    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj75h\" (UniqueName: \"kubernetes.io/projected/5b33fed1-e127-4f1f-8b7f-d56427da0155-kube-api-access-sj75h\") pod \"payload-processing-55fbbbf9dc-slt99\" (UID: \"5b33fed1-e127-4f1f-8b7f-d56427da0155\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-slt99"
Nov 02 19:37:34.022027 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:34.021971    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-slt99"
Nov 02 19:37:34.293949 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:37:34.293817    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 19:37:34.385666 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:34.385635    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-slt99"]
Nov 02 19:37:34.389308 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:37:34.389278    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b33fed1_e127_4f1f_8b7f_d56427da0155.slice/crio-5b6f9d543aced71b3751d9a36ffcc6660d9721d6e253cd87fcc8ebde6866959f WatchSource:0}: Error finding container 5b6f9d543aced71b3751d9a36ffcc6660d9721d6e253cd87fcc8ebde6866959f: Status 404 returned error can't find the container with id 5b6f9d543aced71b3751d9a36ffcc6660d9721d6e253cd87fcc8ebde6866959f
Nov 02 19:37:34.748509 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:34.748475    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-slt99" event={"ID":"5b33fed1-e127-4f1f-8b7f-d56427da0155","Type":"ContainerStarted","Data":"5b6f9d543aced71b3751d9a36ffcc6660d9721d6e253cd87fcc8ebde6866959f"}
Nov 02 19:37:37.765814 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:37.765770    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-slt99" event={"ID":"5b33fed1-e127-4f1f-8b7f-d56427da0155","Type":"ContainerStarted","Data":"79a2c2439d5fd34927be5ea4cc5ec4d316851627f475d3d1d18c651e4499d97d"}
Nov 02 19:37:37.766214 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:37.765835    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-slt99"
Nov 02 19:37:37.871508 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:37.871452    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-slt99" podStartSLOduration=2.126077804 podStartE2EDuration="4.871439014s" totalImagesPullingTime="2.74536121s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:37:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:37:37.87012876 +0000 UTC m=+564.477162542" watchObservedRunningTime="2026-11-02 19:37:37.871439014 +0000 UTC m=+564.478472782"
Nov 02 19:37:48.773691 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:37:48.773659    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-slt99"
Nov 02 19:38:04.293520 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:38:04.293477    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 19:38:34.294278 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:38:34.294237    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 19:38:45.287908 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:45.287870    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-fh467"]
Nov 02 19:38:45.291365 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:45.291348    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-fh467"
Nov 02 19:38:45.360581 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:45.360550    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-43ceaa-openshift-default-dockercfg-nlhql\""
Nov 02 19:38:45.388068 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:45.388032    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-fh467"]
Nov 02 19:38:45.451427 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:45.451391    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f76ece2-703b-41b2-8833-f909342a8390-workload-socket\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-fh467\" (UID: \"0f76ece2-703b-41b2-8833-f909342a8390\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-fh467"
Nov 02 19:38:45.451427 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:45.451428    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f76ece2-703b-41b2-8833-f909342a8390-workload-certs\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-fh467\" (UID: \"0f76ece2-703b-41b2-8833-f909342a8390\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-fh467"
Nov 02 19:38:45.451678 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:45.451452    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xp7d\" (UniqueName: \"kubernetes.io/projected/0f76ece2-703b-41b2-8833-f909342a8390-kube-api-access-9xp7d\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-fh467\" (UID: \"0f76ece2-703b-41b2-8833-f909342a8390\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-fh467"
Nov 02 19:38:45.451678 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:45.451592    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f76ece2-703b-41b2-8833-f909342a8390-istiod-ca-cert\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-fh467\" (UID: \"0f76ece2-703b-41b2-8833-f909342a8390\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-fh467"
Nov 02 19:38:45.451678 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:45.451638    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f76ece2-703b-41b2-8833-f909342a8390-istio-podinfo\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-fh467\" (UID: \"0f76ece2-703b-41b2-8833-f909342a8390\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-fh467"
Nov 02 19:38:45.451808 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:45.451701    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f76ece2-703b-41b2-8833-f909342a8390-istio-data\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-fh467\" (UID: \"0f76ece2-703b-41b2-8833-f909342a8390\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-fh467"
Nov 02 19:38:45.451808 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:45.451751    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f76ece2-703b-41b2-8833-f909342a8390-istio-token\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-fh467\" (UID: \"0f76ece2-703b-41b2-8833-f909342a8390\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-fh467"
Nov 02 19:38:45.451808 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:45.451799    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f76ece2-703b-41b2-8833-f909342a8390-istio-envoy\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-fh467\" (UID: \"0f76ece2-703b-41b2-8833-f909342a8390\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-fh467"
Nov 02 19:38:45.451959 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:45.451863    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f76ece2-703b-41b2-8833-f909342a8390-credential-socket\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-fh467\" (UID: \"0f76ece2-703b-41b2-8833-f909342a8390\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-fh467"
Nov 02 19:38:45.552749 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:45.552666    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f76ece2-703b-41b2-8833-f909342a8390-credential-socket\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-fh467\" (UID: \"0f76ece2-703b-41b2-8833-f909342a8390\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-fh467"
Nov 02 19:38:45.552749 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:45.552732    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f76ece2-703b-41b2-8833-f909342a8390-workload-socket\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-fh467\" (UID: \"0f76ece2-703b-41b2-8833-f909342a8390\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-fh467"
Nov 02 19:38:45.552976 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:45.552758    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f76ece2-703b-41b2-8833-f909342a8390-workload-certs\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-fh467\" (UID: \"0f76ece2-703b-41b2-8833-f909342a8390\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-fh467"
Nov 02 19:38:45.552976 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:45.552779    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xp7d\" (UniqueName: \"kubernetes.io/projected/0f76ece2-703b-41b2-8833-f909342a8390-kube-api-access-9xp7d\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-fh467\" (UID: \"0f76ece2-703b-41b2-8833-f909342a8390\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-fh467"
Nov 02 19:38:45.552976 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:45.552931    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f76ece2-703b-41b2-8833-f909342a8390-istiod-ca-cert\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-fh467\" (UID: \"0f76ece2-703b-41b2-8833-f909342a8390\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-fh467"
Nov 02 19:38:45.553205 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:45.552990    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f76ece2-703b-41b2-8833-f909342a8390-istio-podinfo\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-fh467\" (UID: \"0f76ece2-703b-41b2-8833-f909342a8390\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-fh467"
Nov 02 19:38:45.553205 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:45.553057    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f76ece2-703b-41b2-8833-f909342a8390-istio-data\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-fh467\" (UID: \"0f76ece2-703b-41b2-8833-f909342a8390\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-fh467"
Nov 02 19:38:45.553205 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:45.553094    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f76ece2-703b-41b2-8833-f909342a8390-credential-socket\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-fh467\" (UID: \"0f76ece2-703b-41b2-8833-f909342a8390\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-fh467"
Nov 02 19:38:45.553205 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:45.553102    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f76ece2-703b-41b2-8833-f909342a8390-istio-token\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-fh467\" (UID: \"0f76ece2-703b-41b2-8833-f909342a8390\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-fh467"
Nov 02 19:38:45.553205 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:45.553145    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f76ece2-703b-41b2-8833-f909342a8390-workload-socket\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-fh467\" (UID: \"0f76ece2-703b-41b2-8833-f909342a8390\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-fh467"
Nov 02 19:38:45.553384 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:45.553207    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f76ece2-703b-41b2-8833-f909342a8390-istio-envoy\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-fh467\" (UID: \"0f76ece2-703b-41b2-8833-f909342a8390\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-fh467"
Nov 02 19:38:45.553384 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:45.553220    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f76ece2-703b-41b2-8833-f909342a8390-workload-certs\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-fh467\" (UID: \"0f76ece2-703b-41b2-8833-f909342a8390\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-fh467"
Nov 02 19:38:45.553466 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:45.553413    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f76ece2-703b-41b2-8833-f909342a8390-istio-data\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-fh467\" (UID: \"0f76ece2-703b-41b2-8833-f909342a8390\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-fh467"
Nov 02 19:38:45.553642 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:45.553623    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f76ece2-703b-41b2-8833-f909342a8390-istiod-ca-cert\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-fh467\" (UID: \"0f76ece2-703b-41b2-8833-f909342a8390\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-fh467"
Nov 02 19:38:45.555388 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:45.555366    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f76ece2-703b-41b2-8833-f909342a8390-istio-envoy\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-fh467\" (UID: \"0f76ece2-703b-41b2-8833-f909342a8390\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-fh467"
Nov 02 19:38:45.555520 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:45.555502    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f76ece2-703b-41b2-8833-f909342a8390-istio-podinfo\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-fh467\" (UID: \"0f76ece2-703b-41b2-8833-f909342a8390\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-fh467"
Nov 02 19:38:45.613753 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:45.613719    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xp7d\" (UniqueName: \"kubernetes.io/projected/0f76ece2-703b-41b2-8833-f909342a8390-kube-api-access-9xp7d\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-fh467\" (UID: \"0f76ece2-703b-41b2-8833-f909342a8390\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-fh467"
Nov 02 19:38:45.615117 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:45.615092    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f76ece2-703b-41b2-8833-f909342a8390-istio-token\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-fh467\" (UID: \"0f76ece2-703b-41b2-8833-f909342a8390\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-fh467"
Nov 02 19:38:45.903493 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:45.903403    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-fh467"
Nov 02 19:38:45.961404 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:45.961370    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-2zrqj"]
Nov 02 19:38:45.966721 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:45.966221    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-2zrqj"
Nov 02 19:38:46.020461 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:46.020264    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-5c46af-openshift-default-dockercfg-c97j7\""
Nov 02 19:38:46.022880 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:46.022833    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-2zrqj"]
Nov 02 19:38:46.061272 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:46.061238    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlf5m\" (UniqueName: \"kubernetes.io/projected/a8216408-95d0-413b-a609-2ca8cf3299b6-kube-api-access-qlf5m\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-2zrqj\" (UID: \"a8216408-95d0-413b-a609-2ca8cf3299b6\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-2zrqj"
Nov 02 19:38:46.061451 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:46.061286    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8216408-95d0-413b-a609-2ca8cf3299b6-istio-data\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-2zrqj\" (UID: \"a8216408-95d0-413b-a609-2ca8cf3299b6\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-2zrqj"
Nov 02 19:38:46.061451 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:46.061408    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8216408-95d0-413b-a609-2ca8cf3299b6-istio-podinfo\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-2zrqj\" (UID: \"a8216408-95d0-413b-a609-2ca8cf3299b6\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-2zrqj"
Nov 02 19:38:46.061451 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:46.061441    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8216408-95d0-413b-a609-2ca8cf3299b6-istiod-ca-cert\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-2zrqj\" (UID: \"a8216408-95d0-413b-a609-2ca8cf3299b6\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-2zrqj"
Nov 02 19:38:46.061577 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:46.061488    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8216408-95d0-413b-a609-2ca8cf3299b6-istio-envoy\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-2zrqj\" (UID: \"a8216408-95d0-413b-a609-2ca8cf3299b6\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-2zrqj"
Nov 02 19:38:46.061577 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:46.061564    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8216408-95d0-413b-a609-2ca8cf3299b6-istio-token\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-2zrqj\" (UID: \"a8216408-95d0-413b-a609-2ca8cf3299b6\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-2zrqj"
Nov 02 19:38:46.061639 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:46.061598    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8216408-95d0-413b-a609-2ca8cf3299b6-workload-socket\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-2zrqj\" (UID: \"a8216408-95d0-413b-a609-2ca8cf3299b6\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-2zrqj"
Nov 02 19:38:46.061639 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:46.061629    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8216408-95d0-413b-a609-2ca8cf3299b6-credential-socket\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-2zrqj\" (UID: \"a8216408-95d0-413b-a609-2ca8cf3299b6\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-2zrqj"
Nov 02 19:38:46.061707 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:46.061652    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8216408-95d0-413b-a609-2ca8cf3299b6-workload-certs\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-2zrqj\" (UID: \"a8216408-95d0-413b-a609-2ca8cf3299b6\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-2zrqj"
Nov 02 19:38:46.163211 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:46.163143    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8216408-95d0-413b-a609-2ca8cf3299b6-credential-socket\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-2zrqj\" (UID: \"a8216408-95d0-413b-a609-2ca8cf3299b6\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-2zrqj"
Nov 02 19:38:46.163211 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:46.163194    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8216408-95d0-413b-a609-2ca8cf3299b6-workload-certs\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-2zrqj\" (UID: \"a8216408-95d0-413b-a609-2ca8cf3299b6\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-2zrqj"
Nov 02 19:38:46.163411 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:46.163213    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlf5m\" (UniqueName: \"kubernetes.io/projected/a8216408-95d0-413b-a609-2ca8cf3299b6-kube-api-access-qlf5m\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-2zrqj\" (UID: \"a8216408-95d0-413b-a609-2ca8cf3299b6\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-2zrqj"
Nov 02 19:38:46.163411 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:46.163233    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8216408-95d0-413b-a609-2ca8cf3299b6-istio-data\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-2zrqj\" (UID: \"a8216408-95d0-413b-a609-2ca8cf3299b6\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-2zrqj"
Nov 02 19:38:46.163411 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:46.163267    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8216408-95d0-413b-a609-2ca8cf3299b6-istio-podinfo\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-2zrqj\" (UID: \"a8216408-95d0-413b-a609-2ca8cf3299b6\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-2zrqj"
Nov 02 19:38:46.163411 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:46.163286    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8216408-95d0-413b-a609-2ca8cf3299b6-istiod-ca-cert\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-2zrqj\" (UID: \"a8216408-95d0-413b-a609-2ca8cf3299b6\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-2zrqj"
Nov 02 19:38:46.163411 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:46.163345    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8216408-95d0-413b-a609-2ca8cf3299b6-istio-envoy\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-2zrqj\" (UID: \"a8216408-95d0-413b-a609-2ca8cf3299b6\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-2zrqj"
Nov 02 19:38:46.163411 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:46.163384    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8216408-95d0-413b-a609-2ca8cf3299b6-istio-token\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-2zrqj\" (UID: \"a8216408-95d0-413b-a609-2ca8cf3299b6\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-2zrqj"
Nov 02 19:38:46.163785 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:46.163433    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8216408-95d0-413b-a609-2ca8cf3299b6-workload-socket\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-2zrqj\" (UID: \"a8216408-95d0-413b-a609-2ca8cf3299b6\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-2zrqj"
Nov 02 19:38:46.163785 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:46.163616    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8216408-95d0-413b-a609-2ca8cf3299b6-workload-certs\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-2zrqj\" (UID: \"a8216408-95d0-413b-a609-2ca8cf3299b6\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-2zrqj"
Nov 02 19:38:46.163884 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:46.163785    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8216408-95d0-413b-a609-2ca8cf3299b6-istio-data\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-2zrqj\" (UID: \"a8216408-95d0-413b-a609-2ca8cf3299b6\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-2zrqj"
Nov 02 19:38:46.163884 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:46.163813    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8216408-95d0-413b-a609-2ca8cf3299b6-credential-socket\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-2zrqj\" (UID: \"a8216408-95d0-413b-a609-2ca8cf3299b6\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-2zrqj"
Nov 02 19:38:46.163884 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:46.163826    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8216408-95d0-413b-a609-2ca8cf3299b6-workload-socket\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-2zrqj\" (UID: \"a8216408-95d0-413b-a609-2ca8cf3299b6\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-2zrqj"
Nov 02 19:38:46.164145 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:46.164128    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8216408-95d0-413b-a609-2ca8cf3299b6-istiod-ca-cert\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-2zrqj\" (UID: \"a8216408-95d0-413b-a609-2ca8cf3299b6\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-2zrqj"
Nov 02 19:38:46.165437 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:46.165419    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8216408-95d0-413b-a609-2ca8cf3299b6-istio-envoy\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-2zrqj\" (UID: \"a8216408-95d0-413b-a609-2ca8cf3299b6\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-2zrqj"
Nov 02 19:38:46.165548 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:46.165533    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8216408-95d0-413b-a609-2ca8cf3299b6-istio-podinfo\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-2zrqj\" (UID: \"a8216408-95d0-413b-a609-2ca8cf3299b6\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-2zrqj"
Nov 02 19:38:46.216413 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:46.216378    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8216408-95d0-413b-a609-2ca8cf3299b6-istio-token\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-2zrqj\" (UID: \"a8216408-95d0-413b-a609-2ca8cf3299b6\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-2zrqj"
Nov 02 19:38:46.216786 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:46.216764    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlf5m\" (UniqueName: \"kubernetes.io/projected/a8216408-95d0-413b-a609-2ca8cf3299b6-kube-api-access-qlf5m\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-2zrqj\" (UID: \"a8216408-95d0-413b-a609-2ca8cf3299b6\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-2zrqj"
Nov 02 19:38:46.282691 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:46.282652    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-2zrqj"
Nov 02 19:38:46.318068 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:46.316413    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-fh467"]
Nov 02 19:38:46.320661 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:38:46.320599    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f76ece2_703b_41b2_8833_f909342a8390.slice/crio-5202a2211b34d4b6cca958f5bf26e17b10fc9b99b6e03f16d66ba3f69f6b5dfb WatchSource:0}: Error finding container 5202a2211b34d4b6cca958f5bf26e17b10fc9b99b6e03f16d66ba3f69f6b5dfb: Status 404 returned error can't find the container with id 5202a2211b34d4b6cca958f5bf26e17b10fc9b99b6e03f16d66ba3f69f6b5dfb
Nov 02 19:38:46.326152 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:46.324179    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:38:46.326152 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:46.324275    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:38:46.326152 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:46.324320    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:38:46.672357 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:46.672320    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-2zrqj"]
Nov 02 19:38:46.678343 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:38:46.678310    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8216408_95d0_413b_a609_2ca8cf3299b6.slice/crio-b486ffccbc9e9095dffa4ae2b260b109089598ac6276ef7362508dbc5c095ae2 WatchSource:0}: Error finding container b486ffccbc9e9095dffa4ae2b260b109089598ac6276ef7362508dbc5c095ae2: Status 404 returned error can't find the container with id b486ffccbc9e9095dffa4ae2b260b109089598ac6276ef7362508dbc5c095ae2
Nov 02 19:38:46.683219 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:46.681729    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:38:46.683219 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:46.681820    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:38:46.683219 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:46.681862    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:38:46.954971 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:46.954897    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9ndrz"]
Nov 02 19:38:46.958482 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:46.958463    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9ndrz"
Nov 02 19:38:47.007378 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:47.007346    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-e9c144-openshift-default-dockercfg-7slsb\""
Nov 02 19:38:47.016908 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:47.016619    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9ndrz"]
Nov 02 19:38:47.072558 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:47.072516    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-2zrqj" event={"ID":"a8216408-95d0-413b-a609-2ca8cf3299b6","Type":"ContainerStarted","Data":"22cf3df17c532e6521ea22300285e0798bc8f9e96c1c973b1d851db1f9dd597c"}
Nov 02 19:38:47.072733 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:47.072571    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-2zrqj" event={"ID":"a8216408-95d0-413b-a609-2ca8cf3299b6","Type":"ContainerStarted","Data":"b486ffccbc9e9095dffa4ae2b260b109089598ac6276ef7362508dbc5c095ae2"}
Nov 02 19:38:47.073179 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:47.073147    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5734864-6a35-46c3-80b8-bdac768e1de1-credential-socket\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9ndrz\" (UID: \"f5734864-6a35-46c3-80b8-bdac768e1de1\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9ndrz"
Nov 02 19:38:47.073279 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:47.073203    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5734864-6a35-46c3-80b8-bdac768e1de1-istio-envoy\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9ndrz\" (UID: \"f5734864-6a35-46c3-80b8-bdac768e1de1\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9ndrz"
Nov 02 19:38:47.073327 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:47.073294    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5734864-6a35-46c3-80b8-bdac768e1de1-istio-data\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9ndrz\" (UID: \"f5734864-6a35-46c3-80b8-bdac768e1de1\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9ndrz"
Nov 02 19:38:47.073367 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:47.073329    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5734864-6a35-46c3-80b8-bdac768e1de1-istiod-ca-cert\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9ndrz\" (UID: \"f5734864-6a35-46c3-80b8-bdac768e1de1\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9ndrz"
Nov 02 19:38:47.073402 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:47.073375    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsfwd\" (UniqueName: \"kubernetes.io/projected/f5734864-6a35-46c3-80b8-bdac768e1de1-kube-api-access-tsfwd\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9ndrz\" (UID: \"f5734864-6a35-46c3-80b8-bdac768e1de1\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9ndrz"
Nov 02 19:38:47.073483 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:47.073406    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5734864-6a35-46c3-80b8-bdac768e1de1-istio-token\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9ndrz\" (UID: \"f5734864-6a35-46c3-80b8-bdac768e1de1\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9ndrz"
Nov 02 19:38:47.073483 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:47.073475    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5734864-6a35-46c3-80b8-bdac768e1de1-istio-podinfo\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9ndrz\" (UID: \"f5734864-6a35-46c3-80b8-bdac768e1de1\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9ndrz"
Nov 02 19:38:47.073554 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:47.073508    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5734864-6a35-46c3-80b8-bdac768e1de1-workload-socket\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9ndrz\" (UID: \"f5734864-6a35-46c3-80b8-bdac768e1de1\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9ndrz"
Nov 02 19:38:47.073554 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:47.073527    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5734864-6a35-46c3-80b8-bdac768e1de1-workload-certs\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9ndrz\" (UID: \"f5734864-6a35-46c3-80b8-bdac768e1de1\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9ndrz"
Nov 02 19:38:47.074192 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:47.074170    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-fh467" event={"ID":"0f76ece2-703b-41b2-8833-f909342a8390","Type":"ContainerStarted","Data":"a5850cb61c7e98c613c433a5ac1dc76279be71e87e4c7fa07c31a4c86a55e91d"}
Nov 02 19:38:47.074257 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:47.074213    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-fh467" event={"ID":"0f76ece2-703b-41b2-8833-f909342a8390","Type":"ContainerStarted","Data":"5202a2211b34d4b6cca958f5bf26e17b10fc9b99b6e03f16d66ba3f69f6b5dfb"}
Nov 02 19:38:47.174473 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:47.174434    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5734864-6a35-46c3-80b8-bdac768e1de1-istio-podinfo\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9ndrz\" (UID: \"f5734864-6a35-46c3-80b8-bdac768e1de1\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9ndrz"
Nov 02 19:38:47.174658 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:47.174500    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5734864-6a35-46c3-80b8-bdac768e1de1-workload-socket\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9ndrz\" (UID: \"f5734864-6a35-46c3-80b8-bdac768e1de1\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9ndrz"
Nov 02 19:38:47.174725 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:47.174696    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5734864-6a35-46c3-80b8-bdac768e1de1-workload-certs\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9ndrz\" (UID: \"f5734864-6a35-46c3-80b8-bdac768e1de1\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9ndrz"
Nov 02 19:38:47.175502 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:47.174828    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5734864-6a35-46c3-80b8-bdac768e1de1-credential-socket\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9ndrz\" (UID: \"f5734864-6a35-46c3-80b8-bdac768e1de1\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9ndrz"
Nov 02 19:38:47.175502 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:47.174893    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5734864-6a35-46c3-80b8-bdac768e1de1-workload-socket\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9ndrz\" (UID: \"f5734864-6a35-46c3-80b8-bdac768e1de1\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9ndrz"
Nov 02 19:38:47.175502 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:47.174933    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5734864-6a35-46c3-80b8-bdac768e1de1-istio-envoy\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9ndrz\" (UID: \"f5734864-6a35-46c3-80b8-bdac768e1de1\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9ndrz"
Nov 02 19:38:47.175502 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:47.175032    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5734864-6a35-46c3-80b8-bdac768e1de1-istio-data\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9ndrz\" (UID: \"f5734864-6a35-46c3-80b8-bdac768e1de1\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9ndrz"
Nov 02 19:38:47.175502 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:47.175043    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5734864-6a35-46c3-80b8-bdac768e1de1-workload-certs\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9ndrz\" (UID: \"f5734864-6a35-46c3-80b8-bdac768e1de1\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9ndrz"
Nov 02 19:38:47.175502 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:47.175062    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5734864-6a35-46c3-80b8-bdac768e1de1-istiod-ca-cert\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9ndrz\" (UID: \"f5734864-6a35-46c3-80b8-bdac768e1de1\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9ndrz"
Nov 02 19:38:47.175502 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:47.175098    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsfwd\" (UniqueName: \"kubernetes.io/projected/f5734864-6a35-46c3-80b8-bdac768e1de1-kube-api-access-tsfwd\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9ndrz\" (UID: \"f5734864-6a35-46c3-80b8-bdac768e1de1\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9ndrz"
Nov 02 19:38:47.175502 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:47.175150    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5734864-6a35-46c3-80b8-bdac768e1de1-istio-token\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9ndrz\" (UID: \"f5734864-6a35-46c3-80b8-bdac768e1de1\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9ndrz"
Nov 02 19:38:47.175502 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:47.175165    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5734864-6a35-46c3-80b8-bdac768e1de1-credential-socket\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9ndrz\" (UID: \"f5734864-6a35-46c3-80b8-bdac768e1de1\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9ndrz"
Nov 02 19:38:47.175502 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:47.175475    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5734864-6a35-46c3-80b8-bdac768e1de1-istio-data\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9ndrz\" (UID: \"f5734864-6a35-46c3-80b8-bdac768e1de1\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9ndrz"
Nov 02 19:38:47.176122 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:47.176101    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5734864-6a35-46c3-80b8-bdac768e1de1-istiod-ca-cert\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9ndrz\" (UID: \"f5734864-6a35-46c3-80b8-bdac768e1de1\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9ndrz"
Nov 02 19:38:47.177368 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:47.177346    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5734864-6a35-46c3-80b8-bdac768e1de1-istio-podinfo\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9ndrz\" (UID: \"f5734864-6a35-46c3-80b8-bdac768e1de1\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9ndrz"
Nov 02 19:38:47.177600 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:47.177579    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5734864-6a35-46c3-80b8-bdac768e1de1-istio-envoy\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9ndrz\" (UID: \"f5734864-6a35-46c3-80b8-bdac768e1de1\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9ndrz"
Nov 02 19:38:47.186213 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:47.186176    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-2zrqj" podStartSLOduration=2.186162687 podStartE2EDuration="2.186162687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:38:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:38:47.181645186 +0000 UTC m=+633.788678955" watchObservedRunningTime="2026-11-02 19:38:47.186162687 +0000 UTC m=+633.793196456"
Nov 02 19:38:47.228546 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:47.228470    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5734864-6a35-46c3-80b8-bdac768e1de1-istio-token\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9ndrz\" (UID: \"f5734864-6a35-46c3-80b8-bdac768e1de1\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9ndrz"
Nov 02 19:38:47.228686 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:47.228568    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsfwd\" (UniqueName: \"kubernetes.io/projected/f5734864-6a35-46c3-80b8-bdac768e1de1-kube-api-access-tsfwd\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9ndrz\" (UID: \"f5734864-6a35-46c3-80b8-bdac768e1de1\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9ndrz"
Nov 02 19:38:47.271618 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:47.271584    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9ndrz"
Nov 02 19:38:47.283929 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:47.283852    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-2zrqj"
Nov 02 19:38:47.290124 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:47.290098    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-2zrqj"
Nov 02 19:38:47.295799 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:47.295755    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-fh467" podStartSLOduration=2.295742629 podStartE2EDuration="2.295742629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:38:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:38:47.2944172 +0000 UTC m=+633.901450980" watchObservedRunningTime="2026-11-02 19:38:47.295742629 +0000 UTC m=+633.902776397"
Nov 02 19:38:47.642624 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:47.642597    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9ndrz"]
Nov 02 19:38:47.644742 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:38:47.644714    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5734864_6a35_46c3_80b8_bdac768e1de1.slice/crio-916c8d53bb355408adcb9a051f33408077e90d26fccd696eea82acee250bbbf9 WatchSource:0}: Error finding container 916c8d53bb355408adcb9a051f33408077e90d26fccd696eea82acee250bbbf9: Status 404 returned error can't find the container with id 916c8d53bb355408adcb9a051f33408077e90d26fccd696eea82acee250bbbf9
Nov 02 19:38:47.647071 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:47.647031    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:38:47.647174 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:47.647099    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:38:47.647174 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:47.647127    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:38:47.904115 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:47.904035    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-fh467"
Nov 02 19:38:47.909598 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:47.909563    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-fh467"
Nov 02 19:38:48.080604 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:48.080565    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9ndrz" event={"ID":"f5734864-6a35-46c3-80b8-bdac768e1de1","Type":"ContainerStarted","Data":"23387a024c12a6210c34a1f28ef800c18033948f5ff83a61e84ac54b4597fd4b"}
Nov 02 19:38:48.080791 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:48.080625    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9ndrz" event={"ID":"f5734864-6a35-46c3-80b8-bdac768e1de1","Type":"ContainerStarted","Data":"916c8d53bb355408adcb9a051f33408077e90d26fccd696eea82acee250bbbf9"}
Nov 02 19:38:48.081149 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:48.081123    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-2zrqj"
Nov 02 19:38:48.081254 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:48.081175    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-fh467"
Nov 02 19:38:48.081975 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:48.081948    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-fh467"
Nov 02 19:38:48.082104 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:48.082055    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-2zrqj"
Nov 02 19:38:48.189418 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:48.189328    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9ndrz" podStartSLOduration=2.189279419 podStartE2EDuration="2.189279419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:38:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:38:48.188684 +0000 UTC m=+634.795717769" watchObservedRunningTime="2026-11-02 19:38:48.189279419 +0000 UTC m=+634.796313191"
Nov 02 19:38:48.272444 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:48.272408    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9ndrz"
Nov 02 19:38:48.277605 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:48.277576    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9ndrz"
Nov 02 19:38:49.087171 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:49.087138    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9ndrz"
Nov 02 19:38:49.087748 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:49.087730    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9ndrz"
Nov 02 19:38:54.472699 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:54.472668    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-8795bf-66d99fc7c9-nj9gz"]
Nov 02 19:38:54.476378 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:54.476362    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8795bf-66d99fc7c9-nj9gz"
Nov 02 19:38:54.527529 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:54.527503    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-8795bf\""
Nov 02 19:38:54.542558 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:54.542525    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpr6p\" (UniqueName: \"kubernetes.io/projected/4553e306-34a0-48cf-b662-843e8de5bb45-kube-api-access-xpr6p\") pod \"payload-pre-processing-e2e-worker-w4-8795bf-66d99fc7c9-nj9gz\" (UID: \"4553e306-34a0-48cf-b662-843e8de5bb45\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8795bf-66d99fc7c9-nj9gz"
Nov 02 19:38:54.542909 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:54.542882    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4553e306-34a0-48cf-b662-843e8de5bb45-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-8795bf-66d99fc7c9-nj9gz\" (UID: \"4553e306-34a0-48cf-b662-843e8de5bb45\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8795bf-66d99fc7c9-nj9gz"
Nov 02 19:38:54.573862 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:54.573833    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-8795bf-dockercfg-pj9rv\""
Nov 02 19:38:54.575733 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:54.575705    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-8795bf-66d99fc7c9-nj9gz"]
Nov 02 19:38:54.580488 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:54.580464    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-8795bf-7bd68d77d9-5xm6w"]
Nov 02 19:38:54.583964 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:54.583944    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-8795bf-7bd68d77d9-5xm6w"
Nov 02 19:38:54.637758 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:54.637717    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-8795bf-7bd68d77d9-5xm6w"]
Nov 02 19:38:54.644468 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:54.644432    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0368fdf1-1c9f-4ee9-8da1-5dd57731a8fd-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-8795bf-7bd68d77d9-5xm6w\" (UID: \"0368fdf1-1c9f-4ee9-8da1-5dd57731a8fd\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-8795bf-7bd68d77d9-5xm6w"
Nov 02 19:38:54.644627 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:54.644497    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4553e306-34a0-48cf-b662-843e8de5bb45-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-8795bf-66d99fc7c9-nj9gz\" (UID: \"4553e306-34a0-48cf-b662-843e8de5bb45\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8795bf-66d99fc7c9-nj9gz"
Nov 02 19:38:54.644697 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:54.644628    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps895\" (UniqueName: \"kubernetes.io/projected/0368fdf1-1c9f-4ee9-8da1-5dd57731a8fd-kube-api-access-ps895\") pod \"payload-processing-e2e-worker-w4-8795bf-7bd68d77d9-5xm6w\" (UID: \"0368fdf1-1c9f-4ee9-8da1-5dd57731a8fd\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-8795bf-7bd68d77d9-5xm6w"
Nov 02 19:38:54.644753 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:54.644689    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpr6p\" (UniqueName: \"kubernetes.io/projected/4553e306-34a0-48cf-b662-843e8de5bb45-kube-api-access-xpr6p\") pod \"payload-pre-processing-e2e-worker-w4-8795bf-66d99fc7c9-nj9gz\" (UID: \"4553e306-34a0-48cf-b662-843e8de5bb45\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8795bf-66d99fc7c9-nj9gz"
Nov 02 19:38:54.645397 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:54.645370    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4553e306-34a0-48cf-b662-843e8de5bb45-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-8795bf-66d99fc7c9-nj9gz\" (UID: \"4553e306-34a0-48cf-b662-843e8de5bb45\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8795bf-66d99fc7c9-nj9gz"
Nov 02 19:38:54.698581 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:54.698556    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpr6p\" (UniqueName: \"kubernetes.io/projected/4553e306-34a0-48cf-b662-843e8de5bb45-kube-api-access-xpr6p\") pod \"payload-pre-processing-e2e-worker-w4-8795bf-66d99fc7c9-nj9gz\" (UID: \"4553e306-34a0-48cf-b662-843e8de5bb45\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8795bf-66d99fc7c9-nj9gz"
Nov 02 19:38:54.745462 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:54.745390    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0368fdf1-1c9f-4ee9-8da1-5dd57731a8fd-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-8795bf-7bd68d77d9-5xm6w\" (UID: \"0368fdf1-1c9f-4ee9-8da1-5dd57731a8fd\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-8795bf-7bd68d77d9-5xm6w"
Nov 02 19:38:54.745592 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:54.745493    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps895\" (UniqueName: \"kubernetes.io/projected/0368fdf1-1c9f-4ee9-8da1-5dd57731a8fd-kube-api-access-ps895\") pod \"payload-processing-e2e-worker-w4-8795bf-7bd68d77d9-5xm6w\" (UID: \"0368fdf1-1c9f-4ee9-8da1-5dd57731a8fd\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-8795bf-7bd68d77d9-5xm6w"
Nov 02 19:38:54.746065 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:54.746046    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0368fdf1-1c9f-4ee9-8da1-5dd57731a8fd-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-8795bf-7bd68d77d9-5xm6w\" (UID: \"0368fdf1-1c9f-4ee9-8da1-5dd57731a8fd\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-8795bf-7bd68d77d9-5xm6w"
Nov 02 19:38:54.788855 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:54.788828    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8795bf-66d99fc7c9-nj9gz"
Nov 02 19:38:54.798421 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:54.798395    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps895\" (UniqueName: \"kubernetes.io/projected/0368fdf1-1c9f-4ee9-8da1-5dd57731a8fd-kube-api-access-ps895\") pod \"payload-processing-e2e-worker-w4-8795bf-7bd68d77d9-5xm6w\" (UID: \"0368fdf1-1c9f-4ee9-8da1-5dd57731a8fd\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-8795bf-7bd68d77d9-5xm6w"
Nov 02 19:38:54.896618 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:54.896581    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-8795bf-7bd68d77d9-5xm6w"
Nov 02 19:38:55.153610 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:55.153587    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-8795bf-66d99fc7c9-nj9gz"]
Nov 02 19:38:55.155702 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:38:55.155666    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4553e306_34a0_48cf_b662_843e8de5bb45.slice/crio-37b3a61ce10d160ea639dc86295080e9b40e8c4abf99d9f92aac5f2ce0cd177f WatchSource:0}: Error finding container 37b3a61ce10d160ea639dc86295080e9b40e8c4abf99d9f92aac5f2ce0cd177f: Status 404 returned error can't find the container with id 37b3a61ce10d160ea639dc86295080e9b40e8c4abf99d9f92aac5f2ce0cd177f
Nov 02 19:38:55.260733 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:55.260591    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-8795bf-7bd68d77d9-5xm6w"]
Nov 02 19:38:55.262595 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:38:55.262570    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0368fdf1_1c9f_4ee9_8da1_5dd57731a8fd.slice/crio-eb36f80e1fb6ac4d1145320fae725da255756b589c216a743d7511ccfe9abe92 WatchSource:0}: Error finding container eb36f80e1fb6ac4d1145320fae725da255756b589c216a743d7511ccfe9abe92: Status 404 returned error can't find the container with id eb36f80e1fb6ac4d1145320fae725da255756b589c216a743d7511ccfe9abe92
Nov 02 19:38:56.122338 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:56.122299    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-8795bf-7bd68d77d9-5xm6w" event={"ID":"0368fdf1-1c9f-4ee9-8da1-5dd57731a8fd","Type":"ContainerStarted","Data":"867aca871d89fa6ca408b48822438b55aa9c1e9419e7452ba8baa39d821afb71"}
Nov 02 19:38:56.122783 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:56.122353    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-8795bf-7bd68d77d9-5xm6w" event={"ID":"0368fdf1-1c9f-4ee9-8da1-5dd57731a8fd","Type":"ContainerStarted","Data":"eb36f80e1fb6ac4d1145320fae725da255756b589c216a743d7511ccfe9abe92"}
Nov 02 19:38:56.122783 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:56.122394    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-8795bf-7bd68d77d9-5xm6w"
Nov 02 19:38:56.123840 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:56.123816    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8795bf-66d99fc7c9-nj9gz" event={"ID":"4553e306-34a0-48cf-b662-843e8de5bb45","Type":"ContainerStarted","Data":"db579ede9b1a5ed8b51f1953fa0cbdd7f646ad0744547b7d5a7fffd6feb07c19"}
Nov 02 19:38:56.123955 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:56.123861    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8795bf-66d99fc7c9-nj9gz" event={"ID":"4553e306-34a0-48cf-b662-843e8de5bb45","Type":"ContainerStarted","Data":"37b3a61ce10d160ea639dc86295080e9b40e8c4abf99d9f92aac5f2ce0cd177f"}
Nov 02 19:38:56.123955 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:56.123897    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8795bf-66d99fc7c9-nj9gz"
Nov 02 19:38:56.231199 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:56.231157    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-8795bf-7bd68d77d9-5xm6w" podStartSLOduration=2.231144427 podStartE2EDuration="2.231144427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:38:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:38:56.229949831 +0000 UTC m=+642.836983600" watchObservedRunningTime="2026-11-02 19:38:56.231144427 +0000 UTC m=+642.838178195"
Nov 02 19:38:56.336719 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:56.336662    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8795bf-66d99fc7c9-nj9gz" podStartSLOduration=2.336645937 podStartE2EDuration="2.336645937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:38:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:38:56.335039362 +0000 UTC m=+642.942073131" watchObservedRunningTime="2026-11-02 19:38:56.336645937 +0000 UTC m=+642.943679707"
Nov 02 19:38:59.525457 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:59.525423    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-5c46af-7f86f8f5c5-jxhp6"]
Nov 02 19:38:59.528741 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:59.528720    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5c46af-7f86f8f5c5-jxhp6"
Nov 02 19:38:59.591004 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:59.590962    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/058aa7d8-1404-494f-b25b-70e8b702455b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-5c46af-7f86f8f5c5-jxhp6\" (UID: \"058aa7d8-1404-494f-b25b-70e8b702455b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5c46af-7f86f8f5c5-jxhp6"
Nov 02 19:38:59.591193 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:59.591171    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86r4f\" (UniqueName: \"kubernetes.io/projected/058aa7d8-1404-494f-b25b-70e8b702455b-kube-api-access-86r4f\") pod \"payload-pre-processing-e2e-worker-w3-5c46af-7f86f8f5c5-jxhp6\" (UID: \"058aa7d8-1404-494f-b25b-70e8b702455b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5c46af-7f86f8f5c5-jxhp6"
Nov 02 19:38:59.631060 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:59.626090    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-5c46af-dockercfg-wr4v2\""
Nov 02 19:38:59.631060 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:59.626162    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-5c46af\""
Nov 02 19:38:59.631060 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:59.629438    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-5c46af-7f86f8f5c5-jxhp6"]
Nov 02 19:38:59.692222 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:59.692187    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/058aa7d8-1404-494f-b25b-70e8b702455b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-5c46af-7f86f8f5c5-jxhp6\" (UID: \"058aa7d8-1404-494f-b25b-70e8b702455b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5c46af-7f86f8f5c5-jxhp6"
Nov 02 19:38:59.692415 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:59.692269    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86r4f\" (UniqueName: \"kubernetes.io/projected/058aa7d8-1404-494f-b25b-70e8b702455b-kube-api-access-86r4f\") pod \"payload-pre-processing-e2e-worker-w3-5c46af-7f86f8f5c5-jxhp6\" (UID: \"058aa7d8-1404-494f-b25b-70e8b702455b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5c46af-7f86f8f5c5-jxhp6"
Nov 02 19:38:59.692940 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:59.692914    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/058aa7d8-1404-494f-b25b-70e8b702455b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-5c46af-7f86f8f5c5-jxhp6\" (UID: \"058aa7d8-1404-494f-b25b-70e8b702455b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5c46af-7f86f8f5c5-jxhp6"
Nov 02 19:38:59.745294 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:59.745264    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86r4f\" (UniqueName: \"kubernetes.io/projected/058aa7d8-1404-494f-b25b-70e8b702455b-kube-api-access-86r4f\") pod \"payload-pre-processing-e2e-worker-w3-5c46af-7f86f8f5c5-jxhp6\" (UID: \"058aa7d8-1404-494f-b25b-70e8b702455b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5c46af-7f86f8f5c5-jxhp6"
Nov 02 19:38:59.842257 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:38:59.842175    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5c46af-7f86f8f5c5-jxhp6"
Nov 02 19:39:00.205486 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:00.205458    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-5c46af-7f86f8f5c5-jxhp6"]
Nov 02 19:39:00.207985 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:39:00.207954    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod058aa7d8_1404_494f_b25b_70e8b702455b.slice/crio-97b8cef001b3280ead2a46ceeb5f371afbe1aae313f380a51f6f36b4704e1efc WatchSource:0}: Error finding container 97b8cef001b3280ead2a46ceeb5f371afbe1aae313f380a51f6f36b4704e1efc: Status 404 returned error can't find the container with id 97b8cef001b3280ead2a46ceeb5f371afbe1aae313f380a51f6f36b4704e1efc
Nov 02 19:39:01.147886 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:01.147842    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5c46af-7f86f8f5c5-jxhp6" event={"ID":"058aa7d8-1404-494f-b25b-70e8b702455b","Type":"ContainerStarted","Data":"58575358d352bd7bd204f26186106be5d4d3bb920881611d0f6cb31f79938748"}
Nov 02 19:39:01.148360 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:01.147945    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5c46af-7f86f8f5c5-jxhp6" event={"ID":"058aa7d8-1404-494f-b25b-70e8b702455b","Type":"ContainerStarted","Data":"97b8cef001b3280ead2a46ceeb5f371afbe1aae313f380a51f6f36b4704e1efc"}
Nov 02 19:39:01.148360 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:01.147980    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5c46af-7f86f8f5c5-jxhp6"
Nov 02 19:39:01.254831 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:01.254689    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5c46af-7f86f8f5c5-jxhp6" podStartSLOduration=2.254656565 podStartE2EDuration="2.254656565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:38:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:39:01.254036127 +0000 UTC m=+647.861069887" watchObservedRunningTime="2026-11-02 19:39:01.254656565 +0000 UTC m=+647.861690338"
Nov 02 19:39:02.177239 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:02.177211    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-43ceaa-7bf8fd68b7-nstb8"]
Nov 02 19:39:02.180797 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:02.180780    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-43ceaa-7bf8fd68b7-nstb8"
Nov 02 19:39:02.218465 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:02.218432    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30d2bf45-98f1-4394-957b-18c5b768f902-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-43ceaa-7bf8fd68b7-nstb8\" (UID: \"30d2bf45-98f1-4394-957b-18c5b768f902\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-43ceaa-7bf8fd68b7-nstb8"
Nov 02 19:39:02.218650 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:02.218485    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrccq\" (UniqueName: \"kubernetes.io/projected/30d2bf45-98f1-4394-957b-18c5b768f902-kube-api-access-zrccq\") pod \"payload-pre-processing-e2e-worker-w0-43ceaa-7bf8fd68b7-nstb8\" (UID: \"30d2bf45-98f1-4394-957b-18c5b768f902\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-43ceaa-7bf8fd68b7-nstb8"
Nov 02 19:39:02.231139 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:02.231112    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-43ceaa\""
Nov 02 19:39:02.277300 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:02.277246    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-43ceaa-dockercfg-zmpmn\""
Nov 02 19:39:02.280441 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:02.280408    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-43ceaa-7bf8fd68b7-nstb8"]
Nov 02 19:39:02.319483 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:02.319435    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30d2bf45-98f1-4394-957b-18c5b768f902-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-43ceaa-7bf8fd68b7-nstb8\" (UID: \"30d2bf45-98f1-4394-957b-18c5b768f902\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-43ceaa-7bf8fd68b7-nstb8"
Nov 02 19:39:02.319677 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:02.319513    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrccq\" (UniqueName: \"kubernetes.io/projected/30d2bf45-98f1-4394-957b-18c5b768f902-kube-api-access-zrccq\") pod \"payload-pre-processing-e2e-worker-w0-43ceaa-7bf8fd68b7-nstb8\" (UID: \"30d2bf45-98f1-4394-957b-18c5b768f902\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-43ceaa-7bf8fd68b7-nstb8"
Nov 02 19:39:02.320307 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:02.320280    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30d2bf45-98f1-4394-957b-18c5b768f902-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-43ceaa-7bf8fd68b7-nstb8\" (UID: \"30d2bf45-98f1-4394-957b-18c5b768f902\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-43ceaa-7bf8fd68b7-nstb8"
Nov 02 19:39:02.373364 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:02.373291    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrccq\" (UniqueName: \"kubernetes.io/projected/30d2bf45-98f1-4394-957b-18c5b768f902-kube-api-access-zrccq\") pod \"payload-pre-processing-e2e-worker-w0-43ceaa-7bf8fd68b7-nstb8\" (UID: \"30d2bf45-98f1-4394-957b-18c5b768f902\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-43ceaa-7bf8fd68b7-nstb8"
Nov 02 19:39:02.493706 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:02.493615    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-43ceaa-7bf8fd68b7-nstb8"
Nov 02 19:39:02.862537 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:02.862489    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-43ceaa-7bf8fd68b7-nstb8"]
Nov 02 19:39:02.864729 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:39:02.864694    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30d2bf45_98f1_4394_957b_18c5b768f902.slice/crio-735c5718a2a39ebbf8259997da6a4c7489245bb77141a61ff10e8f0462a1223c WatchSource:0}: Error finding container 735c5718a2a39ebbf8259997da6a4c7489245bb77141a61ff10e8f0462a1223c: Status 404 returned error can't find the container with id 735c5718a2a39ebbf8259997da6a4c7489245bb77141a61ff10e8f0462a1223c
Nov 02 19:39:03.158080 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:03.157974    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-43ceaa-7bf8fd68b7-nstb8" event={"ID":"30d2bf45-98f1-4394-957b-18c5b768f902","Type":"ContainerStarted","Data":"0bba31894ee453d16c367d4a3d9a2205e6c826e728fc317058583a603bc2fb0d"}
Nov 02 19:39:03.158080 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:03.158038    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-43ceaa-7bf8fd68b7-nstb8" event={"ID":"30d2bf45-98f1-4394-957b-18c5b768f902","Type":"ContainerStarted","Data":"735c5718a2a39ebbf8259997da6a4c7489245bb77141a61ff10e8f0462a1223c"}
Nov 02 19:39:03.158267 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:03.158144    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-43ceaa-7bf8fd68b7-nstb8"
Nov 02 19:39:03.238967 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:03.238933    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 19:39:03.264120 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:03.264056    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-43ceaa-7bf8fd68b7-nstb8" podStartSLOduration=1.264007616 podStartE2EDuration="1.264007616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:39:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:39:03.263220029 +0000 UTC m=+649.870253798" watchObservedRunningTime="2026-11-02 19:39:03.264007616 +0000 UTC m=+649.871041386"
Nov 02 19:39:04.299576 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:39:04.299540    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 19:39:04.876297 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:04.876268    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-1d869e-7d8f7ccf9f-nhdnm"]
Nov 02 19:39:04.880109 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:04.880092    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-1d869e-7d8f7ccf9f-nhdnm"
Nov 02 19:39:04.927571 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:04.927544    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-1d869e-7d8f7ccf9f-nhdnm"]
Nov 02 19:39:04.947021 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:04.946980    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ebcf427-e9e4-4445-86d8-7d4a32398724-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-1d869e-7d8f7ccf9f-nhdnm\" (UID: \"7ebcf427-e9e4-4445-86d8-7d4a32398724\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-1d869e-7d8f7ccf9f-nhdnm"
Nov 02 19:39:04.947198 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:04.947176    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5gns\" (UniqueName: \"kubernetes.io/projected/7ebcf427-e9e4-4445-86d8-7d4a32398724-kube-api-access-p5gns\") pod \"payload-pre-processing-e2e-worker-w5-1d869e-7d8f7ccf9f-nhdnm\" (UID: \"7ebcf427-e9e4-4445-86d8-7d4a32398724\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-1d869e-7d8f7ccf9f-nhdnm"
Nov 02 19:39:04.973239 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:04.973206    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-1d869e\""
Nov 02 19:39:04.974144 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:04.974121    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-1d869e-dockercfg-7l89t\""
Nov 02 19:39:05.048552 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:05.048517    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5gns\" (UniqueName: \"kubernetes.io/projected/7ebcf427-e9e4-4445-86d8-7d4a32398724-kube-api-access-p5gns\") pod \"payload-pre-processing-e2e-worker-w5-1d869e-7d8f7ccf9f-nhdnm\" (UID: \"7ebcf427-e9e4-4445-86d8-7d4a32398724\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-1d869e-7d8f7ccf9f-nhdnm"
Nov 02 19:39:05.048729 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:05.048559    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ebcf427-e9e4-4445-86d8-7d4a32398724-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-1d869e-7d8f7ccf9f-nhdnm\" (UID: \"7ebcf427-e9e4-4445-86d8-7d4a32398724\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-1d869e-7d8f7ccf9f-nhdnm"
Nov 02 19:39:05.062035 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:05.049369    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ebcf427-e9e4-4445-86d8-7d4a32398724-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-1d869e-7d8f7ccf9f-nhdnm\" (UID: \"7ebcf427-e9e4-4445-86d8-7d4a32398724\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-1d869e-7d8f7ccf9f-nhdnm"
Nov 02 19:39:05.102863 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:05.102829    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5gns\" (UniqueName: \"kubernetes.io/projected/7ebcf427-e9e4-4445-86d8-7d4a32398724-kube-api-access-p5gns\") pod \"payload-pre-processing-e2e-worker-w5-1d869e-7d8f7ccf9f-nhdnm\" (UID: \"7ebcf427-e9e4-4445-86d8-7d4a32398724\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-1d869e-7d8f7ccf9f-nhdnm"
Nov 02 19:39:05.106609 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:05.106578    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-1d869e-7486c47697-4qtff"]
Nov 02 19:39:05.110591 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:05.110572    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-1d869e-7486c47697-4qtff"
Nov 02 19:39:05.169072 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:05.167067    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe88713d-8bbb-4ef3-b106-476db242f9a5-maas-api-tls\") pod \"maas-api-e2e-worker-w5-1d869e-7486c47697-4qtff\" (UID: \"fe88713d-8bbb-4ef3-b106-476db242f9a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-1d869e-7486c47697-4qtff"
Nov 02 19:39:05.169072 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:05.167850    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe88713d-8bbb-4ef3-b106-476db242f9a5-metrics-certs\") pod \"maas-api-e2e-worker-w5-1d869e-7486c47697-4qtff\" (UID: \"fe88713d-8bbb-4ef3-b106-476db242f9a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-1d869e-7486c47697-4qtff"
Nov 02 19:39:05.169072 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:05.167936    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe88713d-8bbb-4ef3-b106-476db242f9a5-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-1d869e-7486c47697-4qtff\" (UID: \"fe88713d-8bbb-4ef3-b106-476db242f9a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-1d869e-7486c47697-4qtff"
Nov 02 19:39:05.169072 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:05.168001    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvnmc\" (UniqueName: \"kubernetes.io/projected/fe88713d-8bbb-4ef3-b106-476db242f9a5-kube-api-access-dvnmc\") pod \"maas-api-e2e-worker-w5-1d869e-7486c47697-4qtff\" (UID: \"fe88713d-8bbb-4ef3-b106-476db242f9a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-1d869e-7486c47697-4qtff"
Nov 02 19:39:05.175189 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:05.170524    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-1d869e-7486c47697-4qtff"]
Nov 02 19:39:05.178693 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:05.178190    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-1d869e-serving-cert\""
Nov 02 19:39:05.193676 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:05.193650    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-1d869e-7d8f7ccf9f-nhdnm"
Nov 02 19:39:05.283823 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:05.283499    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvnmc\" (UniqueName: \"kubernetes.io/projected/fe88713d-8bbb-4ef3-b106-476db242f9a5-kube-api-access-dvnmc\") pod \"maas-api-e2e-worker-w5-1d869e-7486c47697-4qtff\" (UID: \"fe88713d-8bbb-4ef3-b106-476db242f9a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-1d869e-7486c47697-4qtff"
Nov 02 19:39:05.283823 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:05.283567    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe88713d-8bbb-4ef3-b106-476db242f9a5-maas-api-tls\") pod \"maas-api-e2e-worker-w5-1d869e-7486c47697-4qtff\" (UID: \"fe88713d-8bbb-4ef3-b106-476db242f9a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-1d869e-7486c47697-4qtff"
Nov 02 19:39:05.283823 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:05.283597    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe88713d-8bbb-4ef3-b106-476db242f9a5-metrics-certs\") pod \"maas-api-e2e-worker-w5-1d869e-7486c47697-4qtff\" (UID: \"fe88713d-8bbb-4ef3-b106-476db242f9a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-1d869e-7486c47697-4qtff"
Nov 02 19:39:05.283823 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:05.283632    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe88713d-8bbb-4ef3-b106-476db242f9a5-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-1d869e-7486c47697-4qtff\" (UID: \"fe88713d-8bbb-4ef3-b106-476db242f9a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-1d869e-7486c47697-4qtff"
Nov 02 19:39:05.284412 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:05.284385    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe88713d-8bbb-4ef3-b106-476db242f9a5-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-1d869e-7486c47697-4qtff\" (UID: \"fe88713d-8bbb-4ef3-b106-476db242f9a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-1d869e-7486c47697-4qtff"
Nov 02 19:39:05.284524 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:39:05.284505    4463 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w5-1d869e-serving-cert: secret "maas-api-e2e-worker-w5-1d869e-serving-cert" not found
Nov 02 19:39:05.284589 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:39:05.284569    4463 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe88713d-8bbb-4ef3-b106-476db242f9a5-maas-api-tls podName:fe88713d-8bbb-4ef3-b106-476db242f9a5 nodeName:}" failed. No retries permitted until 2026-11-02 19:39:05.784550831 +0000 UTC m=+652.391584593 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fe88713d-8bbb-4ef3-b106-476db242f9a5-maas-api-tls") pod "maas-api-e2e-worker-w5-1d869e-7486c47697-4qtff" (UID: "fe88713d-8bbb-4ef3-b106-476db242f9a5") : secret "maas-api-e2e-worker-w5-1d869e-serving-cert" not found
Nov 02 19:39:05.288527 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:05.288498    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe88713d-8bbb-4ef3-b106-476db242f9a5-metrics-certs\") pod \"maas-api-e2e-worker-w5-1d869e-7486c47697-4qtff\" (UID: \"fe88713d-8bbb-4ef3-b106-476db242f9a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-1d869e-7486c47697-4qtff"
Nov 02 19:39:05.338571 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:05.338352    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvnmc\" (UniqueName: \"kubernetes.io/projected/fe88713d-8bbb-4ef3-b106-476db242f9a5-kube-api-access-dvnmc\") pod \"maas-api-e2e-worker-w5-1d869e-7486c47697-4qtff\" (UID: \"fe88713d-8bbb-4ef3-b106-476db242f9a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-1d869e-7486c47697-4qtff"
Nov 02 19:39:05.597931 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:05.597903    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-1d869e-7d8f7ccf9f-nhdnm"]
Nov 02 19:39:05.600244 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:39:05.600215    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ebcf427_e9e4_4445_86d8_7d4a32398724.slice/crio-48b3ba40a95d0743447c6952c8131723282a95002e66912ce2c718b04fbaecdd WatchSource:0}: Error finding container 48b3ba40a95d0743447c6952c8131723282a95002e66912ce2c718b04fbaecdd: Status 404 returned error can't find the container with id 48b3ba40a95d0743447c6952c8131723282a95002e66912ce2c718b04fbaecdd
Nov 02 19:39:05.789345 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:05.789295    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe88713d-8bbb-4ef3-b106-476db242f9a5-maas-api-tls\") pod \"maas-api-e2e-worker-w5-1d869e-7486c47697-4qtff\" (UID: \"fe88713d-8bbb-4ef3-b106-476db242f9a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-1d869e-7486c47697-4qtff"
Nov 02 19:39:05.791611 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:05.791583    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe88713d-8bbb-4ef3-b106-476db242f9a5-maas-api-tls\") pod \"maas-api-e2e-worker-w5-1d869e-7486c47697-4qtff\" (UID: \"fe88713d-8bbb-4ef3-b106-476db242f9a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-1d869e-7486c47697-4qtff"
Nov 02 19:39:06.023707 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:06.023671    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-1d869e-7486c47697-4qtff"
Nov 02 19:39:06.191399 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:06.191361    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-1d869e-7d8f7ccf9f-nhdnm" event={"ID":"7ebcf427-e9e4-4445-86d8-7d4a32398724","Type":"ContainerStarted","Data":"99b3a6aa46bebea585ad09fe6e48386f4bafd9387866b0af60885be803e2ee75"}
Nov 02 19:39:06.191607 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:06.191419    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-1d869e-7d8f7ccf9f-nhdnm" event={"ID":"7ebcf427-e9e4-4445-86d8-7d4a32398724","Type":"ContainerStarted","Data":"48b3ba40a95d0743447c6952c8131723282a95002e66912ce2c718b04fbaecdd"}
Nov 02 19:39:06.191607 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:06.191478    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-1d869e-7d8f7ccf9f-nhdnm"
Nov 02 19:39:06.300574 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:06.300465    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-1d869e-7d8f7ccf9f-nhdnm" podStartSLOduration=2.300445934 podStartE2EDuration="2.300445934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:39:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:39:06.297802043 +0000 UTC m=+652.904835812" watchObservedRunningTime="2026-11-02 19:39:06.300445934 +0000 UTC m=+652.907479703"
Nov 02 19:39:06.416535 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:06.416078    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-1d869e-7486c47697-4qtff"]
Nov 02 19:39:06.419972 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:39:06.419938    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe88713d_8bbb_4ef3_b106_476db242f9a5.slice/crio-6d7b35f3ed2765d7924b6202a63a92178aafc7bd01dc2199ab0ac4fe869f4238 WatchSource:0}: Error finding container 6d7b35f3ed2765d7924b6202a63a92178aafc7bd01dc2199ab0ac4fe869f4238: Status 404 returned error can't find the container with id 6d7b35f3ed2765d7924b6202a63a92178aafc7bd01dc2199ab0ac4fe869f4238
Nov 02 19:39:07.130752 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:07.130721    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8795bf-66d99fc7c9-nj9gz"
Nov 02 19:39:07.130900 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:07.130784    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-8795bf-7bd68d77d9-5xm6w"
Nov 02 19:39:07.199179 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:07.199141    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-1d869e-7486c47697-4qtff" event={"ID":"fe88713d-8bbb-4ef3-b106-476db242f9a5","Type":"ContainerStarted","Data":"6d7b35f3ed2765d7924b6202a63a92178aafc7bd01dc2199ab0ac4fe869f4238"}
Nov 02 19:39:08.204943 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:08.204905    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-1d869e-7486c47697-4qtff" event={"ID":"fe88713d-8bbb-4ef3-b106-476db242f9a5","Type":"ContainerStarted","Data":"4934e4982e55082b3ebd053d7a9df07a1674225c4e2cc897ea109ff10dc0cef2"}
Nov 02 19:39:08.205356 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:08.204972    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-1d869e-7486c47697-4qtff"
Nov 02 19:39:08.327509 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:08.327457    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-1d869e-7486c47697-4qtff" podStartSLOduration=2.511593589 podStartE2EDuration="3.327444631s" totalImagesPullingTime="815.851042ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:39:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:39:08.325218023 +0000 UTC m=+654.932251797" watchObservedRunningTime="2026-11-02 19:39:08.327444631 +0000 UTC m=+654.934478400"
Nov 02 19:39:10.007359 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:10.007320    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-26cfbe-674f8bdf75-kjvkz"]
Nov 02 19:39:10.011032 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:10.010986    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-26cfbe-674f8bdf75-kjvkz"
Nov 02 19:39:10.059543 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:10.059515    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-26cfbe\""
Nov 02 19:39:10.108683 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:10.108637    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-26cfbe-674f8bdf75-kjvkz"]
Nov 02 19:39:10.136041 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:10.135467    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03ec9733-7e8d-49fa-94d6-6000b30c0e16-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-26cfbe-674f8bdf75-kjvkz\" (UID: \"03ec9733-7e8d-49fa-94d6-6000b30c0e16\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-26cfbe-674f8bdf75-kjvkz"
Nov 02 19:39:10.136041 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:10.135549    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6644h\" (UniqueName: \"kubernetes.io/projected/03ec9733-7e8d-49fa-94d6-6000b30c0e16-kube-api-access-6644h\") pod \"payload-pre-processing-e2e-worker-w1-26cfbe-674f8bdf75-kjvkz\" (UID: \"03ec9733-7e8d-49fa-94d6-6000b30c0e16\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-26cfbe-674f8bdf75-kjvkz"
Nov 02 19:39:10.156042 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:10.152312    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-26cfbe-dockercfg-xfmsw\""
Nov 02 19:39:10.237280 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:10.236818    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6644h\" (UniqueName: \"kubernetes.io/projected/03ec9733-7e8d-49fa-94d6-6000b30c0e16-kube-api-access-6644h\") pod \"payload-pre-processing-e2e-worker-w1-26cfbe-674f8bdf75-kjvkz\" (UID: \"03ec9733-7e8d-49fa-94d6-6000b30c0e16\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-26cfbe-674f8bdf75-kjvkz"
Nov 02 19:39:10.237280 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:10.237031    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03ec9733-7e8d-49fa-94d6-6000b30c0e16-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-26cfbe-674f8bdf75-kjvkz\" (UID: \"03ec9733-7e8d-49fa-94d6-6000b30c0e16\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-26cfbe-674f8bdf75-kjvkz"
Nov 02 19:39:10.238744 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:10.238339    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03ec9733-7e8d-49fa-94d6-6000b30c0e16-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-26cfbe-674f8bdf75-kjvkz\" (UID: \"03ec9733-7e8d-49fa-94d6-6000b30c0e16\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-26cfbe-674f8bdf75-kjvkz"
Nov 02 19:39:10.293067 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:10.292976    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6644h\" (UniqueName: \"kubernetes.io/projected/03ec9733-7e8d-49fa-94d6-6000b30c0e16-kube-api-access-6644h\") pod \"payload-pre-processing-e2e-worker-w1-26cfbe-674f8bdf75-kjvkz\" (UID: \"03ec9733-7e8d-49fa-94d6-6000b30c0e16\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-26cfbe-674f8bdf75-kjvkz"
Nov 02 19:39:10.327890 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:10.327847    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-26cfbe-674f8bdf75-kjvkz"
Nov 02 19:39:10.707834 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:10.707794    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-26cfbe-674f8bdf75-kjvkz"]
Nov 02 19:39:10.710442 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:39:10.710404    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03ec9733_7e8d_49fa_94d6_6000b30c0e16.slice/crio-acf2b941581620028a56e7e91b0c563df9185f5ff4d320e60cd0dabfe1daab9a WatchSource:0}: Error finding container acf2b941581620028a56e7e91b0c563df9185f5ff4d320e60cd0dabfe1daab9a: Status 404 returned error can't find the container with id acf2b941581620028a56e7e91b0c563df9185f5ff4d320e60cd0dabfe1daab9a
Nov 02 19:39:11.224808 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:11.224775    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-26cfbe-674f8bdf75-kjvkz" event={"ID":"03ec9733-7e8d-49fa-94d6-6000b30c0e16","Type":"ContainerStarted","Data":"5df7ddcae246819d46c049c0004141bab0c810f3b8573f22f2b5ac632fbf2565"}
Nov 02 19:39:11.225304 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:11.224834    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-26cfbe-674f8bdf75-kjvkz" event={"ID":"03ec9733-7e8d-49fa-94d6-6000b30c0e16","Type":"ContainerStarted","Data":"acf2b941581620028a56e7e91b0c563df9185f5ff4d320e60cd0dabfe1daab9a"}
Nov 02 19:39:11.225304 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:11.224923    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-26cfbe-674f8bdf75-kjvkz"
Nov 02 19:39:11.337037 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:11.336963    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-26cfbe-674f8bdf75-kjvkz" podStartSLOduration=2.3369482489999998 podStartE2EDuration="2.336948249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:39:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:39:11.334766217 +0000 UTC m=+657.941799991" watchObservedRunningTime="2026-11-02 19:39:11.336948249 +0000 UTC m=+657.943982018"
Nov 02 19:39:11.971185 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:11.971148    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 19:39:12.176036 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:12.175216    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5c46af-7f86f8f5c5-jxhp6"
Nov 02 19:39:13.235274 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:13.235232    4463 generic.go:356] "Generic (PLEG): container finished" podID="a8216408-95d0-413b-a609-2ca8cf3299b6" containerID="22cf3df17c532e6521ea22300285e0798bc8f9e96c1c973b1d851db1f9dd597c" exitCode=0
Nov 02 19:39:13.235821 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:13.235279    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-2zrqj" event={"ID":"a8216408-95d0-413b-a609-2ca8cf3299b6","Type":"ContainerDied","Data":"22cf3df17c532e6521ea22300285e0798bc8f9e96c1c973b1d851db1f9dd597c"}
Nov 02 19:39:13.235821 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:13.235759    4463 scope.go:122] "RemoveContainer" containerID="22cf3df17c532e6521ea22300285e0798bc8f9e96c1c973b1d851db1f9dd597c"
Nov 02 19:39:13.237051 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:13.237022    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:39:13.237153 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:13.237086    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:39:13.237153 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:13.237119    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:39:14.198226 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:14.198194    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-43ceaa-7bf8fd68b7-nstb8"
Nov 02 19:39:14.215212 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:14.215183    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-1d869e-7486c47697-4qtff"
Nov 02 19:39:14.242490 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:14.242465    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-2zrqj" event={"ID":"a8216408-95d0-413b-a609-2ca8cf3299b6","Type":"ContainerStarted","Data":"f56ee5a76c4bf7aeafdf7fb898d4cbccecfe3d39d9505d83fa7eea05d94a0c75"}
Nov 02 19:39:14.287038 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:14.286643    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-2zrqj"
Nov 02 19:39:14.287038 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:14.286800    4463 patch_prober.go:29] interesting pod/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-2zrqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Nov 02 19:39:14.287038 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:14.286848    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-2zrqj" podUID="a8216408-95d0-413b-a609-2ca8cf3299b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Nov 02 19:39:14.819573 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:14.819534    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-26cfbe-76898cf9c6-rlrzb"]
Nov 02 19:39:14.822709 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:14.822690    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-26cfbe-76898cf9c6-rlrzb"
Nov 02 19:39:14.880038 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:14.879745    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-26cfbe-76898cf9c6-rlrzb"]
Nov 02 19:39:14.880038 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:14.879976    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-26cfbe-serving-cert\""
Nov 02 19:39:14.914660 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:14.914620    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dggw4\" (UniqueName: \"kubernetes.io/projected/91cbd659-c151-4a51-a9a9-236ddf601f8d-kube-api-access-dggw4\") pod \"maas-api-e2e-worker-w1-26cfbe-76898cf9c6-rlrzb\" (UID: \"91cbd659-c151-4a51-a9a9-236ddf601f8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-26cfbe-76898cf9c6-rlrzb"
Nov 02 19:39:14.914974 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:14.914948    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91cbd659-c151-4a51-a9a9-236ddf601f8d-maas-api-tls\") pod \"maas-api-e2e-worker-w1-26cfbe-76898cf9c6-rlrzb\" (UID: \"91cbd659-c151-4a51-a9a9-236ddf601f8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-26cfbe-76898cf9c6-rlrzb"
Nov 02 19:39:14.915164 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:14.915144    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91cbd659-c151-4a51-a9a9-236ddf601f8d-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-26cfbe-76898cf9c6-rlrzb\" (UID: \"91cbd659-c151-4a51-a9a9-236ddf601f8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-26cfbe-76898cf9c6-rlrzb"
Nov 02 19:39:14.915353 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:14.915337    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91cbd659-c151-4a51-a9a9-236ddf601f8d-metrics-certs\") pod \"maas-api-e2e-worker-w1-26cfbe-76898cf9c6-rlrzb\" (UID: \"91cbd659-c151-4a51-a9a9-236ddf601f8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-26cfbe-76898cf9c6-rlrzb"
Nov 02 19:39:15.016738 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:15.016711    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dggw4\" (UniqueName: \"kubernetes.io/projected/91cbd659-c151-4a51-a9a9-236ddf601f8d-kube-api-access-dggw4\") pod \"maas-api-e2e-worker-w1-26cfbe-76898cf9c6-rlrzb\" (UID: \"91cbd659-c151-4a51-a9a9-236ddf601f8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-26cfbe-76898cf9c6-rlrzb"
Nov 02 19:39:15.016948 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:15.016935    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91cbd659-c151-4a51-a9a9-236ddf601f8d-maas-api-tls\") pod \"maas-api-e2e-worker-w1-26cfbe-76898cf9c6-rlrzb\" (UID: \"91cbd659-c151-4a51-a9a9-236ddf601f8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-26cfbe-76898cf9c6-rlrzb"
Nov 02 19:39:15.017051 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:15.017041    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91cbd659-c151-4a51-a9a9-236ddf601f8d-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-26cfbe-76898cf9c6-rlrzb\" (UID: \"91cbd659-c151-4a51-a9a9-236ddf601f8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-26cfbe-76898cf9c6-rlrzb"
Nov 02 19:39:15.017217 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:15.017207    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91cbd659-c151-4a51-a9a9-236ddf601f8d-metrics-certs\") pod \"maas-api-e2e-worker-w1-26cfbe-76898cf9c6-rlrzb\" (UID: \"91cbd659-c151-4a51-a9a9-236ddf601f8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-26cfbe-76898cf9c6-rlrzb"
Nov 02 19:39:15.018221 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:39:15.018199    4463 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w1-26cfbe-serving-cert: secret "maas-api-e2e-worker-w1-26cfbe-serving-cert" not found
Nov 02 19:39:15.018454 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:39:15.018437    4463 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91cbd659-c151-4a51-a9a9-236ddf601f8d-maas-api-tls podName:91cbd659-c151-4a51-a9a9-236ddf601f8d nodeName:}" failed. No retries permitted until 2026-11-02 19:39:15.51841474 +0000 UTC m=+662.125448487 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/91cbd659-c151-4a51-a9a9-236ddf601f8d-maas-api-tls") pod "maas-api-e2e-worker-w1-26cfbe-76898cf9c6-rlrzb" (UID: "91cbd659-c151-4a51-a9a9-236ddf601f8d") : secret "maas-api-e2e-worker-w1-26cfbe-serving-cert" not found
Nov 02 19:39:15.018641 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:15.018620    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91cbd659-c151-4a51-a9a9-236ddf601f8d-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-26cfbe-76898cf9c6-rlrzb\" (UID: \"91cbd659-c151-4a51-a9a9-236ddf601f8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-26cfbe-76898cf9c6-rlrzb"
Nov 02 19:39:15.020198 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:15.020177    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91cbd659-c151-4a51-a9a9-236ddf601f8d-metrics-certs\") pod \"maas-api-e2e-worker-w1-26cfbe-76898cf9c6-rlrzb\" (UID: \"91cbd659-c151-4a51-a9a9-236ddf601f8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-26cfbe-76898cf9c6-rlrzb"
Nov 02 19:39:15.120411 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:15.119331    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dggw4\" (UniqueName: \"kubernetes.io/projected/91cbd659-c151-4a51-a9a9-236ddf601f8d-kube-api-access-dggw4\") pod \"maas-api-e2e-worker-w1-26cfbe-76898cf9c6-rlrzb\" (UID: \"91cbd659-c151-4a51-a9a9-236ddf601f8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-26cfbe-76898cf9c6-rlrzb"
Nov 02 19:39:15.288034 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:15.285377    4463 patch_prober.go:29] interesting pod/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-2zrqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Nov 02 19:39:15.288034 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:15.285446    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-2zrqj" podUID="a8216408-95d0-413b-a609-2ca8cf3299b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Nov 02 19:39:15.522539 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:15.522502    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91cbd659-c151-4a51-a9a9-236ddf601f8d-maas-api-tls\") pod \"maas-api-e2e-worker-w1-26cfbe-76898cf9c6-rlrzb\" (UID: \"91cbd659-c151-4a51-a9a9-236ddf601f8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-26cfbe-76898cf9c6-rlrzb"
Nov 02 19:39:15.524876 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:15.524856    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91cbd659-c151-4a51-a9a9-236ddf601f8d-maas-api-tls\") pod \"maas-api-e2e-worker-w1-26cfbe-76898cf9c6-rlrzb\" (UID: \"91cbd659-c151-4a51-a9a9-236ddf601f8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-26cfbe-76898cf9c6-rlrzb"
Nov 02 19:39:15.735140 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:15.735108    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-26cfbe-76898cf9c6-rlrzb"
Nov 02 19:39:16.148997 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:16.148974    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-26cfbe-76898cf9c6-rlrzb"]
Nov 02 19:39:16.162691 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:39:16.162638    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91cbd659_c151_4a51_a9a9_236ddf601f8d.slice/crio-e4cff7acd3cb83eba1a3c2aa3c55237339d5467e59d2283b99ef361c26fe9501 WatchSource:0}: Error finding container e4cff7acd3cb83eba1a3c2aa3c55237339d5467e59d2283b99ef361c26fe9501: Status 404 returned error can't find the container with id e4cff7acd3cb83eba1a3c2aa3c55237339d5467e59d2283b99ef361c26fe9501
Nov 02 19:39:16.295039 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:16.291509    4463 patch_prober.go:29] interesting pod/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-2zrqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Nov 02 19:39:16.295039 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:16.291570    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-2zrqj" podUID="a8216408-95d0-413b-a609-2ca8cf3299b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Nov 02 19:39:16.304143 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:16.304064    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-26cfbe-76898cf9c6-rlrzb" event={"ID":"91cbd659-c151-4a51-a9a9-236ddf601f8d","Type":"ContainerStarted","Data":"e4cff7acd3cb83eba1a3c2aa3c55237339d5467e59d2283b99ef361c26fe9501"}
Nov 02 19:39:17.204427 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:17.204393    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-1d869e-7d8f7ccf9f-nhdnm"
Nov 02 19:39:17.283504 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:17.283415    4463 patch_prober.go:29] interesting pod/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-2zrqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Nov 02 19:39:17.283763 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:17.283734    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-2zrqj" podUID="a8216408-95d0-413b-a609-2ca8cf3299b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Nov 02 19:39:17.310175 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:17.310140    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-26cfbe-76898cf9c6-rlrzb" event={"ID":"91cbd659-c151-4a51-a9a9-236ddf601f8d","Type":"ContainerStarted","Data":"8dd82949eaf1ba814caed6432aa28aa0791d3ffa38521bb0f34fe1b89713b437"}
Nov 02 19:39:17.310701 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:17.310683    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-26cfbe-76898cf9c6-rlrzb"
Nov 02 19:39:17.431137 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:17.431065    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-26cfbe-76898cf9c6-rlrzb" podStartSLOduration=2.576964215 podStartE2EDuration="3.43104403s" totalImagesPullingTime="854.079815ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:39:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:39:17.428300006 +0000 UTC m=+664.035333774" watchObservedRunningTime="2026-11-02 19:39:17.43104403 +0000 UTC m=+664.038077802"
Nov 02 19:39:18.080835 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:18.080758    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-2zrqj"
Nov 02 19:39:18.283546 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:18.283504    4463 patch_prober.go:29] interesting pod/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-2zrqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Nov 02 19:39:18.283702 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:18.283566    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-2zrqj" podUID="a8216408-95d0-413b-a609-2ca8cf3299b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Nov 02 19:39:19.081422 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:19.081382    4463 patch_prober.go:29] interesting pod/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-fh467 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 19:39:19.081887 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:19.081443    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-fh467" podUID="0f76ece2-703b-41b2-8833-f909342a8390" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:39:19.283815 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:19.283770    4463 patch_prober.go:29] interesting pod/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-2zrqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Nov 02 19:39:19.283991 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:19.283842    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-2zrqj" podUID="a8216408-95d0-413b-a609-2ca8cf3299b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Nov 02 19:39:20.287799 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:20.287769    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-2zrqj"
Nov 02 19:39:20.288904 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:20.288878    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-2zrqj"
Nov 02 19:39:21.332914 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:21.332837    4463 generic.go:356] "Generic (PLEG): container finished" podID="0f76ece2-703b-41b2-8833-f909342a8390" containerID="a5850cb61c7e98c613c433a5ac1dc76279be71e87e4c7fa07c31a4c86a55e91d" exitCode=0
Nov 02 19:39:21.332914 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:21.332868    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-fh467" event={"ID":"0f76ece2-703b-41b2-8833-f909342a8390","Type":"ContainerDied","Data":"a5850cb61c7e98c613c433a5ac1dc76279be71e87e4c7fa07c31a4c86a55e91d"}
Nov 02 19:39:21.333386 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:21.333253    4463 scope.go:122] "RemoveContainer" containerID="a5850cb61c7e98c613c433a5ac1dc76279be71e87e4c7fa07c31a4c86a55e91d"
Nov 02 19:39:21.334573 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:21.334551    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:39:21.334632 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:21.334616    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:39:21.334667 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:21.334647    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:39:21.770627 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:21.770581    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 19:39:22.233369 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:22.233342    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-26cfbe-674f8bdf75-kjvkz"
Nov 02 19:39:22.339702 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:22.339665    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-fh467" event={"ID":"0f76ece2-703b-41b2-8833-f909342a8390","Type":"ContainerStarted","Data":"9c469b9a1882050cec9f8501a6de988758cc55964828388b535c7ff949341a01"}
Nov 02 19:39:22.904344 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:22.904297    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-fh467"
Nov 02 19:39:22.904616 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:22.904586    4463 patch_prober.go:29] interesting pod/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-fh467 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Nov 02 19:39:22.904725 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:22.904650    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-fh467" podUID="0f76ece2-703b-41b2-8833-f909342a8390" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Nov 02 19:39:23.904379 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:23.904345    4463 patch_prober.go:29] interesting pod/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-fh467 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Nov 02 19:39:23.904785 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:23.904398    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-fh467" podUID="0f76ece2-703b-41b2-8833-f909342a8390" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Nov 02 19:39:24.327133 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:24.327104    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-26cfbe-76898cf9c6-rlrzb"
Nov 02 19:39:24.904426 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:24.904393    4463 patch_prober.go:29] interesting pod/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-fh467 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Nov 02 19:39:24.904820 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:24.904447    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-fh467" podUID="0f76ece2-703b-41b2-8833-f909342a8390" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Nov 02 19:39:25.904070 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:25.903959    4463 patch_prober.go:29] interesting pod/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-fh467 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Nov 02 19:39:25.904070 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:25.904048    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-fh467" podUID="0f76ece2-703b-41b2-8833-f909342a8390" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Nov 02 19:39:26.904532 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:26.904495    4463 patch_prober.go:29] interesting pod/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-fh467 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Nov 02 19:39:26.904929 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:26.904546    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-fh467" podUID="0f76ece2-703b-41b2-8833-f909342a8390" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Nov 02 19:39:27.907869 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:27.907839    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-fh467"
Nov 02 19:39:27.908285 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:27.908224    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-fh467"
Nov 02 19:39:27.908855 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:27.908836    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-fh467"
Nov 02 19:39:34.108162 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:34.108129    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-28623b-5b7449b6b4-f4ddr"]
Nov 02 19:39:34.111946 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:34.111931    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-28623b-5b7449b6b4-f4ddr"
Nov 02 19:39:34.209412 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:34.209378    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-28623b-serving-cert\""
Nov 02 19:39:34.211559 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:34.211529    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-28623b-5b7449b6b4-f4ddr"]
Nov 02 19:39:34.279519 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:34.279481    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0fdc6d2c-45e4-48be-bf24-93ed75441397-maas-api-tls\") pod \"maas-api-e2e-worker-w2-28623b-5b7449b6b4-f4ddr\" (UID: \"0fdc6d2c-45e4-48be-bf24-93ed75441397\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-28623b-5b7449b6b4-f4ddr"
Nov 02 19:39:34.279705 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:34.279567    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrtdq\" (UniqueName: \"kubernetes.io/projected/0fdc6d2c-45e4-48be-bf24-93ed75441397-kube-api-access-xrtdq\") pod \"maas-api-e2e-worker-w2-28623b-5b7449b6b4-f4ddr\" (UID: \"0fdc6d2c-45e4-48be-bf24-93ed75441397\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-28623b-5b7449b6b4-f4ddr"
Nov 02 19:39:34.279705 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:34.279621    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fdc6d2c-45e4-48be-bf24-93ed75441397-metrics-certs\") pod \"maas-api-e2e-worker-w2-28623b-5b7449b6b4-f4ddr\" (UID: \"0fdc6d2c-45e4-48be-bf24-93ed75441397\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-28623b-5b7449b6b4-f4ddr"
Nov 02 19:39:34.279705 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:34.279671    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fdc6d2c-45e4-48be-bf24-93ed75441397-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-28623b-5b7449b6b4-f4ddr\" (UID: \"0fdc6d2c-45e4-48be-bf24-93ed75441397\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-28623b-5b7449b6b4-f4ddr"
Nov 02 19:39:34.298706 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:39:34.298665    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 19:39:34.380485 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:34.380402    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fdc6d2c-45e4-48be-bf24-93ed75441397-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-28623b-5b7449b6b4-f4ddr\" (UID: \"0fdc6d2c-45e4-48be-bf24-93ed75441397\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-28623b-5b7449b6b4-f4ddr"
Nov 02 19:39:34.380485 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:34.380469    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0fdc6d2c-45e4-48be-bf24-93ed75441397-maas-api-tls\") pod \"maas-api-e2e-worker-w2-28623b-5b7449b6b4-f4ddr\" (UID: \"0fdc6d2c-45e4-48be-bf24-93ed75441397\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-28623b-5b7449b6b4-f4ddr"
Nov 02 19:39:34.380701 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:34.380521    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrtdq\" (UniqueName: \"kubernetes.io/projected/0fdc6d2c-45e4-48be-bf24-93ed75441397-kube-api-access-xrtdq\") pod \"maas-api-e2e-worker-w2-28623b-5b7449b6b4-f4ddr\" (UID: \"0fdc6d2c-45e4-48be-bf24-93ed75441397\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-28623b-5b7449b6b4-f4ddr"
Nov 02 19:39:34.380701 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:34.380575    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fdc6d2c-45e4-48be-bf24-93ed75441397-metrics-certs\") pod \"maas-api-e2e-worker-w2-28623b-5b7449b6b4-f4ddr\" (UID: \"0fdc6d2c-45e4-48be-bf24-93ed75441397\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-28623b-5b7449b6b4-f4ddr"
Nov 02 19:39:34.380701 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:39:34.380606    4463 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w2-28623b-serving-cert: secret "maas-api-e2e-worker-w2-28623b-serving-cert" not found
Nov 02 19:39:34.380701 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:39:34.380685    4463 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fdc6d2c-45e4-48be-bf24-93ed75441397-maas-api-tls podName:0fdc6d2c-45e4-48be-bf24-93ed75441397 nodeName:}" failed. No retries permitted until 2026-11-02 19:39:34.88066262 +0000 UTC m=+681.487696385 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0fdc6d2c-45e4-48be-bf24-93ed75441397-maas-api-tls") pod "maas-api-e2e-worker-w2-28623b-5b7449b6b4-f4ddr" (UID: "0fdc6d2c-45e4-48be-bf24-93ed75441397") : secret "maas-api-e2e-worker-w2-28623b-serving-cert" not found
Nov 02 19:39:34.381094 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:34.381070    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fdc6d2c-45e4-48be-bf24-93ed75441397-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-28623b-5b7449b6b4-f4ddr\" (UID: \"0fdc6d2c-45e4-48be-bf24-93ed75441397\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-28623b-5b7449b6b4-f4ddr"
Nov 02 19:39:34.383151 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:34.383131    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fdc6d2c-45e4-48be-bf24-93ed75441397-metrics-certs\") pod \"maas-api-e2e-worker-w2-28623b-5b7449b6b4-f4ddr\" (UID: \"0fdc6d2c-45e4-48be-bf24-93ed75441397\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-28623b-5b7449b6b4-f4ddr"
Nov 02 19:39:34.434080 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:34.434053    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrtdq\" (UniqueName: \"kubernetes.io/projected/0fdc6d2c-45e4-48be-bf24-93ed75441397-kube-api-access-xrtdq\") pod \"maas-api-e2e-worker-w2-28623b-5b7449b6b4-f4ddr\" (UID: \"0fdc6d2c-45e4-48be-bf24-93ed75441397\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-28623b-5b7449b6b4-f4ddr"
Nov 02 19:39:34.885871 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:34.885823    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0fdc6d2c-45e4-48be-bf24-93ed75441397-maas-api-tls\") pod \"maas-api-e2e-worker-w2-28623b-5b7449b6b4-f4ddr\" (UID: \"0fdc6d2c-45e4-48be-bf24-93ed75441397\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-28623b-5b7449b6b4-f4ddr"
Nov 02 19:39:34.888377 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:34.888343    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0fdc6d2c-45e4-48be-bf24-93ed75441397-maas-api-tls\") pod \"maas-api-e2e-worker-w2-28623b-5b7449b6b4-f4ddr\" (UID: \"0fdc6d2c-45e4-48be-bf24-93ed75441397\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-28623b-5b7449b6b4-f4ddr"
Nov 02 19:39:35.024641 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:35.024594    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-28623b-5b7449b6b4-f4ddr"
Nov 02 19:39:35.388889 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:35.388859    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-28623b-5b7449b6b4-f4ddr"]
Nov 02 19:39:35.392333 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:39:35.392303    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fdc6d2c_45e4_48be_bf24_93ed75441397.slice/crio-fb3aa1e0ab7e4cc49b5305bf96dbdf3b05076cf788eb821c85b4bfc70935b29d WatchSource:0}: Error finding container fb3aa1e0ab7e4cc49b5305bf96dbdf3b05076cf788eb821c85b4bfc70935b29d: Status 404 returned error can't find the container with id fb3aa1e0ab7e4cc49b5305bf96dbdf3b05076cf788eb821c85b4bfc70935b29d
Nov 02 19:39:35.394134 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:35.394112    4463 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 19:39:35.426550 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:35.426508    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-28623b-5b7449b6b4-f4ddr" event={"ID":"0fdc6d2c-45e4-48be-bf24-93ed75441397","Type":"ContainerStarted","Data":"fb3aa1e0ab7e4cc49b5305bf96dbdf3b05076cf788eb821c85b4bfc70935b29d"}
Nov 02 19:39:36.432978 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:36.432944    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-28623b-5b7449b6b4-f4ddr" event={"ID":"0fdc6d2c-45e4-48be-bf24-93ed75441397","Type":"ContainerStarted","Data":"633e6c13ae459cea676d4e97b12e32b857493ccce177d3c765883662a9e77b8d"}
Nov 02 19:39:36.433402 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:36.433142    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-28623b-5b7449b6b4-f4ddr"
Nov 02 19:39:36.540800 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:36.540699    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-28623b-5b7449b6b4-f4ddr" podStartSLOduration=1.702012307 podStartE2EDuration="2.540686889s" totalImagesPullingTime="838.674582ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:39:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:39:36.539067189 +0000 UTC m=+683.146100979" watchObservedRunningTime="2026-11-02 19:39:36.540686889 +0000 UTC m=+683.147720657"
Nov 02 19:39:39.173988 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:39.173945    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 19:39:42.443002 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:42.442973    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-28623b-5b7449b6b4-f4ddr"
Nov 02 19:39:48.531342 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:48.531298    4463 generic.go:356] "Generic (PLEG): container finished" podID="f5734864-6a35-46c3-80b8-bdac768e1de1" containerID="23387a024c12a6210c34a1f28ef800c18033948f5ff83a61e84ac54b4597fd4b" exitCode=0
Nov 02 19:39:48.531754 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:48.531367    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9ndrz" event={"ID":"f5734864-6a35-46c3-80b8-bdac768e1de1","Type":"ContainerDied","Data":"23387a024c12a6210c34a1f28ef800c18033948f5ff83a61e84ac54b4597fd4b"}
Nov 02 19:39:48.531754 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:48.531722    4463 scope.go:122] "RemoveContainer" containerID="23387a024c12a6210c34a1f28ef800c18033948f5ff83a61e84ac54b4597fd4b"
Nov 02 19:39:48.533118 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:48.533092    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:39:48.533202 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:48.533153    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:39:48.533202 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:48.533186    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:39:49.086952 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:49.086921    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9ndrz"
Nov 02 19:39:49.538640 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:49.538608    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9ndrz" event={"ID":"f5734864-6a35-46c3-80b8-bdac768e1de1","Type":"ContainerStarted","Data":"916aecd458917f4668c09a0921566e17197367f37bcc3dd208817f4d230a20c4"}
Nov 02 19:39:50.274246 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:50.274202    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9ndrz"
Nov 02 19:39:50.275677 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:50.275652    4463 patch_prober.go:29] interesting pod/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9ndrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Nov 02 19:39:50.275790 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:50.275702    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9ndrz" podUID="f5734864-6a35-46c3-80b8-bdac768e1de1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Nov 02 19:39:51.274590 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:51.274485    4463 patch_prober.go:29] interesting pod/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9ndrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Nov 02 19:39:51.274590 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:51.274560    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9ndrz" podUID="f5734864-6a35-46c3-80b8-bdac768e1de1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Nov 02 19:39:52.273449 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:52.273409    4463 patch_prober.go:29] interesting pod/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9ndrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Nov 02 19:39:52.273753 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:52.273718    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9ndrz" podUID="f5734864-6a35-46c3-80b8-bdac768e1de1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Nov 02 19:39:53.273035 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:53.272392    4463 patch_prober.go:29] interesting pod/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9ndrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Nov 02 19:39:53.273035 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:53.272466    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9ndrz" podUID="f5734864-6a35-46c3-80b8-bdac768e1de1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Nov 02 19:39:54.274480 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:54.274441    4463 patch_prober.go:29] interesting pod/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9ndrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Nov 02 19:39:54.274870 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:54.274508    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9ndrz" podUID="f5734864-6a35-46c3-80b8-bdac768e1de1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Nov 02 19:39:55.276868 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:55.276836    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9ndrz"
Nov 02 19:39:55.277270 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:55.277209    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9ndrz"
Nov 02 19:39:55.277809 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:39:55.277791    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9ndrz"
Nov 02 19:40:03.649978 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:40:03.649944    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 19:40:04.294441 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:40:04.294403    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 19:40:22.262710 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:40:22.262675    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 19:40:34.293883 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:40:34.293848    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 19:40:36.304563 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:40:36.304527    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-26cfbe/simulator-w1-26cfbe-kserve-74c5578fbd-t79hq"]
Nov 02 19:40:36.308363 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:40:36.308343    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-26cfbe/simulator-w1-26cfbe-kserve-74c5578fbd-t79hq"
Nov 02 19:40:36.379083 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:40:36.379035    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-26cfbe/simulator-w1-26cfbe-kserve-74c5578fbd-t79hq"]
Nov 02 19:40:36.405339 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:40:36.405298    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-26cfbe\"/\"openshift-service-ca.crt\""
Nov 02 19:40:36.405339 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:40:36.405317    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-26cfbe\"/\"kube-root-ca.crt\""
Nov 02 19:40:36.405567 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:40:36.405423    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-26cfbe\"/\"simulator-w1-26cfbe-kserve-self-signed-certs\""
Nov 02 19:40:36.405567 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:40:36.405471    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-26cfbe\"/\"default-dockercfg-dnjgm\""
Nov 02 19:40:36.447094 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:40:36.447057    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50dc76f4-aebd-41f4-9f23-799ddc8c70f4-home\") pod \"simulator-w1-26cfbe-kserve-74c5578fbd-t79hq\" (UID: \"50dc76f4-aebd-41f4-9f23-799ddc8c70f4\") " pod="e2e-models-e2e-worker-w1-26cfbe/simulator-w1-26cfbe-kserve-74c5578fbd-t79hq"
Nov 02 19:40:36.447210 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:40:36.447176    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk5g9\" (UniqueName: \"kubernetes.io/projected/50dc76f4-aebd-41f4-9f23-799ddc8c70f4-kube-api-access-zk5g9\") pod \"simulator-w1-26cfbe-kserve-74c5578fbd-t79hq\" (UID: \"50dc76f4-aebd-41f4-9f23-799ddc8c70f4\") " pod="e2e-models-e2e-worker-w1-26cfbe/simulator-w1-26cfbe-kserve-74c5578fbd-t79hq"
Nov 02 19:40:36.447251 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:40:36.447223    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50dc76f4-aebd-41f4-9f23-799ddc8c70f4-dshm\") pod \"simulator-w1-26cfbe-kserve-74c5578fbd-t79hq\" (UID: \"50dc76f4-aebd-41f4-9f23-799ddc8c70f4\") " pod="e2e-models-e2e-worker-w1-26cfbe/simulator-w1-26cfbe-kserve-74c5578fbd-t79hq"
Nov 02 19:40:36.447284 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:40:36.447253    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50dc76f4-aebd-41f4-9f23-799ddc8c70f4-model-cache\") pod \"simulator-w1-26cfbe-kserve-74c5578fbd-t79hq\" (UID: \"50dc76f4-aebd-41f4-9f23-799ddc8c70f4\") " pod="e2e-models-e2e-worker-w1-26cfbe/simulator-w1-26cfbe-kserve-74c5578fbd-t79hq"
Nov 02 19:40:36.447413 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:40:36.447400    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50dc76f4-aebd-41f4-9f23-799ddc8c70f4-tmp-dir\") pod \"simulator-w1-26cfbe-kserve-74c5578fbd-t79hq\" (UID: \"50dc76f4-aebd-41f4-9f23-799ddc8c70f4\") " pod="e2e-models-e2e-worker-w1-26cfbe/simulator-w1-26cfbe-kserve-74c5578fbd-t79hq"
Nov 02 19:40:36.447448 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:40:36.447419    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50dc76f4-aebd-41f4-9f23-799ddc8c70f4-tls-certs\") pod \"simulator-w1-26cfbe-kserve-74c5578fbd-t79hq\" (UID: \"50dc76f4-aebd-41f4-9f23-799ddc8c70f4\") " pod="e2e-models-e2e-worker-w1-26cfbe/simulator-w1-26cfbe-kserve-74c5578fbd-t79hq"
Nov 02 19:40:36.548328 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:40:36.548296    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50dc76f4-aebd-41f4-9f23-799ddc8c70f4-tmp-dir\") pod \"simulator-w1-26cfbe-kserve-74c5578fbd-t79hq\" (UID: \"50dc76f4-aebd-41f4-9f23-799ddc8c70f4\") " pod="e2e-models-e2e-worker-w1-26cfbe/simulator-w1-26cfbe-kserve-74c5578fbd-t79hq"
Nov 02 19:40:36.548328 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:40:36.548333    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50dc76f4-aebd-41f4-9f23-799ddc8c70f4-tls-certs\") pod \"simulator-w1-26cfbe-kserve-74c5578fbd-t79hq\" (UID: \"50dc76f4-aebd-41f4-9f23-799ddc8c70f4\") " pod="e2e-models-e2e-worker-w1-26cfbe/simulator-w1-26cfbe-kserve-74c5578fbd-t79hq"
Nov 02 19:40:36.548598 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:40:36.548383    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50dc76f4-aebd-41f4-9f23-799ddc8c70f4-home\") pod \"simulator-w1-26cfbe-kserve-74c5578fbd-t79hq\" (UID: \"50dc76f4-aebd-41f4-9f23-799ddc8c70f4\") " pod="e2e-models-e2e-worker-w1-26cfbe/simulator-w1-26cfbe-kserve-74c5578fbd-t79hq"
Nov 02 19:40:36.548598 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:40:36.548459    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk5g9\" (UniqueName: \"kubernetes.io/projected/50dc76f4-aebd-41f4-9f23-799ddc8c70f4-kube-api-access-zk5g9\") pod \"simulator-w1-26cfbe-kserve-74c5578fbd-t79hq\" (UID: \"50dc76f4-aebd-41f4-9f23-799ddc8c70f4\") " pod="e2e-models-e2e-worker-w1-26cfbe/simulator-w1-26cfbe-kserve-74c5578fbd-t79hq"
Nov 02 19:40:36.548598 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:40:36.548504    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50dc76f4-aebd-41f4-9f23-799ddc8c70f4-dshm\") pod \"simulator-w1-26cfbe-kserve-74c5578fbd-t79hq\" (UID: \"50dc76f4-aebd-41f4-9f23-799ddc8c70f4\") " pod="e2e-models-e2e-worker-w1-26cfbe/simulator-w1-26cfbe-kserve-74c5578fbd-t79hq"
Nov 02 19:40:36.548598 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:40:36.548544    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50dc76f4-aebd-41f4-9f23-799ddc8c70f4-model-cache\") pod \"simulator-w1-26cfbe-kserve-74c5578fbd-t79hq\" (UID: \"50dc76f4-aebd-41f4-9f23-799ddc8c70f4\") " pod="e2e-models-e2e-worker-w1-26cfbe/simulator-w1-26cfbe-kserve-74c5578fbd-t79hq"
Nov 02 19:40:36.548791 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:40:36.548759    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50dc76f4-aebd-41f4-9f23-799ddc8c70f4-tmp-dir\") pod \"simulator-w1-26cfbe-kserve-74c5578fbd-t79hq\" (UID: \"50dc76f4-aebd-41f4-9f23-799ddc8c70f4\") " pod="e2e-models-e2e-worker-w1-26cfbe/simulator-w1-26cfbe-kserve-74c5578fbd-t79hq"
Nov 02 19:40:36.548791 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:40:36.548776    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50dc76f4-aebd-41f4-9f23-799ddc8c70f4-home\") pod \"simulator-w1-26cfbe-kserve-74c5578fbd-t79hq\" (UID: \"50dc76f4-aebd-41f4-9f23-799ddc8c70f4\") " pod="e2e-models-e2e-worker-w1-26cfbe/simulator-w1-26cfbe-kserve-74c5578fbd-t79hq"
Nov 02 19:40:36.548964 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:40:36.548939    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50dc76f4-aebd-41f4-9f23-799ddc8c70f4-model-cache\") pod \"simulator-w1-26cfbe-kserve-74c5578fbd-t79hq\" (UID: \"50dc76f4-aebd-41f4-9f23-799ddc8c70f4\") " pod="e2e-models-e2e-worker-w1-26cfbe/simulator-w1-26cfbe-kserve-74c5578fbd-t79hq"
Nov 02 19:40:36.550651 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:40:36.550632    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50dc76f4-aebd-41f4-9f23-799ddc8c70f4-dshm\") pod \"simulator-w1-26cfbe-kserve-74c5578fbd-t79hq\" (UID: \"50dc76f4-aebd-41f4-9f23-799ddc8c70f4\") " pod="e2e-models-e2e-worker-w1-26cfbe/simulator-w1-26cfbe-kserve-74c5578fbd-t79hq"
Nov 02 19:40:36.550820 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:40:36.550803    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50dc76f4-aebd-41f4-9f23-799ddc8c70f4-tls-certs\") pod \"simulator-w1-26cfbe-kserve-74c5578fbd-t79hq\" (UID: \"50dc76f4-aebd-41f4-9f23-799ddc8c70f4\") " pod="e2e-models-e2e-worker-w1-26cfbe/simulator-w1-26cfbe-kserve-74c5578fbd-t79hq"
Nov 02 19:40:36.603136 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:40:36.603068    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk5g9\" (UniqueName: \"kubernetes.io/projected/50dc76f4-aebd-41f4-9f23-799ddc8c70f4-kube-api-access-zk5g9\") pod \"simulator-w1-26cfbe-kserve-74c5578fbd-t79hq\" (UID: \"50dc76f4-aebd-41f4-9f23-799ddc8c70f4\") " pod="e2e-models-e2e-worker-w1-26cfbe/simulator-w1-26cfbe-kserve-74c5578fbd-t79hq"
Nov 02 19:40:36.619910 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:40:36.619883    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-26cfbe/simulator-w1-26cfbe-kserve-74c5578fbd-t79hq"
Nov 02 19:40:36.984305 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:40:36.984278    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-26cfbe/simulator-w1-26cfbe-kserve-74c5578fbd-t79hq"]
Nov 02 19:40:36.986210 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:40:36.986181    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod50dc76f4_aebd_41f4_9f23_799ddc8c70f4.slice/crio-cf889241316dd46e6c7a9420fab5d6d958b2584551424f5fee3fd3fdc95dbd5e WatchSource:0}: Error finding container cf889241316dd46e6c7a9420fab5d6d958b2584551424f5fee3fd3fdc95dbd5e: Status 404 returned error can't find the container with id cf889241316dd46e6c7a9420fab5d6d958b2584551424f5fee3fd3fdc95dbd5e
Nov 02 19:40:36.986485 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:40:36.986467    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 19:40:37.809128 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:40:37.809081    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-26cfbe/simulator-w1-26cfbe-kserve-74c5578fbd-t79hq" event={"ID":"50dc76f4-aebd-41f4-9f23-799ddc8c70f4","Type":"ContainerStarted","Data":"17ea621ec4532ca3d7f1a5ffd28fe026bc1d8b6f532e2139c3dd84cd5817f679"}
Nov 02 19:40:37.809128 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:40:37.809138    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-26cfbe/simulator-w1-26cfbe-kserve-74c5578fbd-t79hq" event={"ID":"50dc76f4-aebd-41f4-9f23-799ddc8c70f4","Type":"ContainerStarted","Data":"cf889241316dd46e6c7a9420fab5d6d958b2584551424f5fee3fd3fdc95dbd5e"}
Nov 02 19:40:37.915820 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:40:37.915772    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-26cfbe/simulator-w1-26cfbe-kserve-74c5578fbd-t79hq" podStartSLOduration=1.915757924 podStartE2EDuration="1.915757924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:40:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:40:37.913935475 +0000 UTC m=+744.520969266" watchObservedRunningTime="2026-11-02 19:40:37.915757924 +0000 UTC m=+744.522791692"
Nov 02 19:40:38.620196 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:40:38.620161    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-26cfbe/simulator-w1-26cfbe-kserve-74c5578fbd-t79hq"
Nov 02 19:40:46.620313 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:40:46.620279    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-26cfbe/simulator-w1-26cfbe-kserve-74c5578fbd-t79hq"
Nov 02 19:40:46.632914 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:40:46.632892    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-26cfbe/simulator-w1-26cfbe-kserve-74c5578fbd-t79hq"
Nov 02 19:40:46.871863 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:40:46.871787    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-26cfbe/simulator-w1-26cfbe-kserve-74c5578fbd-t79hq"
Nov 02 19:40:52.795491 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:40:52.795456    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 19:40:56.829145 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:40:56.829072    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-43ceaa/premium-w0-43ceaa-kserve-67b55659b7-rckfd"]
Nov 02 19:40:56.832761 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:40:56.832745    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-43ceaa/premium-w0-43ceaa-kserve-67b55659b7-rckfd"
Nov 02 19:40:56.861883 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:40:56.861853    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7be32df1-5049-4f79-be23-a827f3f0a85f-model-cache\") pod \"premium-w0-43ceaa-kserve-67b55659b7-rckfd\" (UID: \"7be32df1-5049-4f79-be23-a827f3f0a85f\") " pod="e2e-models-e2e-worker-w0-43ceaa/premium-w0-43ceaa-kserve-67b55659b7-rckfd"
Nov 02 19:40:56.862005 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:40:56.861897    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7be32df1-5049-4f79-be23-a827f3f0a85f-home\") pod \"premium-w0-43ceaa-kserve-67b55659b7-rckfd\" (UID: \"7be32df1-5049-4f79-be23-a827f3f0a85f\") " pod="e2e-models-e2e-worker-w0-43ceaa/premium-w0-43ceaa-kserve-67b55659b7-rckfd"
Nov 02 19:40:56.862139 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:40:56.862117    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7be32df1-5049-4f79-be23-a827f3f0a85f-tmp-dir\") pod \"premium-w0-43ceaa-kserve-67b55659b7-rckfd\" (UID: \"7be32df1-5049-4f79-be23-a827f3f0a85f\") " pod="e2e-models-e2e-worker-w0-43ceaa/premium-w0-43ceaa-kserve-67b55659b7-rckfd"
Nov 02 19:40:56.862211 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:40:56.862157    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7be32df1-5049-4f79-be23-a827f3f0a85f-tls-certs\") pod \"premium-w0-43ceaa-kserve-67b55659b7-rckfd\" (UID: \"7be32df1-5049-4f79-be23-a827f3f0a85f\") " pod="e2e-models-e2e-worker-w0-43ceaa/premium-w0-43ceaa-kserve-67b55659b7-rckfd"
Nov 02 19:40:56.862257 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:40:56.862221    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swpf7\" (UniqueName: \"kubernetes.io/projected/7be32df1-5049-4f79-be23-a827f3f0a85f-kube-api-access-swpf7\") pod \"premium-w0-43ceaa-kserve-67b55659b7-rckfd\" (UID: \"7be32df1-5049-4f79-be23-a827f3f0a85f\") " pod="e2e-models-e2e-worker-w0-43ceaa/premium-w0-43ceaa-kserve-67b55659b7-rckfd"
Nov 02 19:40:56.862315 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:40:56.862289    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7be32df1-5049-4f79-be23-a827f3f0a85f-dshm\") pod \"premium-w0-43ceaa-kserve-67b55659b7-rckfd\" (UID: \"7be32df1-5049-4f79-be23-a827f3f0a85f\") " pod="e2e-models-e2e-worker-w0-43ceaa/premium-w0-43ceaa-kserve-67b55659b7-rckfd"
Nov 02 19:40:56.886816 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:40:56.886793    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-43ceaa/premium-w0-43ceaa-kserve-67b55659b7-rckfd"]
Nov 02 19:40:56.929916 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:40:56.929890    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-43ceaa\"/\"default-dockercfg-824k2\""
Nov 02 19:40:56.930048 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:40:56.929935    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-43ceaa\"/\"openshift-service-ca.crt\""
Nov 02 19:40:56.930048 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:40:56.929996    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-43ceaa\"/\"premium-w0-43ceaa-kserve-self-signed-certs\""
Nov 02 19:40:56.930048 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:40:56.929996    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-43ceaa\"/\"kube-root-ca.crt\""
Nov 02 19:40:56.963280 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:40:56.963257    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7be32df1-5049-4f79-be23-a827f3f0a85f-dshm\") pod \"premium-w0-43ceaa-kserve-67b55659b7-rckfd\" (UID: \"7be32df1-5049-4f79-be23-a827f3f0a85f\") " pod="e2e-models-e2e-worker-w0-43ceaa/premium-w0-43ceaa-kserve-67b55659b7-rckfd"
Nov 02 19:40:56.963370 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:40:56.963293    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7be32df1-5049-4f79-be23-a827f3f0a85f-model-cache\") pod \"premium-w0-43ceaa-kserve-67b55659b7-rckfd\" (UID: \"7be32df1-5049-4f79-be23-a827f3f0a85f\") " pod="e2e-models-e2e-worker-w0-43ceaa/premium-w0-43ceaa-kserve-67b55659b7-rckfd"
Nov 02 19:40:56.963370 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:40:56.963315    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7be32df1-5049-4f79-be23-a827f3f0a85f-home\") pod \"premium-w0-43ceaa-kserve-67b55659b7-rckfd\" (UID: \"7be32df1-5049-4f79-be23-a827f3f0a85f\") " pod="e2e-models-e2e-worker-w0-43ceaa/premium-w0-43ceaa-kserve-67b55659b7-rckfd"
Nov 02 19:40:56.963468 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:40:56.963428    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7be32df1-5049-4f79-be23-a827f3f0a85f-tmp-dir\") pod \"premium-w0-43ceaa-kserve-67b55659b7-rckfd\" (UID: \"7be32df1-5049-4f79-be23-a827f3f0a85f\") " pod="e2e-models-e2e-worker-w0-43ceaa/premium-w0-43ceaa-kserve-67b55659b7-rckfd"
Nov 02 19:40:56.963468 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:40:56.963456    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7be32df1-5049-4f79-be23-a827f3f0a85f-tls-certs\") pod \"premium-w0-43ceaa-kserve-67b55659b7-rckfd\" (UID: \"7be32df1-5049-4f79-be23-a827f3f0a85f\") " pod="e2e-models-e2e-worker-w0-43ceaa/premium-w0-43ceaa-kserve-67b55659b7-rckfd"
Nov 02 19:40:56.963543 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:40:56.963483    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swpf7\" (UniqueName: \"kubernetes.io/projected/7be32df1-5049-4f79-be23-a827f3f0a85f-kube-api-access-swpf7\") pod \"premium-w0-43ceaa-kserve-67b55659b7-rckfd\" (UID: \"7be32df1-5049-4f79-be23-a827f3f0a85f\") " pod="e2e-models-e2e-worker-w0-43ceaa/premium-w0-43ceaa-kserve-67b55659b7-rckfd"
Nov 02 19:40:56.963742 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:40:56.963720    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7be32df1-5049-4f79-be23-a827f3f0a85f-home\") pod \"premium-w0-43ceaa-kserve-67b55659b7-rckfd\" (UID: \"7be32df1-5049-4f79-be23-a827f3f0a85f\") " pod="e2e-models-e2e-worker-w0-43ceaa/premium-w0-43ceaa-kserve-67b55659b7-rckfd"
Nov 02 19:40:56.963843 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:40:56.963781    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7be32df1-5049-4f79-be23-a827f3f0a85f-model-cache\") pod \"premium-w0-43ceaa-kserve-67b55659b7-rckfd\" (UID: \"7be32df1-5049-4f79-be23-a827f3f0a85f\") " pod="e2e-models-e2e-worker-w0-43ceaa/premium-w0-43ceaa-kserve-67b55659b7-rckfd"
Nov 02 19:40:56.963843 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:40:56.963803    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7be32df1-5049-4f79-be23-a827f3f0a85f-tmp-dir\") pod \"premium-w0-43ceaa-kserve-67b55659b7-rckfd\" (UID: \"7be32df1-5049-4f79-be23-a827f3f0a85f\") " pod="e2e-models-e2e-worker-w0-43ceaa/premium-w0-43ceaa-kserve-67b55659b7-rckfd"
Nov 02 19:40:56.965480 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:40:56.965457    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7be32df1-5049-4f79-be23-a827f3f0a85f-dshm\") pod \"premium-w0-43ceaa-kserve-67b55659b7-rckfd\" (UID: \"7be32df1-5049-4f79-be23-a827f3f0a85f\") " pod="e2e-models-e2e-worker-w0-43ceaa/premium-w0-43ceaa-kserve-67b55659b7-rckfd"
Nov 02 19:40:56.965876 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:40:56.965853    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7be32df1-5049-4f79-be23-a827f3f0a85f-tls-certs\") pod \"premium-w0-43ceaa-kserve-67b55659b7-rckfd\" (UID: \"7be32df1-5049-4f79-be23-a827f3f0a85f\") " pod="e2e-models-e2e-worker-w0-43ceaa/premium-w0-43ceaa-kserve-67b55659b7-rckfd"
Nov 02 19:40:57.017165 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:40:57.017137    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swpf7\" (UniqueName: \"kubernetes.io/projected/7be32df1-5049-4f79-be23-a827f3f0a85f-kube-api-access-swpf7\") pod \"premium-w0-43ceaa-kserve-67b55659b7-rckfd\" (UID: \"7be32df1-5049-4f79-be23-a827f3f0a85f\") " pod="e2e-models-e2e-worker-w0-43ceaa/premium-w0-43ceaa-kserve-67b55659b7-rckfd"
Nov 02 19:40:57.144945 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:40:57.144876    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-43ceaa/premium-w0-43ceaa-kserve-67b55659b7-rckfd"
Nov 02 19:40:57.520686 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:40:57.520658    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-43ceaa/premium-w0-43ceaa-kserve-67b55659b7-rckfd"]
Nov 02 19:40:57.522507 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:40:57.522479    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7be32df1_5049_4f79_be23_a827f3f0a85f.slice/crio-5f576ba99d213d2d4757daa276d6d8c96b9456ec757e916fe8b4eea0667dbbce WatchSource:0}: Error finding container 5f576ba99d213d2d4757daa276d6d8c96b9456ec757e916fe8b4eea0667dbbce: Status 404 returned error can't find the container with id 5f576ba99d213d2d4757daa276d6d8c96b9456ec757e916fe8b4eea0667dbbce
Nov 02 19:40:57.920122 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:40:57.920042    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-43ceaa/premium-w0-43ceaa-kserve-67b55659b7-rckfd" event={"ID":"7be32df1-5049-4f79-be23-a827f3f0a85f","Type":"ContainerStarted","Data":"d5da31da419a75e0a447ef9fc079d2874efb0b85c4fbaf0a04237941165a7af7"}
Nov 02 19:40:57.920122 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:40:57.920083    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-43ceaa/premium-w0-43ceaa-kserve-67b55659b7-rckfd" event={"ID":"7be32df1-5049-4f79-be23-a827f3f0a85f","Type":"ContainerStarted","Data":"5f576ba99d213d2d4757daa276d6d8c96b9456ec757e916fe8b4eea0667dbbce"}
Nov 02 19:40:58.027667 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:40:58.027614    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-43ceaa/premium-w0-43ceaa-kserve-67b55659b7-rckfd" podStartSLOduration=2.027597913 podStartE2EDuration="2.027597913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:40:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:40:58.025639702 +0000 UTC m=+764.632673470" watchObservedRunningTime="2026-11-02 19:40:58.027597913 +0000 UTC m=+764.634631682"
Nov 02 19:40:58.145548 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:40:58.145515    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-43ceaa/premium-w0-43ceaa-kserve-67b55659b7-rckfd"
Nov 02 19:41:04.293558 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:41:04.293519    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 19:41:07.145572 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:41:07.145540    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-43ceaa/premium-w0-43ceaa-kserve-67b55659b7-rckfd"
Nov 02 19:41:07.157652 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:41:07.157631    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-43ceaa/premium-w0-43ceaa-kserve-67b55659b7-rckfd"
Nov 02 19:41:07.980936 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:41:07.980909    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-43ceaa/premium-w0-43ceaa-kserve-67b55659b7-rckfd"
Nov 02 19:41:15.622049 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:41:15.621997    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 19:41:34.294251 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:41:34.294209    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 19:41:44.582561 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:41:44.582524    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 19:41:46.103815 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:41:46.103781    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-1d869e/premium-w5-1d869e-kserve-7745799ff8-pv2th"]
Nov 02 19:41:46.107607 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:41:46.107589    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-1d869e/premium-w5-1d869e-kserve-7745799ff8-pv2th"
Nov 02 19:41:46.146028 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:41:46.145988    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d784b10-99e4-4561-bfb7-6614877d2b6d-model-cache\") pod \"premium-w5-1d869e-kserve-7745799ff8-pv2th\" (UID: \"1d784b10-99e4-4561-bfb7-6614877d2b6d\") " pod="e2e-models-e2e-worker-w5-1d869e/premium-w5-1d869e-kserve-7745799ff8-pv2th"
Nov 02 19:41:46.146170 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:41:46.146044    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d784b10-99e4-4561-bfb7-6614877d2b6d-home\") pod \"premium-w5-1d869e-kserve-7745799ff8-pv2th\" (UID: \"1d784b10-99e4-4561-bfb7-6614877d2b6d\") " pod="e2e-models-e2e-worker-w5-1d869e/premium-w5-1d869e-kserve-7745799ff8-pv2th"
Nov 02 19:41:46.146227 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:41:46.146181    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d784b10-99e4-4561-bfb7-6614877d2b6d-tls-certs\") pod \"premium-w5-1d869e-kserve-7745799ff8-pv2th\" (UID: \"1d784b10-99e4-4561-bfb7-6614877d2b6d\") " pod="e2e-models-e2e-worker-w5-1d869e/premium-w5-1d869e-kserve-7745799ff8-pv2th"
Nov 02 19:41:46.146274 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:41:46.146248    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d784b10-99e4-4561-bfb7-6614877d2b6d-tmp-dir\") pod \"premium-w5-1d869e-kserve-7745799ff8-pv2th\" (UID: \"1d784b10-99e4-4561-bfb7-6614877d2b6d\") " pod="e2e-models-e2e-worker-w5-1d869e/premium-w5-1d869e-kserve-7745799ff8-pv2th"
Nov 02 19:41:46.146373 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:41:46.146349    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d784b10-99e4-4561-bfb7-6614877d2b6d-dshm\") pod \"premium-w5-1d869e-kserve-7745799ff8-pv2th\" (UID: \"1d784b10-99e4-4561-bfb7-6614877d2b6d\") " pod="e2e-models-e2e-worker-w5-1d869e/premium-w5-1d869e-kserve-7745799ff8-pv2th"
Nov 02 19:41:46.146433 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:41:46.146417    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p268k\" (UniqueName: \"kubernetes.io/projected/1d784b10-99e4-4561-bfb7-6614877d2b6d-kube-api-access-p268k\") pod \"premium-w5-1d869e-kserve-7745799ff8-pv2th\" (UID: \"1d784b10-99e4-4561-bfb7-6614877d2b6d\") " pod="e2e-models-e2e-worker-w5-1d869e/premium-w5-1d869e-kserve-7745799ff8-pv2th"
Nov 02 19:41:46.156360 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:41:46.156320    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-1d869e\"/\"openshift-service-ca.crt\""
Nov 02 19:41:46.202566 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:41:46.202533    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-1d869e\"/\"default-dockercfg-9c7rx\""
Nov 02 19:41:46.202775 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:41:46.202756    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-1d869e\"/\"premium-w5-1d869e-kserve-self-signed-certs\""
Nov 02 19:41:46.205233 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:41:46.205213    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-1d869e/premium-w5-1d869e-kserve-7745799ff8-pv2th"]
Nov 02 19:41:46.247670 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:41:46.247646    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d784b10-99e4-4561-bfb7-6614877d2b6d-tmp-dir\") pod \"premium-w5-1d869e-kserve-7745799ff8-pv2th\" (UID: \"1d784b10-99e4-4561-bfb7-6614877d2b6d\") " pod="e2e-models-e2e-worker-w5-1d869e/premium-w5-1d869e-kserve-7745799ff8-pv2th"
Nov 02 19:41:46.247808 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:41:46.247688    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d784b10-99e4-4561-bfb7-6614877d2b6d-dshm\") pod \"premium-w5-1d869e-kserve-7745799ff8-pv2th\" (UID: \"1d784b10-99e4-4561-bfb7-6614877d2b6d\") " pod="e2e-models-e2e-worker-w5-1d869e/premium-w5-1d869e-kserve-7745799ff8-pv2th"
Nov 02 19:41:46.247808 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:41:46.247722    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p268k\" (UniqueName: \"kubernetes.io/projected/1d784b10-99e4-4561-bfb7-6614877d2b6d-kube-api-access-p268k\") pod \"premium-w5-1d869e-kserve-7745799ff8-pv2th\" (UID: \"1d784b10-99e4-4561-bfb7-6614877d2b6d\") " pod="e2e-models-e2e-worker-w5-1d869e/premium-w5-1d869e-kserve-7745799ff8-pv2th"
Nov 02 19:41:46.247808 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:41:46.247769    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d784b10-99e4-4561-bfb7-6614877d2b6d-model-cache\") pod \"premium-w5-1d869e-kserve-7745799ff8-pv2th\" (UID: \"1d784b10-99e4-4561-bfb7-6614877d2b6d\") " pod="e2e-models-e2e-worker-w5-1d869e/premium-w5-1d869e-kserve-7745799ff8-pv2th"
Nov 02 19:41:46.247808 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:41:46.247793    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d784b10-99e4-4561-bfb7-6614877d2b6d-home\") pod \"premium-w5-1d869e-kserve-7745799ff8-pv2th\" (UID: \"1d784b10-99e4-4561-bfb7-6614877d2b6d\") " pod="e2e-models-e2e-worker-w5-1d869e/premium-w5-1d869e-kserve-7745799ff8-pv2th"
Nov 02 19:41:46.248116 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:41:46.247845    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d784b10-99e4-4561-bfb7-6614877d2b6d-tls-certs\") pod \"premium-w5-1d869e-kserve-7745799ff8-pv2th\" (UID: \"1d784b10-99e4-4561-bfb7-6614877d2b6d\") " pod="e2e-models-e2e-worker-w5-1d869e/premium-w5-1d869e-kserve-7745799ff8-pv2th"
Nov 02 19:41:46.248206 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:41:46.248178    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d784b10-99e4-4561-bfb7-6614877d2b6d-model-cache\") pod \"premium-w5-1d869e-kserve-7745799ff8-pv2th\" (UID: \"1d784b10-99e4-4561-bfb7-6614877d2b6d\") " pod="e2e-models-e2e-worker-w5-1d869e/premium-w5-1d869e-kserve-7745799ff8-pv2th"
Nov 02 19:41:46.248356 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:41:46.248326    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d784b10-99e4-4561-bfb7-6614877d2b6d-home\") pod \"premium-w5-1d869e-kserve-7745799ff8-pv2th\" (UID: \"1d784b10-99e4-4561-bfb7-6614877d2b6d\") " pod="e2e-models-e2e-worker-w5-1d869e/premium-w5-1d869e-kserve-7745799ff8-pv2th"
Nov 02 19:41:46.248468 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:41:46.248357    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d784b10-99e4-4561-bfb7-6614877d2b6d-tmp-dir\") pod \"premium-w5-1d869e-kserve-7745799ff8-pv2th\" (UID: \"1d784b10-99e4-4561-bfb7-6614877d2b6d\") " pod="e2e-models-e2e-worker-w5-1d869e/premium-w5-1d869e-kserve-7745799ff8-pv2th"
Nov 02 19:41:46.249883 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:41:46.249864    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d784b10-99e4-4561-bfb7-6614877d2b6d-dshm\") pod \"premium-w5-1d869e-kserve-7745799ff8-pv2th\" (UID: \"1d784b10-99e4-4561-bfb7-6614877d2b6d\") " pod="e2e-models-e2e-worker-w5-1d869e/premium-w5-1d869e-kserve-7745799ff8-pv2th"
Nov 02 19:41:46.250354 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:41:46.250334    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d784b10-99e4-4561-bfb7-6614877d2b6d-tls-certs\") pod \"premium-w5-1d869e-kserve-7745799ff8-pv2th\" (UID: \"1d784b10-99e4-4561-bfb7-6614877d2b6d\") " pod="e2e-models-e2e-worker-w5-1d869e/premium-w5-1d869e-kserve-7745799ff8-pv2th"
Nov 02 19:41:46.257029 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:41:46.256993    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-1d869e\"/\"kube-root-ca.crt\""
Nov 02 19:41:46.302667 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:41:46.302628    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p268k\" (UniqueName: \"kubernetes.io/projected/1d784b10-99e4-4561-bfb7-6614877d2b6d-kube-api-access-p268k\") pod \"premium-w5-1d869e-kserve-7745799ff8-pv2th\" (UID: \"1d784b10-99e4-4561-bfb7-6614877d2b6d\") " pod="e2e-models-e2e-worker-w5-1d869e/premium-w5-1d869e-kserve-7745799ff8-pv2th"
Nov 02 19:41:46.418728 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:41:46.418649    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-1d869e/premium-w5-1d869e-kserve-7745799ff8-pv2th"
Nov 02 19:41:46.795797 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:41:46.795771    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-1d869e/premium-w5-1d869e-kserve-7745799ff8-pv2th"]
Nov 02 19:41:46.797673 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:41:46.797641    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1d784b10_99e4_4561_bfb7_6614877d2b6d.slice/crio-9ddf0236e7ca264b47e545a93ebf3896fcfe6d049342579712d2f064e4a5b340 WatchSource:0}: Error finding container 9ddf0236e7ca264b47e545a93ebf3896fcfe6d049342579712d2f064e4a5b340: Status 404 returned error can't find the container with id 9ddf0236e7ca264b47e545a93ebf3896fcfe6d049342579712d2f064e4a5b340
Nov 02 19:41:47.058297 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:41:47.058219    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kpr8z"]
Nov 02 19:41:47.061888 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:41:47.061871    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kpr8z"
Nov 02 19:41:47.111643 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:41:47.111614    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-8795bf-openshift-default-dockercfg-s7q56\""
Nov 02 19:41:47.155065 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:41:47.155039    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kpr8z"]
Nov 02 19:41:47.155891 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:41:47.155872    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c725b384-6c96-444e-a657-d4ebeb0e5685-istio-data\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-kpr8z\" (UID: \"c725b384-6c96-444e-a657-d4ebeb0e5685\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kpr8z"
Nov 02 19:41:47.155981 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:41:47.155913    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c725b384-6c96-444e-a657-d4ebeb0e5685-istiod-ca-cert\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-kpr8z\" (UID: \"c725b384-6c96-444e-a657-d4ebeb0e5685\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kpr8z"
Nov 02 19:41:47.156084 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:41:47.156054    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c725b384-6c96-444e-a657-d4ebeb0e5685-istio-envoy\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-kpr8z\" (UID: \"c725b384-6c96-444e-a657-d4ebeb0e5685\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kpr8z"
Nov 02 19:41:47.156200 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:41:47.156144    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfp7t\" (UniqueName: \"kubernetes.io/projected/c725b384-6c96-444e-a657-d4ebeb0e5685-kube-api-access-lfp7t\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-kpr8z\" (UID: \"c725b384-6c96-444e-a657-d4ebeb0e5685\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kpr8z"
Nov 02 19:41:47.156200 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:41:47.156196    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c725b384-6c96-444e-a657-d4ebeb0e5685-istio-token\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-kpr8z\" (UID: \"c725b384-6c96-444e-a657-d4ebeb0e5685\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kpr8z"
Nov 02 19:41:47.156304 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:41:47.156231    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c725b384-6c96-444e-a657-d4ebeb0e5685-workload-certs\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-kpr8z\" (UID: \"c725b384-6c96-444e-a657-d4ebeb0e5685\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kpr8z"
Nov 02 19:41:47.156341 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:41:47.156311    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c725b384-6c96-444e-a657-d4ebeb0e5685-istio-podinfo\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-kpr8z\" (UID: \"c725b384-6c96-444e-a657-d4ebeb0e5685\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kpr8z"
Nov 02 19:41:47.156440 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:41:47.156422    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c725b384-6c96-444e-a657-d4ebeb0e5685-workload-socket\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-kpr8z\" (UID: \"c725b384-6c96-444e-a657-d4ebeb0e5685\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kpr8z"
Nov 02 19:41:47.156482 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:41:47.156467    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c725b384-6c96-444e-a657-d4ebeb0e5685-credential-socket\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-kpr8z\" (UID: \"c725b384-6c96-444e-a657-d4ebeb0e5685\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kpr8z"
Nov 02 19:41:47.165549 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:41:47.165520    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-1d869e/premium-w5-1d869e-kserve-7745799ff8-pv2th" event={"ID":"1d784b10-99e4-4561-bfb7-6614877d2b6d","Type":"ContainerStarted","Data":"fc910b085bc6f56193e79977d9dd7b30867c6e9144953196d6a6bb57a4ce2058"}
Nov 02 19:41:47.165641 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:41:47.165569    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-1d869e/premium-w5-1d869e-kserve-7745799ff8-pv2th" event={"ID":"1d784b10-99e4-4561-bfb7-6614877d2b6d","Type":"ContainerStarted","Data":"9ddf0236e7ca264b47e545a93ebf3896fcfe6d049342579712d2f064e4a5b340"}
Nov 02 19:41:47.257542 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:41:47.257512    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c725b384-6c96-444e-a657-d4ebeb0e5685-istio-envoy\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-kpr8z\" (UID: \"c725b384-6c96-444e-a657-d4ebeb0e5685\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kpr8z"
Nov 02 19:41:47.257721 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:41:47.257550    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfp7t\" (UniqueName: \"kubernetes.io/projected/c725b384-6c96-444e-a657-d4ebeb0e5685-kube-api-access-lfp7t\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-kpr8z\" (UID: \"c725b384-6c96-444e-a657-d4ebeb0e5685\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kpr8z"
Nov 02 19:41:47.257721 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:41:47.257585    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c725b384-6c96-444e-a657-d4ebeb0e5685-istio-token\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-kpr8z\" (UID: \"c725b384-6c96-444e-a657-d4ebeb0e5685\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kpr8z"
Nov 02 19:41:47.257721 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:41:47.257615    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c725b384-6c96-444e-a657-d4ebeb0e5685-workload-certs\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-kpr8z\" (UID: \"c725b384-6c96-444e-a657-d4ebeb0e5685\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kpr8z"
Nov 02 19:41:47.257721 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:41:47.257639    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c725b384-6c96-444e-a657-d4ebeb0e5685-istio-podinfo\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-kpr8z\" (UID: \"c725b384-6c96-444e-a657-d4ebeb0e5685\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kpr8z"
Nov 02 19:41:47.257721 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:41:47.257678    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c725b384-6c96-444e-a657-d4ebeb0e5685-workload-socket\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-kpr8z\" (UID: \"c725b384-6c96-444e-a657-d4ebeb0e5685\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kpr8z"
Nov 02 19:41:47.257721 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:41:47.257717    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c725b384-6c96-444e-a657-d4ebeb0e5685-credential-socket\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-kpr8z\" (UID: \"c725b384-6c96-444e-a657-d4ebeb0e5685\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kpr8z"
Nov 02 19:41:47.258070 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:41:47.257843    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c725b384-6c96-444e-a657-d4ebeb0e5685-istio-data\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-kpr8z\" (UID: \"c725b384-6c96-444e-a657-d4ebeb0e5685\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kpr8z"
Nov 02 19:41:47.258070 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:41:47.257881    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c725b384-6c96-444e-a657-d4ebeb0e5685-istiod-ca-cert\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-kpr8z\" (UID: \"c725b384-6c96-444e-a657-d4ebeb0e5685\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kpr8z"
Nov 02 19:41:47.258178 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:41:47.258142    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c725b384-6c96-444e-a657-d4ebeb0e5685-workload-certs\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-kpr8z\" (UID: \"c725b384-6c96-444e-a657-d4ebeb0e5685\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kpr8z"
Nov 02 19:41:47.258380 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:41:47.258357    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c725b384-6c96-444e-a657-d4ebeb0e5685-istio-data\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-kpr8z\" (UID: \"c725b384-6c96-444e-a657-d4ebeb0e5685\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kpr8z"
Nov 02 19:41:47.258439 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:41:47.258419    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c725b384-6c96-444e-a657-d4ebeb0e5685-credential-socket\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-kpr8z\" (UID: \"c725b384-6c96-444e-a657-d4ebeb0e5685\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kpr8z"
Nov 02 19:41:47.258547 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:41:47.258522    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c725b384-6c96-444e-a657-d4ebeb0e5685-workload-socket\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-kpr8z\" (UID: \"c725b384-6c96-444e-a657-d4ebeb0e5685\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kpr8z"
Nov 02 19:41:47.258759 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:41:47.258731    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c725b384-6c96-444e-a657-d4ebeb0e5685-istiod-ca-cert\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-kpr8z\" (UID: \"c725b384-6c96-444e-a657-d4ebeb0e5685\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kpr8z"
Nov 02 19:41:47.260146 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:41:47.260127    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c725b384-6c96-444e-a657-d4ebeb0e5685-istio-envoy\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-kpr8z\" (UID: \"c725b384-6c96-444e-a657-d4ebeb0e5685\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kpr8z"
Nov 02 19:41:47.260794 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:41:47.260774    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c725b384-6c96-444e-a657-d4ebeb0e5685-istio-podinfo\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-kpr8z\" (UID: \"c725b384-6c96-444e-a657-d4ebeb0e5685\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kpr8z"
Nov 02 19:41:47.278111 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:41:47.278068    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-1d869e/premium-w5-1d869e-kserve-7745799ff8-pv2th" podStartSLOduration=1.278057088 podStartE2EDuration="1.278057088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:41:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:41:47.275734829 +0000 UTC m=+813.882768598" watchObservedRunningTime="2026-11-02 19:41:47.278057088 +0000 UTC m=+813.885090857"
Nov 02 19:41:47.310786 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:41:47.310733    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c725b384-6c96-444e-a657-d4ebeb0e5685-istio-token\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-kpr8z\" (UID: \"c725b384-6c96-444e-a657-d4ebeb0e5685\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kpr8z"
Nov 02 19:41:47.310786 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:41:47.310746    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfp7t\" (UniqueName: \"kubernetes.io/projected/c725b384-6c96-444e-a657-d4ebeb0e5685-kube-api-access-lfp7t\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-kpr8z\" (UID: \"c725b384-6c96-444e-a657-d4ebeb0e5685\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kpr8z"
Nov 02 19:41:47.374917 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:41:47.374892    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kpr8z"
Nov 02 19:41:47.420644 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:41:47.420573    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-1d869e/premium-w5-1d869e-kserve-7745799ff8-pv2th"
Nov 02 19:41:47.794533 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:41:47.794508    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kpr8z"]
Nov 02 19:41:47.796865 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:41:47.796835    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc725b384_6c96_444e_a657_d4ebeb0e5685.slice/crio-e506e5cf94db1dcddb9ee1cdd5fa3d60c0823c1e2b3392fbc2ee221e317b943d WatchSource:0}: Error finding container e506e5cf94db1dcddb9ee1cdd5fa3d60c0823c1e2b3392fbc2ee221e317b943d: Status 404 returned error can't find the container with id e506e5cf94db1dcddb9ee1cdd5fa3d60c0823c1e2b3392fbc2ee221e317b943d
Nov 02 19:41:47.799126 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:41:47.799091    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:41:47.799246 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:41:47.799167    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:41:47.799246 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:41:47.799210    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:41:48.173289 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:41:48.173251    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kpr8z" event={"ID":"c725b384-6c96-444e-a657-d4ebeb0e5685","Type":"ContainerStarted","Data":"253434d791e65f1934bb94b67c4e2f77546f9f6376672c2b019d7c97861ba244"}
Nov 02 19:41:48.173691 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:41:48.173325    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kpr8z" event={"ID":"c725b384-6c96-444e-a657-d4ebeb0e5685","Type":"ContainerStarted","Data":"e506e5cf94db1dcddb9ee1cdd5fa3d60c0823c1e2b3392fbc2ee221e317b943d"}
Nov 02 19:41:48.283072 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:41:48.283001    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kpr8z" podStartSLOduration=1.28298655 podStartE2EDuration="1.28298655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:41:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:41:48.281255399 +0000 UTC m=+814.888289187" watchObservedRunningTime="2026-11-02 19:41:48.28298655 +0000 UTC m=+814.890020318"
Nov 02 19:41:48.375651 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:41:48.375620    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kpr8z"
Nov 02 19:41:48.377287 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:41:48.377263    4463 patch_prober.go:29] interesting pod/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kpr8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Nov 02 19:41:48.377430 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:41:48.377314    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kpr8z" podUID="c725b384-6c96-444e-a657-d4ebeb0e5685" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Nov 02 19:41:49.376302 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:41:49.376259    4463 patch_prober.go:29] interesting pod/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kpr8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Nov 02 19:41:49.376739 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:41:49.376344    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kpr8z" podUID="c725b384-6c96-444e-a657-d4ebeb0e5685" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Nov 02 19:41:50.375502 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:41:50.375462    4463 patch_prober.go:29] interesting pod/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kpr8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Nov 02 19:41:50.375693 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:41:50.375532    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kpr8z" podUID="c725b384-6c96-444e-a657-d4ebeb0e5685" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Nov 02 19:41:51.375738 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:41:51.375706    4463 patch_prober.go:29] interesting pod/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kpr8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Nov 02 19:41:51.376143 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:41:51.375754    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kpr8z" podUID="c725b384-6c96-444e-a657-d4ebeb0e5685" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Nov 02 19:41:52.375665 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:41:52.375628    4463 patch_prober.go:29] interesting pod/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kpr8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Nov 02 19:41:52.375842 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:41:52.375684    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kpr8z" podUID="c725b384-6c96-444e-a657-d4ebeb0e5685" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Nov 02 19:41:53.375773 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:41:53.375735    4463 patch_prober.go:29] interesting pod/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kpr8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Nov 02 19:41:53.375961 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:41:53.375790    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kpr8z" podUID="c725b384-6c96-444e-a657-d4ebeb0e5685" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Nov 02 19:41:54.375667 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:41:54.375631    4463 patch_prober.go:29] interesting pod/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kpr8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Nov 02 19:41:54.375862 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:41:54.375693    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kpr8z" podUID="c725b384-6c96-444e-a657-d4ebeb0e5685" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Nov 02 19:41:55.380305 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:41:55.380274    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kpr8z"
Nov 02 19:41:55.380778 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:41:55.380524    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kpr8z"
Nov 02 19:41:55.381425 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:41:55.381410    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kpr8z"
Nov 02 19:41:56.419840 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:41:56.419807    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-1d869e/premium-w5-1d869e-kserve-7745799ff8-pv2th"
Nov 02 19:41:56.432651 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:41:56.432626    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-1d869e/premium-w5-1d869e-kserve-7745799ff8-pv2th"
Nov 02 19:41:57.231966 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:41:57.231936    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-1d869e/premium-w5-1d869e-kserve-7745799ff8-pv2th"
Nov 02 19:42:04.293914 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:42:04.293877    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 19:42:10.610784 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:42:10.610751    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 19:42:34.294305 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:42:34.294225    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 19:42:41.384672 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:42:41.384637    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 19:43:03.547732 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:03.547697    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-94hmq"]
Nov 02 19:43:03.551555 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:03.551536    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-94hmq"
Nov 02 19:43:03.600160 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:03.600129    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-1d869e-openshift-default-dockercfg-cg57h\""
Nov 02 19:43:03.644445 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:03.644393    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32d7da41-73b0-4ae5-a8b3-69bcb021cf4a-workload-certs\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-94hmq\" (UID: \"32d7da41-73b0-4ae5-a8b3-69bcb021cf4a\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-94hmq"
Nov 02 19:43:03.644445 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:03.644444    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32d7da41-73b0-4ae5-a8b3-69bcb021cf4a-workload-socket\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-94hmq\" (UID: \"32d7da41-73b0-4ae5-a8b3-69bcb021cf4a\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-94hmq"
Nov 02 19:43:03.644685 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:03.644520    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32d7da41-73b0-4ae5-a8b3-69bcb021cf4a-istio-data\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-94hmq\" (UID: \"32d7da41-73b0-4ae5-a8b3-69bcb021cf4a\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-94hmq"
Nov 02 19:43:03.644751 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:03.644685    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32d7da41-73b0-4ae5-a8b3-69bcb021cf4a-istiod-ca-cert\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-94hmq\" (UID: \"32d7da41-73b0-4ae5-a8b3-69bcb021cf4a\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-94hmq"
Nov 02 19:43:03.644751 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:03.644723    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32d7da41-73b0-4ae5-a8b3-69bcb021cf4a-istio-envoy\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-94hmq\" (UID: \"32d7da41-73b0-4ae5-a8b3-69bcb021cf4a\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-94hmq"
Nov 02 19:43:03.644853 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:03.644803    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32d7da41-73b0-4ae5-a8b3-69bcb021cf4a-istio-podinfo\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-94hmq\" (UID: \"32d7da41-73b0-4ae5-a8b3-69bcb021cf4a\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-94hmq"
Nov 02 19:43:03.644907 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:03.644848    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbnn8\" (UniqueName: \"kubernetes.io/projected/32d7da41-73b0-4ae5-a8b3-69bcb021cf4a-kube-api-access-tbnn8\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-94hmq\" (UID: \"32d7da41-73b0-4ae5-a8b3-69bcb021cf4a\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-94hmq"
Nov 02 19:43:03.644907 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:03.644881    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32d7da41-73b0-4ae5-a8b3-69bcb021cf4a-credential-socket\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-94hmq\" (UID: \"32d7da41-73b0-4ae5-a8b3-69bcb021cf4a\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-94hmq"
Nov 02 19:43:03.645171 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:03.645139    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32d7da41-73b0-4ae5-a8b3-69bcb021cf4a-istio-token\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-94hmq\" (UID: \"32d7da41-73b0-4ae5-a8b3-69bcb021cf4a\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-94hmq"
Nov 02 19:43:03.649199 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:03.649176    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-94hmq"]
Nov 02 19:43:03.745938 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:03.745901    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32d7da41-73b0-4ae5-a8b3-69bcb021cf4a-istio-token\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-94hmq\" (UID: \"32d7da41-73b0-4ae5-a8b3-69bcb021cf4a\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-94hmq"
Nov 02 19:43:03.746156 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:03.745971    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32d7da41-73b0-4ae5-a8b3-69bcb021cf4a-workload-certs\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-94hmq\" (UID: \"32d7da41-73b0-4ae5-a8b3-69bcb021cf4a\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-94hmq"
Nov 02 19:43:03.746156 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:03.746003    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32d7da41-73b0-4ae5-a8b3-69bcb021cf4a-workload-socket\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-94hmq\" (UID: \"32d7da41-73b0-4ae5-a8b3-69bcb021cf4a\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-94hmq"
Nov 02 19:43:03.746156 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:03.746079    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32d7da41-73b0-4ae5-a8b3-69bcb021cf4a-istio-data\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-94hmq\" (UID: \"32d7da41-73b0-4ae5-a8b3-69bcb021cf4a\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-94hmq"
Nov 02 19:43:03.746156 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:03.746109    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32d7da41-73b0-4ae5-a8b3-69bcb021cf4a-istiod-ca-cert\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-94hmq\" (UID: \"32d7da41-73b0-4ae5-a8b3-69bcb021cf4a\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-94hmq"
Nov 02 19:43:03.746156 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:03.746134    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32d7da41-73b0-4ae5-a8b3-69bcb021cf4a-istio-envoy\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-94hmq\" (UID: \"32d7da41-73b0-4ae5-a8b3-69bcb021cf4a\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-94hmq"
Nov 02 19:43:03.746473 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:03.746165    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32d7da41-73b0-4ae5-a8b3-69bcb021cf4a-istio-podinfo\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-94hmq\" (UID: \"32d7da41-73b0-4ae5-a8b3-69bcb021cf4a\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-94hmq"
Nov 02 19:43:03.746473 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:03.746194    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbnn8\" (UniqueName: \"kubernetes.io/projected/32d7da41-73b0-4ae5-a8b3-69bcb021cf4a-kube-api-access-tbnn8\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-94hmq\" (UID: \"32d7da41-73b0-4ae5-a8b3-69bcb021cf4a\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-94hmq"
Nov 02 19:43:03.746473 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:03.746224    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32d7da41-73b0-4ae5-a8b3-69bcb021cf4a-credential-socket\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-94hmq\" (UID: \"32d7da41-73b0-4ae5-a8b3-69bcb021cf4a\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-94hmq"
Nov 02 19:43:03.746473 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:03.746409    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32d7da41-73b0-4ae5-a8b3-69bcb021cf4a-workload-certs\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-94hmq\" (UID: \"32d7da41-73b0-4ae5-a8b3-69bcb021cf4a\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-94hmq"
Nov 02 19:43:03.746473 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:03.746453    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32d7da41-73b0-4ae5-a8b3-69bcb021cf4a-workload-socket\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-94hmq\" (UID: \"32d7da41-73b0-4ae5-a8b3-69bcb021cf4a\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-94hmq"
Nov 02 19:43:03.746729 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:03.746550    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32d7da41-73b0-4ae5-a8b3-69bcb021cf4a-istio-data\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-94hmq\" (UID: \"32d7da41-73b0-4ae5-a8b3-69bcb021cf4a\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-94hmq"
Nov 02 19:43:03.746792 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:03.746734    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32d7da41-73b0-4ae5-a8b3-69bcb021cf4a-credential-socket\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-94hmq\" (UID: \"32d7da41-73b0-4ae5-a8b3-69bcb021cf4a\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-94hmq"
Nov 02 19:43:03.746792 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:03.746768    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32d7da41-73b0-4ae5-a8b3-69bcb021cf4a-istiod-ca-cert\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-94hmq\" (UID: \"32d7da41-73b0-4ae5-a8b3-69bcb021cf4a\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-94hmq"
Nov 02 19:43:03.748619 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:03.748583    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32d7da41-73b0-4ae5-a8b3-69bcb021cf4a-istio-envoy\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-94hmq\" (UID: \"32d7da41-73b0-4ae5-a8b3-69bcb021cf4a\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-94hmq"
Nov 02 19:43:03.748723 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:03.748663    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32d7da41-73b0-4ae5-a8b3-69bcb021cf4a-istio-podinfo\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-94hmq\" (UID: \"32d7da41-73b0-4ae5-a8b3-69bcb021cf4a\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-94hmq"
Nov 02 19:43:03.799870 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:03.799802    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32d7da41-73b0-4ae5-a8b3-69bcb021cf4a-istio-token\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-94hmq\" (UID: \"32d7da41-73b0-4ae5-a8b3-69bcb021cf4a\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-94hmq"
Nov 02 19:43:03.799870 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:03.799864    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbnn8\" (UniqueName: \"kubernetes.io/projected/32d7da41-73b0-4ae5-a8b3-69bcb021cf4a-kube-api-access-tbnn8\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-94hmq\" (UID: \"32d7da41-73b0-4ae5-a8b3-69bcb021cf4a\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-94hmq"
Nov 02 19:43:03.865857 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:03.865821    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-94hmq"
Nov 02 19:43:04.222183 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:04.222144    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-74ft9"]
Nov 02 19:43:04.227094 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:04.227070    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-74ft9"
Nov 02 19:43:04.251439 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:04.251404    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cb6e664-e3c4-498a-978c-6a65ae7ffad9-istio-data\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-74ft9\" (UID: \"1cb6e664-e3c4-498a-978c-6a65ae7ffad9\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-74ft9"
Nov 02 19:43:04.251595 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:04.251442    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cb6e664-e3c4-498a-978c-6a65ae7ffad9-istio-token\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-74ft9\" (UID: \"1cb6e664-e3c4-498a-978c-6a65ae7ffad9\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-74ft9"
Nov 02 19:43:04.251595 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:04.251476    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cb6e664-e3c4-498a-978c-6a65ae7ffad9-istio-envoy\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-74ft9\" (UID: \"1cb6e664-e3c4-498a-978c-6a65ae7ffad9\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-74ft9"
Nov 02 19:43:04.251665 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:04.251621    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tr26s\" (UniqueName: \"kubernetes.io/projected/1cb6e664-e3c4-498a-978c-6a65ae7ffad9-kube-api-access-tr26s\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-74ft9\" (UID: \"1cb6e664-e3c4-498a-978c-6a65ae7ffad9\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-74ft9"
Nov 02 19:43:04.251704 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:04.251664    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cb6e664-e3c4-498a-978c-6a65ae7ffad9-credential-socket\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-74ft9\" (UID: \"1cb6e664-e3c4-498a-978c-6a65ae7ffad9\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-74ft9"
Nov 02 19:43:04.251704 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:04.251684    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cb6e664-e3c4-498a-978c-6a65ae7ffad9-istio-podinfo\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-74ft9\" (UID: \"1cb6e664-e3c4-498a-978c-6a65ae7ffad9\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-74ft9"
Nov 02 19:43:04.251790 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:04.251777    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cb6e664-e3c4-498a-978c-6a65ae7ffad9-istiod-ca-cert\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-74ft9\" (UID: \"1cb6e664-e3c4-498a-978c-6a65ae7ffad9\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-74ft9"
Nov 02 19:43:04.251824 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:04.251812    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cb6e664-e3c4-498a-978c-6a65ae7ffad9-workload-socket\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-74ft9\" (UID: \"1cb6e664-e3c4-498a-978c-6a65ae7ffad9\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-74ft9"
Nov 02 19:43:04.251861 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:04.251845    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cb6e664-e3c4-498a-978c-6a65ae7ffad9-workload-certs\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-74ft9\" (UID: \"1cb6e664-e3c4-498a-978c-6a65ae7ffad9\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-74ft9"
Nov 02 19:43:04.279662 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:04.279629    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-28623b-openshift-default-dockercfg-cxdg7\""
Nov 02 19:43:04.282921 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:04.282896    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-74ft9"]
Nov 02 19:43:04.292358 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:43:04.292326    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32d7da41_73b0_4ae5_a8b3_69bcb021cf4a.slice/crio-9b39cc0b486b08cfcade15b1dedc1a95121be7fa76d558f62ae416801405faec WatchSource:0}: Error finding container 9b39cc0b486b08cfcade15b1dedc1a95121be7fa76d558f62ae416801405faec: Status 404 returned error can't find the container with id 9b39cc0b486b08cfcade15b1dedc1a95121be7fa76d558f62ae416801405faec
Nov 02 19:43:04.295223 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:04.295186    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:43:04.295316 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:04.295297    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:43:04.295361 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:04.295339    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:43:04.300444 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:43:04.300418    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 19:43:04.328504 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:04.327768    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-94hmq"]
Nov 02 19:43:04.353252 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:04.353219    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cb6e664-e3c4-498a-978c-6a65ae7ffad9-workload-socket\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-74ft9\" (UID: \"1cb6e664-e3c4-498a-978c-6a65ae7ffad9\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-74ft9"
Nov 02 19:43:04.353395 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:04.353271    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cb6e664-e3c4-498a-978c-6a65ae7ffad9-workload-certs\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-74ft9\" (UID: \"1cb6e664-e3c4-498a-978c-6a65ae7ffad9\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-74ft9"
Nov 02 19:43:04.353395 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:04.353301    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cb6e664-e3c4-498a-978c-6a65ae7ffad9-istio-data\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-74ft9\" (UID: \"1cb6e664-e3c4-498a-978c-6a65ae7ffad9\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-74ft9"
Nov 02 19:43:04.353395 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:04.353324    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cb6e664-e3c4-498a-978c-6a65ae7ffad9-istio-token\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-74ft9\" (UID: \"1cb6e664-e3c4-498a-978c-6a65ae7ffad9\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-74ft9"
Nov 02 19:43:04.353395 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:04.353350    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cb6e664-e3c4-498a-978c-6a65ae7ffad9-istio-envoy\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-74ft9\" (UID: \"1cb6e664-e3c4-498a-978c-6a65ae7ffad9\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-74ft9"
Nov 02 19:43:04.353583 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:04.353419    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tr26s\" (UniqueName: \"kubernetes.io/projected/1cb6e664-e3c4-498a-978c-6a65ae7ffad9-kube-api-access-tr26s\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-74ft9\" (UID: \"1cb6e664-e3c4-498a-978c-6a65ae7ffad9\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-74ft9"
Nov 02 19:43:04.353583 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:04.353467    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cb6e664-e3c4-498a-978c-6a65ae7ffad9-credential-socket\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-74ft9\" (UID: \"1cb6e664-e3c4-498a-978c-6a65ae7ffad9\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-74ft9"
Nov 02 19:43:04.353583 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:04.353500    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cb6e664-e3c4-498a-978c-6a65ae7ffad9-istio-podinfo\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-74ft9\" (UID: \"1cb6e664-e3c4-498a-978c-6a65ae7ffad9\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-74ft9"
Nov 02 19:43:04.353740 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:04.353583    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cb6e664-e3c4-498a-978c-6a65ae7ffad9-istiod-ca-cert\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-74ft9\" (UID: \"1cb6e664-e3c4-498a-978c-6a65ae7ffad9\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-74ft9"
Nov 02 19:43:04.353740 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:04.353721    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cb6e664-e3c4-498a-978c-6a65ae7ffad9-istio-data\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-74ft9\" (UID: \"1cb6e664-e3c4-498a-978c-6a65ae7ffad9\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-74ft9"
Nov 02 19:43:04.353840 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:04.353745    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cb6e664-e3c4-498a-978c-6a65ae7ffad9-workload-socket\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-74ft9\" (UID: \"1cb6e664-e3c4-498a-978c-6a65ae7ffad9\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-74ft9"
Nov 02 19:43:04.353840 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:04.353821    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cb6e664-e3c4-498a-978c-6a65ae7ffad9-workload-certs\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-74ft9\" (UID: \"1cb6e664-e3c4-498a-978c-6a65ae7ffad9\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-74ft9"
Nov 02 19:43:04.354168 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:04.354146    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cb6e664-e3c4-498a-978c-6a65ae7ffad9-credential-socket\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-74ft9\" (UID: \"1cb6e664-e3c4-498a-978c-6a65ae7ffad9\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-74ft9"
Nov 02 19:43:04.354293 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:04.354275    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cb6e664-e3c4-498a-978c-6a65ae7ffad9-istiod-ca-cert\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-74ft9\" (UID: \"1cb6e664-e3c4-498a-978c-6a65ae7ffad9\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-74ft9"
Nov 02 19:43:04.355793 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:04.355773    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cb6e664-e3c4-498a-978c-6a65ae7ffad9-istio-envoy\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-74ft9\" (UID: \"1cb6e664-e3c4-498a-978c-6a65ae7ffad9\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-74ft9"
Nov 02 19:43:04.356140 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:04.356123    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cb6e664-e3c4-498a-978c-6a65ae7ffad9-istio-podinfo\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-74ft9\" (UID: \"1cb6e664-e3c4-498a-978c-6a65ae7ffad9\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-74ft9"
Nov 02 19:43:04.414764 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:04.414699    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cb6e664-e3c4-498a-978c-6a65ae7ffad9-istio-token\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-74ft9\" (UID: \"1cb6e664-e3c4-498a-978c-6a65ae7ffad9\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-74ft9"
Nov 02 19:43:04.418032 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:04.415405    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tr26s\" (UniqueName: \"kubernetes.io/projected/1cb6e664-e3c4-498a-978c-6a65ae7ffad9-kube-api-access-tr26s\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-74ft9\" (UID: \"1cb6e664-e3c4-498a-978c-6a65ae7ffad9\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-74ft9"
Nov 02 19:43:04.541721 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:04.541634    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-74ft9"
Nov 02 19:43:04.570317 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:04.570272    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-94hmq" event={"ID":"32d7da41-73b0-4ae5-a8b3-69bcb021cf4a","Type":"ContainerStarted","Data":"79a9f1a468225211f945377c7c63e9d2edb050c28f2cac2924d942cbba60f0a8"}
Nov 02 19:43:04.570796 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:04.570344    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-94hmq" event={"ID":"32d7da41-73b0-4ae5-a8b3-69bcb021cf4a","Type":"ContainerStarted","Data":"9b39cc0b486b08cfcade15b1dedc1a95121be7fa76d558f62ae416801405faec"}
Nov 02 19:43:04.681813 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:04.681750    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-94hmq" podStartSLOduration=1.681728353 podStartE2EDuration="1.681728353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:43:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:43:04.679885402 +0000 UTC m=+891.286919188" watchObservedRunningTime="2026-11-02 19:43:04.681728353 +0000 UTC m=+891.288762123"
Nov 02 19:43:04.872096 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:04.871110    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-94hmq"
Nov 02 19:43:04.980662 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:04.978996    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-74ft9"]
Nov 02 19:43:04.997987 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:43:04.997853    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cb6e664_e3c4_498a_978c_6a65ae7ffad9.slice/crio-d558b5681f8218c8dfbb994c76d69966ef37b9ca72b71093d5d26cb5dc3d3253 WatchSource:0}: Error finding container d558b5681f8218c8dfbb994c76d69966ef37b9ca72b71093d5d26cb5dc3d3253: Status 404 returned error can't find the container with id d558b5681f8218c8dfbb994c76d69966ef37b9ca72b71093d5d26cb5dc3d3253
Nov 02 19:43:05.002985 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:05.001761    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:43:05.002985 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:05.001935    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:43:05.002985 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:05.002178    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:43:05.601976 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:05.601931    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-74ft9" event={"ID":"1cb6e664-e3c4-498a-978c-6a65ae7ffad9","Type":"ContainerStarted","Data":"4876240eca1c749c2da9ad82e84f0ef726287db80bd656f4a9aa676cfce5de52"}
Nov 02 19:43:05.602402 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:05.601994    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-74ft9" event={"ID":"1cb6e664-e3c4-498a-978c-6a65ae7ffad9","Type":"ContainerStarted","Data":"d558b5681f8218c8dfbb994c76d69966ef37b9ca72b71093d5d26cb5dc3d3253"}
Nov 02 19:43:05.711381 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:05.711313    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-74ft9" podStartSLOduration=1.7112945910000001 podStartE2EDuration="1.711294591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:43:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:43:05.709694326 +0000 UTC m=+892.316728116" watchObservedRunningTime="2026-11-02 19:43:05.711294591 +0000 UTC m=+892.318328361"
Nov 02 19:43:05.869222 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:05.869123    4463 patch_prober.go:29] interesting pod/e2e-worker-w5-1d869e-openshift-default-56b959b4df-94hmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Nov 02 19:43:05.869479 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:05.869451    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-94hmq" podUID="32d7da41-73b0-4ae5-a8b3-69bcb021cf4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Nov 02 19:43:06.542069 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:06.542032    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-74ft9"
Nov 02 19:43:06.543480 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:06.543456    4463 patch_prober.go:29] interesting pod/e2e-worker-w2-28623b-openshift-default-5b66f6f775-74ft9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Nov 02 19:43:06.543655 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:06.543637    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-74ft9" podUID="1cb6e664-e3c4-498a-978c-6a65ae7ffad9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Nov 02 19:43:06.879096 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:06.878466    4463 patch_prober.go:29] interesting pod/e2e-worker-w5-1d869e-openshift-default-56b959b4df-94hmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Nov 02 19:43:06.879096 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:06.878528    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-94hmq" podUID="32d7da41-73b0-4ae5-a8b3-69bcb021cf4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Nov 02 19:43:07.542512 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:07.542469    4463 patch_prober.go:29] interesting pod/e2e-worker-w2-28623b-openshift-default-5b66f6f775-74ft9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Nov 02 19:43:07.542674 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:07.542531    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-74ft9" podUID="1cb6e664-e3c4-498a-978c-6a65ae7ffad9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Nov 02 19:43:07.871098 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:07.870394    4463 patch_prober.go:29] interesting pod/e2e-worker-w5-1d869e-openshift-default-56b959b4df-94hmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Nov 02 19:43:07.871098 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:07.870460    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-94hmq" podUID="32d7da41-73b0-4ae5-a8b3-69bcb021cf4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Nov 02 19:43:08.542934 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:08.542899    4463 patch_prober.go:29] interesting pod/e2e-worker-w2-28623b-openshift-default-5b66f6f775-74ft9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Nov 02 19:43:08.543358 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:08.542947    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-74ft9" podUID="1cb6e664-e3c4-498a-978c-6a65ae7ffad9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Nov 02 19:43:08.868090 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:08.867491    4463 patch_prober.go:29] interesting pod/e2e-worker-w5-1d869e-openshift-default-56b959b4df-94hmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Nov 02 19:43:08.868090 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:08.867555    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-94hmq" podUID="32d7da41-73b0-4ae5-a8b3-69bcb021cf4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Nov 02 19:43:09.542589 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:09.542555    4463 patch_prober.go:29] interesting pod/e2e-worker-w2-28623b-openshift-default-5b66f6f775-74ft9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Nov 02 19:43:09.542802 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:09.542610    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-74ft9" podUID="1cb6e664-e3c4-498a-978c-6a65ae7ffad9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Nov 02 19:43:09.874094 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:09.868420    4463 patch_prober.go:29] interesting pod/e2e-worker-w5-1d869e-openshift-default-56b959b4df-94hmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Nov 02 19:43:09.874094 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:09.868483    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-94hmq" podUID="32d7da41-73b0-4ae5-a8b3-69bcb021cf4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Nov 02 19:43:10.542654 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:10.542617    4463 patch_prober.go:29] interesting pod/e2e-worker-w2-28623b-openshift-default-5b66f6f775-74ft9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Nov 02 19:43:10.542839 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:10.542678    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-74ft9" podUID="1cb6e664-e3c4-498a-978c-6a65ae7ffad9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Nov 02 19:43:10.868094 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:10.866449    4463 patch_prober.go:29] interesting pod/e2e-worker-w5-1d869e-openshift-default-56b959b4df-94hmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Nov 02 19:43:10.868094 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:10.866512    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-94hmq" podUID="32d7da41-73b0-4ae5-a8b3-69bcb021cf4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Nov 02 19:43:11.542611 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:11.542573    4463 patch_prober.go:29] interesting pod/e2e-worker-w2-28623b-openshift-default-5b66f6f775-74ft9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Nov 02 19:43:11.543154 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:11.542635    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-74ft9" podUID="1cb6e664-e3c4-498a-978c-6a65ae7ffad9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Nov 02 19:43:11.872533 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:11.872442    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-94hmq"
Nov 02 19:43:11.872846 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:11.872823    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-94hmq"
Nov 02 19:43:11.873702 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:11.873684    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-94hmq"
Nov 02 19:43:12.547086 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:12.547056    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-74ft9"
Nov 02 19:43:12.547551 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:12.547378    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-74ft9"
Nov 02 19:43:12.548063 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:12.548042    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-74ft9"
Nov 02 19:43:33.770658 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:43:33.770621    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 19:43:34.303204 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:43:34.303163    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 19:44:04.293985 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:44:04.293947    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 19:44:14.379883 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:14.379846    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 19:44:34.294194 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:44:34.294161    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 19:44:34.377168 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:34.377138    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6v2d5"]
Nov 02 19:44:34.377361 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:34.377341    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6v2d5" podUID="14de05d8-f8d6-4ab0-8671-faa1158ca274" containerName="istio-proxy" containerID="cri-o://f0786da9edfe1c2555abfd654fbb2f3e30c7c610764dc3063b8fc212f4d6cecd" gracePeriod=30
Nov 02 19:44:39.645309 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:39.645254    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6v2d5"
Nov 02 19:44:39.723599 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:39.723574    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14de05d8-f8d6-4ab0-8671-faa1158ca274-istio-token\" (UniqueName: \"kubernetes.io/projected/14de05d8-f8d6-4ab0-8671-faa1158ca274-istio-token\") pod \"14de05d8-f8d6-4ab0-8671-faa1158ca274\" (UID: \"14de05d8-f8d6-4ab0-8671-faa1158ca274\") "
Nov 02 19:44:39.723725 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:39.723669    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/14de05d8-f8d6-4ab0-8671-faa1158ca274-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14de05d8-f8d6-4ab0-8671-faa1158ca274-istio-podinfo\") pod \"14de05d8-f8d6-4ab0-8671-faa1158ca274\" (UID: \"14de05d8-f8d6-4ab0-8671-faa1158ca274\") "
Nov 02 19:44:39.723725 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:39.723709    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14de05d8-f8d6-4ab0-8671-faa1158ca274-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14de05d8-f8d6-4ab0-8671-faa1158ca274-workload-socket\") pod \"14de05d8-f8d6-4ab0-8671-faa1158ca274\" (UID: \"14de05d8-f8d6-4ab0-8671-faa1158ca274\") "
Nov 02 19:44:39.723806 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:39.723752    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14de05d8-f8d6-4ab0-8671-faa1158ca274-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14de05d8-f8d6-4ab0-8671-faa1158ca274-credential-socket\") pod \"14de05d8-f8d6-4ab0-8671-faa1158ca274\" (UID: \"14de05d8-f8d6-4ab0-8671-faa1158ca274\") "
Nov 02 19:44:39.723806 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:39.723781    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14de05d8-f8d6-4ab0-8671-faa1158ca274-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14de05d8-f8d6-4ab0-8671-faa1158ca274-istio-data\") pod \"14de05d8-f8d6-4ab0-8671-faa1158ca274\" (UID: \"14de05d8-f8d6-4ab0-8671-faa1158ca274\") "
Nov 02 19:44:39.723901 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:39.723812    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14de05d8-f8d6-4ab0-8671-faa1158ca274-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14de05d8-f8d6-4ab0-8671-faa1158ca274-istiod-ca-cert\") pod \"14de05d8-f8d6-4ab0-8671-faa1158ca274\" (UID: \"14de05d8-f8d6-4ab0-8671-faa1158ca274\") "
Nov 02 19:44:39.723901 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:39.723860    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14de05d8-f8d6-4ab0-8671-faa1158ca274-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14de05d8-f8d6-4ab0-8671-faa1158ca274-workload-certs\") pod \"14de05d8-f8d6-4ab0-8671-faa1158ca274\" (UID: \"14de05d8-f8d6-4ab0-8671-faa1158ca274\") "
Nov 02 19:44:39.724006 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:39.723900    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14de05d8-f8d6-4ab0-8671-faa1158ca274-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14de05d8-f8d6-4ab0-8671-faa1158ca274-istio-envoy\") pod \"14de05d8-f8d6-4ab0-8671-faa1158ca274\" (UID: \"14de05d8-f8d6-4ab0-8671-faa1158ca274\") "
Nov 02 19:44:39.724006 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:39.723981    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14de05d8-f8d6-4ab0-8671-faa1158ca274-kube-api-access-7zpsc\" (UniqueName: \"kubernetes.io/projected/14de05d8-f8d6-4ab0-8671-faa1158ca274-kube-api-access-7zpsc\") pod \"14de05d8-f8d6-4ab0-8671-faa1158ca274\" (UID: \"14de05d8-f8d6-4ab0-8671-faa1158ca274\") "
Nov 02 19:44:39.724162 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:39.724112    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14de05d8-f8d6-4ab0-8671-faa1158ca274-workload-socket" pod "14de05d8-f8d6-4ab0-8671-faa1158ca274" (UID: "14de05d8-f8d6-4ab0-8671-faa1158ca274"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:44:39.724231 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:39.724172    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14de05d8-f8d6-4ab0-8671-faa1158ca274-istio-data" pod "14de05d8-f8d6-4ab0-8671-faa1158ca274" (UID: "14de05d8-f8d6-4ab0-8671-faa1158ca274"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:44:39.724421 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:39.724393    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14de05d8-f8d6-4ab0-8671-faa1158ca274-workload-certs" pod "14de05d8-f8d6-4ab0-8671-faa1158ca274" (UID: "14de05d8-f8d6-4ab0-8671-faa1158ca274"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:44:39.724421 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:39.724403    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14de05d8-f8d6-4ab0-8671-faa1158ca274-istiod-ca-cert" pod "14de05d8-f8d6-4ab0-8671-faa1158ca274" (UID: "14de05d8-f8d6-4ab0-8671-faa1158ca274"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 19:44:39.724567 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:39.724548    4463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14de05d8-f8d6-4ab0-8671-faa1158ca274-workload-socket\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:44:39.724627 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:39.724575    4463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14de05d8-f8d6-4ab0-8671-faa1158ca274-istio-data\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:44:39.724627 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:39.724592    4463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14de05d8-f8d6-4ab0-8671-faa1158ca274-istiod-ca-cert\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:44:39.724627 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:39.724606    4463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14de05d8-f8d6-4ab0-8671-faa1158ca274-workload-certs\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:44:39.724787 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:39.724664    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14de05d8-f8d6-4ab0-8671-faa1158ca274-credential-socket" pod "14de05d8-f8d6-4ab0-8671-faa1158ca274" (UID: "14de05d8-f8d6-4ab0-8671-faa1158ca274"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:44:39.725911 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:39.725885    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/14de05d8-f8d6-4ab0-8671-faa1158ca274-istio-podinfo" pod "14de05d8-f8d6-4ab0-8671-faa1158ca274" (UID: "14de05d8-f8d6-4ab0-8671-faa1158ca274"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 19:44:39.725911 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:39.725906    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14de05d8-f8d6-4ab0-8671-faa1158ca274-istio-token" pod "14de05d8-f8d6-4ab0-8671-faa1158ca274" (UID: "14de05d8-f8d6-4ab0-8671-faa1158ca274"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:44:39.726179 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:39.726164    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14de05d8-f8d6-4ab0-8671-faa1158ca274-kube-api-access-7zpsc" pod "14de05d8-f8d6-4ab0-8671-faa1158ca274" (UID: "14de05d8-f8d6-4ab0-8671-faa1158ca274"). InnerVolumeSpecName "kube-api-access-7zpsc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:44:39.726179 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:39.726172    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14de05d8-f8d6-4ab0-8671-faa1158ca274-istio-envoy" pod "14de05d8-f8d6-4ab0-8671-faa1158ca274" (UID: "14de05d8-f8d6-4ab0-8671-faa1158ca274"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:44:39.826160 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:39.826119    4463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14de05d8-f8d6-4ab0-8671-faa1158ca274-istio-envoy\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:44:39.826160 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:39.826152    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zpsc\" (UniqueName: \"kubernetes.io/projected/14de05d8-f8d6-4ab0-8671-faa1158ca274-kube-api-access-7zpsc\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:44:39.826160 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:39.826166    4463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14de05d8-f8d6-4ab0-8671-faa1158ca274-istio-token\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:44:39.826392 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:39.826180    4463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14de05d8-f8d6-4ab0-8671-faa1158ca274-istio-podinfo\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:44:39.826392 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:39.826192    4463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14de05d8-f8d6-4ab0-8671-faa1158ca274-credential-socket\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:44:40.119942 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:40.119907    4463 generic.go:356] "Generic (PLEG): container finished" podID="14de05d8-f8d6-4ab0-8671-faa1158ca274" containerID="f0786da9edfe1c2555abfd654fbb2f3e30c7c610764dc3063b8fc212f4d6cecd" exitCode=0
Nov 02 19:44:40.120126 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:40.119955    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6v2d5"
Nov 02 19:44:40.120126 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:40.119993    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6v2d5" event={"ID":"14de05d8-f8d6-4ab0-8671-faa1158ca274","Type":"ContainerDied","Data":"f0786da9edfe1c2555abfd654fbb2f3e30c7c610764dc3063b8fc212f4d6cecd"}
Nov 02 19:44:40.120126 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:40.120053    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6v2d5" event={"ID":"14de05d8-f8d6-4ab0-8671-faa1158ca274","Type":"ContainerDied","Data":"70ad73cfe5853ea47cf1f6b01f7b09a8f9915981a1382b9be4c902d6c64c82c5"}
Nov 02 19:44:40.120126 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:40.120076    4463 scope.go:122] "RemoveContainer" containerID="f0786da9edfe1c2555abfd654fbb2f3e30c7c610764dc3063b8fc212f4d6cecd"
Nov 02 19:44:40.129914 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:40.129896    4463 scope.go:122] "RemoveContainer" containerID="f0786da9edfe1c2555abfd654fbb2f3e30c7c610764dc3063b8fc212f4d6cecd"
Nov 02 19:44:40.130186 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:44:40.130168    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0786da9edfe1c2555abfd654fbb2f3e30c7c610764dc3063b8fc212f4d6cecd\": container with ID starting with f0786da9edfe1c2555abfd654fbb2f3e30c7c610764dc3063b8fc212f4d6cecd not found: ID does not exist" containerID="f0786da9edfe1c2555abfd654fbb2f3e30c7c610764dc3063b8fc212f4d6cecd"
Nov 02 19:44:40.130238 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:40.130195    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0786da9edfe1c2555abfd654fbb2f3e30c7c610764dc3063b8fc212f4d6cecd"} err="failed to get container status \"f0786da9edfe1c2555abfd654fbb2f3e30c7c610764dc3063b8fc212f4d6cecd\": rpc error: code = NotFound desc = could not find container \"f0786da9edfe1c2555abfd654fbb2f3e30c7c610764dc3063b8fc212f4d6cecd\": container with ID starting with f0786da9edfe1c2555abfd654fbb2f3e30c7c610764dc3063b8fc212f4d6cecd not found: ID does not exist"
Nov 02 19:44:40.273780 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:40.273747    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6v2d5"]
Nov 02 19:44:40.319787 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:40.319757    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6v2d5"]
Nov 02 19:44:41.585633 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:41.585595    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14de05d8-f8d6-4ab0-8671-faa1158ca274" path="/var/lib/kubelet/pods/14de05d8-f8d6-4ab0-8671-faa1158ca274/volumes"
Nov 02 19:44:44.288578 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:44.288538    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-8795bf/distinct-w4-8795bf-kserve-64b4796df4-8qgqh"]
Nov 02 19:44:44.289246 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:44.289224    4463 cpu_manager.go:414] "containerMap: removing container" podUID="14de05d8-f8d6-4ab0-8671-faa1158ca274" containerName="istio-proxy"
Nov 02 19:44:44.289349 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:44.289247    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14de05d8-f8d6-4ab0-8671-faa1158ca274" containerName="istio-proxy"
Nov 02 19:44:44.289416 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:44.289405    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="14de05d8-f8d6-4ab0-8671-faa1158ca274" containerName="istio-proxy"
Nov 02 19:44:44.293602 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:44.293580    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8795bf/distinct-w4-8795bf-kserve-64b4796df4-8qgqh"
Nov 02 19:44:44.375038 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:44.374977    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee7d28f4-e9c7-43e2-b983-1427589f1785-model-cache\") pod \"distinct-w4-8795bf-kserve-64b4796df4-8qgqh\" (UID: \"ee7d28f4-e9c7-43e2-b983-1427589f1785\") " pod="e2e-models-e2e-worker-w4-8795bf/distinct-w4-8795bf-kserve-64b4796df4-8qgqh"
Nov 02 19:44:44.375174 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:44.375040    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee7d28f4-e9c7-43e2-b983-1427589f1785-dshm\") pod \"distinct-w4-8795bf-kserve-64b4796df4-8qgqh\" (UID: \"ee7d28f4-e9c7-43e2-b983-1427589f1785\") " pod="e2e-models-e2e-worker-w4-8795bf/distinct-w4-8795bf-kserve-64b4796df4-8qgqh"
Nov 02 19:44:44.375219 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:44.375177    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee7d28f4-e9c7-43e2-b983-1427589f1785-home\") pod \"distinct-w4-8795bf-kserve-64b4796df4-8qgqh\" (UID: \"ee7d28f4-e9c7-43e2-b983-1427589f1785\") " pod="e2e-models-e2e-worker-w4-8795bf/distinct-w4-8795bf-kserve-64b4796df4-8qgqh"
Nov 02 19:44:44.375385 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:44.375366    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee7d28f4-e9c7-43e2-b983-1427589f1785-tls-certs\") pod \"distinct-w4-8795bf-kserve-64b4796df4-8qgqh\" (UID: \"ee7d28f4-e9c7-43e2-b983-1427589f1785\") " pod="e2e-models-e2e-worker-w4-8795bf/distinct-w4-8795bf-kserve-64b4796df4-8qgqh"
Nov 02 19:44:44.375430 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:44.375418    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee7d28f4-e9c7-43e2-b983-1427589f1785-tmp-dir\") pod \"distinct-w4-8795bf-kserve-64b4796df4-8qgqh\" (UID: \"ee7d28f4-e9c7-43e2-b983-1427589f1785\") " pod="e2e-models-e2e-worker-w4-8795bf/distinct-w4-8795bf-kserve-64b4796df4-8qgqh"
Nov 02 19:44:44.375514 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:44.375497    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68xht\" (UniqueName: \"kubernetes.io/projected/ee7d28f4-e9c7-43e2-b983-1427589f1785-kube-api-access-68xht\") pod \"distinct-w4-8795bf-kserve-64b4796df4-8qgqh\" (UID: \"ee7d28f4-e9c7-43e2-b983-1427589f1785\") " pod="e2e-models-e2e-worker-w4-8795bf/distinct-w4-8795bf-kserve-64b4796df4-8qgqh"
Nov 02 19:44:44.388627 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:44.388607    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-8795bf\"/\"openshift-service-ca.crt\""
Nov 02 19:44:44.388767 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:44.388671    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-8795bf\"/\"kube-root-ca.crt\""
Nov 02 19:44:44.388851 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:44.388763    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-8795bf\"/\"distinct-w4-8795bf-kserve-self-signed-certs\""
Nov 02 19:44:44.388851 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:44.388839    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-8795bf\"/\"default-dockercfg-h46nk\""
Nov 02 19:44:44.390879 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:44.390858    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8795bf/distinct-w4-8795bf-kserve-64b4796df4-8qgqh"]
Nov 02 19:44:44.476732 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:44.476697    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68xht\" (UniqueName: \"kubernetes.io/projected/ee7d28f4-e9c7-43e2-b983-1427589f1785-kube-api-access-68xht\") pod \"distinct-w4-8795bf-kserve-64b4796df4-8qgqh\" (UID: \"ee7d28f4-e9c7-43e2-b983-1427589f1785\") " pod="e2e-models-e2e-worker-w4-8795bf/distinct-w4-8795bf-kserve-64b4796df4-8qgqh"
Nov 02 19:44:44.476924 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:44.476741    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee7d28f4-e9c7-43e2-b983-1427589f1785-model-cache\") pod \"distinct-w4-8795bf-kserve-64b4796df4-8qgqh\" (UID: \"ee7d28f4-e9c7-43e2-b983-1427589f1785\") " pod="e2e-models-e2e-worker-w4-8795bf/distinct-w4-8795bf-kserve-64b4796df4-8qgqh"
Nov 02 19:44:44.476924 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:44.476781    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee7d28f4-e9c7-43e2-b983-1427589f1785-dshm\") pod \"distinct-w4-8795bf-kserve-64b4796df4-8qgqh\" (UID: \"ee7d28f4-e9c7-43e2-b983-1427589f1785\") " pod="e2e-models-e2e-worker-w4-8795bf/distinct-w4-8795bf-kserve-64b4796df4-8qgqh"
Nov 02 19:44:44.476924 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:44.476838    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee7d28f4-e9c7-43e2-b983-1427589f1785-home\") pod \"distinct-w4-8795bf-kserve-64b4796df4-8qgqh\" (UID: \"ee7d28f4-e9c7-43e2-b983-1427589f1785\") " pod="e2e-models-e2e-worker-w4-8795bf/distinct-w4-8795bf-kserve-64b4796df4-8qgqh"
Nov 02 19:44:44.477132 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:44.476996    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee7d28f4-e9c7-43e2-b983-1427589f1785-tls-certs\") pod \"distinct-w4-8795bf-kserve-64b4796df4-8qgqh\" (UID: \"ee7d28f4-e9c7-43e2-b983-1427589f1785\") " pod="e2e-models-e2e-worker-w4-8795bf/distinct-w4-8795bf-kserve-64b4796df4-8qgqh"
Nov 02 19:44:44.477132 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:44.477082    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee7d28f4-e9c7-43e2-b983-1427589f1785-tmp-dir\") pod \"distinct-w4-8795bf-kserve-64b4796df4-8qgqh\" (UID: \"ee7d28f4-e9c7-43e2-b983-1427589f1785\") " pod="e2e-models-e2e-worker-w4-8795bf/distinct-w4-8795bf-kserve-64b4796df4-8qgqh"
Nov 02 19:44:44.477346 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:44.477322    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee7d28f4-e9c7-43e2-b983-1427589f1785-model-cache\") pod \"distinct-w4-8795bf-kserve-64b4796df4-8qgqh\" (UID: \"ee7d28f4-e9c7-43e2-b983-1427589f1785\") " pod="e2e-models-e2e-worker-w4-8795bf/distinct-w4-8795bf-kserve-64b4796df4-8qgqh"
Nov 02 19:44:44.477420 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:44.477318    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee7d28f4-e9c7-43e2-b983-1427589f1785-home\") pod \"distinct-w4-8795bf-kserve-64b4796df4-8qgqh\" (UID: \"ee7d28f4-e9c7-43e2-b983-1427589f1785\") " pod="e2e-models-e2e-worker-w4-8795bf/distinct-w4-8795bf-kserve-64b4796df4-8qgqh"
Nov 02 19:44:44.477420 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:44.477396    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee7d28f4-e9c7-43e2-b983-1427589f1785-tmp-dir\") pod \"distinct-w4-8795bf-kserve-64b4796df4-8qgqh\" (UID: \"ee7d28f4-e9c7-43e2-b983-1427589f1785\") " pod="e2e-models-e2e-worker-w4-8795bf/distinct-w4-8795bf-kserve-64b4796df4-8qgqh"
Nov 02 19:44:44.479111 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:44.479093    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee7d28f4-e9c7-43e2-b983-1427589f1785-dshm\") pod \"distinct-w4-8795bf-kserve-64b4796df4-8qgqh\" (UID: \"ee7d28f4-e9c7-43e2-b983-1427589f1785\") " pod="e2e-models-e2e-worker-w4-8795bf/distinct-w4-8795bf-kserve-64b4796df4-8qgqh"
Nov 02 19:44:44.479379 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:44.479363    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee7d28f4-e9c7-43e2-b983-1427589f1785-tls-certs\") pod \"distinct-w4-8795bf-kserve-64b4796df4-8qgqh\" (UID: \"ee7d28f4-e9c7-43e2-b983-1427589f1785\") " pod="e2e-models-e2e-worker-w4-8795bf/distinct-w4-8795bf-kserve-64b4796df4-8qgqh"
Nov 02 19:44:44.529546 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:44.529516    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68xht\" (UniqueName: \"kubernetes.io/projected/ee7d28f4-e9c7-43e2-b983-1427589f1785-kube-api-access-68xht\") pod \"distinct-w4-8795bf-kserve-64b4796df4-8qgqh\" (UID: \"ee7d28f4-e9c7-43e2-b983-1427589f1785\") " pod="e2e-models-e2e-worker-w4-8795bf/distinct-w4-8795bf-kserve-64b4796df4-8qgqh"
Nov 02 19:44:44.605471 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:44.605416    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8795bf/distinct-w4-8795bf-kserve-64b4796df4-8qgqh"
Nov 02 19:44:44.862440 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:44.862357    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 19:44:44.991964 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:44.991941    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8795bf/distinct-w4-8795bf-kserve-64b4796df4-8qgqh"]
Nov 02 19:44:44.994341 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:44:44.994313    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podee7d28f4_e9c7_43e2_b983_1427589f1785.slice/crio-00fb291e44ab7e4c63cf3034fd67466e4b190533f68a5ae9fe1ad254240d241f WatchSource:0}: Error finding container 00fb291e44ab7e4c63cf3034fd67466e4b190533f68a5ae9fe1ad254240d241f: Status 404 returned error can't find the container with id 00fb291e44ab7e4c63cf3034fd67466e4b190533f68a5ae9fe1ad254240d241f
Nov 02 19:44:45.150766 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:45.150671    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8795bf/distinct-w4-8795bf-kserve-64b4796df4-8qgqh" event={"ID":"ee7d28f4-e9c7-43e2-b983-1427589f1785","Type":"ContainerStarted","Data":"f52fdb1337b65d4c6774275aa530bc13f1b35b7ff2e505567c9d660d18c7befb"}
Nov 02 19:44:45.150766 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:45.150717    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8795bf/distinct-w4-8795bf-kserve-64b4796df4-8qgqh" event={"ID":"ee7d28f4-e9c7-43e2-b983-1427589f1785","Type":"ContainerStarted","Data":"00fb291e44ab7e4c63cf3034fd67466e4b190533f68a5ae9fe1ad254240d241f"}
Nov 02 19:44:45.259791 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:45.259736    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-8795bf/distinct-w4-8795bf-kserve-64b4796df4-8qgqh" podStartSLOduration=1.259722843 podStartE2EDuration="1.259722843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:44:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:44:45.257523515 +0000 UTC m=+991.864557319" watchObservedRunningTime="2026-11-02 19:44:45.259722843 +0000 UTC m=+991.866756649"
Nov 02 19:44:45.605851 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:45.605820    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8795bf/distinct-w4-8795bf-kserve-64b4796df4-8qgqh"
Nov 02 19:44:49.596814 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:49.596781    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-26cfbe/distinct-w1-26cfbe-kserve-685985bfb-tkgn2"]
Nov 02 19:44:49.600325 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:49.600304    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-26cfbe/distinct-w1-26cfbe-kserve-685985bfb-tkgn2"
Nov 02 19:44:49.651985 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:49.651952    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-26cfbe\"/\"distinct-w1-26cfbe-kserve-self-signed-certs\""
Nov 02 19:44:49.697274 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:49.697247    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-26cfbe/distinct-w1-26cfbe-kserve-685985bfb-tkgn2"]
Nov 02 19:44:49.739127 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:49.739083    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7415aa62-467c-43af-94c8-d604f57ec1c3-home\") pod \"distinct-w1-26cfbe-kserve-685985bfb-tkgn2\" (UID: \"7415aa62-467c-43af-94c8-d604f57ec1c3\") " pod="e2e-models-e2e-worker-w1-26cfbe/distinct-w1-26cfbe-kserve-685985bfb-tkgn2"
Nov 02 19:44:49.739280 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:49.739241    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7415aa62-467c-43af-94c8-d604f57ec1c3-dshm\") pod \"distinct-w1-26cfbe-kserve-685985bfb-tkgn2\" (UID: \"7415aa62-467c-43af-94c8-d604f57ec1c3\") " pod="e2e-models-e2e-worker-w1-26cfbe/distinct-w1-26cfbe-kserve-685985bfb-tkgn2"
Nov 02 19:44:49.739381 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:49.739363    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9zgq\" (UniqueName: \"kubernetes.io/projected/7415aa62-467c-43af-94c8-d604f57ec1c3-kube-api-access-j9zgq\") pod \"distinct-w1-26cfbe-kserve-685985bfb-tkgn2\" (UID: \"7415aa62-467c-43af-94c8-d604f57ec1c3\") " pod="e2e-models-e2e-worker-w1-26cfbe/distinct-w1-26cfbe-kserve-685985bfb-tkgn2"
Nov 02 19:44:49.739446 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:49.739435    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7415aa62-467c-43af-94c8-d604f57ec1c3-tmp-dir\") pod \"distinct-w1-26cfbe-kserve-685985bfb-tkgn2\" (UID: \"7415aa62-467c-43af-94c8-d604f57ec1c3\") " pod="e2e-models-e2e-worker-w1-26cfbe/distinct-w1-26cfbe-kserve-685985bfb-tkgn2"
Nov 02 19:44:49.739530 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:49.739508    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7415aa62-467c-43af-94c8-d604f57ec1c3-model-cache\") pod \"distinct-w1-26cfbe-kserve-685985bfb-tkgn2\" (UID: \"7415aa62-467c-43af-94c8-d604f57ec1c3\") " pod="e2e-models-e2e-worker-w1-26cfbe/distinct-w1-26cfbe-kserve-685985bfb-tkgn2"
Nov 02 19:44:49.739577 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:49.739547    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7415aa62-467c-43af-94c8-d604f57ec1c3-tls-certs\") pod \"distinct-w1-26cfbe-kserve-685985bfb-tkgn2\" (UID: \"7415aa62-467c-43af-94c8-d604f57ec1c3\") " pod="e2e-models-e2e-worker-w1-26cfbe/distinct-w1-26cfbe-kserve-685985bfb-tkgn2"
Nov 02 19:44:49.841063 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:49.841025    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7415aa62-467c-43af-94c8-d604f57ec1c3-home\") pod \"distinct-w1-26cfbe-kserve-685985bfb-tkgn2\" (UID: \"7415aa62-467c-43af-94c8-d604f57ec1c3\") " pod="e2e-models-e2e-worker-w1-26cfbe/distinct-w1-26cfbe-kserve-685985bfb-tkgn2"
Nov 02 19:44:49.841243 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:49.841075    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7415aa62-467c-43af-94c8-d604f57ec1c3-dshm\") pod \"distinct-w1-26cfbe-kserve-685985bfb-tkgn2\" (UID: \"7415aa62-467c-43af-94c8-d604f57ec1c3\") " pod="e2e-models-e2e-worker-w1-26cfbe/distinct-w1-26cfbe-kserve-685985bfb-tkgn2"
Nov 02 19:44:49.841243 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:49.841116    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9zgq\" (UniqueName: \"kubernetes.io/projected/7415aa62-467c-43af-94c8-d604f57ec1c3-kube-api-access-j9zgq\") pod \"distinct-w1-26cfbe-kserve-685985bfb-tkgn2\" (UID: \"7415aa62-467c-43af-94c8-d604f57ec1c3\") " pod="e2e-models-e2e-worker-w1-26cfbe/distinct-w1-26cfbe-kserve-685985bfb-tkgn2"
Nov 02 19:44:49.841243 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:49.841154    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7415aa62-467c-43af-94c8-d604f57ec1c3-tmp-dir\") pod \"distinct-w1-26cfbe-kserve-685985bfb-tkgn2\" (UID: \"7415aa62-467c-43af-94c8-d604f57ec1c3\") " pod="e2e-models-e2e-worker-w1-26cfbe/distinct-w1-26cfbe-kserve-685985bfb-tkgn2"
Nov 02 19:44:49.841243 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:49.841172    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7415aa62-467c-43af-94c8-d604f57ec1c3-model-cache\") pod \"distinct-w1-26cfbe-kserve-685985bfb-tkgn2\" (UID: \"7415aa62-467c-43af-94c8-d604f57ec1c3\") " pod="e2e-models-e2e-worker-w1-26cfbe/distinct-w1-26cfbe-kserve-685985bfb-tkgn2"
Nov 02 19:44:49.841243 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:49.841188    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7415aa62-467c-43af-94c8-d604f57ec1c3-tls-certs\") pod \"distinct-w1-26cfbe-kserve-685985bfb-tkgn2\" (UID: \"7415aa62-467c-43af-94c8-d604f57ec1c3\") " pod="e2e-models-e2e-worker-w1-26cfbe/distinct-w1-26cfbe-kserve-685985bfb-tkgn2"
Nov 02 19:44:49.841557 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:49.841536    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7415aa62-467c-43af-94c8-d604f57ec1c3-home\") pod \"distinct-w1-26cfbe-kserve-685985bfb-tkgn2\" (UID: \"7415aa62-467c-43af-94c8-d604f57ec1c3\") " pod="e2e-models-e2e-worker-w1-26cfbe/distinct-w1-26cfbe-kserve-685985bfb-tkgn2"
Nov 02 19:44:49.841622 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:49.841561    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7415aa62-467c-43af-94c8-d604f57ec1c3-tmp-dir\") pod \"distinct-w1-26cfbe-kserve-685985bfb-tkgn2\" (UID: \"7415aa62-467c-43af-94c8-d604f57ec1c3\") " pod="e2e-models-e2e-worker-w1-26cfbe/distinct-w1-26cfbe-kserve-685985bfb-tkgn2"
Nov 02 19:44:49.841659 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:49.841617    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7415aa62-467c-43af-94c8-d604f57ec1c3-model-cache\") pod \"distinct-w1-26cfbe-kserve-685985bfb-tkgn2\" (UID: \"7415aa62-467c-43af-94c8-d604f57ec1c3\") " pod="e2e-models-e2e-worker-w1-26cfbe/distinct-w1-26cfbe-kserve-685985bfb-tkgn2"
Nov 02 19:44:49.843448 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:49.843420    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7415aa62-467c-43af-94c8-d604f57ec1c3-dshm\") pod \"distinct-w1-26cfbe-kserve-685985bfb-tkgn2\" (UID: \"7415aa62-467c-43af-94c8-d604f57ec1c3\") " pod="e2e-models-e2e-worker-w1-26cfbe/distinct-w1-26cfbe-kserve-685985bfb-tkgn2"
Nov 02 19:44:49.843733 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:49.843713    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7415aa62-467c-43af-94c8-d604f57ec1c3-tls-certs\") pod \"distinct-w1-26cfbe-kserve-685985bfb-tkgn2\" (UID: \"7415aa62-467c-43af-94c8-d604f57ec1c3\") " pod="e2e-models-e2e-worker-w1-26cfbe/distinct-w1-26cfbe-kserve-685985bfb-tkgn2"
Nov 02 19:44:49.894107 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:49.894048    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9zgq\" (UniqueName: \"kubernetes.io/projected/7415aa62-467c-43af-94c8-d604f57ec1c3-kube-api-access-j9zgq\") pod \"distinct-w1-26cfbe-kserve-685985bfb-tkgn2\" (UID: \"7415aa62-467c-43af-94c8-d604f57ec1c3\") " pod="e2e-models-e2e-worker-w1-26cfbe/distinct-w1-26cfbe-kserve-685985bfb-tkgn2"
Nov 02 19:44:49.916123 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:49.916102    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-26cfbe/distinct-w1-26cfbe-kserve-685985bfb-tkgn2"
Nov 02 19:44:50.287521 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:50.287499    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-26cfbe/distinct-w1-26cfbe-kserve-685985bfb-tkgn2"]
Nov 02 19:44:50.289265 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:44:50.289229    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7415aa62_467c_43af_94c8_d604f57ec1c3.slice/crio-6c1c0ada948d819542f79b806cbec58476f63916e92721246c6a1cdc242cc775 WatchSource:0}: Error finding container 6c1c0ada948d819542f79b806cbec58476f63916e92721246c6a1cdc242cc775: Status 404 returned error can't find the container with id 6c1c0ada948d819542f79b806cbec58476f63916e92721246c6a1cdc242cc775
Nov 02 19:44:51.184902 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:51.184861    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-26cfbe/distinct-w1-26cfbe-kserve-685985bfb-tkgn2" event={"ID":"7415aa62-467c-43af-94c8-d604f57ec1c3","Type":"ContainerStarted","Data":"070cf61c57bcab497189c1d0adb97ea4d06c65d76aef34acbf1bbe8a56fd457f"}
Nov 02 19:44:51.185364 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:51.184909    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-26cfbe/distinct-w1-26cfbe-kserve-685985bfb-tkgn2" event={"ID":"7415aa62-467c-43af-94c8-d604f57ec1c3","Type":"ContainerStarted","Data":"6c1c0ada948d819542f79b806cbec58476f63916e92721246c6a1cdc242cc775"}
Nov 02 19:44:51.294080 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:51.294031    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-26cfbe/distinct-w1-26cfbe-kserve-685985bfb-tkgn2" podStartSLOduration=2.294001428 podStartE2EDuration="2.294001428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:44:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:44:51.293243586 +0000 UTC m=+997.900277373" watchObservedRunningTime="2026-11-02 19:44:51.294001428 +0000 UTC m=+997.901035197"
Nov 02 19:44:51.916355 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:51.916316    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-26cfbe/distinct-w1-26cfbe-kserve-685985bfb-tkgn2"
Nov 02 19:44:54.606269 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:54.606231    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-8795bf/distinct-w4-8795bf-kserve-64b4796df4-8qgqh"
Nov 02 19:44:54.619542 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:54.619512    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-8795bf/distinct-w4-8795bf-kserve-64b4796df4-8qgqh"
Nov 02 19:44:55.220473 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:55.220443    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8795bf/distinct-w4-8795bf-kserve-64b4796df4-8qgqh"
Nov 02 19:44:59.916877 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:59.916846    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-26cfbe/distinct-w1-26cfbe-kserve-685985bfb-tkgn2"
Nov 02 19:44:59.929390 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:44:59.929365    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-26cfbe/distinct-w1-26cfbe-kserve-685985bfb-tkgn2"
Nov 02 19:45:00.247425 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:00.247394    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-26cfbe/distinct-w1-26cfbe-kserve-685985bfb-tkgn2"
Nov 02 19:45:00.292846 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:00.292810    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894145-kl8fq"]
Nov 02 19:45:00.296838 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:00.296820    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894145-kl8fq"
Nov 02 19:45:00.350688 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:00.350652    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894145-kl8fq"]
Nov 02 19:45:00.378826 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:00.378792    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\""
Nov 02 19:45:00.456866 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:00.456824    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca685e0b-4b42-4fc8-8177-bdf4d2f1a2ab-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894145-kl8fq\" (UID: \"ca685e0b-4b42-4fc8-8177-bdf4d2f1a2ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894145-kl8fq"
Nov 02 19:45:00.457072 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:00.456898    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qczzc\" (UniqueName: \"kubernetes.io/projected/ca685e0b-4b42-4fc8-8177-bdf4d2f1a2ab-kube-api-access-qczzc\") pod \"maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894145-kl8fq\" (UID: \"ca685e0b-4b42-4fc8-8177-bdf4d2f1a2ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894145-kl8fq"
Nov 02 19:45:00.557912 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:00.557821    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca685e0b-4b42-4fc8-8177-bdf4d2f1a2ab-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894145-kl8fq\" (UID: \"ca685e0b-4b42-4fc8-8177-bdf4d2f1a2ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894145-kl8fq"
Nov 02 19:45:00.557912 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:00.557883    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qczzc\" (UniqueName: \"kubernetes.io/projected/ca685e0b-4b42-4fc8-8177-bdf4d2f1a2ab-kube-api-access-qczzc\") pod \"maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894145-kl8fq\" (UID: \"ca685e0b-4b42-4fc8-8177-bdf4d2f1a2ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894145-kl8fq"
Nov 02 19:45:00.558526 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:00.558502    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca685e0b-4b42-4fc8-8177-bdf4d2f1a2ab-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894145-kl8fq\" (UID: \"ca685e0b-4b42-4fc8-8177-bdf4d2f1a2ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894145-kl8fq"
Nov 02 19:45:00.610331 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:00.610305    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qczzc\" (UniqueName: \"kubernetes.io/projected/ca685e0b-4b42-4fc8-8177-bdf4d2f1a2ab-kube-api-access-qczzc\") pod \"maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894145-kl8fq\" (UID: \"ca685e0b-4b42-4fc8-8177-bdf4d2f1a2ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894145-kl8fq"
Nov 02 19:45:00.910436 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:00.910355    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894145-kl8fq"
Nov 02 19:45:01.274730 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:01.274707    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894145-kl8fq"]
Nov 02 19:45:01.276914 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:45:01.276884    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca685e0b_4b42_4fc8_8177_bdf4d2f1a2ab.slice/crio-bfd84388f0e5607689423deb95b500b3d227ce3e1fd1f269aae131f0260fe3e1 WatchSource:0}: Error finding container bfd84388f0e5607689423deb95b500b3d227ce3e1fd1f269aae131f0260fe3e1: Status 404 returned error can't find the container with id bfd84388f0e5607689423deb95b500b3d227ce3e1fd1f269aae131f0260fe3e1
Nov 02 19:45:01.278748 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:01.278732    4463 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 19:45:02.248220 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:02.248177    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894145-kl8fq" event={"ID":"ca685e0b-4b42-4fc8-8177-bdf4d2f1a2ab","Type":"ContainerStarted","Data":"bfd84388f0e5607689423deb95b500b3d227ce3e1fd1f269aae131f0260fe3e1"}
Nov 02 19:45:04.298312 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:45:04.298273    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 19:45:06.273962 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:06.273930    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894145-kl8fq" event={"ID":"ca685e0b-4b42-4fc8-8177-bdf4d2f1a2ab","Type":"ContainerStarted","Data":"0a7d25d4ee509a3b38fd77a0fdc0d2f9a5d96fffa86d1f0622890a03da3123b1"}
Nov 02 19:45:06.380074 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:06.380003    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894145-kl8fq" podStartSLOduration=2.135217132 podStartE2EDuration="6.379989541s" totalImagesPullingTime="4.244772409s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:45:06.37825248 +0000 UTC m=+1012.985286249" watchObservedRunningTime="2026-11-02 19:45:06.379989541 +0000 UTC m=+1012.987023307"
Nov 02 19:45:07.281503 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:07.281463    4463 generic.go:356] "Generic (PLEG): container finished" podID="ca685e0b-4b42-4fc8-8177-bdf4d2f1a2ab" containerID="0a7d25d4ee509a3b38fd77a0fdc0d2f9a5d96fffa86d1f0622890a03da3123b1" exitCode=60
Nov 02 19:45:07.281889 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:07.281554    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894145-kl8fq" event={"ID":"ca685e0b-4b42-4fc8-8177-bdf4d2f1a2ab","Type":"ContainerDied","Data":"0a7d25d4ee509a3b38fd77a0fdc0d2f9a5d96fffa86d1f0622890a03da3123b1"}
Nov 02 19:45:07.281889 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:07.281778    4463 scope.go:122] "RemoveContainer" containerID="0a7d25d4ee509a3b38fd77a0fdc0d2f9a5d96fffa86d1f0622890a03da3123b1"
Nov 02 19:45:08.288915 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:08.288875    4463 generic.go:356] "Generic (PLEG): container finished" podID="ca685e0b-4b42-4fc8-8177-bdf4d2f1a2ab" containerID="2e8e95f4be3675bc686ffb24604c9d3779f6b1ba49b7db528a8b93d5aeba6fe8" exitCode=60
Nov 02 19:45:08.289399 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:08.288917    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894145-kl8fq" event={"ID":"ca685e0b-4b42-4fc8-8177-bdf4d2f1a2ab","Type":"ContainerDied","Data":"2e8e95f4be3675bc686ffb24604c9d3779f6b1ba49b7db528a8b93d5aeba6fe8"}
Nov 02 19:45:08.289399 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:08.288991    4463 scope.go:122] "RemoveContainer" containerID="0a7d25d4ee509a3b38fd77a0fdc0d2f9a5d96fffa86d1f0622890a03da3123b1"
Nov 02 19:45:08.289399 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:08.289258    4463 scope.go:122] "RemoveContainer" containerID="2e8e95f4be3675bc686ffb24604c9d3779f6b1ba49b7db528a8b93d5aeba6fe8"
Nov 02 19:45:08.289567 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:45:08.289544    4463 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894145-kl8fq_odh-ai-gateway-infra(ca685e0b-4b42-4fc8-8177-bdf4d2f1a2ab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894145-kl8fq" podUID="ca685e0b-4b42-4fc8-8177-bdf4d2f1a2ab"
Nov 02 19:45:08.518601 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:08.518568    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-5c46af/distinct-w3-5c46af-kserve-68b6dff7f5-bp59f"]
Nov 02 19:45:08.523506 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:08.523487    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5c46af/distinct-w3-5c46af-kserve-68b6dff7f5-bp59f"
Nov 02 19:45:08.539983 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:08.539957    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dd6362e-7dbd-489b-b6d8-0d9caab0fa69-tls-certs\") pod \"distinct-w3-5c46af-kserve-68b6dff7f5-bp59f\" (UID: \"8dd6362e-7dbd-489b-b6d8-0d9caab0fa69\") " pod="e2e-models-e2e-worker-w3-5c46af/distinct-w3-5c46af-kserve-68b6dff7f5-bp59f"
Nov 02 19:45:08.540096 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:08.540028    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8dd6362e-7dbd-489b-b6d8-0d9caab0fa69-tmp-dir\") pod \"distinct-w3-5c46af-kserve-68b6dff7f5-bp59f\" (UID: \"8dd6362e-7dbd-489b-b6d8-0d9caab0fa69\") " pod="e2e-models-e2e-worker-w3-5c46af/distinct-w3-5c46af-kserve-68b6dff7f5-bp59f"
Nov 02 19:45:08.540148 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:08.540128    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7gwp\" (UniqueName: \"kubernetes.io/projected/8dd6362e-7dbd-489b-b6d8-0d9caab0fa69-kube-api-access-k7gwp\") pod \"distinct-w3-5c46af-kserve-68b6dff7f5-bp59f\" (UID: \"8dd6362e-7dbd-489b-b6d8-0d9caab0fa69\") " pod="e2e-models-e2e-worker-w3-5c46af/distinct-w3-5c46af-kserve-68b6dff7f5-bp59f"
Nov 02 19:45:08.540190 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:08.540178    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8dd6362e-7dbd-489b-b6d8-0d9caab0fa69-home\") pod \"distinct-w3-5c46af-kserve-68b6dff7f5-bp59f\" (UID: \"8dd6362e-7dbd-489b-b6d8-0d9caab0fa69\") " pod="e2e-models-e2e-worker-w3-5c46af/distinct-w3-5c46af-kserve-68b6dff7f5-bp59f"
Nov 02 19:45:08.540353 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:08.540337    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8dd6362e-7dbd-489b-b6d8-0d9caab0fa69-dshm\") pod \"distinct-w3-5c46af-kserve-68b6dff7f5-bp59f\" (UID: \"8dd6362e-7dbd-489b-b6d8-0d9caab0fa69\") " pod="e2e-models-e2e-worker-w3-5c46af/distinct-w3-5c46af-kserve-68b6dff7f5-bp59f"
Nov 02 19:45:08.540400 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:08.540387    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dd6362e-7dbd-489b-b6d8-0d9caab0fa69-model-cache\") pod \"distinct-w3-5c46af-kserve-68b6dff7f5-bp59f\" (UID: \"8dd6362e-7dbd-489b-b6d8-0d9caab0fa69\") " pod="e2e-models-e2e-worker-w3-5c46af/distinct-w3-5c46af-kserve-68b6dff7f5-bp59f"
Nov 02 19:45:08.573896 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:08.573869    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-5c46af\"/\"kube-root-ca.crt\""
Nov 02 19:45:08.574503 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:08.573914    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-5c46af\"/\"openshift-service-ca.crt\""
Nov 02 19:45:08.574503 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:08.574048    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-5c46af\"/\"distinct-w3-5c46af-kserve-self-signed-certs\""
Nov 02 19:45:08.574683 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:08.574669    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-5c46af\"/\"default-dockercfg-f89sw\""
Nov 02 19:45:08.578388 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:08.578361    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-5c46af/distinct-w3-5c46af-kserve-68b6dff7f5-bp59f"]
Nov 02 19:45:08.641752 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:08.641716    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8dd6362e-7dbd-489b-b6d8-0d9caab0fa69-dshm\") pod \"distinct-w3-5c46af-kserve-68b6dff7f5-bp59f\" (UID: \"8dd6362e-7dbd-489b-b6d8-0d9caab0fa69\") " pod="e2e-models-e2e-worker-w3-5c46af/distinct-w3-5c46af-kserve-68b6dff7f5-bp59f"
Nov 02 19:45:08.641927 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:08.641771    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dd6362e-7dbd-489b-b6d8-0d9caab0fa69-model-cache\") pod \"distinct-w3-5c46af-kserve-68b6dff7f5-bp59f\" (UID: \"8dd6362e-7dbd-489b-b6d8-0d9caab0fa69\") " pod="e2e-models-e2e-worker-w3-5c46af/distinct-w3-5c46af-kserve-68b6dff7f5-bp59f"
Nov 02 19:45:08.641927 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:08.641811    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dd6362e-7dbd-489b-b6d8-0d9caab0fa69-tls-certs\") pod \"distinct-w3-5c46af-kserve-68b6dff7f5-bp59f\" (UID: \"8dd6362e-7dbd-489b-b6d8-0d9caab0fa69\") " pod="e2e-models-e2e-worker-w3-5c46af/distinct-w3-5c46af-kserve-68b6dff7f5-bp59f"
Nov 02 19:45:08.642084 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:08.641960    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8dd6362e-7dbd-489b-b6d8-0d9caab0fa69-tmp-dir\") pod \"distinct-w3-5c46af-kserve-68b6dff7f5-bp59f\" (UID: \"8dd6362e-7dbd-489b-b6d8-0d9caab0fa69\") " pod="e2e-models-e2e-worker-w3-5c46af/distinct-w3-5c46af-kserve-68b6dff7f5-bp59f"
Nov 02 19:45:08.642153 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:08.642085    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7gwp\" (UniqueName: \"kubernetes.io/projected/8dd6362e-7dbd-489b-b6d8-0d9caab0fa69-kube-api-access-k7gwp\") pod \"distinct-w3-5c46af-kserve-68b6dff7f5-bp59f\" (UID: \"8dd6362e-7dbd-489b-b6d8-0d9caab0fa69\") " pod="e2e-models-e2e-worker-w3-5c46af/distinct-w3-5c46af-kserve-68b6dff7f5-bp59f"
Nov 02 19:45:08.642153 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:08.642143    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8dd6362e-7dbd-489b-b6d8-0d9caab0fa69-home\") pod \"distinct-w3-5c46af-kserve-68b6dff7f5-bp59f\" (UID: \"8dd6362e-7dbd-489b-b6d8-0d9caab0fa69\") " pod="e2e-models-e2e-worker-w3-5c46af/distinct-w3-5c46af-kserve-68b6dff7f5-bp59f"
Nov 02 19:45:08.642256 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:08.642240    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dd6362e-7dbd-489b-b6d8-0d9caab0fa69-model-cache\") pod \"distinct-w3-5c46af-kserve-68b6dff7f5-bp59f\" (UID: \"8dd6362e-7dbd-489b-b6d8-0d9caab0fa69\") " pod="e2e-models-e2e-worker-w3-5c46af/distinct-w3-5c46af-kserve-68b6dff7f5-bp59f"
Nov 02 19:45:08.642312 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:08.642299    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8dd6362e-7dbd-489b-b6d8-0d9caab0fa69-tmp-dir\") pod \"distinct-w3-5c46af-kserve-68b6dff7f5-bp59f\" (UID: \"8dd6362e-7dbd-489b-b6d8-0d9caab0fa69\") " pod="e2e-models-e2e-worker-w3-5c46af/distinct-w3-5c46af-kserve-68b6dff7f5-bp59f"
Nov 02 19:45:08.642471 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:08.642450    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8dd6362e-7dbd-489b-b6d8-0d9caab0fa69-home\") pod \"distinct-w3-5c46af-kserve-68b6dff7f5-bp59f\" (UID: \"8dd6362e-7dbd-489b-b6d8-0d9caab0fa69\") " pod="e2e-models-e2e-worker-w3-5c46af/distinct-w3-5c46af-kserve-68b6dff7f5-bp59f"
Nov 02 19:45:08.644080 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:08.644058    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8dd6362e-7dbd-489b-b6d8-0d9caab0fa69-dshm\") pod \"distinct-w3-5c46af-kserve-68b6dff7f5-bp59f\" (UID: \"8dd6362e-7dbd-489b-b6d8-0d9caab0fa69\") " pod="e2e-models-e2e-worker-w3-5c46af/distinct-w3-5c46af-kserve-68b6dff7f5-bp59f"
Nov 02 19:45:08.644349 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:08.644332    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dd6362e-7dbd-489b-b6d8-0d9caab0fa69-tls-certs\") pod \"distinct-w3-5c46af-kserve-68b6dff7f5-bp59f\" (UID: \"8dd6362e-7dbd-489b-b6d8-0d9caab0fa69\") " pod="e2e-models-e2e-worker-w3-5c46af/distinct-w3-5c46af-kserve-68b6dff7f5-bp59f"
Nov 02 19:45:08.696349 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:08.696314    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7gwp\" (UniqueName: \"kubernetes.io/projected/8dd6362e-7dbd-489b-b6d8-0d9caab0fa69-kube-api-access-k7gwp\") pod \"distinct-w3-5c46af-kserve-68b6dff7f5-bp59f\" (UID: \"8dd6362e-7dbd-489b-b6d8-0d9caab0fa69\") " pod="e2e-models-e2e-worker-w3-5c46af/distinct-w3-5c46af-kserve-68b6dff7f5-bp59f"
Nov 02 19:45:08.835563 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:08.835470    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5c46af/distinct-w3-5c46af-kserve-68b6dff7f5-bp59f"
Nov 02 19:45:09.205343 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:09.205314    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-5c46af/distinct-w3-5c46af-kserve-68b6dff7f5-bp59f"]
Nov 02 19:45:09.206405 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:45:09.206378    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8dd6362e_7dbd_489b_b6d8_0d9caab0fa69.slice/crio-360f3139739280293c1813c029b8b3876efe9f5bf38f82fcfd3ccbaa2d4fc518 WatchSource:0}: Error finding container 360f3139739280293c1813c029b8b3876efe9f5bf38f82fcfd3ccbaa2d4fc518: Status 404 returned error can't find the container with id 360f3139739280293c1813c029b8b3876efe9f5bf38f82fcfd3ccbaa2d4fc518
Nov 02 19:45:09.296938 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:09.296898    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5c46af/distinct-w3-5c46af-kserve-68b6dff7f5-bp59f" event={"ID":"8dd6362e-7dbd-489b-b6d8-0d9caab0fa69","Type":"ContainerStarted","Data":"84eaf8eee9246dfdc63b511f35cf6cf110b28c0ed17a025bb941243d3ab38b9b"}
Nov 02 19:45:09.297417 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:09.296950    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5c46af/distinct-w3-5c46af-kserve-68b6dff7f5-bp59f" event={"ID":"8dd6362e-7dbd-489b-b6d8-0d9caab0fa69","Type":"ContainerStarted","Data":"360f3139739280293c1813c029b8b3876efe9f5bf38f82fcfd3ccbaa2d4fc518"}
Nov 02 19:45:09.298750 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:09.298732    4463 scope.go:122] "RemoveContainer" containerID="2e8e95f4be3675bc686ffb24604c9d3779f6b1ba49b7db528a8b93d5aeba6fe8"
Nov 02 19:45:09.298970 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:45:09.298952    4463 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894145-kl8fq_odh-ai-gateway-infra(ca685e0b-4b42-4fc8-8177-bdf4d2f1a2ab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894145-kl8fq" podUID="ca685e0b-4b42-4fc8-8177-bdf4d2f1a2ab"
Nov 02 19:45:09.405779 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:09.405724    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-5c46af/distinct-w3-5c46af-kserve-68b6dff7f5-bp59f" podStartSLOduration=1.405707904 podStartE2EDuration="1.405707904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:45:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:45:09.4030054 +0000 UTC m=+1016.010039178" watchObservedRunningTime="2026-11-02 19:45:09.405707904 +0000 UTC m=+1016.012741674"
Nov 02 19:45:09.836379 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:09.836324    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5c46af/distinct-w3-5c46af-kserve-68b6dff7f5-bp59f"
Nov 02 19:45:18.442657 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:18.442623    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 19:45:18.596382 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:18.596349    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-k7k7k"]
Nov 02 19:45:18.602055 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:18.602033    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-k7k7k"
Nov 02 19:45:18.657669 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:18.657646    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-k7k7k"]
Nov 02 19:45:18.750090 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:18.749996    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb05ff45-67c9-44fb-995d-5f9e0b4059e9-workload-socket\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-k7k7k\" (UID: \"bb05ff45-67c9-44fb-995d-5f9e0b4059e9\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-k7k7k"
Nov 02 19:45:18.750090 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:18.750063    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb05ff45-67c9-44fb-995d-5f9e0b4059e9-istio-token\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-k7k7k\" (UID: \"bb05ff45-67c9-44fb-995d-5f9e0b4059e9\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-k7k7k"
Nov 02 19:45:18.750311 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:18.750170    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb05ff45-67c9-44fb-995d-5f9e0b4059e9-istiod-ca-cert\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-k7k7k\" (UID: \"bb05ff45-67c9-44fb-995d-5f9e0b4059e9\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-k7k7k"
Nov 02 19:45:18.750311 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:18.750239    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb05ff45-67c9-44fb-995d-5f9e0b4059e9-istio-podinfo\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-k7k7k\" (UID: \"bb05ff45-67c9-44fb-995d-5f9e0b4059e9\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-k7k7k"
Nov 02 19:45:18.750311 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:18.750272    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5vnk\" (UniqueName: \"kubernetes.io/projected/bb05ff45-67c9-44fb-995d-5f9e0b4059e9-kube-api-access-k5vnk\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-k7k7k\" (UID: \"bb05ff45-67c9-44fb-995d-5f9e0b4059e9\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-k7k7k"
Nov 02 19:45:18.750411 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:18.750327    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb05ff45-67c9-44fb-995d-5f9e0b4059e9-credential-socket\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-k7k7k\" (UID: \"bb05ff45-67c9-44fb-995d-5f9e0b4059e9\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-k7k7k"
Nov 02 19:45:18.750411 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:18.750356    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb05ff45-67c9-44fb-995d-5f9e0b4059e9-workload-certs\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-k7k7k\" (UID: \"bb05ff45-67c9-44fb-995d-5f9e0b4059e9\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-k7k7k"
Nov 02 19:45:18.750411 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:18.750375    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb05ff45-67c9-44fb-995d-5f9e0b4059e9-istio-data\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-k7k7k\" (UID: \"bb05ff45-67c9-44fb-995d-5f9e0b4059e9\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-k7k7k"
Nov 02 19:45:18.750506 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:18.750481    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb05ff45-67c9-44fb-995d-5f9e0b4059e9-istio-envoy\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-k7k7k\" (UID: \"bb05ff45-67c9-44fb-995d-5f9e0b4059e9\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-k7k7k"
Nov 02 19:45:18.836314 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:18.836288    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-5c46af/distinct-w3-5c46af-kserve-68b6dff7f5-bp59f"
Nov 02 19:45:18.848723 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:18.848703    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-5c46af/distinct-w3-5c46af-kserve-68b6dff7f5-bp59f"
Nov 02 19:45:18.851235 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:18.851206    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb05ff45-67c9-44fb-995d-5f9e0b4059e9-istio-envoy\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-k7k7k\" (UID: \"bb05ff45-67c9-44fb-995d-5f9e0b4059e9\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-k7k7k"
Nov 02 19:45:18.851316 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:18.851291    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb05ff45-67c9-44fb-995d-5f9e0b4059e9-workload-socket\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-k7k7k\" (UID: \"bb05ff45-67c9-44fb-995d-5f9e0b4059e9\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-k7k7k"
Nov 02 19:45:18.851353 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:18.851328    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb05ff45-67c9-44fb-995d-5f9e0b4059e9-istio-token\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-k7k7k\" (UID: \"bb05ff45-67c9-44fb-995d-5f9e0b4059e9\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-k7k7k"
Nov 02 19:45:18.851405 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:18.851348    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb05ff45-67c9-44fb-995d-5f9e0b4059e9-istiod-ca-cert\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-k7k7k\" (UID: \"bb05ff45-67c9-44fb-995d-5f9e0b4059e9\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-k7k7k"
Nov 02 19:45:18.851457 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:18.851401    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb05ff45-67c9-44fb-995d-5f9e0b4059e9-istio-podinfo\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-k7k7k\" (UID: \"bb05ff45-67c9-44fb-995d-5f9e0b4059e9\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-k7k7k"
Nov 02 19:45:18.851457 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:18.851436    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5vnk\" (UniqueName: \"kubernetes.io/projected/bb05ff45-67c9-44fb-995d-5f9e0b4059e9-kube-api-access-k5vnk\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-k7k7k\" (UID: \"bb05ff45-67c9-44fb-995d-5f9e0b4059e9\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-k7k7k"
Nov 02 19:45:18.851562 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:18.851475    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb05ff45-67c9-44fb-995d-5f9e0b4059e9-credential-socket\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-k7k7k\" (UID: \"bb05ff45-67c9-44fb-995d-5f9e0b4059e9\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-k7k7k"
Nov 02 19:45:18.851615 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:18.851516    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb05ff45-67c9-44fb-995d-5f9e0b4059e9-workload-certs\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-k7k7k\" (UID: \"bb05ff45-67c9-44fb-995d-5f9e0b4059e9\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-k7k7k"
Nov 02 19:45:18.851671 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:18.851637    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb05ff45-67c9-44fb-995d-5f9e0b4059e9-istio-data\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-k7k7k\" (UID: \"bb05ff45-67c9-44fb-995d-5f9e0b4059e9\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-k7k7k"
Nov 02 19:45:18.851821 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:18.851794    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb05ff45-67c9-44fb-995d-5f9e0b4059e9-credential-socket\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-k7k7k\" (UID: \"bb05ff45-67c9-44fb-995d-5f9e0b4059e9\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-k7k7k"
Nov 02 19:45:18.851989 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:18.851964    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb05ff45-67c9-44fb-995d-5f9e0b4059e9-workload-socket\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-k7k7k\" (UID: \"bb05ff45-67c9-44fb-995d-5f9e0b4059e9\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-k7k7k"
Nov 02 19:45:18.851989 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:18.851986    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb05ff45-67c9-44fb-995d-5f9e0b4059e9-workload-certs\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-k7k7k\" (UID: \"bb05ff45-67c9-44fb-995d-5f9e0b4059e9\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-k7k7k"
Nov 02 19:45:18.852169 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:18.852074    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb05ff45-67c9-44fb-995d-5f9e0b4059e9-istiod-ca-cert\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-k7k7k\" (UID: \"bb05ff45-67c9-44fb-995d-5f9e0b4059e9\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-k7k7k"
Nov 02 19:45:18.852169 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:18.852129    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb05ff45-67c9-44fb-995d-5f9e0b4059e9-istio-data\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-k7k7k\" (UID: \"bb05ff45-67c9-44fb-995d-5f9e0b4059e9\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-k7k7k"
Nov 02 19:45:18.853440 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:18.853416    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb05ff45-67c9-44fb-995d-5f9e0b4059e9-istio-envoy\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-k7k7k\" (UID: \"bb05ff45-67c9-44fb-995d-5f9e0b4059e9\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-k7k7k"
Nov 02 19:45:18.853740 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:18.853720    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb05ff45-67c9-44fb-995d-5f9e0b4059e9-istio-podinfo\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-k7k7k\" (UID: \"bb05ff45-67c9-44fb-995d-5f9e0b4059e9\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-k7k7k"
Nov 02 19:45:18.904950 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:18.904925    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb05ff45-67c9-44fb-995d-5f9e0b4059e9-istio-token\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-k7k7k\" (UID: \"bb05ff45-67c9-44fb-995d-5f9e0b4059e9\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-k7k7k"
Nov 02 19:45:18.905255 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:18.905231    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5vnk\" (UniqueName: \"kubernetes.io/projected/bb05ff45-67c9-44fb-995d-5f9e0b4059e9-kube-api-access-k5vnk\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-k7k7k\" (UID: \"bb05ff45-67c9-44fb-995d-5f9e0b4059e9\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-k7k7k"
Nov 02 19:45:18.929036 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:18.925248    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-k7k7k"
Nov 02 19:45:19.397055 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:19.396987    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:45:19.397183 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:19.397160    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:45:19.397395 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:19.397323    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:45:19.427674 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:19.427642    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-k7k7k"]
Nov 02 19:45:19.439650 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:19.439625    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5c46af/distinct-w3-5c46af-kserve-68b6dff7f5-bp59f"
Nov 02 19:45:20.429202 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:20.429160    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-k7k7k" event={"ID":"bb05ff45-67c9-44fb-995d-5f9e0b4059e9","Type":"ContainerStarted","Data":"9e77367018fb447387d1a4222f5204d4bc156185e6f0ffec132c0a35da61c123"}
Nov 02 19:45:20.429601 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:20.429218    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-k7k7k" event={"ID":"bb05ff45-67c9-44fb-995d-5f9e0b4059e9","Type":"ContainerStarted","Data":"1325e35f2e4c41fa53b13b2736f9f549ea6534a4775f20426df4c33de68139f1"}
Nov 02 19:45:20.587329 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:20.587282    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-k7k7k" podStartSLOduration=2.5872689639999997 podStartE2EDuration="2.587268964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:45:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:45:20.583707437 +0000 UTC m=+1027.190741386" watchObservedRunningTime="2026-11-02 19:45:20.587268964 +0000 UTC m=+1027.194302733"
Nov 02 19:45:20.930066 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:20.930040    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-k7k7k"
Nov 02 19:45:20.931448 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:20.931422    4463 patch_prober.go:29] interesting pod/e2e-worker-w4-8795bf-openshift-default-c67bc566c-k7k7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 02 19:45:20.931598 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:20.931464    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-k7k7k" podUID="bb05ff45-67c9-44fb-995d-5f9e0b4059e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 02 19:45:21.930108 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:21.930072    4463 patch_prober.go:29] interesting pod/e2e-worker-w4-8795bf-openshift-default-c67bc566c-k7k7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 02 19:45:21.930505 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:21.930124    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-k7k7k" podUID="bb05ff45-67c9-44fb-995d-5f9e0b4059e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 02 19:45:22.577504 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:22.577471    4463 scope.go:122] "RemoveContainer" containerID="2e8e95f4be3675bc686ffb24604c9d3779f6b1ba49b7db528a8b93d5aeba6fe8"
Nov 02 19:45:22.930070 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:22.930039    4463 patch_prober.go:29] interesting pod/e2e-worker-w4-8795bf-openshift-default-c67bc566c-k7k7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 02 19:45:22.930218 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:22.930104    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-k7k7k" podUID="bb05ff45-67c9-44fb-995d-5f9e0b4059e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 02 19:45:23.448005 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:23.447971    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894145-kl8fq" event={"ID":"ca685e0b-4b42-4fc8-8177-bdf4d2f1a2ab","Type":"ContainerStarted","Data":"00bd444e4b071507e226c55b164425a02aafaf9e68594bf1de36e2f1ab78039c"}
Nov 02 19:45:23.930159 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:23.930132    4463 patch_prober.go:29] interesting pod/e2e-worker-w4-8795bf-openshift-default-c67bc566c-k7k7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 02 19:45:23.930322 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:23.930207    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-k7k7k" podUID="bb05ff45-67c9-44fb-995d-5f9e0b4059e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 02 19:45:24.499161 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:24.499120    4463 generic.go:356] "Generic (PLEG): container finished" podID="ca685e0b-4b42-4fc8-8177-bdf4d2f1a2ab" containerID="00bd444e4b071507e226c55b164425a02aafaf9e68594bf1de36e2f1ab78039c" exitCode=60
Nov 02 19:45:24.499427 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:24.499346    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894145-kl8fq" event={"ID":"ca685e0b-4b42-4fc8-8177-bdf4d2f1a2ab","Type":"ContainerDied","Data":"00bd444e4b071507e226c55b164425a02aafaf9e68594bf1de36e2f1ab78039c"}
Nov 02 19:45:24.499490 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:24.499438    4463 scope.go:122] "RemoveContainer" containerID="2e8e95f4be3675bc686ffb24604c9d3779f6b1ba49b7db528a8b93d5aeba6fe8"
Nov 02 19:45:24.499820 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:24.499796    4463 scope.go:122] "RemoveContainer" containerID="00bd444e4b071507e226c55b164425a02aafaf9e68594bf1de36e2f1ab78039c"
Nov 02 19:45:24.500150 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:45:24.500126    4463 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894145-kl8fq_odh-ai-gateway-infra(ca685e0b-4b42-4fc8-8177-bdf4d2f1a2ab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894145-kl8fq" podUID="ca685e0b-4b42-4fc8-8177-bdf4d2f1a2ab"
Nov 02 19:45:24.575170 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:24.575082    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894145-kl8fq"]
Nov 02 19:45:24.930874 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:24.930839    4463 patch_prober.go:29] interesting pod/e2e-worker-w4-8795bf-openshift-default-c67bc566c-k7k7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 02 19:45:24.931272 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:24.930889    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-k7k7k" podUID="bb05ff45-67c9-44fb-995d-5f9e0b4059e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 02 19:45:25.690912 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:25.690882    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894145-kl8fq"
Nov 02 19:45:25.793810 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:25.793730    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca685e0b-4b42-4fc8-8177-bdf4d2f1a2ab-kube-api-access-qczzc\" (UniqueName: \"kubernetes.io/projected/ca685e0b-4b42-4fc8-8177-bdf4d2f1a2ab-kube-api-access-qczzc\") pod \"ca685e0b-4b42-4fc8-8177-bdf4d2f1a2ab\" (UID: \"ca685e0b-4b42-4fc8-8177-bdf4d2f1a2ab\") "
Nov 02 19:45:25.793978 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:25.793900    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca685e0b-4b42-4fc8-8177-bdf4d2f1a2ab-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca685e0b-4b42-4fc8-8177-bdf4d2f1a2ab-maas-api-service-ca\") pod \"ca685e0b-4b42-4fc8-8177-bdf4d2f1a2ab\" (UID: \"ca685e0b-4b42-4fc8-8177-bdf4d2f1a2ab\") "
Nov 02 19:45:25.794310 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:25.794285    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca685e0b-4b42-4fc8-8177-bdf4d2f1a2ab-maas-api-service-ca" pod "ca685e0b-4b42-4fc8-8177-bdf4d2f1a2ab" (UID: "ca685e0b-4b42-4fc8-8177-bdf4d2f1a2ab"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 19:45:25.795834 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:25.795814    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca685e0b-4b42-4fc8-8177-bdf4d2f1a2ab-kube-api-access-qczzc" pod "ca685e0b-4b42-4fc8-8177-bdf4d2f1a2ab" (UID: "ca685e0b-4b42-4fc8-8177-bdf4d2f1a2ab"). InnerVolumeSpecName "kube-api-access-qczzc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:45:25.895519 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:25.895430    4463 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca685e0b-4b42-4fc8-8177-bdf4d2f1a2ab-maas-api-service-ca\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:45:25.895519 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:25.895463    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qczzc\" (UniqueName: \"kubernetes.io/projected/ca685e0b-4b42-4fc8-8177-bdf4d2f1a2ab-kube-api-access-qczzc\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:45:25.930832 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:25.930797    4463 patch_prober.go:29] interesting pod/e2e-worker-w4-8795bf-openshift-default-c67bc566c-k7k7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 02 19:45:25.930956 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:25.930843    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-k7k7k" podUID="bb05ff45-67c9-44fb-995d-5f9e0b4059e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 02 19:45:26.516212 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:26.516175    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894145-kl8fq"
Nov 02 19:45:26.516410 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:26.516219    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894145-kl8fq" event={"ID":"ca685e0b-4b42-4fc8-8177-bdf4d2f1a2ab","Type":"ContainerDied","Data":"bfd84388f0e5607689423deb95b500b3d227ce3e1fd1f269aae131f0260fe3e1"}
Nov 02 19:45:26.516410 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:26.516276    4463 scope.go:122] "RemoveContainer" containerID="00bd444e4b071507e226c55b164425a02aafaf9e68594bf1de36e2f1ab78039c"
Nov 02 19:45:26.669704 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:26.669673    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894145-kl8fq"]
Nov 02 19:45:26.673024 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:26.672988    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894145-kl8fq"]
Nov 02 19:45:26.934414 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:26.934389    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-k7k7k"
Nov 02 19:45:26.934804 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:26.934766    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-k7k7k"
Nov 02 19:45:26.935430 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:26.935409    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-k7k7k"
Nov 02 19:45:27.585008 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:27.584976    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca685e0b-4b42-4fc8-8177-bdf4d2f1a2ab" path="/var/lib/kubelet/pods/ca685e0b-4b42-4fc8-8177-bdf4d2f1a2ab/volumes"
Nov 02 19:45:34.293577 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:45:34.293541    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 19:45:55.172637 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:45:55.172603    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 19:46:04.295531 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:46:04.295493    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 19:46:18.370766 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:18.370730    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-f4vxc-qgtpp"]
Nov 02 19:46:18.371226 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:18.371212    4463 cpu_manager.go:414] "containerMap: removing container" podUID="ca685e0b-4b42-4fc8-8177-bdf4d2f1a2ab" containerName="cleanup"
Nov 02 19:46:18.371268 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:18.371226    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca685e0b-4b42-4fc8-8177-bdf4d2f1a2ab" containerName="cleanup"
Nov 02 19:46:18.371268 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:18.371264    4463 cpu_manager.go:414] "containerMap: removing container" podUID="ca685e0b-4b42-4fc8-8177-bdf4d2f1a2ab" containerName="cleanup"
Nov 02 19:46:18.371329 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:18.371269    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca685e0b-4b42-4fc8-8177-bdf4d2f1a2ab" containerName="cleanup"
Nov 02 19:46:18.371329 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:18.371278    4463 cpu_manager.go:414] "containerMap: removing container" podUID="ca685e0b-4b42-4fc8-8177-bdf4d2f1a2ab" containerName="cleanup"
Nov 02 19:46:18.371329 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:18.371283    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca685e0b-4b42-4fc8-8177-bdf4d2f1a2ab" containerName="cleanup"
Nov 02 19:46:18.371421 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:18.371353    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca685e0b-4b42-4fc8-8177-bdf4d2f1a2ab" containerName="cleanup"
Nov 02 19:46:18.371421 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:18.371365    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca685e0b-4b42-4fc8-8177-bdf4d2f1a2ab" containerName="cleanup"
Nov 02 19:46:18.375824 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:18.375803    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-f4vxc-qgtpp"
Nov 02 19:46:18.426634 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:18.426603    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-f4vxc-qgtpp"]
Nov 02 19:46:18.442037 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:18.441987    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49094ceb-3222-4695-8186-7b87940fd7be-service-ca-bundle\") pod \"periodic-gathering-f4vxc-qgtpp\" (UID: \"49094ceb-3222-4695-8186-7b87940fd7be\") " pod="openshift-insights/periodic-gathering-f4vxc-qgtpp"
Nov 02 19:46:18.442152 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:18.442127    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49094ceb-3222-4695-8186-7b87940fd7be-trusted-ca-bundle\") pod \"periodic-gathering-f4vxc-qgtpp\" (UID: \"49094ceb-3222-4695-8186-7b87940fd7be\") " pod="openshift-insights/periodic-gathering-f4vxc-qgtpp"
Nov 02 19:46:18.442279 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:18.442262    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qr227\" (UniqueName: \"kubernetes.io/projected/49094ceb-3222-4695-8186-7b87940fd7be-kube-api-access-qr227\") pod \"periodic-gathering-f4vxc-qgtpp\" (UID: \"49094ceb-3222-4695-8186-7b87940fd7be\") " pod="openshift-insights/periodic-gathering-f4vxc-qgtpp"
Nov 02 19:46:18.442342 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:18.442299    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/49094ceb-3222-4695-8186-7b87940fd7be-archives-path\") pod \"periodic-gathering-f4vxc-qgtpp\" (UID: \"49094ceb-3222-4695-8186-7b87940fd7be\") " pod="openshift-insights/periodic-gathering-f4vxc-qgtpp"
Nov 02 19:46:18.469832 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:18.469804    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-rnxqc\""
Nov 02 19:46:18.470125 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:18.470099    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 02 19:46:18.543267 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:18.543242    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49094ceb-3222-4695-8186-7b87940fd7be-trusted-ca-bundle\") pod \"periodic-gathering-f4vxc-qgtpp\" (UID: \"49094ceb-3222-4695-8186-7b87940fd7be\") " pod="openshift-insights/periodic-gathering-f4vxc-qgtpp"
Nov 02 19:46:18.543387 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:18.543318    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qr227\" (UniqueName: \"kubernetes.io/projected/49094ceb-3222-4695-8186-7b87940fd7be-kube-api-access-qr227\") pod \"periodic-gathering-f4vxc-qgtpp\" (UID: \"49094ceb-3222-4695-8186-7b87940fd7be\") " pod="openshift-insights/periodic-gathering-f4vxc-qgtpp"
Nov 02 19:46:18.543387 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:18.543345    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/49094ceb-3222-4695-8186-7b87940fd7be-archives-path\") pod \"periodic-gathering-f4vxc-qgtpp\" (UID: \"49094ceb-3222-4695-8186-7b87940fd7be\") " pod="openshift-insights/periodic-gathering-f4vxc-qgtpp"
Nov 02 19:46:18.543459 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:18.543405    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49094ceb-3222-4695-8186-7b87940fd7be-service-ca-bundle\") pod \"periodic-gathering-f4vxc-qgtpp\" (UID: \"49094ceb-3222-4695-8186-7b87940fd7be\") " pod="openshift-insights/periodic-gathering-f4vxc-qgtpp"
Nov 02 19:46:18.543723 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:18.543708    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/49094ceb-3222-4695-8186-7b87940fd7be-archives-path\") pod \"periodic-gathering-f4vxc-qgtpp\" (UID: \"49094ceb-3222-4695-8186-7b87940fd7be\") " pod="openshift-insights/periodic-gathering-f4vxc-qgtpp"
Nov 02 19:46:18.543981 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:18.543966    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49094ceb-3222-4695-8186-7b87940fd7be-service-ca-bundle\") pod \"periodic-gathering-f4vxc-qgtpp\" (UID: \"49094ceb-3222-4695-8186-7b87940fd7be\") " pod="openshift-insights/periodic-gathering-f4vxc-qgtpp"
Nov 02 19:46:18.563769 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:18.563749    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 02 19:46:18.574602 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:18.574580    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49094ceb-3222-4695-8186-7b87940fd7be-trusted-ca-bundle\") pod \"periodic-gathering-f4vxc-qgtpp\" (UID: \"49094ceb-3222-4695-8186-7b87940fd7be\") " pod="openshift-insights/periodic-gathering-f4vxc-qgtpp"
Nov 02 19:46:18.611131 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:18.611106    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qr227\" (UniqueName: \"kubernetes.io/projected/49094ceb-3222-4695-8186-7b87940fd7be-kube-api-access-qr227\") pod \"periodic-gathering-f4vxc-qgtpp\" (UID: \"49094ceb-3222-4695-8186-7b87940fd7be\") " pod="openshift-insights/periodic-gathering-f4vxc-qgtpp"
Nov 02 19:46:18.687710 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:18.687689    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-f4vxc-qgtpp"
Nov 02 19:46:19.054795 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:19.054774    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-f4vxc-qgtpp"]
Nov 02 19:46:19.806848 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:19.806811    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-f4vxc-qgtpp" event={"ID":"49094ceb-3222-4695-8186-7b87940fd7be","Type":"ContainerStarted","Data":"71153b7238a181475e3d3d47352e89e363755ce0c073511da98eda6ba084e78b"}
Nov 02 19:46:19.807259 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:19.806870    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-f4vxc-qgtpp" event={"ID":"49094ceb-3222-4695-8186-7b87940fd7be","Type":"ContainerStarted","Data":"f2abc288f7a3d21778c5dc66db0e474eb55cb08a3bbe3697a9d9cf6106dd0efd"}
Nov 02 19:46:19.828437 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:19.828409    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-q9v28"]
Nov 02 19:46:19.828997 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:19.828982    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca685e0b-4b42-4fc8-8177-bdf4d2f1a2ab" containerName="cleanup"
Nov 02 19:46:19.832267 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:19.832252    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-q9v28"
Nov 02 19:46:19.924545 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:19.923985    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-f4vxc-qgtpp" podStartSLOduration=1.9239679440000002 podStartE2EDuration="1.923967944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:46:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:46:19.914718739 +0000 UTC m=+1086.521752506" watchObservedRunningTime="2026-11-02 19:46:19.923967944 +0000 UTC m=+1086.531001714"
Nov 02 19:46:19.935151 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:19.935120    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-q9v28"]
Nov 02 19:46:19.959004 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:19.958976    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9bfbdea-9214-414e-84af-e5c9ea6406ff-istio-envoy\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-q9v28\" (UID: \"b9bfbdea-9214-414e-84af-e5c9ea6406ff\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-q9v28"
Nov 02 19:46:19.959147 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:19.959062    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9bfbdea-9214-414e-84af-e5c9ea6406ff-istiod-ca-cert\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-q9v28\" (UID: \"b9bfbdea-9214-414e-84af-e5c9ea6406ff\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-q9v28"
Nov 02 19:46:19.959147 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:19.959115    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9bfbdea-9214-414e-84af-e5c9ea6406ff-istio-data\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-q9v28\" (UID: \"b9bfbdea-9214-414e-84af-e5c9ea6406ff\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-q9v28"
Nov 02 19:46:19.959247 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:19.959146    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9bfbdea-9214-414e-84af-e5c9ea6406ff-istio-token\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-q9v28\" (UID: \"b9bfbdea-9214-414e-84af-e5c9ea6406ff\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-q9v28"
Nov 02 19:46:19.959247 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:19.959176    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9bfbdea-9214-414e-84af-e5c9ea6406ff-credential-socket\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-q9v28\" (UID: \"b9bfbdea-9214-414e-84af-e5c9ea6406ff\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-q9v28"
Nov 02 19:46:19.959514 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:19.959350    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4xhh\" (UniqueName: \"kubernetes.io/projected/b9bfbdea-9214-414e-84af-e5c9ea6406ff-kube-api-access-t4xhh\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-q9v28\" (UID: \"b9bfbdea-9214-414e-84af-e5c9ea6406ff\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-q9v28"
Nov 02 19:46:19.959625 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:19.959525    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9bfbdea-9214-414e-84af-e5c9ea6406ff-istio-podinfo\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-q9v28\" (UID: \"b9bfbdea-9214-414e-84af-e5c9ea6406ff\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-q9v28"
Nov 02 19:46:19.959625 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:19.959570    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9bfbdea-9214-414e-84af-e5c9ea6406ff-workload-certs\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-q9v28\" (UID: \"b9bfbdea-9214-414e-84af-e5c9ea6406ff\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-q9v28"
Nov 02 19:46:19.959746 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:19.959684    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9bfbdea-9214-414e-84af-e5c9ea6406ff-workload-socket\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-q9v28\" (UID: \"b9bfbdea-9214-414e-84af-e5c9ea6406ff\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-q9v28"
Nov 02 19:46:20.061256 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:20.061190    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9bfbdea-9214-414e-84af-e5c9ea6406ff-workload-socket\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-q9v28\" (UID: \"b9bfbdea-9214-414e-84af-e5c9ea6406ff\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-q9v28"
Nov 02 19:46:20.061256 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:20.061230    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9bfbdea-9214-414e-84af-e5c9ea6406ff-istio-envoy\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-q9v28\" (UID: \"b9bfbdea-9214-414e-84af-e5c9ea6406ff\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-q9v28"
Nov 02 19:46:20.061406 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:20.061348    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9bfbdea-9214-414e-84af-e5c9ea6406ff-istiod-ca-cert\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-q9v28\" (UID: \"b9bfbdea-9214-414e-84af-e5c9ea6406ff\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-q9v28"
Nov 02 19:46:20.061406 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:20.061393    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9bfbdea-9214-414e-84af-e5c9ea6406ff-istio-data\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-q9v28\" (UID: \"b9bfbdea-9214-414e-84af-e5c9ea6406ff\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-q9v28"
Nov 02 19:46:20.061511 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:20.061434    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9bfbdea-9214-414e-84af-e5c9ea6406ff-istio-token\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-q9v28\" (UID: \"b9bfbdea-9214-414e-84af-e5c9ea6406ff\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-q9v28"
Nov 02 19:46:20.061511 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:20.061463    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9bfbdea-9214-414e-84af-e5c9ea6406ff-credential-socket\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-q9v28\" (UID: \"b9bfbdea-9214-414e-84af-e5c9ea6406ff\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-q9v28"
Nov 02 19:46:20.061644 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:20.061621    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4xhh\" (UniqueName: \"kubernetes.io/projected/b9bfbdea-9214-414e-84af-e5c9ea6406ff-kube-api-access-t4xhh\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-q9v28\" (UID: \"b9bfbdea-9214-414e-84af-e5c9ea6406ff\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-q9v28"
Nov 02 19:46:20.061714 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:20.061699    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9bfbdea-9214-414e-84af-e5c9ea6406ff-istio-podinfo\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-q9v28\" (UID: \"b9bfbdea-9214-414e-84af-e5c9ea6406ff\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-q9v28"
Nov 02 19:46:20.061768 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:20.061759    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9bfbdea-9214-414e-84af-e5c9ea6406ff-workload-certs\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-q9v28\" (UID: \"b9bfbdea-9214-414e-84af-e5c9ea6406ff\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-q9v28"
Nov 02 19:46:20.061890 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:20.061868    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9bfbdea-9214-414e-84af-e5c9ea6406ff-istio-data\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-q9v28\" (UID: \"b9bfbdea-9214-414e-84af-e5c9ea6406ff\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-q9v28"
Nov 02 19:46:20.061952 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:20.061905    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9bfbdea-9214-414e-84af-e5c9ea6406ff-credential-socket\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-q9v28\" (UID: \"b9bfbdea-9214-414e-84af-e5c9ea6406ff\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-q9v28"
Nov 02 19:46:20.061952 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:20.061925    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9bfbdea-9214-414e-84af-e5c9ea6406ff-workload-socket\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-q9v28\" (UID: \"b9bfbdea-9214-414e-84af-e5c9ea6406ff\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-q9v28"
Nov 02 19:46:20.062071 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:20.062055    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9bfbdea-9214-414e-84af-e5c9ea6406ff-istiod-ca-cert\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-q9v28\" (UID: \"b9bfbdea-9214-414e-84af-e5c9ea6406ff\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-q9v28"
Nov 02 19:46:20.062155 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:20.062135    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9bfbdea-9214-414e-84af-e5c9ea6406ff-workload-certs\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-q9v28\" (UID: \"b9bfbdea-9214-414e-84af-e5c9ea6406ff\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-q9v28"
Nov 02 19:46:20.063444 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:20.063419    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9bfbdea-9214-414e-84af-e5c9ea6406ff-istio-envoy\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-q9v28\" (UID: \"b9bfbdea-9214-414e-84af-e5c9ea6406ff\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-q9v28"
Nov 02 19:46:20.063947 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:20.063930    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9bfbdea-9214-414e-84af-e5c9ea6406ff-istio-podinfo\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-q9v28\" (UID: \"b9bfbdea-9214-414e-84af-e5c9ea6406ff\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-q9v28"
Nov 02 19:46:20.116884 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:20.116848    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9bfbdea-9214-414e-84af-e5c9ea6406ff-istio-token\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-q9v28\" (UID: \"b9bfbdea-9214-414e-84af-e5c9ea6406ff\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-q9v28"
Nov 02 19:46:20.117036 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:20.117003    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4xhh\" (UniqueName: \"kubernetes.io/projected/b9bfbdea-9214-414e-84af-e5c9ea6406ff-kube-api-access-t4xhh\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-q9v28\" (UID: \"b9bfbdea-9214-414e-84af-e5c9ea6406ff\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-q9v28"
Nov 02 19:46:20.146832 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:20.146809    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-q9v28"
Nov 02 19:46:20.561552 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:20.561520    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-q9v28"]
Nov 02 19:46:20.564549 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:46:20.564524    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9bfbdea_9214_414e_84af_e5c9ea6406ff.slice/crio-b49c60078e8ee9145658abc4d4717a06efa3ce2404157b5943791e8283207fb9 WatchSource:0}: Error finding container b49c60078e8ee9145658abc4d4717a06efa3ce2404157b5943791e8283207fb9: Status 404 returned error can't find the container with id b49c60078e8ee9145658abc4d4717a06efa3ce2404157b5943791e8283207fb9
Nov 02 19:46:20.566892 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:20.566866    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:46:20.566957 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:20.566928    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:46:20.567003 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:20.566957    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:46:20.814745 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:20.814649    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-q9v28" event={"ID":"b9bfbdea-9214-414e-84af-e5c9ea6406ff","Type":"ContainerStarted","Data":"65a2df16c2f73565703bcf2da8604827789c6fe32c195fd1064ff05c05221c32"}
Nov 02 19:46:20.814745 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:20.814710    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-q9v28" event={"ID":"b9bfbdea-9214-414e-84af-e5c9ea6406ff","Type":"ContainerStarted","Data":"b49c60078e8ee9145658abc4d4717a06efa3ce2404157b5943791e8283207fb9"}
Nov 02 19:46:20.927354 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:20.927283    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-q9v28" podStartSLOduration=1.927264699 podStartE2EDuration="1.927264699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:46:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:46:20.92580571 +0000 UTC m=+1087.532839501" watchObservedRunningTime="2026-11-02 19:46:20.927264699 +0000 UTC m=+1087.534298469"
Nov 02 19:46:21.147424 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:21.147351    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-q9v28"
Nov 02 19:46:21.150048 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:21.149176    4463 patch_prober.go:29] interesting pod/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-q9v28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Nov 02 19:46:21.150048 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:21.149244    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-q9v28" podUID="b9bfbdea-9214-414e-84af-e5c9ea6406ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Nov 02 19:46:22.147814 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:22.147782    4463 patch_prober.go:29] interesting pod/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-q9v28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Nov 02 19:46:22.148217 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:22.147836    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-q9v28" podUID="b9bfbdea-9214-414e-84af-e5c9ea6406ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Nov 02 19:46:23.148149 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:23.148117    4463 patch_prober.go:29] interesting pod/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-q9v28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Nov 02 19:46:23.148533 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:23.148173    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-q9v28" podUID="b9bfbdea-9214-414e-84af-e5c9ea6406ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Nov 02 19:46:24.147731 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:24.147698    4463 patch_prober.go:29] interesting pod/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-q9v28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Nov 02 19:46:24.147901 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:24.147747    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-q9v28" podUID="b9bfbdea-9214-414e-84af-e5c9ea6406ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Nov 02 19:46:25.147412 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:25.147376    4463 patch_prober.go:29] interesting pod/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-q9v28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Nov 02 19:46:25.147807 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:25.147428    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-q9v28" podUID="b9bfbdea-9214-414e-84af-e5c9ea6406ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Nov 02 19:46:26.147905 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:26.147877    4463 patch_prober.go:29] interesting pod/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-q9v28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Nov 02 19:46:26.148320 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:26.147933    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-q9v28" podUID="b9bfbdea-9214-414e-84af-e5c9ea6406ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Nov 02 19:46:27.147265 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:27.147228    4463 patch_prober.go:29] interesting pod/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-q9v28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Nov 02 19:46:27.147453 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:27.147280    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-q9v28" podUID="b9bfbdea-9214-414e-84af-e5c9ea6406ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Nov 02 19:46:28.152037 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:28.151994    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-q9v28"
Nov 02 19:46:28.152445 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:28.152424    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-q9v28"
Nov 02 19:46:28.153207 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:28.153189    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-q9v28"
Nov 02 19:46:34.293988 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:46:34.293951    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 19:46:35.600192 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:35.591601    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 19:46:54.210133 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:54.210098    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-8795bf/distinct-2-w4-8795bf-kserve-69c8b6b885-2chhn"]
Nov 02 19:46:54.215717 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:54.215693    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8795bf/distinct-2-w4-8795bf-kserve-69c8b6b885-2chhn"
Nov 02 19:46:54.267716 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:54.267683    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8795bf/distinct-2-w4-8795bf-kserve-69c8b6b885-2chhn"]
Nov 02 19:46:54.310218 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:54.310186    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-8795bf\"/\"distinct-2-w4-8795bf-kserve-self-signed-certs\""
Nov 02 19:46:54.310528 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:54.310506    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f430131-4938-47ca-ba86-b6a236f8cc3b-model-cache\") pod \"distinct-2-w4-8795bf-kserve-69c8b6b885-2chhn\" (UID: \"9f430131-4938-47ca-ba86-b6a236f8cc3b\") " pod="e2e-models-e2e-worker-w4-8795bf/distinct-2-w4-8795bf-kserve-69c8b6b885-2chhn"
Nov 02 19:46:54.310586 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:54.310562    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb9qr\" (UniqueName: \"kubernetes.io/projected/9f430131-4938-47ca-ba86-b6a236f8cc3b-kube-api-access-rb9qr\") pod \"distinct-2-w4-8795bf-kserve-69c8b6b885-2chhn\" (UID: \"9f430131-4938-47ca-ba86-b6a236f8cc3b\") " pod="e2e-models-e2e-worker-w4-8795bf/distinct-2-w4-8795bf-kserve-69c8b6b885-2chhn"
Nov 02 19:46:54.310720 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:54.310701    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f430131-4938-47ca-ba86-b6a236f8cc3b-dshm\") pod \"distinct-2-w4-8795bf-kserve-69c8b6b885-2chhn\" (UID: \"9f430131-4938-47ca-ba86-b6a236f8cc3b\") " pod="e2e-models-e2e-worker-w4-8795bf/distinct-2-w4-8795bf-kserve-69c8b6b885-2chhn"
Nov 02 19:46:54.310771 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:54.310743    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f430131-4938-47ca-ba86-b6a236f8cc3b-tls-certs\") pod \"distinct-2-w4-8795bf-kserve-69c8b6b885-2chhn\" (UID: \"9f430131-4938-47ca-ba86-b6a236f8cc3b\") " pod="e2e-models-e2e-worker-w4-8795bf/distinct-2-w4-8795bf-kserve-69c8b6b885-2chhn"
Nov 02 19:46:54.310808 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:54.310766    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f430131-4938-47ca-ba86-b6a236f8cc3b-tmp-dir\") pod \"distinct-2-w4-8795bf-kserve-69c8b6b885-2chhn\" (UID: \"9f430131-4938-47ca-ba86-b6a236f8cc3b\") " pod="e2e-models-e2e-worker-w4-8795bf/distinct-2-w4-8795bf-kserve-69c8b6b885-2chhn"
Nov 02 19:46:54.310858 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:54.310841    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f430131-4938-47ca-ba86-b6a236f8cc3b-home\") pod \"distinct-2-w4-8795bf-kserve-69c8b6b885-2chhn\" (UID: \"9f430131-4938-47ca-ba86-b6a236f8cc3b\") " pod="e2e-models-e2e-worker-w4-8795bf/distinct-2-w4-8795bf-kserve-69c8b6b885-2chhn"
Nov 02 19:46:54.412769 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:54.412731    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f430131-4938-47ca-ba86-b6a236f8cc3b-model-cache\") pod \"distinct-2-w4-8795bf-kserve-69c8b6b885-2chhn\" (UID: \"9f430131-4938-47ca-ba86-b6a236f8cc3b\") " pod="e2e-models-e2e-worker-w4-8795bf/distinct-2-w4-8795bf-kserve-69c8b6b885-2chhn"
Nov 02 19:46:54.412974 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:54.412818    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb9qr\" (UniqueName: \"kubernetes.io/projected/9f430131-4938-47ca-ba86-b6a236f8cc3b-kube-api-access-rb9qr\") pod \"distinct-2-w4-8795bf-kserve-69c8b6b885-2chhn\" (UID: \"9f430131-4938-47ca-ba86-b6a236f8cc3b\") " pod="e2e-models-e2e-worker-w4-8795bf/distinct-2-w4-8795bf-kserve-69c8b6b885-2chhn"
Nov 02 19:46:54.412974 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:54.412905    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f430131-4938-47ca-ba86-b6a236f8cc3b-dshm\") pod \"distinct-2-w4-8795bf-kserve-69c8b6b885-2chhn\" (UID: \"9f430131-4938-47ca-ba86-b6a236f8cc3b\") " pod="e2e-models-e2e-worker-w4-8795bf/distinct-2-w4-8795bf-kserve-69c8b6b885-2chhn"
Nov 02 19:46:54.412974 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:54.412952    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f430131-4938-47ca-ba86-b6a236f8cc3b-tls-certs\") pod \"distinct-2-w4-8795bf-kserve-69c8b6b885-2chhn\" (UID: \"9f430131-4938-47ca-ba86-b6a236f8cc3b\") " pod="e2e-models-e2e-worker-w4-8795bf/distinct-2-w4-8795bf-kserve-69c8b6b885-2chhn"
Nov 02 19:46:54.413164 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:54.412978    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f430131-4938-47ca-ba86-b6a236f8cc3b-tmp-dir\") pod \"distinct-2-w4-8795bf-kserve-69c8b6b885-2chhn\" (UID: \"9f430131-4938-47ca-ba86-b6a236f8cc3b\") " pod="e2e-models-e2e-worker-w4-8795bf/distinct-2-w4-8795bf-kserve-69c8b6b885-2chhn"
Nov 02 19:46:54.413164 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:54.413059    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f430131-4938-47ca-ba86-b6a236f8cc3b-home\") pod \"distinct-2-w4-8795bf-kserve-69c8b6b885-2chhn\" (UID: \"9f430131-4938-47ca-ba86-b6a236f8cc3b\") " pod="e2e-models-e2e-worker-w4-8795bf/distinct-2-w4-8795bf-kserve-69c8b6b885-2chhn"
Nov 02 19:46:54.413271 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:54.413199    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f430131-4938-47ca-ba86-b6a236f8cc3b-model-cache\") pod \"distinct-2-w4-8795bf-kserve-69c8b6b885-2chhn\" (UID: \"9f430131-4938-47ca-ba86-b6a236f8cc3b\") " pod="e2e-models-e2e-worker-w4-8795bf/distinct-2-w4-8795bf-kserve-69c8b6b885-2chhn"
Nov 02 19:46:54.413403 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:54.413367    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f430131-4938-47ca-ba86-b6a236f8cc3b-tmp-dir\") pod \"distinct-2-w4-8795bf-kserve-69c8b6b885-2chhn\" (UID: \"9f430131-4938-47ca-ba86-b6a236f8cc3b\") " pod="e2e-models-e2e-worker-w4-8795bf/distinct-2-w4-8795bf-kserve-69c8b6b885-2chhn"
Nov 02 19:46:54.413403 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:54.413404    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f430131-4938-47ca-ba86-b6a236f8cc3b-home\") pod \"distinct-2-w4-8795bf-kserve-69c8b6b885-2chhn\" (UID: \"9f430131-4938-47ca-ba86-b6a236f8cc3b\") " pod="e2e-models-e2e-worker-w4-8795bf/distinct-2-w4-8795bf-kserve-69c8b6b885-2chhn"
Nov 02 19:46:54.415501 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:54.415475    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f430131-4938-47ca-ba86-b6a236f8cc3b-dshm\") pod \"distinct-2-w4-8795bf-kserve-69c8b6b885-2chhn\" (UID: \"9f430131-4938-47ca-ba86-b6a236f8cc3b\") " pod="e2e-models-e2e-worker-w4-8795bf/distinct-2-w4-8795bf-kserve-69c8b6b885-2chhn"
Nov 02 19:46:54.415821 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:54.415803    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f430131-4938-47ca-ba86-b6a236f8cc3b-tls-certs\") pod \"distinct-2-w4-8795bf-kserve-69c8b6b885-2chhn\" (UID: \"9f430131-4938-47ca-ba86-b6a236f8cc3b\") " pod="e2e-models-e2e-worker-w4-8795bf/distinct-2-w4-8795bf-kserve-69c8b6b885-2chhn"
Nov 02 19:46:54.469633 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:54.469559    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb9qr\" (UniqueName: \"kubernetes.io/projected/9f430131-4938-47ca-ba86-b6a236f8cc3b-kube-api-access-rb9qr\") pod \"distinct-2-w4-8795bf-kserve-69c8b6b885-2chhn\" (UID: \"9f430131-4938-47ca-ba86-b6a236f8cc3b\") " pod="e2e-models-e2e-worker-w4-8795bf/distinct-2-w4-8795bf-kserve-69c8b6b885-2chhn"
Nov 02 19:46:54.530138 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:54.530103    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8795bf/distinct-2-w4-8795bf-kserve-69c8b6b885-2chhn"
Nov 02 19:46:54.924483 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:54.924454    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8795bf/distinct-2-w4-8795bf-kserve-69c8b6b885-2chhn"]
Nov 02 19:46:54.926205 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:46:54.926179    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9f430131_4938_47ca_ba86_b6a236f8cc3b.slice/crio-4226485cc1abdd71f7999c4bc55d2528908345f45083e327bd3ebe012687af61 WatchSource:0}: Error finding container 4226485cc1abdd71f7999c4bc55d2528908345f45083e327bd3ebe012687af61: Status 404 returned error can't find the container with id 4226485cc1abdd71f7999c4bc55d2528908345f45083e327bd3ebe012687af61
Nov 02 19:46:55.035383 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:55.035349    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8795bf/distinct-2-w4-8795bf-kserve-69c8b6b885-2chhn" event={"ID":"9f430131-4938-47ca-ba86-b6a236f8cc3b","Type":"ContainerStarted","Data":"776b5f17729e872597f4d31a2c6f0b826c106f88742ae67f05214246255cc9a9"}
Nov 02 19:46:55.035383 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:55.035387    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8795bf/distinct-2-w4-8795bf-kserve-69c8b6b885-2chhn" event={"ID":"9f430131-4938-47ca-ba86-b6a236f8cc3b","Type":"ContainerStarted","Data":"4226485cc1abdd71f7999c4bc55d2528908345f45083e327bd3ebe012687af61"}
Nov 02 19:46:55.142798 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:55.142735    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-8795bf/distinct-2-w4-8795bf-kserve-69c8b6b885-2chhn" podStartSLOduration=1.142721036 podStartE2EDuration="1.142721036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:46:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:46:55.141077752 +0000 UTC m=+1121.748111520" watchObservedRunningTime="2026-11-02 19:46:55.142721036 +0000 UTC m=+1121.749754829"
Nov 02 19:46:55.531195 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:46:55.531160    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8795bf/distinct-2-w4-8795bf-kserve-69c8b6b885-2chhn"
Nov 02 19:47:04.294375 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:47:04.294340    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 19:47:04.530394 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:04.530361    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-8795bf/distinct-2-w4-8795bf-kserve-69c8b6b885-2chhn"
Nov 02 19:47:04.543274 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:04.543246    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-8795bf/distinct-2-w4-8795bf-kserve-69c8b6b885-2chhn"
Nov 02 19:47:04.931267 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:04.931232    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-w5gw2"]
Nov 02 19:47:04.935267 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:04.935247    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-w5gw2"
Nov 02 19:47:05.018509 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:05.018471    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-w5gw2"]
Nov 02 19:47:05.040969 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:05.040937    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d09ad81c-4422-4b1a-b845-a616f5c9e93c-credential-socket\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-w5gw2\" (UID: \"d09ad81c-4422-4b1a-b845-a616f5c9e93c\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-w5gw2"
Nov 02 19:47:05.041167 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:05.040980    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d09ad81c-4422-4b1a-b845-a616f5c9e93c-istio-podinfo\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-w5gw2\" (UID: \"d09ad81c-4422-4b1a-b845-a616f5c9e93c\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-w5gw2"
Nov 02 19:47:05.041167 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:05.041038    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d09ad81c-4422-4b1a-b845-a616f5c9e93c-workload-socket\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-w5gw2\" (UID: \"d09ad81c-4422-4b1a-b845-a616f5c9e93c\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-w5gw2"
Nov 02 19:47:05.041265 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:05.041155    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d09ad81c-4422-4b1a-b845-a616f5c9e93c-istio-envoy\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-w5gw2\" (UID: \"d09ad81c-4422-4b1a-b845-a616f5c9e93c\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-w5gw2"
Nov 02 19:47:05.041265 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:05.041220    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hphl6\" (UniqueName: \"kubernetes.io/projected/d09ad81c-4422-4b1a-b845-a616f5c9e93c-kube-api-access-hphl6\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-w5gw2\" (UID: \"d09ad81c-4422-4b1a-b845-a616f5c9e93c\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-w5gw2"
Nov 02 19:47:05.041265 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:05.041262    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d09ad81c-4422-4b1a-b845-a616f5c9e93c-workload-certs\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-w5gw2\" (UID: \"d09ad81c-4422-4b1a-b845-a616f5c9e93c\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-w5gw2"
Nov 02 19:47:05.041367 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:05.041296    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d09ad81c-4422-4b1a-b845-a616f5c9e93c-istiod-ca-cert\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-w5gw2\" (UID: \"d09ad81c-4422-4b1a-b845-a616f5c9e93c\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-w5gw2"
Nov 02 19:47:05.041367 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:05.041341    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d09ad81c-4422-4b1a-b845-a616f5c9e93c-istio-data\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-w5gw2\" (UID: \"d09ad81c-4422-4b1a-b845-a616f5c9e93c\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-w5gw2"
Nov 02 19:47:05.041492 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:05.041411    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d09ad81c-4422-4b1a-b845-a616f5c9e93c-istio-token\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-w5gw2\" (UID: \"d09ad81c-4422-4b1a-b845-a616f5c9e93c\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-w5gw2"
Nov 02 19:47:05.102828 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:05.102797    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8795bf/distinct-2-w4-8795bf-kserve-69c8b6b885-2chhn"
Nov 02 19:47:05.142397 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:05.142363    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d09ad81c-4422-4b1a-b845-a616f5c9e93c-workload-certs\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-w5gw2\" (UID: \"d09ad81c-4422-4b1a-b845-a616f5c9e93c\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-w5gw2"
Nov 02 19:47:05.142571 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:05.142420    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d09ad81c-4422-4b1a-b845-a616f5c9e93c-istiod-ca-cert\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-w5gw2\" (UID: \"d09ad81c-4422-4b1a-b845-a616f5c9e93c\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-w5gw2"
Nov 02 19:47:05.142571 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:05.142458    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d09ad81c-4422-4b1a-b845-a616f5c9e93c-istio-data\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-w5gw2\" (UID: \"d09ad81c-4422-4b1a-b845-a616f5c9e93c\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-w5gw2"
Nov 02 19:47:05.142571 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:05.142497    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d09ad81c-4422-4b1a-b845-a616f5c9e93c-istio-token\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-w5gw2\" (UID: \"d09ad81c-4422-4b1a-b845-a616f5c9e93c\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-w5gw2"
Nov 02 19:47:05.142744 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:05.142591    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d09ad81c-4422-4b1a-b845-a616f5c9e93c-credential-socket\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-w5gw2\" (UID: \"d09ad81c-4422-4b1a-b845-a616f5c9e93c\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-w5gw2"
Nov 02 19:47:05.142744 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:05.142627    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d09ad81c-4422-4b1a-b845-a616f5c9e93c-istio-podinfo\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-w5gw2\" (UID: \"d09ad81c-4422-4b1a-b845-a616f5c9e93c\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-w5gw2"
Nov 02 19:47:05.142744 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:05.142680    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d09ad81c-4422-4b1a-b845-a616f5c9e93c-workload-socket\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-w5gw2\" (UID: \"d09ad81c-4422-4b1a-b845-a616f5c9e93c\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-w5gw2"
Nov 02 19:47:05.142895 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:05.142746    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d09ad81c-4422-4b1a-b845-a616f5c9e93c-istio-envoy\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-w5gw2\" (UID: \"d09ad81c-4422-4b1a-b845-a616f5c9e93c\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-w5gw2"
Nov 02 19:47:05.142895 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:05.142766    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d09ad81c-4422-4b1a-b845-a616f5c9e93c-workload-certs\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-w5gw2\" (UID: \"d09ad81c-4422-4b1a-b845-a616f5c9e93c\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-w5gw2"
Nov 02 19:47:05.142895 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:05.142806    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hphl6\" (UniqueName: \"kubernetes.io/projected/d09ad81c-4422-4b1a-b845-a616f5c9e93c-kube-api-access-hphl6\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-w5gw2\" (UID: \"d09ad81c-4422-4b1a-b845-a616f5c9e93c\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-w5gw2"
Nov 02 19:47:05.142895 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:05.142811    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d09ad81c-4422-4b1a-b845-a616f5c9e93c-istio-data\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-w5gw2\" (UID: \"d09ad81c-4422-4b1a-b845-a616f5c9e93c\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-w5gw2"
Nov 02 19:47:05.143144 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:05.142968    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d09ad81c-4422-4b1a-b845-a616f5c9e93c-credential-socket\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-w5gw2\" (UID: \"d09ad81c-4422-4b1a-b845-a616f5c9e93c\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-w5gw2"
Nov 02 19:47:05.143144 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:05.143106    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d09ad81c-4422-4b1a-b845-a616f5c9e93c-workload-socket\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-w5gw2\" (UID: \"d09ad81c-4422-4b1a-b845-a616f5c9e93c\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-w5gw2"
Nov 02 19:47:05.143430 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:05.143408    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d09ad81c-4422-4b1a-b845-a616f5c9e93c-istiod-ca-cert\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-w5gw2\" (UID: \"d09ad81c-4422-4b1a-b845-a616f5c9e93c\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-w5gw2"
Nov 02 19:47:05.145058 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:05.145028    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d09ad81c-4422-4b1a-b845-a616f5c9e93c-istio-envoy\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-w5gw2\" (UID: \"d09ad81c-4422-4b1a-b845-a616f5c9e93c\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-w5gw2"
Nov 02 19:47:05.145187 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:05.145169    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d09ad81c-4422-4b1a-b845-a616f5c9e93c-istio-podinfo\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-w5gw2\" (UID: \"d09ad81c-4422-4b1a-b845-a616f5c9e93c\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-w5gw2"
Nov 02 19:47:05.196135 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:05.196072    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d09ad81c-4422-4b1a-b845-a616f5c9e93c-istio-token\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-w5gw2\" (UID: \"d09ad81c-4422-4b1a-b845-a616f5c9e93c\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-w5gw2"
Nov 02 19:47:05.196277 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:05.196136    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hphl6\" (UniqueName: \"kubernetes.io/projected/d09ad81c-4422-4b1a-b845-a616f5c9e93c-kube-api-access-hphl6\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-w5gw2\" (UID: \"d09ad81c-4422-4b1a-b845-a616f5c9e93c\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-w5gw2"
Nov 02 19:47:05.248215 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:05.248179    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-w5gw2"
Nov 02 19:47:05.665483 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:05.665460    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-w5gw2"]
Nov 02 19:47:05.666697 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:47:05.666671    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd09ad81c_4422_4b1a_b845_a616f5c9e93c.slice/crio-61c34e5c1adb7cca8a2c36281bd73cc1db60899d300d132a434ee1a8f2b26f09 WatchSource:0}: Error finding container 61c34e5c1adb7cca8a2c36281bd73cc1db60899d300d132a434ee1a8f2b26f09: Status 404 returned error can't find the container with id 61c34e5c1adb7cca8a2c36281bd73cc1db60899d300d132a434ee1a8f2b26f09
Nov 02 19:47:05.668951 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:05.668920    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:47:05.669041 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:05.668984    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:47:05.669041 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:05.669028    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:47:06.108467 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:06.108428    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-w5gw2" event={"ID":"d09ad81c-4422-4b1a-b845-a616f5c9e93c","Type":"ContainerStarted","Data":"a02110035b528ab4aad6fd5a33abe1f2af48f93bd083a52279beddd7e1521b01"}
Nov 02 19:47:06.108654 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:06.108518    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-w5gw2" event={"ID":"d09ad81c-4422-4b1a-b845-a616f5c9e93c","Type":"ContainerStarted","Data":"61c34e5c1adb7cca8a2c36281bd73cc1db60899d300d132a434ee1a8f2b26f09"}
Nov 02 19:47:06.219101 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:06.219007    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-w5gw2" podStartSLOduration=2.218989149 podStartE2EDuration="2.218989149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:47:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:47:06.21785003 +0000 UTC m=+1132.824883819" watchObservedRunningTime="2026-11-02 19:47:06.218989149 +0000 UTC m=+1132.826022918"
Nov 02 19:47:06.248617 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:06.248582    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-w5gw2"
Nov 02 19:47:06.250319 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:06.250298    4463 patch_prober.go:29] interesting pod/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-w5gw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Nov 02 19:47:06.250389 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:06.250345    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-w5gw2" podUID="d09ad81c-4422-4b1a-b845-a616f5c9e93c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Nov 02 19:47:07.261037 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:07.251105    4463 patch_prober.go:29] interesting pod/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-w5gw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Nov 02 19:47:07.261037 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:07.251171    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-w5gw2" podUID="d09ad81c-4422-4b1a-b845-a616f5c9e93c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Nov 02 19:47:08.248767 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:08.248734    4463 patch_prober.go:29] interesting pod/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-w5gw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Nov 02 19:47:08.248941 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:08.248783    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-w5gw2" podUID="d09ad81c-4422-4b1a-b845-a616f5c9e93c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Nov 02 19:47:09.269143 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:09.269104    4463 patch_prober.go:29] interesting pod/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-w5gw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Nov 02 19:47:09.269729 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:09.269699    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-w5gw2" podUID="d09ad81c-4422-4b1a-b845-a616f5c9e93c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Nov 02 19:47:10.248623 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:10.248590    4463 patch_prober.go:29] interesting pod/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-w5gw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Nov 02 19:47:10.248802 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:10.248649    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-w5gw2" podUID="d09ad81c-4422-4b1a-b845-a616f5c9e93c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Nov 02 19:47:11.248865 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:11.248836    4463 patch_prober.go:29] interesting pod/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-w5gw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Nov 02 19:47:11.249359 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:11.248884    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-w5gw2" podUID="d09ad81c-4422-4b1a-b845-a616f5c9e93c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Nov 02 19:47:11.737677 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:11.737637    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 19:47:12.248912 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:12.248878    4463 patch_prober.go:29] interesting pod/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-w5gw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Nov 02 19:47:12.249396 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:12.248934    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-w5gw2" podUID="d09ad81c-4422-4b1a-b845-a616f5c9e93c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Nov 02 19:47:14.304149 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:14.304105    4463 patch_prober.go:29] interesting pod/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-w5gw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 19:47:14.304765 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:14.304730    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-w5gw2" podUID="d09ad81c-4422-4b1a-b845-a616f5c9e93c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:47:14.967117 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:14.967088    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-w5gw2"
Nov 02 19:47:14.967607 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:14.967589    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-w5gw2"
Nov 02 19:47:14.968444 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:14.968421    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-w5gw2"
Nov 02 19:47:16.481340 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:16.481311    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-1d869e/distinct-2-w5-1d869e-kserve-574f55db46-4cv47"]
Nov 02 19:47:16.485478 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:16.485460    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-1d869e/distinct-2-w5-1d869e-kserve-574f55db46-4cv47"
Nov 02 19:47:16.581055 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:16.580998    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-1d869e\"/\"distinct-2-w5-1d869e-kserve-self-signed-certs\""
Nov 02 19:47:16.584238 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:16.584209    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-1d869e/distinct-2-w5-1d869e-kserve-574f55db46-4cv47"]
Nov 02 19:47:16.664618 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:16.664573    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e236108-328b-4170-be20-13f16b56c4fc-tmp-dir\") pod \"distinct-2-w5-1d869e-kserve-574f55db46-4cv47\" (UID: \"7e236108-328b-4170-be20-13f16b56c4fc\") " pod="e2e-models-e2e-worker-w5-1d869e/distinct-2-w5-1d869e-kserve-574f55db46-4cv47"
Nov 02 19:47:16.664816 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:16.664640    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e236108-328b-4170-be20-13f16b56c4fc-model-cache\") pod \"distinct-2-w5-1d869e-kserve-574f55db46-4cv47\" (UID: \"7e236108-328b-4170-be20-13f16b56c4fc\") " pod="e2e-models-e2e-worker-w5-1d869e/distinct-2-w5-1d869e-kserve-574f55db46-4cv47"
Nov 02 19:47:16.664816 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:16.664669    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e236108-328b-4170-be20-13f16b56c4fc-tls-certs\") pod \"distinct-2-w5-1d869e-kserve-574f55db46-4cv47\" (UID: \"7e236108-328b-4170-be20-13f16b56c4fc\") " pod="e2e-models-e2e-worker-w5-1d869e/distinct-2-w5-1d869e-kserve-574f55db46-4cv47"
Nov 02 19:47:16.664816 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:16.664747    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcqwf\" (UniqueName: \"kubernetes.io/projected/7e236108-328b-4170-be20-13f16b56c4fc-kube-api-access-zcqwf\") pod \"distinct-2-w5-1d869e-kserve-574f55db46-4cv47\" (UID: \"7e236108-328b-4170-be20-13f16b56c4fc\") " pod="e2e-models-e2e-worker-w5-1d869e/distinct-2-w5-1d869e-kserve-574f55db46-4cv47"
Nov 02 19:47:16.664816 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:16.664797    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e236108-328b-4170-be20-13f16b56c4fc-home\") pod \"distinct-2-w5-1d869e-kserve-574f55db46-4cv47\" (UID: \"7e236108-328b-4170-be20-13f16b56c4fc\") " pod="e2e-models-e2e-worker-w5-1d869e/distinct-2-w5-1d869e-kserve-574f55db46-4cv47"
Nov 02 19:47:16.665123 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:16.665103    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e236108-328b-4170-be20-13f16b56c4fc-dshm\") pod \"distinct-2-w5-1d869e-kserve-574f55db46-4cv47\" (UID: \"7e236108-328b-4170-be20-13f16b56c4fc\") " pod="e2e-models-e2e-worker-w5-1d869e/distinct-2-w5-1d869e-kserve-574f55db46-4cv47"
Nov 02 19:47:16.765896 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:16.765807    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e236108-328b-4170-be20-13f16b56c4fc-tmp-dir\") pod \"distinct-2-w5-1d869e-kserve-574f55db46-4cv47\" (UID: \"7e236108-328b-4170-be20-13f16b56c4fc\") " pod="e2e-models-e2e-worker-w5-1d869e/distinct-2-w5-1d869e-kserve-574f55db46-4cv47"
Nov 02 19:47:16.765896 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:16.765848    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e236108-328b-4170-be20-13f16b56c4fc-model-cache\") pod \"distinct-2-w5-1d869e-kserve-574f55db46-4cv47\" (UID: \"7e236108-328b-4170-be20-13f16b56c4fc\") " pod="e2e-models-e2e-worker-w5-1d869e/distinct-2-w5-1d869e-kserve-574f55db46-4cv47"
Nov 02 19:47:16.765896 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:16.765867    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e236108-328b-4170-be20-13f16b56c4fc-tls-certs\") pod \"distinct-2-w5-1d869e-kserve-574f55db46-4cv47\" (UID: \"7e236108-328b-4170-be20-13f16b56c4fc\") " pod="e2e-models-e2e-worker-w5-1d869e/distinct-2-w5-1d869e-kserve-574f55db46-4cv47"
Nov 02 19:47:16.766217 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:16.765998    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcqwf\" (UniqueName: \"kubernetes.io/projected/7e236108-328b-4170-be20-13f16b56c4fc-kube-api-access-zcqwf\") pod \"distinct-2-w5-1d869e-kserve-574f55db46-4cv47\" (UID: \"7e236108-328b-4170-be20-13f16b56c4fc\") " pod="e2e-models-e2e-worker-w5-1d869e/distinct-2-w5-1d869e-kserve-574f55db46-4cv47"
Nov 02 19:47:16.766217 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:16.766067    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e236108-328b-4170-be20-13f16b56c4fc-home\") pod \"distinct-2-w5-1d869e-kserve-574f55db46-4cv47\" (UID: \"7e236108-328b-4170-be20-13f16b56c4fc\") " pod="e2e-models-e2e-worker-w5-1d869e/distinct-2-w5-1d869e-kserve-574f55db46-4cv47"
Nov 02 19:47:16.766217 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:16.766200    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e236108-328b-4170-be20-13f16b56c4fc-dshm\") pod \"distinct-2-w5-1d869e-kserve-574f55db46-4cv47\" (UID: \"7e236108-328b-4170-be20-13f16b56c4fc\") " pod="e2e-models-e2e-worker-w5-1d869e/distinct-2-w5-1d869e-kserve-574f55db46-4cv47"
Nov 02 19:47:16.766424 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:16.766266    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e236108-328b-4170-be20-13f16b56c4fc-tmp-dir\") pod \"distinct-2-w5-1d869e-kserve-574f55db46-4cv47\" (UID: \"7e236108-328b-4170-be20-13f16b56c4fc\") " pod="e2e-models-e2e-worker-w5-1d869e/distinct-2-w5-1d869e-kserve-574f55db46-4cv47"
Nov 02 19:47:16.766424 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:16.766413    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e236108-328b-4170-be20-13f16b56c4fc-home\") pod \"distinct-2-w5-1d869e-kserve-574f55db46-4cv47\" (UID: \"7e236108-328b-4170-be20-13f16b56c4fc\") " pod="e2e-models-e2e-worker-w5-1d869e/distinct-2-w5-1d869e-kserve-574f55db46-4cv47"
Nov 02 19:47:16.766488 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:16.766305    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e236108-328b-4170-be20-13f16b56c4fc-model-cache\") pod \"distinct-2-w5-1d869e-kserve-574f55db46-4cv47\" (UID: \"7e236108-328b-4170-be20-13f16b56c4fc\") " pod="e2e-models-e2e-worker-w5-1d869e/distinct-2-w5-1d869e-kserve-574f55db46-4cv47"
Nov 02 19:47:16.768377 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:16.768354    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e236108-328b-4170-be20-13f16b56c4fc-dshm\") pod \"distinct-2-w5-1d869e-kserve-574f55db46-4cv47\" (UID: \"7e236108-328b-4170-be20-13f16b56c4fc\") " pod="e2e-models-e2e-worker-w5-1d869e/distinct-2-w5-1d869e-kserve-574f55db46-4cv47"
Nov 02 19:47:16.768493 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:16.768479    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e236108-328b-4170-be20-13f16b56c4fc-tls-certs\") pod \"distinct-2-w5-1d869e-kserve-574f55db46-4cv47\" (UID: \"7e236108-328b-4170-be20-13f16b56c4fc\") " pod="e2e-models-e2e-worker-w5-1d869e/distinct-2-w5-1d869e-kserve-574f55db46-4cv47"
Nov 02 19:47:16.819249 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:16.819223    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcqwf\" (UniqueName: \"kubernetes.io/projected/7e236108-328b-4170-be20-13f16b56c4fc-kube-api-access-zcqwf\") pod \"distinct-2-w5-1d869e-kserve-574f55db46-4cv47\" (UID: \"7e236108-328b-4170-be20-13f16b56c4fc\") " pod="e2e-models-e2e-worker-w5-1d869e/distinct-2-w5-1d869e-kserve-574f55db46-4cv47"
Nov 02 19:47:17.097093 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:17.096998    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-1d869e/distinct-2-w5-1d869e-kserve-574f55db46-4cv47"
Nov 02 19:47:17.466959 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:47:17.466930    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7e236108_328b_4170_be20_13f16b56c4fc.slice/crio-74100345d41ddc8241a0ff8ab3b6a61a0fb4167e0136f5d72ca8f7c27b59b740 WatchSource:0}: Error finding container 74100345d41ddc8241a0ff8ab3b6a61a0fb4167e0136f5d72ca8f7c27b59b740: Status 404 returned error can't find the container with id 74100345d41ddc8241a0ff8ab3b6a61a0fb4167e0136f5d72ca8f7c27b59b740
Nov 02 19:47:17.502136 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:17.502111    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-1d869e/distinct-2-w5-1d869e-kserve-574f55db46-4cv47"]
Nov 02 19:47:18.192787 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:18.192750    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-1d869e/distinct-2-w5-1d869e-kserve-574f55db46-4cv47" event={"ID":"7e236108-328b-4170-be20-13f16b56c4fc","Type":"ContainerStarted","Data":"710ec917c9760293985d1402b9aa8aac118aeb304ae914e8f9a310db1485730b"}
Nov 02 19:47:18.192939 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:18.192800    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-1d869e/distinct-2-w5-1d869e-kserve-574f55db46-4cv47" event={"ID":"7e236108-328b-4170-be20-13f16b56c4fc","Type":"ContainerStarted","Data":"74100345d41ddc8241a0ff8ab3b6a61a0fb4167e0136f5d72ca8f7c27b59b740"}
Nov 02 19:47:18.300587 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:18.300536    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-1d869e/distinct-2-w5-1d869e-kserve-574f55db46-4cv47" podStartSLOduration=2.300523037 podStartE2EDuration="2.300523037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:47:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:47:18.299218941 +0000 UTC m=+1144.906252721" watchObservedRunningTime="2026-11-02 19:47:18.300523037 +0000 UTC m=+1144.907556807"
Nov 02 19:47:19.097196 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:19.097165    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-1d869e/distinct-2-w5-1d869e-kserve-574f55db46-4cv47"
Nov 02 19:47:27.097684 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:27.097642    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-1d869e/distinct-2-w5-1d869e-kserve-574f55db46-4cv47"
Nov 02 19:47:27.110081 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:27.110053    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-1d869e/distinct-2-w5-1d869e-kserve-574f55db46-4cv47"
Nov 02 19:47:27.261334 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:27.261295    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-1d869e/distinct-2-w5-1d869e-kserve-574f55db46-4cv47"
Nov 02 19:47:34.295187 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:47:34.295152    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 19:47:34.597598 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:34.597522    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-lkfn9"]
Nov 02 19:47:34.601965 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:34.601943    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-lkfn9"
Nov 02 19:47:34.670911 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:34.670884    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b33f5f29-4f1a-4e70-a821-df5018260ad7-istio-data\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-lkfn9\" (UID: \"b33f5f29-4f1a-4e70-a821-df5018260ad7\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-lkfn9"
Nov 02 19:47:34.671051 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:34.670944    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b33f5f29-4f1a-4e70-a821-df5018260ad7-istio-token\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-lkfn9\" (UID: \"b33f5f29-4f1a-4e70-a821-df5018260ad7\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-lkfn9"
Nov 02 19:47:34.671051 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:34.671031    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b33f5f29-4f1a-4e70-a821-df5018260ad7-istio-envoy\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-lkfn9\" (UID: \"b33f5f29-4f1a-4e70-a821-df5018260ad7\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-lkfn9"
Nov 02 19:47:34.671134 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:34.671085    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b33f5f29-4f1a-4e70-a821-df5018260ad7-workload-certs\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-lkfn9\" (UID: \"b33f5f29-4f1a-4e70-a821-df5018260ad7\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-lkfn9"
Nov 02 19:47:34.671225 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:34.671206    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b33f5f29-4f1a-4e70-a821-df5018260ad7-istiod-ca-cert\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-lkfn9\" (UID: \"b33f5f29-4f1a-4e70-a821-df5018260ad7\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-lkfn9"
Nov 02 19:47:34.671316 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:34.671300    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5x72b\" (UniqueName: \"kubernetes.io/projected/b33f5f29-4f1a-4e70-a821-df5018260ad7-kube-api-access-5x72b\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-lkfn9\" (UID: \"b33f5f29-4f1a-4e70-a821-df5018260ad7\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-lkfn9"
Nov 02 19:47:34.671372 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:34.671358    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b33f5f29-4f1a-4e70-a821-df5018260ad7-workload-socket\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-lkfn9\" (UID: \"b33f5f29-4f1a-4e70-a821-df5018260ad7\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-lkfn9"
Nov 02 19:47:34.671409 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:34.671388    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b33f5f29-4f1a-4e70-a821-df5018260ad7-credential-socket\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-lkfn9\" (UID: \"b33f5f29-4f1a-4e70-a821-df5018260ad7\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-lkfn9"
Nov 02 19:47:34.671442 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:34.671419    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b33f5f29-4f1a-4e70-a821-df5018260ad7-istio-podinfo\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-lkfn9\" (UID: \"b33f5f29-4f1a-4e70-a821-df5018260ad7\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-lkfn9"
Nov 02 19:47:34.772992 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:34.772944    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5x72b\" (UniqueName: \"kubernetes.io/projected/b33f5f29-4f1a-4e70-a821-df5018260ad7-kube-api-access-5x72b\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-lkfn9\" (UID: \"b33f5f29-4f1a-4e70-a821-df5018260ad7\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-lkfn9"
Nov 02 19:47:34.773194 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:34.773035    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b33f5f29-4f1a-4e70-a821-df5018260ad7-workload-socket\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-lkfn9\" (UID: \"b33f5f29-4f1a-4e70-a821-df5018260ad7\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-lkfn9"
Nov 02 19:47:34.773194 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:34.773080    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b33f5f29-4f1a-4e70-a821-df5018260ad7-credential-socket\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-lkfn9\" (UID: \"b33f5f29-4f1a-4e70-a821-df5018260ad7\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-lkfn9"
Nov 02 19:47:34.773194 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:34.773129    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b33f5f29-4f1a-4e70-a821-df5018260ad7-istio-podinfo\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-lkfn9\" (UID: \"b33f5f29-4f1a-4e70-a821-df5018260ad7\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-lkfn9"
Nov 02 19:47:34.773357 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:34.773244    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b33f5f29-4f1a-4e70-a821-df5018260ad7-istio-data\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-lkfn9\" (UID: \"b33f5f29-4f1a-4e70-a821-df5018260ad7\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-lkfn9"
Nov 02 19:47:34.773357 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:34.773313    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b33f5f29-4f1a-4e70-a821-df5018260ad7-istio-token\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-lkfn9\" (UID: \"b33f5f29-4f1a-4e70-a821-df5018260ad7\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-lkfn9"
Nov 02 19:47:34.773357 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:34.773340    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b33f5f29-4f1a-4e70-a821-df5018260ad7-istio-envoy\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-lkfn9\" (UID: \"b33f5f29-4f1a-4e70-a821-df5018260ad7\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-lkfn9"
Nov 02 19:47:34.773576 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:34.773379    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b33f5f29-4f1a-4e70-a821-df5018260ad7-workload-certs\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-lkfn9\" (UID: \"b33f5f29-4f1a-4e70-a821-df5018260ad7\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-lkfn9"
Nov 02 19:47:34.773576 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:34.773450    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b33f5f29-4f1a-4e70-a821-df5018260ad7-istiod-ca-cert\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-lkfn9\" (UID: \"b33f5f29-4f1a-4e70-a821-df5018260ad7\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-lkfn9"
Nov 02 19:47:34.773576 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:34.773465    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b33f5f29-4f1a-4e70-a821-df5018260ad7-workload-socket\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-lkfn9\" (UID: \"b33f5f29-4f1a-4e70-a821-df5018260ad7\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-lkfn9"
Nov 02 19:47:34.773749 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:34.773726    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b33f5f29-4f1a-4e70-a821-df5018260ad7-istio-data\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-lkfn9\" (UID: \"b33f5f29-4f1a-4e70-a821-df5018260ad7\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-lkfn9"
Nov 02 19:47:34.773828 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:34.773809    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b33f5f29-4f1a-4e70-a821-df5018260ad7-credential-socket\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-lkfn9\" (UID: \"b33f5f29-4f1a-4e70-a821-df5018260ad7\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-lkfn9"
Nov 02 19:47:34.773903 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:34.773882    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b33f5f29-4f1a-4e70-a821-df5018260ad7-workload-certs\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-lkfn9\" (UID: \"b33f5f29-4f1a-4e70-a821-df5018260ad7\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-lkfn9"
Nov 02 19:47:34.774317 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:34.774293    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b33f5f29-4f1a-4e70-a821-df5018260ad7-istiod-ca-cert\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-lkfn9\" (UID: \"b33f5f29-4f1a-4e70-a821-df5018260ad7\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-lkfn9"
Nov 02 19:47:34.775650 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:34.775621    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b33f5f29-4f1a-4e70-a821-df5018260ad7-istio-envoy\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-lkfn9\" (UID: \"b33f5f29-4f1a-4e70-a821-df5018260ad7\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-lkfn9"
Nov 02 19:47:34.775762 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:34.775747    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b33f5f29-4f1a-4e70-a821-df5018260ad7-istio-podinfo\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-lkfn9\" (UID: \"b33f5f29-4f1a-4e70-a821-df5018260ad7\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-lkfn9"
Nov 02 19:47:34.793836 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:34.793810    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-lkfn9"]
Nov 02 19:47:34.833805 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:34.833770    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b33f5f29-4f1a-4e70-a821-df5018260ad7-istio-token\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-lkfn9\" (UID: \"b33f5f29-4f1a-4e70-a821-df5018260ad7\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-lkfn9"
Nov 02 19:47:34.833960 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:34.833854    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5x72b\" (UniqueName: \"kubernetes.io/projected/b33f5f29-4f1a-4e70-a821-df5018260ad7-kube-api-access-5x72b\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-lkfn9\" (UID: \"b33f5f29-4f1a-4e70-a821-df5018260ad7\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-lkfn9"
Nov 02 19:47:34.916208 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:34.916127    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-lkfn9"
Nov 02 19:47:35.335845 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:35.335821    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-lkfn9"]
Nov 02 19:47:35.337369 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:47:35.337341    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb33f5f29_4f1a_4e70_a821_df5018260ad7.slice/crio-f15aab927713ea3c47fd5b59d6a96e9930426ccfc9af3923a46c54a4ff628c51 WatchSource:0}: Error finding container f15aab927713ea3c47fd5b59d6a96e9930426ccfc9af3923a46c54a4ff628c51: Status 404 returned error can't find the container with id f15aab927713ea3c47fd5b59d6a96e9930426ccfc9af3923a46c54a4ff628c51
Nov 02 19:47:35.339834 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:35.339797    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:47:35.339904 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:35.339880    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:47:35.339941 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:35.339925    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:47:36.302228 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:36.302194    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-lkfn9" event={"ID":"b33f5f29-4f1a-4e70-a821-df5018260ad7","Type":"ContainerStarted","Data":"7133490961b3a61bd67e108fd1984df415ac2839fef434550c4cbc2f85b9dc86"}
Nov 02 19:47:36.302408 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:36.302251    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-lkfn9" event={"ID":"b33f5f29-4f1a-4e70-a821-df5018260ad7","Type":"ContainerStarted","Data":"f15aab927713ea3c47fd5b59d6a96e9930426ccfc9af3923a46c54a4ff628c51"}
Nov 02 19:47:36.528693 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:36.528642    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-lkfn9" podStartSLOduration=2.528628512 podStartE2EDuration="2.528628512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:47:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:47:36.481885167 +0000 UTC m=+1163.088918965" watchObservedRunningTime="2026-11-02 19:47:36.528628512 +0000 UTC m=+1163.135662280"
Nov 02 19:47:36.917242 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:36.917211    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-lkfn9"
Nov 02 19:47:36.918957 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:36.918920    4463 patch_prober.go:29] interesting pod/e2e-worker-w4-8795bf-openshift-default-c67bc566c-lkfn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Nov 02 19:47:36.919120 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:36.918983    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-lkfn9" podUID="b33f5f29-4f1a-4e70-a821-df5018260ad7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Nov 02 19:47:37.916917 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:37.916880    4463 patch_prober.go:29] interesting pod/e2e-worker-w4-8795bf-openshift-default-c67bc566c-lkfn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Nov 02 19:47:37.917345 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:37.916940    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-lkfn9" podUID="b33f5f29-4f1a-4e70-a821-df5018260ad7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Nov 02 19:47:38.917118 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:38.917077    4463 patch_prober.go:29] interesting pod/e2e-worker-w4-8795bf-openshift-default-c67bc566c-lkfn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Nov 02 19:47:38.917507 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:38.917136    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-lkfn9" podUID="b33f5f29-4f1a-4e70-a821-df5018260ad7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Nov 02 19:47:39.916570 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:39.916535    4463 patch_prober.go:29] interesting pod/e2e-worker-w4-8795bf-openshift-default-c67bc566c-lkfn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Nov 02 19:47:39.916761 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:39.916594    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-lkfn9" podUID="b33f5f29-4f1a-4e70-a821-df5018260ad7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Nov 02 19:47:40.916776 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:40.916739    4463 patch_prober.go:29] interesting pod/e2e-worker-w4-8795bf-openshift-default-c67bc566c-lkfn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Nov 02 19:47:40.917205 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:40.916813    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-lkfn9" podUID="b33f5f29-4f1a-4e70-a821-df5018260ad7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Nov 02 19:47:41.916866 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:41.916827    4463 patch_prober.go:29] interesting pod/e2e-worker-w4-8795bf-openshift-default-c67bc566c-lkfn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Nov 02 19:47:41.917256 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:41.916898    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-lkfn9" podUID="b33f5f29-4f1a-4e70-a821-df5018260ad7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Nov 02 19:47:42.920558 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:42.920525    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-lkfn9"
Nov 02 19:47:42.921039 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:42.920911    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-lkfn9"
Nov 02 19:47:42.921473 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:42.921453    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-lkfn9"
Nov 02 19:47:49.711097 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:49.711061    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kglqn"]
Nov 02 19:47:49.716111 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:49.716090    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kglqn"
Nov 02 19:47:49.794963 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:49.794923    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c25aace-5449-4ed4-a07a-dd08f8eb4a4e-workload-socket\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-kglqn\" (UID: \"8c25aace-5449-4ed4-a07a-dd08f8eb4a4e\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kglqn"
Nov 02 19:47:49.795171 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:49.794985    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c25aace-5449-4ed4-a07a-dd08f8eb4a4e-istio-token\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-kglqn\" (UID: \"8c25aace-5449-4ed4-a07a-dd08f8eb4a4e\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kglqn"
Nov 02 19:47:49.795225 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:49.795169    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c25aace-5449-4ed4-a07a-dd08f8eb4a4e-workload-certs\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-kglqn\" (UID: \"8c25aace-5449-4ed4-a07a-dd08f8eb4a4e\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kglqn"
Nov 02 19:47:49.795273 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:49.795230    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c25aace-5449-4ed4-a07a-dd08f8eb4a4e-istiod-ca-cert\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-kglqn\" (UID: \"8c25aace-5449-4ed4-a07a-dd08f8eb4a4e\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kglqn"
Nov 02 19:47:49.795325 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:49.795305    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c25aace-5449-4ed4-a07a-dd08f8eb4a4e-credential-socket\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-kglqn\" (UID: \"8c25aace-5449-4ed4-a07a-dd08f8eb4a4e\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kglqn"
Nov 02 19:47:49.795360 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:49.795332    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c25aace-5449-4ed4-a07a-dd08f8eb4a4e-istio-data\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-kglqn\" (UID: \"8c25aace-5449-4ed4-a07a-dd08f8eb4a4e\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kglqn"
Nov 02 19:47:49.795399 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:49.795358    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c25aace-5449-4ed4-a07a-dd08f8eb4a4e-istio-podinfo\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-kglqn\" (UID: \"8c25aace-5449-4ed4-a07a-dd08f8eb4a4e\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kglqn"
Nov 02 19:47:49.795432 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:49.795398    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c25aace-5449-4ed4-a07a-dd08f8eb4a4e-istio-envoy\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-kglqn\" (UID: \"8c25aace-5449-4ed4-a07a-dd08f8eb4a4e\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kglqn"
Nov 02 19:47:49.795500 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:49.795488    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wd7r\" (UniqueName: \"kubernetes.io/projected/8c25aace-5449-4ed4-a07a-dd08f8eb4a4e-kube-api-access-6wd7r\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-kglqn\" (UID: \"8c25aace-5449-4ed4-a07a-dd08f8eb4a4e\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kglqn"
Nov 02 19:47:49.821459 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:49.821425    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kglqn"]
Nov 02 19:47:49.897161 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:49.897124    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c25aace-5449-4ed4-a07a-dd08f8eb4a4e-istio-token\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-kglqn\" (UID: \"8c25aace-5449-4ed4-a07a-dd08f8eb4a4e\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kglqn"
Nov 02 19:47:49.897349 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:49.897202    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c25aace-5449-4ed4-a07a-dd08f8eb4a4e-workload-certs\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-kglqn\" (UID: \"8c25aace-5449-4ed4-a07a-dd08f8eb4a4e\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kglqn"
Nov 02 19:47:49.897349 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:49.897241    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c25aace-5449-4ed4-a07a-dd08f8eb4a4e-istiod-ca-cert\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-kglqn\" (UID: \"8c25aace-5449-4ed4-a07a-dd08f8eb4a4e\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kglqn"
Nov 02 19:47:49.897349 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:49.897263    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c25aace-5449-4ed4-a07a-dd08f8eb4a4e-credential-socket\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-kglqn\" (UID: \"8c25aace-5449-4ed4-a07a-dd08f8eb4a4e\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kglqn"
Nov 02 19:47:49.897349 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:49.897280    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c25aace-5449-4ed4-a07a-dd08f8eb4a4e-istio-data\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-kglqn\" (UID: \"8c25aace-5449-4ed4-a07a-dd08f8eb4a4e\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kglqn"
Nov 02 19:47:49.897349 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:49.897306    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c25aace-5449-4ed4-a07a-dd08f8eb4a4e-istio-podinfo\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-kglqn\" (UID: \"8c25aace-5449-4ed4-a07a-dd08f8eb4a4e\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kglqn"
Nov 02 19:47:49.897349 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:49.897332    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c25aace-5449-4ed4-a07a-dd08f8eb4a4e-istio-envoy\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-kglqn\" (UID: \"8c25aace-5449-4ed4-a07a-dd08f8eb4a4e\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kglqn"
Nov 02 19:47:49.897645 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:49.897367    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wd7r\" (UniqueName: \"kubernetes.io/projected/8c25aace-5449-4ed4-a07a-dd08f8eb4a4e-kube-api-access-6wd7r\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-kglqn\" (UID: \"8c25aace-5449-4ed4-a07a-dd08f8eb4a4e\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kglqn"
Nov 02 19:47:49.897645 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:49.897480    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c25aace-5449-4ed4-a07a-dd08f8eb4a4e-workload-socket\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-kglqn\" (UID: \"8c25aace-5449-4ed4-a07a-dd08f8eb4a4e\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kglqn"
Nov 02 19:47:49.897743 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:49.897663    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c25aace-5449-4ed4-a07a-dd08f8eb4a4e-workload-certs\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-kglqn\" (UID: \"8c25aace-5449-4ed4-a07a-dd08f8eb4a4e\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kglqn"
Nov 02 19:47:49.897743 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:49.897732    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c25aace-5449-4ed4-a07a-dd08f8eb4a4e-credential-socket\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-kglqn\" (UID: \"8c25aace-5449-4ed4-a07a-dd08f8eb4a4e\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kglqn"
Nov 02 19:47:49.897849 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:49.897815    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c25aace-5449-4ed4-a07a-dd08f8eb4a4e-istio-data\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-kglqn\" (UID: \"8c25aace-5449-4ed4-a07a-dd08f8eb4a4e\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kglqn"
Nov 02 19:47:49.897910 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:49.897867    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c25aace-5449-4ed4-a07a-dd08f8eb4a4e-workload-socket\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-kglqn\" (UID: \"8c25aace-5449-4ed4-a07a-dd08f8eb4a4e\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kglqn"
Nov 02 19:47:49.898173 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:49.898149    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c25aace-5449-4ed4-a07a-dd08f8eb4a4e-istiod-ca-cert\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-kglqn\" (UID: \"8c25aace-5449-4ed4-a07a-dd08f8eb4a4e\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kglqn"
Nov 02 19:47:49.899690 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:49.899670    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c25aace-5449-4ed4-a07a-dd08f8eb4a4e-istio-envoy\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-kglqn\" (UID: \"8c25aace-5449-4ed4-a07a-dd08f8eb4a4e\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kglqn"
Nov 02 19:47:49.899847 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:49.899828    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c25aace-5449-4ed4-a07a-dd08f8eb4a4e-istio-podinfo\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-kglqn\" (UID: \"8c25aace-5449-4ed4-a07a-dd08f8eb4a4e\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kglqn"
Nov 02 19:47:49.950892 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:49.950857    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c25aace-5449-4ed4-a07a-dd08f8eb4a4e-istio-token\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-kglqn\" (UID: \"8c25aace-5449-4ed4-a07a-dd08f8eb4a4e\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kglqn"
Nov 02 19:47:49.951082 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:49.950946    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wd7r\" (UniqueName: \"kubernetes.io/projected/8c25aace-5449-4ed4-a07a-dd08f8eb4a4e-kube-api-access-6wd7r\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-kglqn\" (UID: \"8c25aace-5449-4ed4-a07a-dd08f8eb4a4e\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kglqn"
Nov 02 19:47:50.030059 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:50.029941    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kglqn"
Nov 02 19:47:50.448215 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:50.448175    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kglqn"]
Nov 02 19:47:50.449779 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:47:50.449748    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c25aace_5449_4ed4_a07a_dd08f8eb4a4e.slice/crio-ebd9fe405d5ef0e7ea06d91b0cf55bceb342f53ecd825f02d0a4aa22c789bfc9 WatchSource:0}: Error finding container ebd9fe405d5ef0e7ea06d91b0cf55bceb342f53ecd825f02d0a4aa22c789bfc9: Status 404 returned error can't find the container with id ebd9fe405d5ef0e7ea06d91b0cf55bceb342f53ecd825f02d0a4aa22c789bfc9
Nov 02 19:47:50.452277 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:50.452246    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:47:50.452350 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:50.452311    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:47:50.452350 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:50.452345    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:47:51.402655 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:51.402617    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kglqn" event={"ID":"8c25aace-5449-4ed4-a07a-dd08f8eb4a4e","Type":"ContainerStarted","Data":"77e675d907ae53155138da5ad3f3ffdd046434507979aff1ec567d11b4da7706"}
Nov 02 19:47:51.403065 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:51.402677    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kglqn" event={"ID":"8c25aace-5449-4ed4-a07a-dd08f8eb4a4e","Type":"ContainerStarted","Data":"ebd9fe405d5ef0e7ea06d91b0cf55bceb342f53ecd825f02d0a4aa22c789bfc9"}
Nov 02 19:47:51.629322 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:51.629272    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kglqn" podStartSLOduration=2.629259994 podStartE2EDuration="2.629259994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:47:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:47:51.581984507 +0000 UTC m=+1178.189018293" watchObservedRunningTime="2026-11-02 19:47:51.629259994 +0000 UTC m=+1178.236293762"
Nov 02 19:47:52.031215 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:52.031167    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kglqn"
Nov 02 19:47:52.032868 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:52.032833    4463 patch_prober.go:29] interesting pod/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kglqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 02 19:47:52.033009 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:52.032913    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kglqn" podUID="8c25aace-5449-4ed4-a07a-dd08f8eb4a4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 02 19:47:53.031041 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:53.030996    4463 patch_prober.go:29] interesting pod/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kglqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 02 19:47:53.031432 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:53.031069    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kglqn" podUID="8c25aace-5449-4ed4-a07a-dd08f8eb4a4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 02 19:47:54.030309 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:54.030266    4463 patch_prober.go:29] interesting pod/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kglqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 02 19:47:54.030495 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:54.030342    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kglqn" podUID="8c25aace-5449-4ed4-a07a-dd08f8eb4a4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 02 19:47:55.030653 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:55.030618    4463 patch_prober.go:29] interesting pod/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kglqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 02 19:47:55.031075 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:55.030670    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kglqn" podUID="8c25aace-5449-4ed4-a07a-dd08f8eb4a4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 02 19:47:55.427547 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:55.427511    4463 generic.go:356] "Generic (PLEG): container finished" podID="49094ceb-3222-4695-8186-7b87940fd7be" containerID="71153b7238a181475e3d3d47352e89e363755ce0c073511da98eda6ba084e78b" exitCode=0
Nov 02 19:47:55.427708 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:55.427580    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-f4vxc-qgtpp" event={"ID":"49094ceb-3222-4695-8186-7b87940fd7be","Type":"ContainerDied","Data":"71153b7238a181475e3d3d47352e89e363755ce0c073511da98eda6ba084e78b"}
Nov 02 19:47:56.030497 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:56.030463    4463 patch_prober.go:29] interesting pod/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kglqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 02 19:47:56.030661 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:56.030529    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kglqn" podUID="8c25aace-5449-4ed4-a07a-dd08f8eb4a4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 02 19:47:56.621573 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:56.621544    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-f4vxc-qgtpp"
Nov 02 19:47:56.704198 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:56.704125    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49094ceb-3222-4695-8186-7b87940fd7be-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49094ceb-3222-4695-8186-7b87940fd7be-service-ca-bundle\") pod \"49094ceb-3222-4695-8186-7b87940fd7be\" (UID: \"49094ceb-3222-4695-8186-7b87940fd7be\") "
Nov 02 19:47:56.704331 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:56.704272    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49094ceb-3222-4695-8186-7b87940fd7be-kube-api-access-qr227\" (UniqueName: \"kubernetes.io/projected/49094ceb-3222-4695-8186-7b87940fd7be-kube-api-access-qr227\") pod \"49094ceb-3222-4695-8186-7b87940fd7be\" (UID: \"49094ceb-3222-4695-8186-7b87940fd7be\") "
Nov 02 19:47:56.704391 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:56.704368    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49094ceb-3222-4695-8186-7b87940fd7be-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49094ceb-3222-4695-8186-7b87940fd7be-trusted-ca-bundle\") pod \"49094ceb-3222-4695-8186-7b87940fd7be\" (UID: \"49094ceb-3222-4695-8186-7b87940fd7be\") "
Nov 02 19:47:56.704449 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:56.704405    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49094ceb-3222-4695-8186-7b87940fd7be-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/49094ceb-3222-4695-8186-7b87940fd7be-archives-path\") pod \"49094ceb-3222-4695-8186-7b87940fd7be\" (UID: \"49094ceb-3222-4695-8186-7b87940fd7be\") "
Nov 02 19:47:56.704572 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:56.704543    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49094ceb-3222-4695-8186-7b87940fd7be-service-ca-bundle" pod "49094ceb-3222-4695-8186-7b87940fd7be" (UID: "49094ceb-3222-4695-8186-7b87940fd7be"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 19:47:56.704747 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:56.704717    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49094ceb-3222-4695-8186-7b87940fd7be-archives-path" pod "49094ceb-3222-4695-8186-7b87940fd7be" (UID: "49094ceb-3222-4695-8186-7b87940fd7be"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:47:56.704747 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:56.704734    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49094ceb-3222-4695-8186-7b87940fd7be-trusted-ca-bundle" pod "49094ceb-3222-4695-8186-7b87940fd7be" (UID: "49094ceb-3222-4695-8186-7b87940fd7be"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 19:47:56.705225 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:56.705201    4463 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49094ceb-3222-4695-8186-7b87940fd7be-trusted-ca-bundle\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:47:56.705225 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:56.705225    4463 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/49094ceb-3222-4695-8186-7b87940fd7be-archives-path\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:47:56.705377 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:56.705239    4463 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49094ceb-3222-4695-8186-7b87940fd7be-service-ca-bundle\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:47:56.706589 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:56.706569    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49094ceb-3222-4695-8186-7b87940fd7be-kube-api-access-qr227" pod "49094ceb-3222-4695-8186-7b87940fd7be" (UID: "49094ceb-3222-4695-8186-7b87940fd7be"). InnerVolumeSpecName "kube-api-access-qr227". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:47:56.806813 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:56.806781    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qr227\" (UniqueName: \"kubernetes.io/projected/49094ceb-3222-4695-8186-7b87940fd7be-kube-api-access-qr227\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:47:57.031132 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:57.031008    4463 patch_prober.go:29] interesting pod/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kglqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 02 19:47:57.031132 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:57.031104    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kglqn" podUID="8c25aace-5449-4ed4-a07a-dd08f8eb4a4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 02 19:47:57.441620 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:57.441576    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-f4vxc-qgtpp"
Nov 02 19:47:57.441813 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:57.441615    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-f4vxc-qgtpp" event={"ID":"49094ceb-3222-4695-8186-7b87940fd7be","Type":"ContainerDied","Data":"f2abc288f7a3d21778c5dc66db0e474eb55cb08a3bbe3697a9d9cf6106dd0efd"}
Nov 02 19:47:57.441813 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:57.441661    4463 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f2abc288f7a3d21778c5dc66db0e474eb55cb08a3bbe3697a9d9cf6106dd0efd"
Nov 02 19:47:58.035261 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:58.035229    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kglqn"
Nov 02 19:47:58.035666 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:58.035493    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kglqn"
Nov 02 19:47:58.036250 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:47:58.036227    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kglqn"
Nov 02 19:48:04.294129 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:48:04.294095    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 19:48:04.809604 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:04.809568    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z74mv"]
Nov 02 19:48:04.809829 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:04.809806    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z74mv" podUID="658d1f56-a03e-4a38-8911-229de81a3d29" containerName="istio-proxy" containerID="cri-o://70d39895a3906c23298558bb65c6c6c121df351bd839a8e20943d293db6d8e2b" gracePeriod=30
Nov 02 19:48:04.908449 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:04.908418    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-4c5vf"]
Nov 02 19:48:04.908966 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:04.908952    4463 cpu_manager.go:414] "containerMap: removing container" podUID="49094ceb-3222-4695-8186-7b87940fd7be" containerName="insights-gathering"
Nov 02 19:48:04.909045 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:04.908968    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49094ceb-3222-4695-8186-7b87940fd7be" containerName="insights-gathering"
Nov 02 19:48:04.909101 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:04.909095    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="49094ceb-3222-4695-8186-7b87940fd7be" containerName="insights-gathering"
Nov 02 19:48:04.912410 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:04.912393    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-hqc7w"]
Nov 02 19:48:04.912523 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:04.912505    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-4c5vf"
Nov 02 19:48:04.915824 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:04.915802    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-hqc7w"
Nov 02 19:48:04.930811 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:04.930788    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-4c5vf"]
Nov 02 19:48:04.959449 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:04.959393    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-z7nwb"]
Nov 02 19:48:04.964129 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:04.964106    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5hkwc"]
Nov 02 19:48:04.964277 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:04.964246    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-z7nwb"
Nov 02 19:48:04.967743 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:04.967719    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-26cfbe-openshift-default-dockercfg-msctx\""
Nov 02 19:48:04.967899 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:04.967848    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5hkwc"
Nov 02 19:48:04.971715 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:04.971692    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-hqc7w"]
Nov 02 19:48:05.005516 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.005484    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f6ba8f5-7ed9-4807-8435-6794197fee7c-credential-socket\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-hqc7w\" (UID: \"9f6ba8f5-7ed9-4807-8435-6794197fee7c\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-hqc7w"
Nov 02 19:48:05.005663 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.005539    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d288950b-84fb-4cef-bb78-0a713bcd14c0-workload-certs\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-5hkwc\" (UID: \"d288950b-84fb-4cef-bb78-0a713bcd14c0\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5hkwc"
Nov 02 19:48:05.005663 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.005644    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/630de598-0bde-4b3e-af80-b014ca9db567-istio-data\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-4c5vf\" (UID: \"630de598-0bde-4b3e-af80-b014ca9db567\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-4c5vf"
Nov 02 19:48:05.005795 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.005726    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d288950b-84fb-4cef-bb78-0a713bcd14c0-istio-podinfo\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-5hkwc\" (UID: \"d288950b-84fb-4cef-bb78-0a713bcd14c0\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5hkwc"
Nov 02 19:48:05.005852 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.005801    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d288950b-84fb-4cef-bb78-0a713bcd14c0-workload-socket\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-5hkwc\" (UID: \"d288950b-84fb-4cef-bb78-0a713bcd14c0\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5hkwc"
Nov 02 19:48:05.005907 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.005850    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d288950b-84fb-4cef-bb78-0a713bcd14c0-istio-data\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-5hkwc\" (UID: \"d288950b-84fb-4cef-bb78-0a713bcd14c0\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5hkwc"
Nov 02 19:48:05.005907 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.005875    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f40668dc-8c36-4c34-ae69-c87d8c71e338-istio-token\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-z7nwb\" (UID: \"f40668dc-8c36-4c34-ae69-c87d8c71e338\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-z7nwb"
Nov 02 19:48:05.006055 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.005920    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d288950b-84fb-4cef-bb78-0a713bcd14c0-credential-socket\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-5hkwc\" (UID: \"d288950b-84fb-4cef-bb78-0a713bcd14c0\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5hkwc"
Nov 02 19:48:05.006055 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.005947    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dfmc\" (UniqueName: \"kubernetes.io/projected/9f6ba8f5-7ed9-4807-8435-6794197fee7c-kube-api-access-8dfmc\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-hqc7w\" (UID: \"9f6ba8f5-7ed9-4807-8435-6794197fee7c\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-hqc7w"
Nov 02 19:48:05.006055 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.005977    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f40668dc-8c36-4c34-ae69-c87d8c71e338-workload-socket\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-z7nwb\" (UID: \"f40668dc-8c36-4c34-ae69-c87d8c71e338\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-z7nwb"
Nov 02 19:48:05.006055 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.005999    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68hrw\" (UniqueName: \"kubernetes.io/projected/f40668dc-8c36-4c34-ae69-c87d8c71e338-kube-api-access-68hrw\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-z7nwb\" (UID: \"f40668dc-8c36-4c34-ae69-c87d8c71e338\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-z7nwb"
Nov 02 19:48:05.006055 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.006052    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/630de598-0bde-4b3e-af80-b014ca9db567-istio-token\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-4c5vf\" (UID: \"630de598-0bde-4b3e-af80-b014ca9db567\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-4c5vf"
Nov 02 19:48:05.006353 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.006088    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f40668dc-8c36-4c34-ae69-c87d8c71e338-istio-envoy\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-z7nwb\" (UID: \"f40668dc-8c36-4c34-ae69-c87d8c71e338\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-z7nwb"
Nov 02 19:48:05.006353 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.006116    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/630de598-0bde-4b3e-af80-b014ca9db567-istiod-ca-cert\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-4c5vf\" (UID: \"630de598-0bde-4b3e-af80-b014ca9db567\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-4c5vf"
Nov 02 19:48:05.006353 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.006150    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f6ba8f5-7ed9-4807-8435-6794197fee7c-istio-envoy\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-hqc7w\" (UID: \"9f6ba8f5-7ed9-4807-8435-6794197fee7c\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-hqc7w"
Nov 02 19:48:05.006353 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.006186    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f6ba8f5-7ed9-4807-8435-6794197fee7c-istiod-ca-cert\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-hqc7w\" (UID: \"9f6ba8f5-7ed9-4807-8435-6794197fee7c\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-hqc7w"
Nov 02 19:48:05.006353 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.006213    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f40668dc-8c36-4c34-ae69-c87d8c71e338-istiod-ca-cert\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-z7nwb\" (UID: \"f40668dc-8c36-4c34-ae69-c87d8c71e338\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-z7nwb"
Nov 02 19:48:05.006353 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.006255    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f40668dc-8c36-4c34-ae69-c87d8c71e338-workload-certs\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-z7nwb\" (UID: \"f40668dc-8c36-4c34-ae69-c87d8c71e338\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-z7nwb"
Nov 02 19:48:05.006353 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.006281    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f6ba8f5-7ed9-4807-8435-6794197fee7c-istio-data\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-hqc7w\" (UID: \"9f6ba8f5-7ed9-4807-8435-6794197fee7c\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-hqc7w"
Nov 02 19:48:05.006353 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.006303    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d288950b-84fb-4cef-bb78-0a713bcd14c0-istio-envoy\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-5hkwc\" (UID: \"d288950b-84fb-4cef-bb78-0a713bcd14c0\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5hkwc"
Nov 02 19:48:05.006761 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.006360    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w68q\" (UniqueName: \"kubernetes.io/projected/630de598-0bde-4b3e-af80-b014ca9db567-kube-api-access-5w68q\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-4c5vf\" (UID: \"630de598-0bde-4b3e-af80-b014ca9db567\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-4c5vf"
Nov 02 19:48:05.006761 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.006393    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f6ba8f5-7ed9-4807-8435-6794197fee7c-workload-socket\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-hqc7w\" (UID: \"9f6ba8f5-7ed9-4807-8435-6794197fee7c\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-hqc7w"
Nov 02 19:48:05.006761 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.006417    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d288950b-84fb-4cef-bb78-0a713bcd14c0-istio-token\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-5hkwc\" (UID: \"d288950b-84fb-4cef-bb78-0a713bcd14c0\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5hkwc"
Nov 02 19:48:05.006761 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.006468    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/630de598-0bde-4b3e-af80-b014ca9db567-workload-socket\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-4c5vf\" (UID: \"630de598-0bde-4b3e-af80-b014ca9db567\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-4c5vf"
Nov 02 19:48:05.006761 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.006492    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/630de598-0bde-4b3e-af80-b014ca9db567-istio-envoy\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-4c5vf\" (UID: \"630de598-0bde-4b3e-af80-b014ca9db567\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-4c5vf"
Nov 02 19:48:05.006761 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.006520    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjzb8\" (UniqueName: \"kubernetes.io/projected/d288950b-84fb-4cef-bb78-0a713bcd14c0-kube-api-access-zjzb8\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-5hkwc\" (UID: \"d288950b-84fb-4cef-bb78-0a713bcd14c0\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5hkwc"
Nov 02 19:48:05.006761 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.006555    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f40668dc-8c36-4c34-ae69-c87d8c71e338-istio-data\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-z7nwb\" (UID: \"f40668dc-8c36-4c34-ae69-c87d8c71e338\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-z7nwb"
Nov 02 19:48:05.006761 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.006592    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f6ba8f5-7ed9-4807-8435-6794197fee7c-workload-certs\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-hqc7w\" (UID: \"9f6ba8f5-7ed9-4807-8435-6794197fee7c\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-hqc7w"
Nov 02 19:48:05.006761 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.006615    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f40668dc-8c36-4c34-ae69-c87d8c71e338-istio-podinfo\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-z7nwb\" (UID: \"f40668dc-8c36-4c34-ae69-c87d8c71e338\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-z7nwb"
Nov 02 19:48:05.006761 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.006651    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f6ba8f5-7ed9-4807-8435-6794197fee7c-istio-podinfo\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-hqc7w\" (UID: \"9f6ba8f5-7ed9-4807-8435-6794197fee7c\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-hqc7w"
Nov 02 19:48:05.006761 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.006688    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/630de598-0bde-4b3e-af80-b014ca9db567-credential-socket\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-4c5vf\" (UID: \"630de598-0bde-4b3e-af80-b014ca9db567\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-4c5vf"
Nov 02 19:48:05.006761 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.006713    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f6ba8f5-7ed9-4807-8435-6794197fee7c-istio-token\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-hqc7w\" (UID: \"9f6ba8f5-7ed9-4807-8435-6794197fee7c\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-hqc7w"
Nov 02 19:48:05.006761 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.006759    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d288950b-84fb-4cef-bb78-0a713bcd14c0-istiod-ca-cert\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-5hkwc\" (UID: \"d288950b-84fb-4cef-bb78-0a713bcd14c0\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5hkwc"
Nov 02 19:48:05.007467 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.006787    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f40668dc-8c36-4c34-ae69-c87d8c71e338-credential-socket\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-z7nwb\" (UID: \"f40668dc-8c36-4c34-ae69-c87d8c71e338\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-z7nwb"
Nov 02 19:48:05.007467 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.006812    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/630de598-0bde-4b3e-af80-b014ca9db567-workload-certs\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-4c5vf\" (UID: \"630de598-0bde-4b3e-af80-b014ca9db567\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-4c5vf"
Nov 02 19:48:05.007467 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.006843    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/630de598-0bde-4b3e-af80-b014ca9db567-istio-podinfo\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-4c5vf\" (UID: \"630de598-0bde-4b3e-af80-b014ca9db567\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-4c5vf"
Nov 02 19:48:05.043250 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.043220    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-z7nwb"]
Nov 02 19:48:05.058006 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.057978    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5hkwc"]
Nov 02 19:48:05.108383 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.108323    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f6ba8f5-7ed9-4807-8435-6794197fee7c-istio-podinfo\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-hqc7w\" (UID: \"9f6ba8f5-7ed9-4807-8435-6794197fee7c\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-hqc7w"
Nov 02 19:48:05.108383 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.108361    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/630de598-0bde-4b3e-af80-b014ca9db567-credential-socket\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-4c5vf\" (UID: \"630de598-0bde-4b3e-af80-b014ca9db567\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-4c5vf"
Nov 02 19:48:05.108549 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.108388    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f6ba8f5-7ed9-4807-8435-6794197fee7c-istio-token\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-hqc7w\" (UID: \"9f6ba8f5-7ed9-4807-8435-6794197fee7c\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-hqc7w"
Nov 02 19:48:05.108549 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.108425    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d288950b-84fb-4cef-bb78-0a713bcd14c0-istiod-ca-cert\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-5hkwc\" (UID: \"d288950b-84fb-4cef-bb78-0a713bcd14c0\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5hkwc"
Nov 02 19:48:05.108549 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.108453    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f40668dc-8c36-4c34-ae69-c87d8c71e338-credential-socket\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-z7nwb\" (UID: \"f40668dc-8c36-4c34-ae69-c87d8c71e338\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-z7nwb"
Nov 02 19:48:05.108549 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.108476    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/630de598-0bde-4b3e-af80-b014ca9db567-workload-certs\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-4c5vf\" (UID: \"630de598-0bde-4b3e-af80-b014ca9db567\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-4c5vf"
Nov 02 19:48:05.108759 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.108634    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/630de598-0bde-4b3e-af80-b014ca9db567-istio-podinfo\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-4c5vf\" (UID: \"630de598-0bde-4b3e-af80-b014ca9db567\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-4c5vf"
Nov 02 19:48:05.108759 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.108684    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f6ba8f5-7ed9-4807-8435-6794197fee7c-credential-socket\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-hqc7w\" (UID: \"9f6ba8f5-7ed9-4807-8435-6794197fee7c\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-hqc7w"
Nov 02 19:48:05.108759 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.108725    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d288950b-84fb-4cef-bb78-0a713bcd14c0-workload-certs\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-5hkwc\" (UID: \"d288950b-84fb-4cef-bb78-0a713bcd14c0\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5hkwc"
Nov 02 19:48:05.108759 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.108741    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/630de598-0bde-4b3e-af80-b014ca9db567-credential-socket\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-4c5vf\" (UID: \"630de598-0bde-4b3e-af80-b014ca9db567\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-4c5vf"
Nov 02 19:48:05.108759 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.108750    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/630de598-0bde-4b3e-af80-b014ca9db567-istio-data\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-4c5vf\" (UID: \"630de598-0bde-4b3e-af80-b014ca9db567\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-4c5vf"
Nov 02 19:48:05.109033 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.108803    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d288950b-84fb-4cef-bb78-0a713bcd14c0-istio-podinfo\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-5hkwc\" (UID: \"d288950b-84fb-4cef-bb78-0a713bcd14c0\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5hkwc"
Nov 02 19:48:05.109033 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.108830    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f40668dc-8c36-4c34-ae69-c87d8c71e338-credential-socket\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-z7nwb\" (UID: \"f40668dc-8c36-4c34-ae69-c87d8c71e338\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-z7nwb"
Nov 02 19:48:05.109033 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.108897    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d288950b-84fb-4cef-bb78-0a713bcd14c0-workload-socket\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-5hkwc\" (UID: \"d288950b-84fb-4cef-bb78-0a713bcd14c0\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5hkwc"
Nov 02 19:48:05.109033 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.108929    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/630de598-0bde-4b3e-af80-b014ca9db567-workload-certs\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-4c5vf\" (UID: \"630de598-0bde-4b3e-af80-b014ca9db567\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-4c5vf"
Nov 02 19:48:05.109033 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.108954    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d288950b-84fb-4cef-bb78-0a713bcd14c0-istio-data\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-5hkwc\" (UID: \"d288950b-84fb-4cef-bb78-0a713bcd14c0\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5hkwc"
Nov 02 19:48:05.109302 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.109033    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/630de598-0bde-4b3e-af80-b014ca9db567-istio-data\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-4c5vf\" (UID: \"630de598-0bde-4b3e-af80-b014ca9db567\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-4c5vf"
Nov 02 19:48:05.109302 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.109266    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d288950b-84fb-4cef-bb78-0a713bcd14c0-istio-data\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-5hkwc\" (UID: \"d288950b-84fb-4cef-bb78-0a713bcd14c0\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5hkwc"
Nov 02 19:48:05.109402 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.109309    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f40668dc-8c36-4c34-ae69-c87d8c71e338-istio-token\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-z7nwb\" (UID: \"f40668dc-8c36-4c34-ae69-c87d8c71e338\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-z7nwb"
Nov 02 19:48:05.109402 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.109329    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f6ba8f5-7ed9-4807-8435-6794197fee7c-credential-socket\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-hqc7w\" (UID: \"9f6ba8f5-7ed9-4807-8435-6794197fee7c\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-hqc7w"
Nov 02 19:48:05.109402 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.109342    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d288950b-84fb-4cef-bb78-0a713bcd14c0-workload-certs\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-5hkwc\" (UID: \"d288950b-84fb-4cef-bb78-0a713bcd14c0\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5hkwc"
Nov 02 19:48:05.109402 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.109347    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d288950b-84fb-4cef-bb78-0a713bcd14c0-istiod-ca-cert\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-5hkwc\" (UID: \"d288950b-84fb-4cef-bb78-0a713bcd14c0\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5hkwc"
Nov 02 19:48:05.109402 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.109370    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d288950b-84fb-4cef-bb78-0a713bcd14c0-credential-socket\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-5hkwc\" (UID: \"d288950b-84fb-4cef-bb78-0a713bcd14c0\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5hkwc"
Nov 02 19:48:05.109629 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.109411    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dfmc\" (UniqueName: \"kubernetes.io/projected/9f6ba8f5-7ed9-4807-8435-6794197fee7c-kube-api-access-8dfmc\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-hqc7w\" (UID: \"9f6ba8f5-7ed9-4807-8435-6794197fee7c\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-hqc7w"
Nov 02 19:48:05.109629 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.109447    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f40668dc-8c36-4c34-ae69-c87d8c71e338-workload-socket\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-z7nwb\" (UID: \"f40668dc-8c36-4c34-ae69-c87d8c71e338\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-z7nwb"
Nov 02 19:48:05.109629 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.109471    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68hrw\" (UniqueName: \"kubernetes.io/projected/f40668dc-8c36-4c34-ae69-c87d8c71e338-kube-api-access-68hrw\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-z7nwb\" (UID: \"f40668dc-8c36-4c34-ae69-c87d8c71e338\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-z7nwb"
Nov 02 19:48:05.109629 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.109416    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d288950b-84fb-4cef-bb78-0a713bcd14c0-workload-socket\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-5hkwc\" (UID: \"d288950b-84fb-4cef-bb78-0a713bcd14c0\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5hkwc"
Nov 02 19:48:05.109629 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.109502    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/630de598-0bde-4b3e-af80-b014ca9db567-istio-token\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-4c5vf\" (UID: \"630de598-0bde-4b3e-af80-b014ca9db567\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-4c5vf"
Nov 02 19:48:05.109629 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.109546    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f40668dc-8c36-4c34-ae69-c87d8c71e338-istio-envoy\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-z7nwb\" (UID: \"f40668dc-8c36-4c34-ae69-c87d8c71e338\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-z7nwb"
Nov 02 19:48:05.109629 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.109573    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/630de598-0bde-4b3e-af80-b014ca9db567-istiod-ca-cert\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-4c5vf\" (UID: \"630de598-0bde-4b3e-af80-b014ca9db567\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-4c5vf"
Nov 02 19:48:05.109629 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.109610    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f6ba8f5-7ed9-4807-8435-6794197fee7c-istio-envoy\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-hqc7w\" (UID: \"9f6ba8f5-7ed9-4807-8435-6794197fee7c\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-hqc7w"
Nov 02 19:48:05.109629 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.109613    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d288950b-84fb-4cef-bb78-0a713bcd14c0-credential-socket\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-5hkwc\" (UID: \"d288950b-84fb-4cef-bb78-0a713bcd14c0\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5hkwc"
Nov 02 19:48:05.110121 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.109661    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f6ba8f5-7ed9-4807-8435-6794197fee7c-istiod-ca-cert\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-hqc7w\" (UID: \"9f6ba8f5-7ed9-4807-8435-6794197fee7c\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-hqc7w"
Nov 02 19:48:05.110121 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.109695    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f40668dc-8c36-4c34-ae69-c87d8c71e338-istiod-ca-cert\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-z7nwb\" (UID: \"f40668dc-8c36-4c34-ae69-c87d8c71e338\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-z7nwb"
Nov 02 19:48:05.110121 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.109735    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f40668dc-8c36-4c34-ae69-c87d8c71e338-workload-certs\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-z7nwb\" (UID: \"f40668dc-8c36-4c34-ae69-c87d8c71e338\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-z7nwb"
Nov 02 19:48:05.110121 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.109761    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f6ba8f5-7ed9-4807-8435-6794197fee7c-istio-data\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-hqc7w\" (UID: \"9f6ba8f5-7ed9-4807-8435-6794197fee7c\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-hqc7w"
Nov 02 19:48:05.110121 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.109775    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f40668dc-8c36-4c34-ae69-c87d8c71e338-workload-socket\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-z7nwb\" (UID: \"f40668dc-8c36-4c34-ae69-c87d8c71e338\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-z7nwb"
Nov 02 19:48:05.110121 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.109788    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d288950b-84fb-4cef-bb78-0a713bcd14c0-istio-envoy\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-5hkwc\" (UID: \"d288950b-84fb-4cef-bb78-0a713bcd14c0\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5hkwc"
Nov 02 19:48:05.110121 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.109865    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w68q\" (UniqueName: \"kubernetes.io/projected/630de598-0bde-4b3e-af80-b014ca9db567-kube-api-access-5w68q\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-4c5vf\" (UID: \"630de598-0bde-4b3e-af80-b014ca9db567\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-4c5vf"
Nov 02 19:48:05.110121 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.109902    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f6ba8f5-7ed9-4807-8435-6794197fee7c-workload-socket\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-hqc7w\" (UID: \"9f6ba8f5-7ed9-4807-8435-6794197fee7c\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-hqc7w"
Nov 02 19:48:05.110121 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.109929    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d288950b-84fb-4cef-bb78-0a713bcd14c0-istio-token\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-5hkwc\" (UID: \"d288950b-84fb-4cef-bb78-0a713bcd14c0\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5hkwc"
Nov 02 19:48:05.110121 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.109985    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/630de598-0bde-4b3e-af80-b014ca9db567-workload-socket\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-4c5vf\" (UID: \"630de598-0bde-4b3e-af80-b014ca9db567\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-4c5vf"
Nov 02 19:48:05.110121 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.110039    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/630de598-0bde-4b3e-af80-b014ca9db567-istio-envoy\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-4c5vf\" (UID: \"630de598-0bde-4b3e-af80-b014ca9db567\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-4c5vf"
Nov 02 19:48:05.110121 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.110072    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjzb8\" (UniqueName: \"kubernetes.io/projected/d288950b-84fb-4cef-bb78-0a713bcd14c0-kube-api-access-zjzb8\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-5hkwc\" (UID: \"d288950b-84fb-4cef-bb78-0a713bcd14c0\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5hkwc"
Nov 02 19:48:05.110121 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.110107    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f40668dc-8c36-4c34-ae69-c87d8c71e338-istio-data\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-z7nwb\" (UID: \"f40668dc-8c36-4c34-ae69-c87d8c71e338\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-z7nwb"
Nov 02 19:48:05.110782 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.110145    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f6ba8f5-7ed9-4807-8435-6794197fee7c-workload-certs\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-hqc7w\" (UID: \"9f6ba8f5-7ed9-4807-8435-6794197fee7c\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-hqc7w"
Nov 02 19:48:05.110782 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.110173    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f40668dc-8c36-4c34-ae69-c87d8c71e338-istio-podinfo\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-z7nwb\" (UID: \"f40668dc-8c36-4c34-ae69-c87d8c71e338\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-z7nwb"
Nov 02 19:48:05.110782 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.110326    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/630de598-0bde-4b3e-af80-b014ca9db567-istiod-ca-cert\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-4c5vf\" (UID: \"630de598-0bde-4b3e-af80-b014ca9db567\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-4c5vf"
Nov 02 19:48:05.110782 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.110383    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f6ba8f5-7ed9-4807-8435-6794197fee7c-istiod-ca-cert\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-hqc7w\" (UID: \"9f6ba8f5-7ed9-4807-8435-6794197fee7c\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-hqc7w"
Nov 02 19:48:05.110782 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.110591    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f40668dc-8c36-4c34-ae69-c87d8c71e338-workload-certs\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-z7nwb\" (UID: \"f40668dc-8c36-4c34-ae69-c87d8c71e338\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-z7nwb"
Nov 02 19:48:05.110782 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.110702    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/630de598-0bde-4b3e-af80-b014ca9db567-workload-socket\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-4c5vf\" (UID: \"630de598-0bde-4b3e-af80-b014ca9db567\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-4c5vf"
Nov 02 19:48:05.111224 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.111199    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f6ba8f5-7ed9-4807-8435-6794197fee7c-workload-socket\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-hqc7w\" (UID: \"9f6ba8f5-7ed9-4807-8435-6794197fee7c\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-hqc7w"
Nov 02 19:48:05.111622 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.111599    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f6ba8f5-7ed9-4807-8435-6794197fee7c-istio-data\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-hqc7w\" (UID: \"9f6ba8f5-7ed9-4807-8435-6794197fee7c\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-hqc7w"
Nov 02 19:48:05.111724 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.111604    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f6ba8f5-7ed9-4807-8435-6794197fee7c-istio-podinfo\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-hqc7w\" (UID: \"9f6ba8f5-7ed9-4807-8435-6794197fee7c\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-hqc7w"
Nov 02 19:48:05.111923 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.111897    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f40668dc-8c36-4c34-ae69-c87d8c71e338-istio-data\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-z7nwb\" (UID: \"f40668dc-8c36-4c34-ae69-c87d8c71e338\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-z7nwb"
Nov 02 19:48:05.112007 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.111900    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f6ba8f5-7ed9-4807-8435-6794197fee7c-workload-certs\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-hqc7w\" (UID: \"9f6ba8f5-7ed9-4807-8435-6794197fee7c\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-hqc7w"
Nov 02 19:48:05.112007 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.111912    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f40668dc-8c36-4c34-ae69-c87d8c71e338-istiod-ca-cert\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-z7nwb\" (UID: \"f40668dc-8c36-4c34-ae69-c87d8c71e338\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-z7nwb"
Nov 02 19:48:05.112369 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.112345    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d288950b-84fb-4cef-bb78-0a713bcd14c0-istio-podinfo\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-5hkwc\" (UID: \"d288950b-84fb-4cef-bb78-0a713bcd14c0\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5hkwc"
Nov 02 19:48:05.112525 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.112503    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d288950b-84fb-4cef-bb78-0a713bcd14c0-istio-envoy\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-5hkwc\" (UID: \"d288950b-84fb-4cef-bb78-0a713bcd14c0\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5hkwc"
Nov 02 19:48:05.112966 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.112942    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f40668dc-8c36-4c34-ae69-c87d8c71e338-istio-podinfo\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-z7nwb\" (UID: \"f40668dc-8c36-4c34-ae69-c87d8c71e338\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-z7nwb"
Nov 02 19:48:05.113084 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.113064    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f40668dc-8c36-4c34-ae69-c87d8c71e338-istio-envoy\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-z7nwb\" (UID: \"f40668dc-8c36-4c34-ae69-c87d8c71e338\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-z7nwb"
Nov 02 19:48:05.113984 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.113960    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/630de598-0bde-4b3e-af80-b014ca9db567-istio-podinfo\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-4c5vf\" (UID: \"630de598-0bde-4b3e-af80-b014ca9db567\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-4c5vf"
Nov 02 19:48:05.114148 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.114125    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/630de598-0bde-4b3e-af80-b014ca9db567-istio-envoy\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-4c5vf\" (UID: \"630de598-0bde-4b3e-af80-b014ca9db567\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-4c5vf"
Nov 02 19:48:05.114192 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.114183    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f6ba8f5-7ed9-4807-8435-6794197fee7c-istio-envoy\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-hqc7w\" (UID: \"9f6ba8f5-7ed9-4807-8435-6794197fee7c\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-hqc7w"
Nov 02 19:48:05.162788 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.162765    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f6ba8f5-7ed9-4807-8435-6794197fee7c-istio-token\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-hqc7w\" (UID: \"9f6ba8f5-7ed9-4807-8435-6794197fee7c\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-hqc7w"
Nov 02 19:48:05.163820 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.163794    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/630de598-0bde-4b3e-af80-b014ca9db567-istio-token\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-4c5vf\" (UID: \"630de598-0bde-4b3e-af80-b014ca9db567\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-4c5vf"
Nov 02 19:48:05.166227 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.166206    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d288950b-84fb-4cef-bb78-0a713bcd14c0-istio-token\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-5hkwc\" (UID: \"d288950b-84fb-4cef-bb78-0a713bcd14c0\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5hkwc"
Nov 02 19:48:05.166457 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.166433    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f40668dc-8c36-4c34-ae69-c87d8c71e338-istio-token\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-z7nwb\" (UID: \"f40668dc-8c36-4c34-ae69-c87d8c71e338\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-z7nwb"
Nov 02 19:48:05.166608 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.166480    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68hrw\" (UniqueName: \"kubernetes.io/projected/f40668dc-8c36-4c34-ae69-c87d8c71e338-kube-api-access-68hrw\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-z7nwb\" (UID: \"f40668dc-8c36-4c34-ae69-c87d8c71e338\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-z7nwb"
Nov 02 19:48:05.166608 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.166571    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w68q\" (UniqueName: \"kubernetes.io/projected/630de598-0bde-4b3e-af80-b014ca9db567-kube-api-access-5w68q\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-4c5vf\" (UID: \"630de598-0bde-4b3e-af80-b014ca9db567\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-4c5vf"
Nov 02 19:48:05.166883 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.166864    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dfmc\" (UniqueName: \"kubernetes.io/projected/9f6ba8f5-7ed9-4807-8435-6794197fee7c-kube-api-access-8dfmc\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-hqc7w\" (UID: \"9f6ba8f5-7ed9-4807-8435-6794197fee7c\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-hqc7w"
Nov 02 19:48:05.167000 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.166983    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjzb8\" (UniqueName: \"kubernetes.io/projected/d288950b-84fb-4cef-bb78-0a713bcd14c0-kube-api-access-zjzb8\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-5hkwc\" (UID: \"d288950b-84fb-4cef-bb78-0a713bcd14c0\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5hkwc"
Nov 02 19:48:05.227535 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.227510    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-4c5vf"
Nov 02 19:48:05.234312 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.234292    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-hqc7w"
Nov 02 19:48:05.286382 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.286341    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-z7nwb"
Nov 02 19:48:05.291898 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.291871    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5hkwc"
Nov 02 19:48:05.603037 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.602986    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-4c5vf"]
Nov 02 19:48:05.603428 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:48:05.603322    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod630de598_0bde_4b3e_af80_b014ca9db567.slice/crio-f566e9f2747506766595b6f202f0ef3c498a29e5ce1b0ac2eba1d2d3a89d395f WatchSource:0}: Error finding container f566e9f2747506766595b6f202f0ef3c498a29e5ce1b0ac2eba1d2d3a89d395f: Status 404 returned error can't find the container with id f566e9f2747506766595b6f202f0ef3c498a29e5ce1b0ac2eba1d2d3a89d395f
Nov 02 19:48:05.605665 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.605631    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:48:05.605785 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.605700    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:48:05.605785 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.605743    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:48:05.665816 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.665782    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-hqc7w"]
Nov 02 19:48:05.669476 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:48:05.669447    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f6ba8f5_7ed9_4807_8435_6794197fee7c.slice/crio-c1f1b84e23a42ad93320f827d3bcc0bfa0315a6032390bf0b0a986a1a918e8a1 WatchSource:0}: Error finding container c1f1b84e23a42ad93320f827d3bcc0bfa0315a6032390bf0b0a986a1a918e8a1: Status 404 returned error can't find the container with id c1f1b84e23a42ad93320f827d3bcc0bfa0315a6032390bf0b0a986a1a918e8a1
Nov 02 19:48:05.671936 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.671910    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:48:05.672003 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.671987    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:48:05.672090 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.672043    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:48:05.708797 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.708771    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5hkwc"]
Nov 02 19:48:05.715767 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.715215    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-z7nwb"]
Nov 02 19:48:05.720051 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.720004    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:48:05.720198 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.720093    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:48:05.720198 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.720129    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:48:05.721178 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.720700    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:48:05.721178 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.720773    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:48:05.721178 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:05.720877    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:48:06.519111 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:06.519062    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-hqc7w" event={"ID":"9f6ba8f5-7ed9-4807-8435-6794197fee7c","Type":"ContainerStarted","Data":"3ac50a5aa4058ba8746476e950d0eeae572205867e243451c92c523c21a29942"}
Nov 02 19:48:06.519361 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:06.519168    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-hqc7w" event={"ID":"9f6ba8f5-7ed9-4807-8435-6794197fee7c","Type":"ContainerStarted","Data":"c1f1b84e23a42ad93320f827d3bcc0bfa0315a6032390bf0b0a986a1a918e8a1"}
Nov 02 19:48:06.522449 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:06.522358    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5hkwc" event={"ID":"d288950b-84fb-4cef-bb78-0a713bcd14c0","Type":"ContainerStarted","Data":"9b44e60b60bba2de29171381c72094b7a1d37cc7a93f575d15756f7c4b1f6ad1"}
Nov 02 19:48:06.522449 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:06.522412    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5hkwc" event={"ID":"d288950b-84fb-4cef-bb78-0a713bcd14c0","Type":"ContainerStarted","Data":"3dfa420e211f7a5f23c4d4198766667fdc88c1a5875287fb2ffe1b9f2675d4a1"}
Nov 02 19:48:06.524787 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:06.524755    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-4c5vf" event={"ID":"630de598-0bde-4b3e-af80-b014ca9db567","Type":"ContainerStarted","Data":"f63a3558901bb7ac91366545060140770db4cd271caad3969940a232c422d1fc"}
Nov 02 19:48:06.524918 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:06.524809    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-4c5vf" event={"ID":"630de598-0bde-4b3e-af80-b014ca9db567","Type":"ContainerStarted","Data":"f566e9f2747506766595b6f202f0ef3c498a29e5ce1b0ac2eba1d2d3a89d395f"}
Nov 02 19:48:06.526766 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:06.526743    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-z7nwb" event={"ID":"f40668dc-8c36-4c34-ae69-c87d8c71e338","Type":"ContainerStarted","Data":"1753e7925ff7ea4b4e8379df3666fb56c09ea49085d3ee5899a6f36484615ba2"}
Nov 02 19:48:06.526908 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:06.526895    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-z7nwb" event={"ID":"f40668dc-8c36-4c34-ae69-c87d8c71e338","Type":"ContainerStarted","Data":"f5e99d5d0348109dbb82ffde362d4d12f538bbc335af21e81e7e8db7aac32471"}
Nov 02 19:48:06.672405 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:06.672342    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-hqc7w" podStartSLOduration=2.6723163469999998 podStartE2EDuration="2.672316347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:48:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:48:06.6487828 +0000 UTC m=+1193.255816580" watchObservedRunningTime="2026-11-02 19:48:06.672316347 +0000 UTC m=+1193.279350117"
Nov 02 19:48:06.901097 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:06.892898    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-z7nwb" podStartSLOduration=2.892878776 podStartE2EDuration="2.892878776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:48:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:48:06.770350355 +0000 UTC m=+1193.377384128" watchObservedRunningTime="2026-11-02 19:48:06.892878776 +0000 UTC m=+1193.499912546"
Nov 02 19:48:06.914059 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:06.910991    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-4c5vf" podStartSLOduration=2.910969149 podStartE2EDuration="2.910969149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:48:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:48:06.891626761 +0000 UTC m=+1193.498660557" watchObservedRunningTime="2026-11-02 19:48:06.910969149 +0000 UTC m=+1193.518002919"
Nov 02 19:48:07.023037 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:07.022270    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5hkwc" podStartSLOduration=3.022245451 podStartE2EDuration="3.022245451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:48:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:48:07.016694461 +0000 UTC m=+1193.623728231" watchObservedRunningTime="2026-11-02 19:48:07.022245451 +0000 UTC m=+1193.629279222"
Nov 02 19:48:07.247100 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:07.247065    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-hqc7w"
Nov 02 19:48:07.247386 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:07.247369    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-4c5vf"
Nov 02 19:48:07.247548 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:07.247529    4463 patch_prober.go:29] interesting pod/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-4c5vf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 02 19:48:07.247675 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:07.247654    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-4c5vf" podUID="630de598-0bde-4b3e-af80-b014ca9db567" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 02 19:48:07.248073 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:07.247751    4463 patch_prober.go:29] interesting pod/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-hqc7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 02 19:48:07.248229 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:07.248208    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-hqc7w" podUID="9f6ba8f5-7ed9-4807-8435-6794197fee7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 02 19:48:07.288776 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:07.288742    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-z7nwb"
Nov 02 19:48:07.289169 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:07.289149    4463 patch_prober.go:29] interesting pod/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-z7nwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Nov 02 19:48:07.289341 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:07.289318    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-z7nwb" podUID="f40668dc-8c36-4c34-ae69-c87d8c71e338" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Nov 02 19:48:07.295744 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:07.295715    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5hkwc"
Nov 02 19:48:07.296045 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:07.296006    4463 patch_prober.go:29] interesting pod/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5hkwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Nov 02 19:48:07.296248 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:07.296221    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5hkwc" podUID="d288950b-84fb-4cef-bb78-0a713bcd14c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Nov 02 19:48:07.664234 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:07.664118    4463 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-26cfbe/simulator-w1-26cfbe-kserve-74c5578fbd-t79hq" podUID="50dc76f4-aebd-41f4-9f23-799ddc8c70f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:48:07.877033 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:07.864106    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-26cfbe/simulator-w1-26cfbe-kserve-74c5578fbd-t79hq" podUID="50dc76f4-aebd-41f4-9f23-799ddc8c70f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:48:08.023048 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:08.020257    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-43ceaa/premium-w0-43ceaa-kserve-67b55659b7-rckfd" podUID="7be32df1-5049-4f79-be23-a827f3f0a85f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:48:08.149154 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:08.149100    4463 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-43ceaa/premium-w0-43ceaa-kserve-67b55659b7-rckfd" podUID="7be32df1-5049-4f79-be23-a827f3f0a85f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": context deadline exceeded"
Nov 02 19:48:08.233047 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:08.225091    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d869e/premium-w5-1d869e-kserve-7745799ff8-pv2th" podUID="1d784b10-99e4-4561-bfb7-6614877d2b6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:48:08.234820 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:08.234630    4463 patch_prober.go:29] interesting pod/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-4c5vf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 02 19:48:08.234820 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:08.234686    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-4c5vf" podUID="630de598-0bde-4b3e-af80-b014ca9db567" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 02 19:48:08.234820 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:08.234760    4463 patch_prober.go:29] interesting pod/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-hqc7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 02 19:48:08.234820 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:08.234782    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-hqc7w" podUID="9f6ba8f5-7ed9-4807-8435-6794197fee7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 02 19:48:08.243325 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:08.243292    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-26cfbe/distinct-w1-26cfbe-kserve-685985bfb-tkgn2" podUID="7415aa62-467c-43af-94c8-d604f57ec1c3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:48:08.290462 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:08.290371    4463 patch_prober.go:29] interesting pod/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-z7nwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Nov 02 19:48:08.290697 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:08.290669    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-z7nwb" podUID="f40668dc-8c36-4c34-ae69-c87d8c71e338" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Nov 02 19:48:08.295725 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:08.295702    4463 patch_prober.go:29] interesting pod/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5hkwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Nov 02 19:48:08.295912 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:08.295888    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5hkwc" podUID="d288950b-84fb-4cef-bb78-0a713bcd14c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Nov 02 19:48:08.431176 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:08.431134    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5c46af/distinct-w3-5c46af-kserve-68b6dff7f5-bp59f" podUID="8dd6362e-7dbd-489b-b6d8-0d9caab0fa69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:48:09.027324 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:09.027279    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-43ceaa/premium-w0-43ceaa-kserve-67b55659b7-rckfd" podUID="7be32df1-5049-4f79-be23-a827f3f0a85f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:48:09.043042 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:09.028499    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-43ceaa/premium-w0-43ceaa-kserve-67b55659b7-rckfd"
Nov 02 19:48:09.092182 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:09.092131    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8795bf/distinct-2-w4-8795bf-kserve-69c8b6b885-2chhn" podUID="9f430131-4938-47ca-ba86-b6a236f8cc3b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:48:09.340077 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:09.310486    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8795bf/distinct-w4-8795bf-kserve-64b4796df4-8qgqh" podUID="ee7d28f4-e9c7-43e2-b983-1427589f1785" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Nov 02 19:48:09.438488 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:09.422800    4463 patch_prober.go:29] interesting pod/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-z7nwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Nov 02 19:48:09.438488 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:09.422895    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-z7nwb" podUID="f40668dc-8c36-4c34-ae69-c87d8c71e338" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Nov 02 19:48:09.471043 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:09.438877    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-26cfbe/distinct-w1-26cfbe-kserve-685985bfb-tkgn2" podUID="7415aa62-467c-43af-94c8-d604f57ec1c3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Nov 02 19:48:09.471043 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:09.440297    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-26cfbe/distinct-w1-26cfbe-kserve-685985bfb-tkgn2"
Nov 02 19:48:09.471043 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:09.468565    4463 patch_prober.go:29] interesting pod/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-hqc7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 02 19:48:09.471043 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:09.469088    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-hqc7w" podUID="9f6ba8f5-7ed9-4807-8435-6794197fee7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 02 19:48:09.471043 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:09.469222    4463 patch_prober.go:29] interesting pod/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-4c5vf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 02 19:48:09.471043 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:09.469256    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-4c5vf" podUID="630de598-0bde-4b3e-af80-b014ca9db567" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 02 19:48:09.550889 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:09.550124    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d869e/distinct-2-w5-1d869e-kserve-574f55db46-4cv47" podUID="7e236108-328b-4170-be20-13f16b56c4fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:48:09.612133 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:09.580367    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d869e/premium-w5-1d869e-kserve-7745799ff8-pv2th" podUID="1d784b10-99e4-4561-bfb7-6614877d2b6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 02 19:48:09.612133 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:09.581257    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5c46af/distinct-w3-5c46af-kserve-68b6dff7f5-bp59f" podUID="8dd6362e-7dbd-489b-b6d8-0d9caab0fa69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Nov 02 19:48:09.641070 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:09.638363    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-1d869e/premium-w5-1d869e-kserve-7745799ff8-pv2th"
Nov 02 19:48:09.641070 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:09.638540    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5c46af/distinct-w3-5c46af-kserve-68b6dff7f5-bp59f"
Nov 02 19:48:09.859709 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:09.847436    4463 patch_prober.go:29] interesting pod/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5hkwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Nov 02 19:48:09.859709 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:09.847505    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5hkwc" podUID="d288950b-84fb-4cef-bb78-0a713bcd14c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Nov 02 19:48:09.895070 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:09.885819    4463 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5c46af/distinct-w3-5c46af-kserve-68b6dff7f5-bp59f" podUID="8dd6362e-7dbd-489b-b6d8-0d9caab0fa69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:48:09.895070 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:09.886600    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-26cfbe/simulator-w1-26cfbe-kserve-74c5578fbd-t79hq" podUID="50dc76f4-aebd-41f4-9f23-799ddc8c70f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:48:10.106036 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:10.105003    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8795bf/distinct-2-w4-8795bf-kserve-69c8b6b885-2chhn" podUID="9f430131-4938-47ca-ba86-b6a236f8cc3b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:48:10.239039 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:10.223202    4463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-z74mv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Nov 02 19:48:10.239039 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:10.223282    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z74mv" podUID="658d1f56-a03e-4a38-8911-229de81a3d29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Nov 02 19:48:10.260600 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:10.252659    4463 patch_prober.go:29] interesting pod/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-4c5vf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 02 19:48:10.260600 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:10.252719    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-4c5vf" podUID="630de598-0bde-4b3e-af80-b014ca9db567" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 02 19:48:10.260600 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:10.252800    4463 patch_prober.go:29] interesting pod/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-hqc7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 02 19:48:10.260600 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:10.252821    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-hqc7w" podUID="9f6ba8f5-7ed9-4807-8435-6794197fee7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 02 19:48:10.324067 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:10.292105    4463 patch_prober.go:29] interesting pod/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-z7nwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Nov 02 19:48:10.324067 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:10.292175    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-z7nwb" podUID="f40668dc-8c36-4c34-ae69-c87d8c71e338" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Nov 02 19:48:10.471914 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:10.443389    4463 patch_prober.go:29] interesting pod/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5hkwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Nov 02 19:48:10.471914 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:10.443470    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5hkwc" podUID="d288950b-84fb-4cef-bb78-0a713bcd14c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Nov 02 19:48:10.471914 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:10.445515    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-26cfbe/distinct-w1-26cfbe-kserve-685985bfb-tkgn2" podUID="7415aa62-467c-43af-94c8-d604f57ec1c3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Nov 02 19:48:10.524394 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:10.510677    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-43ceaa/premium-w0-43ceaa-kserve-67b55659b7-rckfd"
Nov 02 19:48:10.524394 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:10.510751    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8795bf/distinct-2-w4-8795bf-kserve-69c8b6b885-2chhn"
Nov 02 19:48:10.571293 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:10.571249    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d869e/distinct-2-w5-1d869e-kserve-574f55db46-4cv47" podUID="7e236108-328b-4170-be20-13f16b56c4fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Nov 02 19:48:10.572713 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:10.572689    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-1d869e/distinct-2-w5-1d869e-kserve-574f55db46-4cv47"
Nov 02 19:48:10.613151 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:10.577084    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-cthc4" podUID="27b07c53-9c7a-4cc1-a0cd-8a9cf982d73c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:48:10.660038 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:10.631410    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8795bf/distinct-w4-8795bf-kserve-64b4796df4-8qgqh" podUID="ee7d28f4-e9c7-43e2-b983-1427589f1785" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:48:10.660038 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:10.633113    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8795bf/distinct-w4-8795bf-kserve-64b4796df4-8qgqh"
Nov 02 19:48:10.681287 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:10.681228    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5c46af/distinct-w3-5c46af-kserve-68b6dff7f5-bp59f" podUID="8dd6362e-7dbd-489b-b6d8-0d9caab0fa69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:48:10.682587 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:10.681588    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d869e/premium-w5-1d869e-kserve-7745799ff8-pv2th" podUID="1d784b10-99e4-4561-bfb7-6614877d2b6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 02 19:48:10.894336 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:10.894215    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-26cfbe/simulator-w1-26cfbe-kserve-74c5578fbd-t79hq" podUID="50dc76f4-aebd-41f4-9f23-799ddc8c70f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:48:10.894766 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:10.894742    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-26cfbe/simulator-w1-26cfbe-kserve-74c5578fbd-t79hq"
Nov 02 19:48:10.979633 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:10.977100    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-43ceaa/premium-w0-43ceaa-kserve-67b55659b7-rckfd" podUID="7be32df1-5049-4f79-be23-a827f3f0a85f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:48:11.045164 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:11.045113    4463 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-26cfbe/distinct-w1-26cfbe-kserve-685985bfb-tkgn2" podUID="7415aa62-467c-43af-94c8-d604f57ec1c3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:48:11.108153 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:11.108108    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8795bf/distinct-2-w4-8795bf-kserve-69c8b6b885-2chhn" podUID="9f430131-4938-47ca-ba86-b6a236f8cc3b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:48:11.326038 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:11.312974    4463 patch_prober.go:29] interesting pod/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-4c5vf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 02 19:48:11.326038 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:11.313060    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-4c5vf" podUID="630de598-0bde-4b3e-af80-b014ca9db567" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 02 19:48:11.326038 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:11.314694    4463 patch_prober.go:29] interesting pod/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-hqc7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 02 19:48:11.326038 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:11.314756    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-hqc7w" podUID="9f6ba8f5-7ed9-4807-8435-6794197fee7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 02 19:48:11.346416 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:11.332126    4463 patch_prober.go:29] interesting pod/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-z7nwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Nov 02 19:48:11.360066 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:11.332179    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-z7nwb" podUID="f40668dc-8c36-4c34-ae69-c87d8c71e338" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Nov 02 19:48:11.387538 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:11.387479    4463 generic.go:356] "Generic (PLEG): container finished" podID="658d1f56-a03e-4a38-8911-229de81a3d29" containerID="70d39895a3906c23298558bb65c6c6c121df351bd839a8e20943d293db6d8e2b" exitCode=0
Nov 02 19:48:11.387701 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:11.387549    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z74mv" event={"ID":"658d1f56-a03e-4a38-8911-229de81a3d29","Type":"ContainerDied","Data":"70d39895a3906c23298558bb65c6c6c121df351bd839a8e20943d293db6d8e2b"}
Nov 02 19:48:11.425066 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:11.419354    4463 patch_prober.go:29] interesting pod/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5hkwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Nov 02 19:48:11.425066 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:11.419420    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5hkwc" podUID="d288950b-84fb-4cef-bb78-0a713bcd14c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Nov 02 19:48:11.458047 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:11.453729    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-26cfbe/distinct-w1-26cfbe-kserve-685985bfb-tkgn2" podUID="7415aa62-467c-43af-94c8-d604f57ec1c3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:48:11.716098 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:11.620158    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d869e/distinct-2-w5-1d869e-kserve-574f55db46-4cv47" podUID="7e236108-328b-4170-be20-13f16b56c4fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:48:11.716098 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:11.652699    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8795bf/distinct-w4-8795bf-kserve-64b4796df4-8qgqh" podUID="ee7d28f4-e9c7-43e2-b983-1427589f1785" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Nov 02 19:48:11.716098 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:11.701122    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5c46af/distinct-w3-5c46af-kserve-68b6dff7f5-bp59f" podUID="8dd6362e-7dbd-489b-b6d8-0d9caab0fa69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Nov 02 19:48:11.716098 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:11.702175    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d869e/premium-w5-1d869e-kserve-7745799ff8-pv2th" podUID="1d784b10-99e4-4561-bfb7-6614877d2b6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:48:11.902895 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:11.896097    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-26cfbe/simulator-w1-26cfbe-kserve-74c5578fbd-t79hq" podUID="50dc76f4-aebd-41f4-9f23-799ddc8c70f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:48:12.072111 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:12.063854    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-43ceaa/premium-w0-43ceaa-kserve-67b55659b7-rckfd" podUID="7be32df1-5049-4f79-be23-a827f3f0a85f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:48:12.086037 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:12.078802    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-43ceaa/premium-w0-43ceaa-kserve-67b55659b7-rckfd"
Nov 02 19:48:12.136432 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:12.136385    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8795bf/distinct-2-w4-8795bf-kserve-69c8b6b885-2chhn" podUID="9f430131-4938-47ca-ba86-b6a236f8cc3b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:48:12.357193 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:12.326250    4463 patch_prober.go:29] interesting pod/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-z7nwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Nov 02 19:48:12.357193 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:12.326306    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-z7nwb" podUID="f40668dc-8c36-4c34-ae69-c87d8c71e338" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Nov 02 19:48:12.357193 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:12.326383    4463 patch_prober.go:29] interesting pod/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-hqc7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 02 19:48:12.357193 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:12.326405    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-hqc7w" podUID="9f6ba8f5-7ed9-4807-8435-6794197fee7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 02 19:48:12.357193 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:12.326457    4463 patch_prober.go:29] interesting pod/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-4c5vf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 02 19:48:12.357193 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:12.326477    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-4c5vf" podUID="630de598-0bde-4b3e-af80-b014ca9db567" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 02 19:48:12.357193 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:12.328878    4463 patch_prober.go:29] interesting pod/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5hkwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Nov 02 19:48:12.357193 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:12.328920    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5hkwc" podUID="d288950b-84fb-4cef-bb78-0a713bcd14c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Nov 02 19:48:12.440035 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:12.424619    4463 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-wmp44" podUID="c5fcfd2d-ed14-45ea-a811-dbe2237490eb" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.20:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:48:12.465519 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:12.457557    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-26cfbe/distinct-w1-26cfbe-kserve-685985bfb-tkgn2" podUID="7415aa62-467c-43af-94c8-d604f57ec1c3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:48:12.513034 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:12.512189    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z74mv" event={"ID":"658d1f56-a03e-4a38-8911-229de81a3d29","Type":"ContainerDied","Data":"a1b5df7718e9e6274ed8f95c9086794e8c5693c83fe81d1357b3ab414d8dfb81"}
Nov 02 19:48:12.513034 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:12.512263    4463 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a1b5df7718e9e6274ed8f95c9086794e8c5693c83fe81d1357b3ab414d8dfb81"
Nov 02 19:48:12.513034 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:12.512265    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z74mv"
Nov 02 19:48:12.633096 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:12.628969    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d869e/distinct-2-w5-1d869e-kserve-574f55db46-4cv47" podUID="7e236108-328b-4170-be20-13f16b56c4fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:48:12.684158 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:12.680297    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8795bf/distinct-w4-8795bf-kserve-64b4796df4-8qgqh" podUID="ee7d28f4-e9c7-43e2-b983-1427589f1785" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Nov 02 19:48:12.685941 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:12.681407    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/658d1f56-a03e-4a38-8911-229de81a3d29-istio-token\" (UniqueName: \"kubernetes.io/projected/658d1f56-a03e-4a38-8911-229de81a3d29-istio-token\") pod \"658d1f56-a03e-4a38-8911-229de81a3d29\" (UID: \"658d1f56-a03e-4a38-8911-229de81a3d29\") "
Nov 02 19:48:12.685941 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:12.684622    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/658d1f56-a03e-4a38-8911-229de81a3d29-kube-api-access-cptpz\" (UniqueName: \"kubernetes.io/projected/658d1f56-a03e-4a38-8911-229de81a3d29-kube-api-access-cptpz\") pod \"658d1f56-a03e-4a38-8911-229de81a3d29\" (UID: \"658d1f56-a03e-4a38-8911-229de81a3d29\") "
Nov 02 19:48:12.685941 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:12.684675    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/658d1f56-a03e-4a38-8911-229de81a3d29-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/658d1f56-a03e-4a38-8911-229de81a3d29-workload-certs\") pod \"658d1f56-a03e-4a38-8911-229de81a3d29\" (UID: \"658d1f56-a03e-4a38-8911-229de81a3d29\") "
Nov 02 19:48:12.685941 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:12.684723    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/658d1f56-a03e-4a38-8911-229de81a3d29-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/658d1f56-a03e-4a38-8911-229de81a3d29-istio-podinfo\") pod \"658d1f56-a03e-4a38-8911-229de81a3d29\" (UID: \"658d1f56-a03e-4a38-8911-229de81a3d29\") "
Nov 02 19:48:12.685941 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:12.684761    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/658d1f56-a03e-4a38-8911-229de81a3d29-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/658d1f56-a03e-4a38-8911-229de81a3d29-istiod-ca-cert\") pod \"658d1f56-a03e-4a38-8911-229de81a3d29\" (UID: \"658d1f56-a03e-4a38-8911-229de81a3d29\") "
Nov 02 19:48:12.685941 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:12.684796    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/658d1f56-a03e-4a38-8911-229de81a3d29-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/658d1f56-a03e-4a38-8911-229de81a3d29-istio-envoy\") pod \"658d1f56-a03e-4a38-8911-229de81a3d29\" (UID: \"658d1f56-a03e-4a38-8911-229de81a3d29\") "
Nov 02 19:48:12.685941 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:12.684864    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/658d1f56-a03e-4a38-8911-229de81a3d29-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/658d1f56-a03e-4a38-8911-229de81a3d29-workload-socket\") pod \"658d1f56-a03e-4a38-8911-229de81a3d29\" (UID: \"658d1f56-a03e-4a38-8911-229de81a3d29\") "
Nov 02 19:48:12.685941 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:12.685049    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/658d1f56-a03e-4a38-8911-229de81a3d29-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/658d1f56-a03e-4a38-8911-229de81a3d29-credential-socket\") pod \"658d1f56-a03e-4a38-8911-229de81a3d29\" (UID: \"658d1f56-a03e-4a38-8911-229de81a3d29\") "
Nov 02 19:48:12.685941 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:12.685114    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/658d1f56-a03e-4a38-8911-229de81a3d29-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/658d1f56-a03e-4a38-8911-229de81a3d29-istio-data\") pod \"658d1f56-a03e-4a38-8911-229de81a3d29\" (UID: \"658d1f56-a03e-4a38-8911-229de81a3d29\") "
Nov 02 19:48:12.688537 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:12.687831    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/658d1f56-a03e-4a38-8911-229de81a3d29-istio-data" pod "658d1f56-a03e-4a38-8911-229de81a3d29" (UID: "658d1f56-a03e-4a38-8911-229de81a3d29"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:48:12.688537 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:12.688230    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/658d1f56-a03e-4a38-8911-229de81a3d29-istiod-ca-cert" pod "658d1f56-a03e-4a38-8911-229de81a3d29" (UID: "658d1f56-a03e-4a38-8911-229de81a3d29"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 19:48:12.693061 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:12.689061    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/658d1f56-a03e-4a38-8911-229de81a3d29-workload-certs" pod "658d1f56-a03e-4a38-8911-229de81a3d29" (UID: "658d1f56-a03e-4a38-8911-229de81a3d29"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:48:12.700047 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:12.694747    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/658d1f56-a03e-4a38-8911-229de81a3d29-workload-socket" pod "658d1f56-a03e-4a38-8911-229de81a3d29" (UID: "658d1f56-a03e-4a38-8911-229de81a3d29"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:48:12.700618 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:12.700594    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/658d1f56-a03e-4a38-8911-229de81a3d29-credential-socket" pod "658d1f56-a03e-4a38-8911-229de81a3d29" (UID: "658d1f56-a03e-4a38-8911-229de81a3d29"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:48:12.725942 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:12.701954    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5c46af/distinct-w3-5c46af-kserve-68b6dff7f5-bp59f" podUID="8dd6362e-7dbd-489b-b6d8-0d9caab0fa69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:48:12.730776 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:12.729739    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d869e/premium-w5-1d869e-kserve-7745799ff8-pv2th" podUID="1d784b10-99e4-4561-bfb7-6614877d2b6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 02 19:48:12.767154 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:12.761798    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/658d1f56-a03e-4a38-8911-229de81a3d29-kube-api-access-cptpz" pod "658d1f56-a03e-4a38-8911-229de81a3d29" (UID: "658d1f56-a03e-4a38-8911-229de81a3d29"). InnerVolumeSpecName "kube-api-access-cptpz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:48:12.782572 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:12.782518    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/658d1f56-a03e-4a38-8911-229de81a3d29-istio-envoy" pod "658d1f56-a03e-4a38-8911-229de81a3d29" (UID: "658d1f56-a03e-4a38-8911-229de81a3d29"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:48:12.782987 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:12.782966    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/658d1f56-a03e-4a38-8911-229de81a3d29-istio-token" pod "658d1f56-a03e-4a38-8911-229de81a3d29" (UID: "658d1f56-a03e-4a38-8911-229de81a3d29"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:48:12.798123 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:12.790832    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cptpz\" (UniqueName: \"kubernetes.io/projected/658d1f56-a03e-4a38-8911-229de81a3d29-kube-api-access-cptpz\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:48:12.798123 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:12.790867    4463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/658d1f56-a03e-4a38-8911-229de81a3d29-workload-certs\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:48:12.798123 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:12.790882    4463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/658d1f56-a03e-4a38-8911-229de81a3d29-istiod-ca-cert\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:48:12.798123 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:12.790897    4463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/658d1f56-a03e-4a38-8911-229de81a3d29-istio-envoy\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:48:12.798123 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:12.790910    4463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/658d1f56-a03e-4a38-8911-229de81a3d29-workload-socket\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:48:12.798123 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:12.790923    4463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/658d1f56-a03e-4a38-8911-229de81a3d29-credential-socket\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:48:12.798123 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:12.790939    4463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/658d1f56-a03e-4a38-8911-229de81a3d29-istio-data\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:48:12.798123 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:12.790952    4463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/658d1f56-a03e-4a38-8911-229de81a3d29-istio-token\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:48:12.814367 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:12.814159    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/658d1f56-a03e-4a38-8911-229de81a3d29-istio-podinfo" pod "658d1f56-a03e-4a38-8911-229de81a3d29" (UID: "658d1f56-a03e-4a38-8911-229de81a3d29"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 19:48:12.886140 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:12.874657    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-43ceaa/premium-w0-43ceaa-kserve-67b55659b7-rckfd"
Nov 02 19:48:12.895523 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:12.894103    4463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/658d1f56-a03e-4a38-8911-229de81a3d29-istio-podinfo\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:48:12.903052 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:12.896552    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-26cfbe/simulator-w1-26cfbe-kserve-74c5578fbd-t79hq" podUID="50dc76f4-aebd-41f4-9f23-799ddc8c70f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:48:12.904119 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:12.903434    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8795bf/distinct-w4-8795bf-kserve-64b4796df4-8qgqh"
Nov 02 19:48:12.919388 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:12.918696    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-26cfbe/simulator-w1-26cfbe-kserve-74c5578fbd-t79hq"
Nov 02 19:48:12.939214 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:12.939197    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5c46af/distinct-w3-5c46af-kserve-68b6dff7f5-bp59f"
Nov 02 19:48:12.948318 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:12.947187    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-1d869e/distinct-2-w5-1d869e-kserve-574f55db46-4cv47"
Nov 02 19:48:12.999310 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:12.999278    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8795bf/distinct-2-w4-8795bf-kserve-69c8b6b885-2chhn"
Nov 02 19:48:13.014078 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:13.014038    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-1d869e/premium-w5-1d869e-kserve-7745799ff8-pv2th"
Nov 02 19:48:13.017540 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:13.017518    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-26cfbe/distinct-w1-26cfbe-kserve-685985bfb-tkgn2"
Nov 02 19:48:13.228824 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:13.228789    4463 patch_prober.go:29] interesting pod/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-4c5vf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 02 19:48:13.229289 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:13.228849    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-4c5vf" podUID="630de598-0bde-4b3e-af80-b014ca9db567" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 02 19:48:13.235337 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:13.235307    4463 patch_prober.go:29] interesting pod/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-hqc7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 02 19:48:13.235541 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:13.235515    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-hqc7w" podUID="9f6ba8f5-7ed9-4807-8435-6794197fee7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 02 19:48:13.287985 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:13.287953    4463 patch_prober.go:29] interesting pod/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-z7nwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Nov 02 19:48:13.288139 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:13.288025    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-z7nwb" podUID="f40668dc-8c36-4c34-ae69-c87d8c71e338" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Nov 02 19:48:13.292340 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:13.292311    4463 patch_prober.go:29] interesting pod/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5hkwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Nov 02 19:48:13.292479 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:13.292362    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5hkwc" podUID="d288950b-84fb-4cef-bb78-0a713bcd14c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Nov 02 19:48:13.524065 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:13.523967    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z74mv"
Nov 02 19:48:13.802680 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:13.802589    4463 scope.go:122] "RemoveContainer" containerID="70d39895a3906c23298558bb65c6c6c121df351bd839a8e20943d293db6d8e2b"
Nov 02 19:48:13.923369 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:13.923327    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z74mv"]
Nov 02 19:48:13.945547 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:13.945499    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z74mv"]
Nov 02 19:48:14.292802 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:14.292754    4463 patch_prober.go:29] interesting pod/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5hkwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Nov 02 19:48:14.293401 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:14.293363    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5hkwc" podUID="d288950b-84fb-4cef-bb78-0a713bcd14c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Nov 02 19:48:14.586834 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:14.582416    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-z7nwb"
Nov 02 19:48:14.586834 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:14.584539    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-z7nwb"
Nov 02 19:48:14.592058 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:14.590087    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-z7nwb"
Nov 02 19:48:14.638360 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:14.636627    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-hqc7w"
Nov 02 19:48:14.640229 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:14.640196    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-hqc7w"
Nov 02 19:48:14.640382 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:14.640346    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-4c5vf"
Nov 02 19:48:14.641194 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:14.641127    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-4c5vf"
Nov 02 19:48:14.643238 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:14.643219    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-hqc7w"
Nov 02 19:48:14.645706 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:14.645680    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-4c5vf"
Nov 02 19:48:15.297176 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:15.297142    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5hkwc"
Nov 02 19:48:15.297534 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:15.297431    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5hkwc"
Nov 02 19:48:15.298008 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:15.297988    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5hkwc"
Nov 02 19:48:15.589518 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:15.589436    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="658d1f56-a03e-4a38-8911-229de81a3d29" path="/var/lib/kubelet/pods/658d1f56-a03e-4a38-8911-229de81a3d29/volumes"
Nov 02 19:48:19.241685 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:19.241648    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 19:48:19.824678 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:19.824644    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-26hn5"]
Nov 02 19:48:19.825283 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:19.825266    4463 cpu_manager.go:414] "containerMap: removing container" podUID="658d1f56-a03e-4a38-8911-229de81a3d29" containerName="istio-proxy"
Nov 02 19:48:19.825354 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:19.825284    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="658d1f56-a03e-4a38-8911-229de81a3d29" containerName="istio-proxy"
Nov 02 19:48:19.825397 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:19.825390    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="658d1f56-a03e-4a38-8911-229de81a3d29" containerName="istio-proxy"
Nov 02 19:48:19.831264 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:19.831238    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-26hn5"
Nov 02 19:48:19.927034 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:19.925166    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-26hn5"]
Nov 02 19:48:19.938325 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:19.938285    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d535f89-ff88-4a74-af33-99c15a1b05d4-credential-socket\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-26hn5\" (UID: \"1d535f89-ff88-4a74-af33-99c15a1b05d4\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-26hn5"
Nov 02 19:48:19.938488 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:19.938356    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d535f89-ff88-4a74-af33-99c15a1b05d4-istio-podinfo\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-26hn5\" (UID: \"1d535f89-ff88-4a74-af33-99c15a1b05d4\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-26hn5"
Nov 02 19:48:19.938488 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:19.938390    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d535f89-ff88-4a74-af33-99c15a1b05d4-workload-socket\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-26hn5\" (UID: \"1d535f89-ff88-4a74-af33-99c15a1b05d4\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-26hn5"
Nov 02 19:48:19.938623 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:19.938506    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d535f89-ff88-4a74-af33-99c15a1b05d4-istio-envoy\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-26hn5\" (UID: \"1d535f89-ff88-4a74-af33-99c15a1b05d4\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-26hn5"
Nov 02 19:48:19.938623 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:19.938558    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d535f89-ff88-4a74-af33-99c15a1b05d4-istiod-ca-cert\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-26hn5\" (UID: \"1d535f89-ff88-4a74-af33-99c15a1b05d4\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-26hn5"
Nov 02 19:48:19.938722 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:19.938694    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d535f89-ff88-4a74-af33-99c15a1b05d4-workload-certs\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-26hn5\" (UID: \"1d535f89-ff88-4a74-af33-99c15a1b05d4\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-26hn5"
Nov 02 19:48:19.938771 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:19.938721    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d535f89-ff88-4a74-af33-99c15a1b05d4-istio-token\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-26hn5\" (UID: \"1d535f89-ff88-4a74-af33-99c15a1b05d4\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-26hn5"
Nov 02 19:48:19.938826 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:19.938770    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7t4t\" (UniqueName: \"kubernetes.io/projected/1d535f89-ff88-4a74-af33-99c15a1b05d4-kube-api-access-d7t4t\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-26hn5\" (UID: \"1d535f89-ff88-4a74-af33-99c15a1b05d4\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-26hn5"
Nov 02 19:48:19.938826 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:19.938811    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d535f89-ff88-4a74-af33-99c15a1b05d4-istio-data\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-26hn5\" (UID: \"1d535f89-ff88-4a74-af33-99c15a1b05d4\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-26hn5"
Nov 02 19:48:20.040067 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:20.039985    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d535f89-ff88-4a74-af33-99c15a1b05d4-istio-envoy\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-26hn5\" (UID: \"1d535f89-ff88-4a74-af33-99c15a1b05d4\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-26hn5"
Nov 02 19:48:20.040401 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:20.040101    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d535f89-ff88-4a74-af33-99c15a1b05d4-istiod-ca-cert\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-26hn5\" (UID: \"1d535f89-ff88-4a74-af33-99c15a1b05d4\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-26hn5"
Nov 02 19:48:20.040401 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:20.040233    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d535f89-ff88-4a74-af33-99c15a1b05d4-workload-certs\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-26hn5\" (UID: \"1d535f89-ff88-4a74-af33-99c15a1b05d4\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-26hn5"
Nov 02 19:48:20.040401 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:20.040262    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d535f89-ff88-4a74-af33-99c15a1b05d4-istio-token\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-26hn5\" (UID: \"1d535f89-ff88-4a74-af33-99c15a1b05d4\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-26hn5"
Nov 02 19:48:20.040401 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:20.040337    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7t4t\" (UniqueName: \"kubernetes.io/projected/1d535f89-ff88-4a74-af33-99c15a1b05d4-kube-api-access-d7t4t\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-26hn5\" (UID: \"1d535f89-ff88-4a74-af33-99c15a1b05d4\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-26hn5"
Nov 02 19:48:20.040401 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:20.040375    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d535f89-ff88-4a74-af33-99c15a1b05d4-istio-data\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-26hn5\" (UID: \"1d535f89-ff88-4a74-af33-99c15a1b05d4\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-26hn5"
Nov 02 19:48:20.040733 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:20.040432    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d535f89-ff88-4a74-af33-99c15a1b05d4-credential-socket\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-26hn5\" (UID: \"1d535f89-ff88-4a74-af33-99c15a1b05d4\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-26hn5"
Nov 02 19:48:20.040733 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:20.040490    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d535f89-ff88-4a74-af33-99c15a1b05d4-istio-podinfo\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-26hn5\" (UID: \"1d535f89-ff88-4a74-af33-99c15a1b05d4\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-26hn5"
Nov 02 19:48:20.040733 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:20.040518    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d535f89-ff88-4a74-af33-99c15a1b05d4-workload-socket\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-26hn5\" (UID: \"1d535f89-ff88-4a74-af33-99c15a1b05d4\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-26hn5"
Nov 02 19:48:20.040952 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:20.040924    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d535f89-ff88-4a74-af33-99c15a1b05d4-workload-certs\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-26hn5\" (UID: \"1d535f89-ff88-4a74-af33-99c15a1b05d4\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-26hn5"
Nov 02 19:48:20.041005 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:20.040970    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d535f89-ff88-4a74-af33-99c15a1b05d4-istiod-ca-cert\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-26hn5\" (UID: \"1d535f89-ff88-4a74-af33-99c15a1b05d4\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-26hn5"
Nov 02 19:48:20.041094 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:20.040967    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d535f89-ff88-4a74-af33-99c15a1b05d4-workload-socket\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-26hn5\" (UID: \"1d535f89-ff88-4a74-af33-99c15a1b05d4\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-26hn5"
Nov 02 19:48:20.041254 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:20.041230    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d535f89-ff88-4a74-af33-99c15a1b05d4-credential-socket\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-26hn5\" (UID: \"1d535f89-ff88-4a74-af33-99c15a1b05d4\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-26hn5"
Nov 02 19:48:20.041487 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:20.041460    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d535f89-ff88-4a74-af33-99c15a1b05d4-istio-data\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-26hn5\" (UID: \"1d535f89-ff88-4a74-af33-99c15a1b05d4\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-26hn5"
Nov 02 19:48:20.043508 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:20.043485    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d535f89-ff88-4a74-af33-99c15a1b05d4-istio-envoy\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-26hn5\" (UID: \"1d535f89-ff88-4a74-af33-99c15a1b05d4\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-26hn5"
Nov 02 19:48:20.043678 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:20.043656    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d535f89-ff88-4a74-af33-99c15a1b05d4-istio-podinfo\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-26hn5\" (UID: \"1d535f89-ff88-4a74-af33-99c15a1b05d4\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-26hn5"
Nov 02 19:48:20.097933 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:20.097855    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d535f89-ff88-4a74-af33-99c15a1b05d4-istio-token\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-26hn5\" (UID: \"1d535f89-ff88-4a74-af33-99c15a1b05d4\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-26hn5"
Nov 02 19:48:20.102129 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:20.102104    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7t4t\" (UniqueName: \"kubernetes.io/projected/1d535f89-ff88-4a74-af33-99c15a1b05d4-kube-api-access-d7t4t\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-26hn5\" (UID: \"1d535f89-ff88-4a74-af33-99c15a1b05d4\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-26hn5"
Nov 02 19:48:20.148727 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:20.148696    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-26hn5"
Nov 02 19:48:20.630051 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:48:20.628353    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d535f89_ff88_4a74_af33_99c15a1b05d4.slice/crio-e54e4a0cb29c06165d7f1dafc25d44a0674b6d723c845deaa1992b04db81b6aa WatchSource:0}: Error finding container e54e4a0cb29c06165d7f1dafc25d44a0674b6d723c845deaa1992b04db81b6aa: Status 404 returned error can't find the container with id e54e4a0cb29c06165d7f1dafc25d44a0674b6d723c845deaa1992b04db81b6aa
Nov 02 19:48:20.653033 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:20.649323    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:48:20.702366 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:20.702299    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:48:20.702748 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:20.702707    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:48:20.715769 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:20.715741    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-26hn5"]
Nov 02 19:48:21.719777 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:21.719737    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-26hn5" event={"ID":"1d535f89-ff88-4a74-af33-99c15a1b05d4","Type":"ContainerStarted","Data":"a2f9386caab420175aec3a114b6daaecffbf1c42c609af1c0721a2186b1542de"}
Nov 02 19:48:21.720428 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:21.720401    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-26hn5" event={"ID":"1d535f89-ff88-4a74-af33-99c15a1b05d4","Type":"ContainerStarted","Data":"e54e4a0cb29c06165d7f1dafc25d44a0674b6d723c845deaa1992b04db81b6aa"}
Nov 02 19:48:21.952034 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:21.951514    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-26hn5" podStartSLOduration=2.951492828 podStartE2EDuration="2.951492828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:48:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:48:21.90143906 +0000 UTC m=+1208.508472870" watchObservedRunningTime="2026-11-02 19:48:21.951492828 +0000 UTC m=+1208.558526598"
Nov 02 19:48:22.162127 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:22.156240    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-26hn5"
Nov 02 19:48:22.166034 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:22.162434    4463 patch_prober.go:29] interesting pod/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-26hn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 02 19:48:22.166034 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:22.162523    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-26hn5" podUID="1d535f89-ff88-4a74-af33-99c15a1b05d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 02 19:48:22.515035 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:22.514495    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-26cfbe/distinct-w1-26cfbe-kserve-685985bfb-tkgn2" podUID="7415aa62-467c-43af-94c8-d604f57ec1c3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:48:23.167037 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:23.160382    4463 patch_prober.go:29] interesting pod/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-26hn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 02 19:48:23.167037 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:23.160446    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-26hn5" podUID="1d535f89-ff88-4a74-af33-99c15a1b05d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 02 19:48:24.171391 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:24.171353    4463 patch_prober.go:29] interesting pod/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-26hn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 02 19:48:24.171965 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:24.171934    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-26hn5" podUID="1d535f89-ff88-4a74-af33-99c15a1b05d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 02 19:48:25.168437 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:25.168396    4463 patch_prober.go:29] interesting pod/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-26hn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 02 19:48:25.168750 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:25.168712    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-26hn5" podUID="1d535f89-ff88-4a74-af33-99c15a1b05d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 02 19:48:26.163003 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:26.162403    4463 patch_prober.go:29] interesting pod/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-26hn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 02 19:48:26.163003 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:26.162462    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-26hn5" podUID="1d535f89-ff88-4a74-af33-99c15a1b05d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 02 19:48:27.149486 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:27.149063    4463 patch_prober.go:29] interesting pod/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-26hn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 02 19:48:27.149486 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:27.149131    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-26hn5" podUID="1d535f89-ff88-4a74-af33-99c15a1b05d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 02 19:48:28.149433 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:28.149397    4463 patch_prober.go:29] interesting pod/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-26hn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 02 19:48:28.149806 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:28.149446    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-26hn5" podUID="1d535f89-ff88-4a74-af33-99c15a1b05d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 02 19:48:29.154380 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:29.154350    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-26hn5"
Nov 02 19:48:29.154754 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:29.154633    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-26hn5"
Nov 02 19:48:29.155298 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:29.155278    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-26hn5"
Nov 02 19:48:34.294445 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:48:34.294405    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 19:48:34.866844 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:34.866813    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-8mkqq"]
Nov 02 19:48:34.870791 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:34.870773    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-8mkqq"
Nov 02 19:48:34.965720 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:34.965688    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-8mkqq"]
Nov 02 19:48:35.014790 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:35.014756    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dca3f84e-1191-48bb-a59c-8456088614ac-workload-socket\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-8mkqq\" (UID: \"dca3f84e-1191-48bb-a59c-8456088614ac\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-8mkqq"
Nov 02 19:48:35.014980 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:35.014796    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dca3f84e-1191-48bb-a59c-8456088614ac-workload-certs\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-8mkqq\" (UID: \"dca3f84e-1191-48bb-a59c-8456088614ac\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-8mkqq"
Nov 02 19:48:35.014980 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:35.014833    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dca3f84e-1191-48bb-a59c-8456088614ac-istiod-ca-cert\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-8mkqq\" (UID: \"dca3f84e-1191-48bb-a59c-8456088614ac\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-8mkqq"
Nov 02 19:48:35.014980 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:35.014875    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hllx4\" (UniqueName: \"kubernetes.io/projected/dca3f84e-1191-48bb-a59c-8456088614ac-kube-api-access-hllx4\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-8mkqq\" (UID: \"dca3f84e-1191-48bb-a59c-8456088614ac\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-8mkqq"
Nov 02 19:48:35.015154 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:35.014975    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dca3f84e-1191-48bb-a59c-8456088614ac-istio-envoy\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-8mkqq\" (UID: \"dca3f84e-1191-48bb-a59c-8456088614ac\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-8mkqq"
Nov 02 19:48:35.015154 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:35.015097    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dca3f84e-1191-48bb-a59c-8456088614ac-istio-podinfo\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-8mkqq\" (UID: \"dca3f84e-1191-48bb-a59c-8456088614ac\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-8mkqq"
Nov 02 19:48:35.015154 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:35.015137    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dca3f84e-1191-48bb-a59c-8456088614ac-credential-socket\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-8mkqq\" (UID: \"dca3f84e-1191-48bb-a59c-8456088614ac\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-8mkqq"
Nov 02 19:48:35.015247 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:35.015160    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dca3f84e-1191-48bb-a59c-8456088614ac-istio-token\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-8mkqq\" (UID: \"dca3f84e-1191-48bb-a59c-8456088614ac\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-8mkqq"
Nov 02 19:48:35.015247 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:35.015237    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dca3f84e-1191-48bb-a59c-8456088614ac-istio-data\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-8mkqq\" (UID: \"dca3f84e-1191-48bb-a59c-8456088614ac\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-8mkqq"
Nov 02 19:48:35.116709 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:35.116680    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dca3f84e-1191-48bb-a59c-8456088614ac-workload-socket\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-8mkqq\" (UID: \"dca3f84e-1191-48bb-a59c-8456088614ac\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-8mkqq"
Nov 02 19:48:35.116992 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:35.116723    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dca3f84e-1191-48bb-a59c-8456088614ac-workload-certs\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-8mkqq\" (UID: \"dca3f84e-1191-48bb-a59c-8456088614ac\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-8mkqq"
Nov 02 19:48:35.116992 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:35.116770    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dca3f84e-1191-48bb-a59c-8456088614ac-istiod-ca-cert\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-8mkqq\" (UID: \"dca3f84e-1191-48bb-a59c-8456088614ac\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-8mkqq"
Nov 02 19:48:35.116992 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:35.116793    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hllx4\" (UniqueName: \"kubernetes.io/projected/dca3f84e-1191-48bb-a59c-8456088614ac-kube-api-access-hllx4\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-8mkqq\" (UID: \"dca3f84e-1191-48bb-a59c-8456088614ac\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-8mkqq"
Nov 02 19:48:35.116992 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:35.116822    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dca3f84e-1191-48bb-a59c-8456088614ac-istio-envoy\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-8mkqq\" (UID: \"dca3f84e-1191-48bb-a59c-8456088614ac\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-8mkqq"
Nov 02 19:48:35.116992 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:35.116905    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dca3f84e-1191-48bb-a59c-8456088614ac-istio-podinfo\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-8mkqq\" (UID: \"dca3f84e-1191-48bb-a59c-8456088614ac\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-8mkqq"
Nov 02 19:48:35.116992 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:35.116981    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dca3f84e-1191-48bb-a59c-8456088614ac-credential-socket\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-8mkqq\" (UID: \"dca3f84e-1191-48bb-a59c-8456088614ac\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-8mkqq"
Nov 02 19:48:35.117390 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:35.117075    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dca3f84e-1191-48bb-a59c-8456088614ac-istio-token\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-8mkqq\" (UID: \"dca3f84e-1191-48bb-a59c-8456088614ac\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-8mkqq"
Nov 02 19:48:35.117390 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:35.117142    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dca3f84e-1191-48bb-a59c-8456088614ac-workload-socket\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-8mkqq\" (UID: \"dca3f84e-1191-48bb-a59c-8456088614ac\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-8mkqq"
Nov 02 19:48:35.117390 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:35.117205    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dca3f84e-1191-48bb-a59c-8456088614ac-workload-certs\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-8mkqq\" (UID: \"dca3f84e-1191-48bb-a59c-8456088614ac\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-8mkqq"
Nov 02 19:48:35.117552 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:35.117411    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dca3f84e-1191-48bb-a59c-8456088614ac-credential-socket\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-8mkqq\" (UID: \"dca3f84e-1191-48bb-a59c-8456088614ac\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-8mkqq"
Nov 02 19:48:35.117552 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:35.117488    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dca3f84e-1191-48bb-a59c-8456088614ac-istiod-ca-cert\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-8mkqq\" (UID: \"dca3f84e-1191-48bb-a59c-8456088614ac\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-8mkqq"
Nov 02 19:48:35.117552 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:35.117547    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dca3f84e-1191-48bb-a59c-8456088614ac-istio-data\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-8mkqq\" (UID: \"dca3f84e-1191-48bb-a59c-8456088614ac\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-8mkqq"
Nov 02 19:48:35.117834 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:35.117810    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dca3f84e-1191-48bb-a59c-8456088614ac-istio-data\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-8mkqq\" (UID: \"dca3f84e-1191-48bb-a59c-8456088614ac\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-8mkqq"
Nov 02 19:48:35.119220 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:35.119197    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dca3f84e-1191-48bb-a59c-8456088614ac-istio-envoy\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-8mkqq\" (UID: \"dca3f84e-1191-48bb-a59c-8456088614ac\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-8mkqq"
Nov 02 19:48:35.119435 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:35.119415    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dca3f84e-1191-48bb-a59c-8456088614ac-istio-podinfo\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-8mkqq\" (UID: \"dca3f84e-1191-48bb-a59c-8456088614ac\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-8mkqq"
Nov 02 19:48:35.170209 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:35.170177    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dca3f84e-1191-48bb-a59c-8456088614ac-istio-token\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-8mkqq\" (UID: \"dca3f84e-1191-48bb-a59c-8456088614ac\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-8mkqq"
Nov 02 19:48:35.170307 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:35.170216    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hllx4\" (UniqueName: \"kubernetes.io/projected/dca3f84e-1191-48bb-a59c-8456088614ac-kube-api-access-hllx4\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-8mkqq\" (UID: \"dca3f84e-1191-48bb-a59c-8456088614ac\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-8mkqq"
Nov 02 19:48:35.184811 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:35.184790    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-8mkqq"
Nov 02 19:48:35.602389 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:35.602363    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-8mkqq"]
Nov 02 19:48:35.603601 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:48:35.603576    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddca3f84e_1191_48bb_a59c_8456088614ac.slice/crio-0cd25feb44969d243509cf3925b8a424df5c0f646016103a017b5e2b9da71666 WatchSource:0}: Error finding container 0cd25feb44969d243509cf3925b8a424df5c0f646016103a017b5e2b9da71666: Status 404 returned error can't find the container with id 0cd25feb44969d243509cf3925b8a424df5c0f646016103a017b5e2b9da71666
Nov 02 19:48:35.605985 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:35.605951    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:48:35.606090 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:35.606031    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:48:35.606090 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:35.606061    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:48:35.848580 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:35.848541    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-8mkqq" event={"ID":"dca3f84e-1191-48bb-a59c-8456088614ac","Type":"ContainerStarted","Data":"3b618c0735a004eb627ce924fb547cfcf4ed904ae3d9732bf021b460a04c2d8b"}
Nov 02 19:48:35.848741 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:35.848630    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-8mkqq" event={"ID":"dca3f84e-1191-48bb-a59c-8456088614ac","Type":"ContainerStarted","Data":"0cd25feb44969d243509cf3925b8a424df5c0f646016103a017b5e2b9da71666"}
Nov 02 19:48:35.962746 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:35.962683    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-8mkqq" podStartSLOduration=1.962668727 podStartE2EDuration="1.962668727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:48:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:48:35.960778191 +0000 UTC m=+1222.567811971" watchObservedRunningTime="2026-11-02 19:48:35.962668727 +0000 UTC m=+1222.569702496"
Nov 02 19:48:36.184989 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:36.184950    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-8mkqq"
Nov 02 19:48:36.186553 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:36.186528    4463 patch_prober.go:29] interesting pod/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-8mkqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 02 19:48:36.186685 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:36.186577    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-8mkqq" podUID="dca3f84e-1191-48bb-a59c-8456088614ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 02 19:48:37.185350 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:37.185319    4463 patch_prober.go:29] interesting pod/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-8mkqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 02 19:48:37.185758 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:37.185371    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-8mkqq" podUID="dca3f84e-1191-48bb-a59c-8456088614ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 02 19:48:38.185141 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:38.185102    4463 patch_prober.go:29] interesting pod/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-8mkqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 02 19:48:38.185342 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:38.185152    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-8mkqq" podUID="dca3f84e-1191-48bb-a59c-8456088614ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 02 19:48:39.185576 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:39.185543    4463 patch_prober.go:29] interesting pod/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-8mkqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 02 19:48:39.186139 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:39.185598    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-8mkqq" podUID="dca3f84e-1191-48bb-a59c-8456088614ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 02 19:48:40.185490 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:40.185456    4463 patch_prober.go:29] interesting pod/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-8mkqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 02 19:48:40.185675 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:40.185508    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-8mkqq" podUID="dca3f84e-1191-48bb-a59c-8456088614ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 02 19:48:41.185512 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:41.185476    4463 patch_prober.go:29] interesting pod/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-8mkqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 02 19:48:41.185680 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:41.185548    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-8mkqq" podUID="dca3f84e-1191-48bb-a59c-8456088614ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 02 19:48:42.185774 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:42.185736    4463 patch_prober.go:29] interesting pod/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-8mkqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 02 19:48:42.186218 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:42.185790    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-8mkqq" podUID="dca3f84e-1191-48bb-a59c-8456088614ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 02 19:48:43.189656 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:43.189628    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-8mkqq"
Nov 02 19:48:43.190057 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:43.189886    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-8mkqq"
Nov 02 19:48:43.190661 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:43.190641    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-8mkqq"
Nov 02 19:48:50.160646 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:50.144205    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-kfx7k"]
Nov 02 19:48:50.187704 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:50.186184    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-kfx7k"
Nov 02 19:48:50.331058 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:50.305923    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0714f5c6-adfb-434e-8dd9-9bada5fbed29-istio-data\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-kfx7k\" (UID: \"0714f5c6-adfb-434e-8dd9-9bada5fbed29\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-kfx7k"
Nov 02 19:48:50.332043 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:50.331990    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kjvg\" (UniqueName: \"kubernetes.io/projected/0714f5c6-adfb-434e-8dd9-9bada5fbed29-kube-api-access-4kjvg\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-kfx7k\" (UID: \"0714f5c6-adfb-434e-8dd9-9bada5fbed29\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-kfx7k"
Nov 02 19:48:50.332280 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:50.332258    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0714f5c6-adfb-434e-8dd9-9bada5fbed29-credential-socket\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-kfx7k\" (UID: \"0714f5c6-adfb-434e-8dd9-9bada5fbed29\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-kfx7k"
Nov 02 19:48:50.332436 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:50.332417    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0714f5c6-adfb-434e-8dd9-9bada5fbed29-istiod-ca-cert\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-kfx7k\" (UID: \"0714f5c6-adfb-434e-8dd9-9bada5fbed29\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-kfx7k"
Nov 02 19:48:50.332638 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:50.332621    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0714f5c6-adfb-434e-8dd9-9bada5fbed29-istio-token\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-kfx7k\" (UID: \"0714f5c6-adfb-434e-8dd9-9bada5fbed29\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-kfx7k"
Nov 02 19:48:50.332766 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:50.332751    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0714f5c6-adfb-434e-8dd9-9bada5fbed29-workload-socket\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-kfx7k\" (UID: \"0714f5c6-adfb-434e-8dd9-9bada5fbed29\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-kfx7k"
Nov 02 19:48:50.332879 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:50.332864    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0714f5c6-adfb-434e-8dd9-9bada5fbed29-istio-podinfo\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-kfx7k\" (UID: \"0714f5c6-adfb-434e-8dd9-9bada5fbed29\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-kfx7k"
Nov 02 19:48:50.333036 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:50.333000    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0714f5c6-adfb-434e-8dd9-9bada5fbed29-workload-certs\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-kfx7k\" (UID: \"0714f5c6-adfb-434e-8dd9-9bada5fbed29\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-kfx7k"
Nov 02 19:48:50.333365 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:50.333346    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0714f5c6-adfb-434e-8dd9-9bada5fbed29-istio-envoy\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-kfx7k\" (UID: \"0714f5c6-adfb-434e-8dd9-9bada5fbed29\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-kfx7k"
Nov 02 19:48:50.384158 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:50.384062    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-kfx7k"]
Nov 02 19:48:50.440036 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:50.436856    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0714f5c6-adfb-434e-8dd9-9bada5fbed29-istio-envoy\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-kfx7k\" (UID: \"0714f5c6-adfb-434e-8dd9-9bada5fbed29\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-kfx7k"
Nov 02 19:48:50.440036 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:50.436914    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0714f5c6-adfb-434e-8dd9-9bada5fbed29-istio-data\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-kfx7k\" (UID: \"0714f5c6-adfb-434e-8dd9-9bada5fbed29\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-kfx7k"
Nov 02 19:48:50.440036 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:50.436963    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kjvg\" (UniqueName: \"kubernetes.io/projected/0714f5c6-adfb-434e-8dd9-9bada5fbed29-kube-api-access-4kjvg\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-kfx7k\" (UID: \"0714f5c6-adfb-434e-8dd9-9bada5fbed29\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-kfx7k"
Nov 02 19:48:50.440036 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:50.437027    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0714f5c6-adfb-434e-8dd9-9bada5fbed29-credential-socket\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-kfx7k\" (UID: \"0714f5c6-adfb-434e-8dd9-9bada5fbed29\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-kfx7k"
Nov 02 19:48:50.440036 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:50.437065    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0714f5c6-adfb-434e-8dd9-9bada5fbed29-istiod-ca-cert\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-kfx7k\" (UID: \"0714f5c6-adfb-434e-8dd9-9bada5fbed29\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-kfx7k"
Nov 02 19:48:50.440036 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:50.437156    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0714f5c6-adfb-434e-8dd9-9bada5fbed29-istio-token\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-kfx7k\" (UID: \"0714f5c6-adfb-434e-8dd9-9bada5fbed29\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-kfx7k"
Nov 02 19:48:50.440036 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:50.437194    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0714f5c6-adfb-434e-8dd9-9bada5fbed29-workload-socket\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-kfx7k\" (UID: \"0714f5c6-adfb-434e-8dd9-9bada5fbed29\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-kfx7k"
Nov 02 19:48:50.440036 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:50.437223    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0714f5c6-adfb-434e-8dd9-9bada5fbed29-istio-podinfo\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-kfx7k\" (UID: \"0714f5c6-adfb-434e-8dd9-9bada5fbed29\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-kfx7k"
Nov 02 19:48:50.440036 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:50.437271    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0714f5c6-adfb-434e-8dd9-9bada5fbed29-workload-certs\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-kfx7k\" (UID: \"0714f5c6-adfb-434e-8dd9-9bada5fbed29\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-kfx7k"
Nov 02 19:48:50.440036 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:50.437871    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0714f5c6-adfb-434e-8dd9-9bada5fbed29-workload-certs\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-kfx7k\" (UID: \"0714f5c6-adfb-434e-8dd9-9bada5fbed29\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-kfx7k"
Nov 02 19:48:50.441107 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:50.441081    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0714f5c6-adfb-434e-8dd9-9bada5fbed29-workload-socket\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-kfx7k\" (UID: \"0714f5c6-adfb-434e-8dd9-9bada5fbed29\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-kfx7k"
Nov 02 19:48:50.442454 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:50.442416    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0714f5c6-adfb-434e-8dd9-9bada5fbed29-istio-data\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-kfx7k\" (UID: \"0714f5c6-adfb-434e-8dd9-9bada5fbed29\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-kfx7k"
Nov 02 19:48:50.442911 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:50.442859    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0714f5c6-adfb-434e-8dd9-9bada5fbed29-credential-socket\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-kfx7k\" (UID: \"0714f5c6-adfb-434e-8dd9-9bada5fbed29\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-kfx7k"
Nov 02 19:48:50.467396 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:50.443682    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0714f5c6-adfb-434e-8dd9-9bada5fbed29-istiod-ca-cert\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-kfx7k\" (UID: \"0714f5c6-adfb-434e-8dd9-9bada5fbed29\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-kfx7k"
Nov 02 19:48:50.469574 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:50.469545    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0714f5c6-adfb-434e-8dd9-9bada5fbed29-istio-envoy\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-kfx7k\" (UID: \"0714f5c6-adfb-434e-8dd9-9bada5fbed29\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-kfx7k"
Nov 02 19:48:50.479033 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:50.472175    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0714f5c6-adfb-434e-8dd9-9bada5fbed29-istio-podinfo\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-kfx7k\" (UID: \"0714f5c6-adfb-434e-8dd9-9bada5fbed29\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-kfx7k"
Nov 02 19:48:50.539038 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:50.525119    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kjvg\" (UniqueName: \"kubernetes.io/projected/0714f5c6-adfb-434e-8dd9-9bada5fbed29-kube-api-access-4kjvg\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-kfx7k\" (UID: \"0714f5c6-adfb-434e-8dd9-9bada5fbed29\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-kfx7k"
Nov 02 19:48:50.562870 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:50.562555    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0714f5c6-adfb-434e-8dd9-9bada5fbed29-istio-token\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-kfx7k\" (UID: \"0714f5c6-adfb-434e-8dd9-9bada5fbed29\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-kfx7k"
Nov 02 19:48:50.816309 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:50.816209    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-kfx7k"
Nov 02 19:48:50.901360 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:50.901256    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-26cfbe/simulator-w1-26cfbe-kserve-74c5578fbd-t79hq" podUID="50dc76f4-aebd-41f4-9f23-799ddc8c70f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:48:50.950136 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:50.950093    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d869e/distinct-2-w5-1d869e-kserve-574f55db46-4cv47" podUID="7e236108-328b-4170-be20-13f16b56c4fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:48:51.985036 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:51.984026    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-43ceaa/premium-w0-43ceaa-kserve-67b55659b7-rckfd" podUID="7be32df1-5049-4f79-be23-a827f3f0a85f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 02 19:48:52.009380 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:51.999616    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d869e/distinct-2-w5-1d869e-kserve-574f55db46-4cv47" podUID="7e236108-328b-4170-be20-13f16b56c4fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:48:52.016043 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:52.012435    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-1d869e/distinct-2-w5-1d869e-kserve-574f55db46-4cv47"
Nov 02 19:48:52.092301 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:52.092256    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-26cfbe/simulator-w1-26cfbe-kserve-74c5578fbd-t79hq" podUID="50dc76f4-aebd-41f4-9f23-799ddc8c70f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:48:52.183080 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:52.146333    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-26cfbe/simulator-w1-26cfbe-kserve-74c5578fbd-t79hq"
Nov 02 19:48:52.187809 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:52.187787    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-kfx7k"]
Nov 02 19:48:52.527043 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:52.516301    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:48:52.527481 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:52.527427    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:48:52.531037 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:52.527723    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:48:52.805053 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:52.796401    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d869e/premium-w5-1d869e-kserve-7745799ff8-pv2th" podUID="1d784b10-99e4-4561-bfb7-6614877d2b6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 02 19:48:52.928054 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:52.922794    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-1d869e/distinct-2-w5-1d869e-kserve-574f55db46-4cv47"
Nov 02 19:48:52.961273 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:52.954615    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-kfx7k" event={"ID":"0714f5c6-adfb-434e-8dd9-9bada5fbed29","Type":"ContainerStarted","Data":"7fc900106940f9b3102f9499150872572164b1bf175f852d73a86c92e12dcd62"}
Nov 02 19:48:52.970416 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:52.967236    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-26cfbe/simulator-w1-26cfbe-kserve-74c5578fbd-t79hq"
Nov 02 19:48:53.981574 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:53.981536    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-kfx7k" event={"ID":"0714f5c6-adfb-434e-8dd9-9bada5fbed29","Type":"ContainerStarted","Data":"802ced2434f4911faa946da31c34e5d042f675a5e745cbd07869139eab1971a0"}
Nov 02 19:48:54.185699 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:54.185650    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-kfx7k" podStartSLOduration=5.185637134 podStartE2EDuration="5.185637134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:48:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:48:54.136489854 +0000 UTC m=+1240.743523624" watchObservedRunningTime="2026-11-02 19:48:54.185637134 +0000 UTC m=+1240.792670902"
Nov 02 19:48:54.817372 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:54.817340    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-kfx7k"
Nov 02 19:48:54.818683 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:54.818658    4463 patch_prober.go:29] interesting pod/e2e-worker-w2-28623b-openshift-default-5b66f6f775-kfx7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 02 19:48:54.818827 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:54.818706    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-kfx7k" podUID="0714f5c6-adfb-434e-8dd9-9bada5fbed29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 02 19:48:55.817666 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:55.817633    4463 patch_prober.go:29] interesting pod/e2e-worker-w2-28623b-openshift-default-5b66f6f775-kfx7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 02 19:48:55.818090 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:55.817692    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-kfx7k" podUID="0714f5c6-adfb-434e-8dd9-9bada5fbed29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 02 19:48:56.817781 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:56.817750    4463 patch_prober.go:29] interesting pod/e2e-worker-w2-28623b-openshift-default-5b66f6f775-kfx7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 02 19:48:56.818239 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:56.818209    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-kfx7k" podUID="0714f5c6-adfb-434e-8dd9-9bada5fbed29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 02 19:48:57.817893 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:57.817860    4463 patch_prober.go:29] interesting pod/e2e-worker-w2-28623b-openshift-default-5b66f6f775-kfx7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 02 19:48:57.818274 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:57.817916    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-kfx7k" podUID="0714f5c6-adfb-434e-8dd9-9bada5fbed29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 02 19:48:58.817235 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:58.817203    4463 patch_prober.go:29] interesting pod/e2e-worker-w2-28623b-openshift-default-5b66f6f775-kfx7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 02 19:48:58.817410 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:48:58.817254    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-kfx7k" podUID="0714f5c6-adfb-434e-8dd9-9bada5fbed29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 02 19:49:00.232876 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:00.232828    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-kfx7k"
Nov 02 19:49:00.233540 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:00.233513    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-kfx7k"
Nov 02 19:49:00.234336 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:00.234316    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-kfx7k"
Nov 02 19:49:04.294795 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:49:04.294760    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 19:49:05.068364 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:05.068330    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-csdm8"]
Nov 02 19:49:05.072401 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:05.072384    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-csdm8"
Nov 02 19:49:05.170415 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:05.170383    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-csdm8"]
Nov 02 19:49:05.223634 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:05.223605    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b51fb09-651f-4dc4-afbd-8734da30febe-credential-socket\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-csdm8\" (UID: \"6b51fb09-651f-4dc4-afbd-8734da30febe\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-csdm8"
Nov 02 19:49:05.223786 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:05.223658    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b51fb09-651f-4dc4-afbd-8734da30febe-workload-certs\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-csdm8\" (UID: \"6b51fb09-651f-4dc4-afbd-8734da30febe\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-csdm8"
Nov 02 19:49:05.223786 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:05.223677    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jggsm\" (UniqueName: \"kubernetes.io/projected/6b51fb09-651f-4dc4-afbd-8734da30febe-kube-api-access-jggsm\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-csdm8\" (UID: \"6b51fb09-651f-4dc4-afbd-8734da30febe\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-csdm8"
Nov 02 19:49:05.223859 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:05.223786    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b51fb09-651f-4dc4-afbd-8734da30febe-istio-podinfo\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-csdm8\" (UID: \"6b51fb09-651f-4dc4-afbd-8734da30febe\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-csdm8"
Nov 02 19:49:05.223894 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:05.223885    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b51fb09-651f-4dc4-afbd-8734da30febe-istio-data\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-csdm8\" (UID: \"6b51fb09-651f-4dc4-afbd-8734da30febe\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-csdm8"
Nov 02 19:49:05.223931 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:05.223906    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b51fb09-651f-4dc4-afbd-8734da30febe-istio-token\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-csdm8\" (UID: \"6b51fb09-651f-4dc4-afbd-8734da30febe\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-csdm8"
Nov 02 19:49:05.224053 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:05.224036    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b51fb09-651f-4dc4-afbd-8734da30febe-istio-envoy\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-csdm8\" (UID: \"6b51fb09-651f-4dc4-afbd-8734da30febe\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-csdm8"
Nov 02 19:49:05.224092 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:05.224069    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b51fb09-651f-4dc4-afbd-8734da30febe-workload-socket\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-csdm8\" (UID: \"6b51fb09-651f-4dc4-afbd-8734da30febe\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-csdm8"
Nov 02 19:49:05.224125 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:05.224092    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b51fb09-651f-4dc4-afbd-8734da30febe-istiod-ca-cert\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-csdm8\" (UID: \"6b51fb09-651f-4dc4-afbd-8734da30febe\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-csdm8"
Nov 02 19:49:05.325267 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:05.325183    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b51fb09-651f-4dc4-afbd-8734da30febe-istio-envoy\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-csdm8\" (UID: \"6b51fb09-651f-4dc4-afbd-8734da30febe\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-csdm8"
Nov 02 19:49:05.325267 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:05.325227    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b51fb09-651f-4dc4-afbd-8734da30febe-workload-socket\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-csdm8\" (UID: \"6b51fb09-651f-4dc4-afbd-8734da30febe\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-csdm8"
Nov 02 19:49:05.325743 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:05.325340    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b51fb09-651f-4dc4-afbd-8734da30febe-istiod-ca-cert\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-csdm8\" (UID: \"6b51fb09-651f-4dc4-afbd-8734da30febe\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-csdm8"
Nov 02 19:49:05.325743 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:05.325456    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b51fb09-651f-4dc4-afbd-8734da30febe-credential-socket\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-csdm8\" (UID: \"6b51fb09-651f-4dc4-afbd-8734da30febe\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-csdm8"
Nov 02 19:49:05.325743 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:05.325531    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b51fb09-651f-4dc4-afbd-8734da30febe-workload-certs\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-csdm8\" (UID: \"6b51fb09-651f-4dc4-afbd-8734da30febe\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-csdm8"
Nov 02 19:49:05.325743 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:05.325557    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jggsm\" (UniqueName: \"kubernetes.io/projected/6b51fb09-651f-4dc4-afbd-8734da30febe-kube-api-access-jggsm\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-csdm8\" (UID: \"6b51fb09-651f-4dc4-afbd-8734da30febe\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-csdm8"
Nov 02 19:49:05.325743 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:05.325592    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b51fb09-651f-4dc4-afbd-8734da30febe-workload-socket\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-csdm8\" (UID: \"6b51fb09-651f-4dc4-afbd-8734da30febe\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-csdm8"
Nov 02 19:49:05.325743 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:05.325644    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b51fb09-651f-4dc4-afbd-8734da30febe-istio-podinfo\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-csdm8\" (UID: \"6b51fb09-651f-4dc4-afbd-8734da30febe\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-csdm8"
Nov 02 19:49:05.326116 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:05.325765    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b51fb09-651f-4dc4-afbd-8734da30febe-istio-data\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-csdm8\" (UID: \"6b51fb09-651f-4dc4-afbd-8734da30febe\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-csdm8"
Nov 02 19:49:05.326116 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:05.325797    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b51fb09-651f-4dc4-afbd-8734da30febe-istio-token\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-csdm8\" (UID: \"6b51fb09-651f-4dc4-afbd-8734da30febe\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-csdm8"
Nov 02 19:49:05.326116 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:05.325763    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b51fb09-651f-4dc4-afbd-8734da30febe-credential-socket\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-csdm8\" (UID: \"6b51fb09-651f-4dc4-afbd-8734da30febe\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-csdm8"
Nov 02 19:49:05.326116 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:05.325923    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b51fb09-651f-4dc4-afbd-8734da30febe-workload-certs\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-csdm8\" (UID: \"6b51fb09-651f-4dc4-afbd-8734da30febe\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-csdm8"
Nov 02 19:49:05.334174 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:05.326122    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b51fb09-651f-4dc4-afbd-8734da30febe-istio-data\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-csdm8\" (UID: \"6b51fb09-651f-4dc4-afbd-8734da30febe\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-csdm8"
Nov 02 19:49:05.334174 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:05.327079    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b51fb09-651f-4dc4-afbd-8734da30febe-istiod-ca-cert\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-csdm8\" (UID: \"6b51fb09-651f-4dc4-afbd-8734da30febe\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-csdm8"
Nov 02 19:49:05.334174 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:05.330832    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b51fb09-651f-4dc4-afbd-8734da30febe-istio-envoy\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-csdm8\" (UID: \"6b51fb09-651f-4dc4-afbd-8734da30febe\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-csdm8"
Nov 02 19:49:05.334174 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:05.331342    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b51fb09-651f-4dc4-afbd-8734da30febe-istio-podinfo\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-csdm8\" (UID: \"6b51fb09-651f-4dc4-afbd-8734da30febe\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-csdm8"
Nov 02 19:49:05.379997 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:05.379966    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b51fb09-651f-4dc4-afbd-8734da30febe-istio-token\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-csdm8\" (UID: \"6b51fb09-651f-4dc4-afbd-8734da30febe\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-csdm8"
Nov 02 19:49:05.380171 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:05.380154    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jggsm\" (UniqueName: \"kubernetes.io/projected/6b51fb09-651f-4dc4-afbd-8734da30febe-kube-api-access-jggsm\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-csdm8\" (UID: \"6b51fb09-651f-4dc4-afbd-8734da30febe\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-csdm8"
Nov 02 19:49:05.384988 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:05.384970    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-csdm8"
Nov 02 19:49:05.805255 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:05.805226    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-csdm8"]
Nov 02 19:49:05.806260 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:49:05.806236    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b51fb09_651f_4dc4_afbd_8734da30febe.slice/crio-e05444dc75bda49409548c3c40ae9f5ec3006fdd182fc21c8cebf8bdc36dc4f5 WatchSource:0}: Error finding container e05444dc75bda49409548c3c40ae9f5ec3006fdd182fc21c8cebf8bdc36dc4f5: Status 404 returned error can't find the container with id e05444dc75bda49409548c3c40ae9f5ec3006fdd182fc21c8cebf8bdc36dc4f5
Nov 02 19:49:05.808788 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:05.808758    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:49:05.808873 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:05.808821    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:49:05.808873 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:05.808851    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:49:06.070364 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:06.070273    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-csdm8" event={"ID":"6b51fb09-651f-4dc4-afbd-8734da30febe","Type":"ContainerStarted","Data":"1a90b50b9718d62aeebd3d406eff50c97330d794e7bf083bf26e8f3b1bf4962a"}
Nov 02 19:49:06.070364 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:06.070338    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-csdm8" event={"ID":"6b51fb09-651f-4dc4-afbd-8734da30febe","Type":"ContainerStarted","Data":"e05444dc75bda49409548c3c40ae9f5ec3006fdd182fc21c8cebf8bdc36dc4f5"}
Nov 02 19:49:06.184351 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:06.184296    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-csdm8" podStartSLOduration=1.184281676 podStartE2EDuration="1.184281676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:49:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:49:06.1797107 +0000 UTC m=+1252.786744489" watchObservedRunningTime="2026-11-02 19:49:06.184281676 +0000 UTC m=+1252.791315446"
Nov 02 19:49:06.386101 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:06.385976    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-csdm8"
Nov 02 19:49:06.388447 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:06.388424    4463 patch_prober.go:29] interesting pod/e2e-worker-w2-28623b-openshift-default-5b66f6f775-csdm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 02 19:49:06.388584 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:06.388470    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-csdm8" podUID="6b51fb09-651f-4dc4-afbd-8734da30febe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 02 19:49:07.385681 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:07.385640    4463 patch_prober.go:29] interesting pod/e2e-worker-w2-28623b-openshift-default-5b66f6f775-csdm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 02 19:49:07.385867 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:07.385705    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-csdm8" podUID="6b51fb09-651f-4dc4-afbd-8734da30febe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 02 19:49:08.385681 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:08.385645    4463 patch_prober.go:29] interesting pod/e2e-worker-w2-28623b-openshift-default-5b66f6f775-csdm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 02 19:49:08.386194 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:08.385705    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-csdm8" podUID="6b51fb09-651f-4dc4-afbd-8734da30febe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 02 19:49:09.385910 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:09.385874    4463 patch_prober.go:29] interesting pod/e2e-worker-w2-28623b-openshift-default-5b66f6f775-csdm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 02 19:49:09.386315 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:09.385928    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-csdm8" podUID="6b51fb09-651f-4dc4-afbd-8734da30febe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 02 19:49:10.385671 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:10.385638    4463 patch_prober.go:29] interesting pod/e2e-worker-w2-28623b-openshift-default-5b66f6f775-csdm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 02 19:49:10.385852 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:10.385684    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-csdm8" podUID="6b51fb09-651f-4dc4-afbd-8734da30febe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 02 19:49:11.385295 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:11.385259    4463 patch_prober.go:29] interesting pod/e2e-worker-w2-28623b-openshift-default-5b66f6f775-csdm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 02 19:49:11.385691 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:11.385309    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-csdm8" podUID="6b51fb09-651f-4dc4-afbd-8734da30febe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 02 19:49:12.385766 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:12.385728    4463 patch_prober.go:29] interesting pod/e2e-worker-w2-28623b-openshift-default-5b66f6f775-csdm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 02 19:49:12.386182 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:12.385790    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-csdm8" podUID="6b51fb09-651f-4dc4-afbd-8734da30febe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 02 19:49:13.093253 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:13.093216    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 19:49:13.390522 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:13.390448    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-csdm8"
Nov 02 19:49:13.390980 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:13.390938    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-csdm8"
Nov 02 19:49:13.391720 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:13.391687    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-csdm8"
Nov 02 19:49:15.565459 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:15.564545    4463 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-8795bf/distinct-2-w4-8795bf-kserve-69c8b6b885-2chhn" podUID="9f430131-4938-47ca-ba86-b6a236f8cc3b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": context deadline exceeded"
Nov 02 19:49:15.661237 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:15.660806    4463 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-8795bf/distinct-w4-8795bf-kserve-64b4796df4-8qgqh" podUID="ee7d28f4-e9c7-43e2-b983-1427589f1785" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": context deadline exceeded"
Nov 02 19:49:15.705048 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:15.703030    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d869e/premium-w5-1d869e-kserve-7745799ff8-pv2th" podUID="1d784b10-99e4-4561-bfb7-6614877d2b6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 02 19:49:15.705048 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:15.703428    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8795bf/distinct-w4-8795bf-kserve-64b4796df4-8qgqh" podUID="ee7d28f4-e9c7-43e2-b983-1427589f1785" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Nov 02 19:49:15.705564 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:15.705536    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5c46af/distinct-w3-5c46af-kserve-68b6dff7f5-bp59f" podUID="8dd6362e-7dbd-489b-b6d8-0d9caab0fa69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:49:15.885566 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:15.885471    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-43ceaa/premium-w0-43ceaa-kserve-67b55659b7-rckfd" podUID="7be32df1-5049-4f79-be23-a827f3f0a85f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 02 19:49:15.930215 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:15.930171    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d869e/distinct-2-w5-1d869e-kserve-574f55db46-4cv47" podUID="7e236108-328b-4170-be20-13f16b56c4fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:49:15.974971 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:15.974933    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-26cfbe/simulator-w1-26cfbe-kserve-74c5578fbd-t79hq" podUID="50dc76f4-aebd-41f4-9f23-799ddc8c70f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 02 19:49:16.015487 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:16.015452    4463 patch_prober.go:29] interesting pod/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-w5gw2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 19:49:16.015767 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:16.015739    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-w5gw2" podUID="d09ad81c-4422-4b1a-b845-a616f5c9e93c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:49:16.136230 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:16.136103    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8795bf/distinct-2-w4-8795bf-kserve-69c8b6b885-2chhn" podUID="9f430131-4938-47ca-ba86-b6a236f8cc3b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:49:16.460798 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:16.460749    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-26cfbe/distinct-w1-26cfbe-kserve-685985bfb-tkgn2" podUID="7415aa62-467c-43af-94c8-d604f57ec1c3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:49:16.706131 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:16.706084    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8795bf/distinct-w4-8795bf-kserve-64b4796df4-8qgqh" podUID="ee7d28f4-e9c7-43e2-b983-1427589f1785" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:49:16.708480 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:16.708452    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d869e/premium-w5-1d869e-kserve-7745799ff8-pv2th" podUID="1d784b10-99e4-4561-bfb7-6614877d2b6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 02 19:49:16.717419 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:16.709089    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5c46af/distinct-w3-5c46af-kserve-68b6dff7f5-bp59f" podUID="8dd6362e-7dbd-489b-b6d8-0d9caab0fa69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Nov 02 19:49:16.717419 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:16.709499    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8795bf/distinct-w4-8795bf-kserve-64b4796df4-8qgqh"
Nov 02 19:49:16.717419 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:16.709555    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-1d869e/premium-w5-1d869e-kserve-7745799ff8-pv2th"
Nov 02 19:49:16.717419 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:16.709592    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5c46af/distinct-w3-5c46af-kserve-68b6dff7f5-bp59f"
Nov 02 19:49:16.784814 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:16.782943    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5c46af/distinct-w3-5c46af-kserve-68b6dff7f5-bp59f"
Nov 02 19:49:16.831033 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:16.830471    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-1d869e/premium-w5-1d869e-kserve-7745799ff8-pv2th"
Nov 02 19:49:16.995100 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:16.985967    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8795bf/distinct-w4-8795bf-kserve-64b4796df4-8qgqh"
Nov 02 19:49:19.935763 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:19.935731    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fjnvm"]
Nov 02 19:49:19.941290 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:19.941265    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fjnvm"
Nov 02 19:49:19.994683 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:19.994649    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48eb73cb-231a-4462-9236-da01f4734826-workload-socket\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-fjnvm\" (UID: \"48eb73cb-231a-4462-9236-da01f4734826\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fjnvm"
Nov 02 19:49:19.994804 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:19.994773    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48eb73cb-231a-4462-9236-da01f4734826-istio-envoy\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-fjnvm\" (UID: \"48eb73cb-231a-4462-9236-da01f4734826\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fjnvm"
Nov 02 19:49:19.994975 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:19.994948    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48eb73cb-231a-4462-9236-da01f4734826-workload-certs\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-fjnvm\" (UID: \"48eb73cb-231a-4462-9236-da01f4734826\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fjnvm"
Nov 02 19:49:19.995136 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:19.995044    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48eb73cb-231a-4462-9236-da01f4734826-istiod-ca-cert\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-fjnvm\" (UID: \"48eb73cb-231a-4462-9236-da01f4734826\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fjnvm"
Nov 02 19:49:19.995136 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:19.995089    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2p4c\" (UniqueName: \"kubernetes.io/projected/48eb73cb-231a-4462-9236-da01f4734826-kube-api-access-l2p4c\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-fjnvm\" (UID: \"48eb73cb-231a-4462-9236-da01f4734826\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fjnvm"
Nov 02 19:49:19.995256 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:19.995156    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48eb73cb-231a-4462-9236-da01f4734826-istio-podinfo\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-fjnvm\" (UID: \"48eb73cb-231a-4462-9236-da01f4734826\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fjnvm"
Nov 02 19:49:19.995256 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:19.995190    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48eb73cb-231a-4462-9236-da01f4734826-istio-token\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-fjnvm\" (UID: \"48eb73cb-231a-4462-9236-da01f4734826\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fjnvm"
Nov 02 19:49:19.995256 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:19.995216    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48eb73cb-231a-4462-9236-da01f4734826-credential-socket\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-fjnvm\" (UID: \"48eb73cb-231a-4462-9236-da01f4734826\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fjnvm"
Nov 02 19:49:19.995406 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:19.995253    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48eb73cb-231a-4462-9236-da01f4734826-istio-data\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-fjnvm\" (UID: \"48eb73cb-231a-4462-9236-da01f4734826\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fjnvm"
Nov 02 19:49:20.044755 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:20.044725    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fjnvm"]
Nov 02 19:49:20.096809 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:20.096778    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48eb73cb-231a-4462-9236-da01f4734826-workload-certs\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-fjnvm\" (UID: \"48eb73cb-231a-4462-9236-da01f4734826\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fjnvm"
Nov 02 19:49:20.096932 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:20.096839    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48eb73cb-231a-4462-9236-da01f4734826-istiod-ca-cert\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-fjnvm\" (UID: \"48eb73cb-231a-4462-9236-da01f4734826\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fjnvm"
Nov 02 19:49:20.096932 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:20.096874    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2p4c\" (UniqueName: \"kubernetes.io/projected/48eb73cb-231a-4462-9236-da01f4734826-kube-api-access-l2p4c\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-fjnvm\" (UID: \"48eb73cb-231a-4462-9236-da01f4734826\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fjnvm"
Nov 02 19:49:20.096932 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:20.096920    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48eb73cb-231a-4462-9236-da01f4734826-istio-podinfo\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-fjnvm\" (UID: \"48eb73cb-231a-4462-9236-da01f4734826\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fjnvm"
Nov 02 19:49:20.097115 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:20.096947    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48eb73cb-231a-4462-9236-da01f4734826-istio-token\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-fjnvm\" (UID: \"48eb73cb-231a-4462-9236-da01f4734826\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fjnvm"
Nov 02 19:49:20.097115 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:20.097047    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48eb73cb-231a-4462-9236-da01f4734826-credential-socket\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-fjnvm\" (UID: \"48eb73cb-231a-4462-9236-da01f4734826\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fjnvm"
Nov 02 19:49:20.097115 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:20.097094    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48eb73cb-231a-4462-9236-da01f4734826-istio-data\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-fjnvm\" (UID: \"48eb73cb-231a-4462-9236-da01f4734826\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fjnvm"
Nov 02 19:49:20.097272 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:20.097137    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48eb73cb-231a-4462-9236-da01f4734826-workload-socket\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-fjnvm\" (UID: \"48eb73cb-231a-4462-9236-da01f4734826\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fjnvm"
Nov 02 19:49:20.097272 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:20.097217    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48eb73cb-231a-4462-9236-da01f4734826-istio-envoy\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-fjnvm\" (UID: \"48eb73cb-231a-4462-9236-da01f4734826\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fjnvm"
Nov 02 19:49:20.097272 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:20.097228    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48eb73cb-231a-4462-9236-da01f4734826-workload-certs\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-fjnvm\" (UID: \"48eb73cb-231a-4462-9236-da01f4734826\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fjnvm"
Nov 02 19:49:20.097432 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:20.097375    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48eb73cb-231a-4462-9236-da01f4734826-credential-socket\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-fjnvm\" (UID: \"48eb73cb-231a-4462-9236-da01f4734826\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fjnvm"
Nov 02 19:49:20.097599 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:20.097545    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48eb73cb-231a-4462-9236-da01f4734826-istio-data\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-fjnvm\" (UID: \"48eb73cb-231a-4462-9236-da01f4734826\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fjnvm"
Nov 02 19:49:20.097661 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:20.097614    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48eb73cb-231a-4462-9236-da01f4734826-istiod-ca-cert\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-fjnvm\" (UID: \"48eb73cb-231a-4462-9236-da01f4734826\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fjnvm"
Nov 02 19:49:20.097714 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:20.097646    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48eb73cb-231a-4462-9236-da01f4734826-workload-socket\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-fjnvm\" (UID: \"48eb73cb-231a-4462-9236-da01f4734826\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fjnvm"
Nov 02 19:49:20.099804 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:20.099779    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48eb73cb-231a-4462-9236-da01f4734826-istio-envoy\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-fjnvm\" (UID: \"48eb73cb-231a-4462-9236-da01f4734826\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fjnvm"
Nov 02 19:49:20.099986 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:20.099963    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48eb73cb-231a-4462-9236-da01f4734826-istio-podinfo\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-fjnvm\" (UID: \"48eb73cb-231a-4462-9236-da01f4734826\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fjnvm"
Nov 02 19:49:20.150359 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:20.150329    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48eb73cb-231a-4462-9236-da01f4734826-istio-token\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-fjnvm\" (UID: \"48eb73cb-231a-4462-9236-da01f4734826\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fjnvm"
Nov 02 19:49:20.150545 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:20.150517    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2p4c\" (UniqueName: \"kubernetes.io/projected/48eb73cb-231a-4462-9236-da01f4734826-kube-api-access-l2p4c\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-fjnvm\" (UID: \"48eb73cb-231a-4462-9236-da01f4734826\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fjnvm"
Nov 02 19:49:20.254603 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:20.254531    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fjnvm"
Nov 02 19:49:20.674575 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:20.674541    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fjnvm"]
Nov 02 19:49:20.675554 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:49:20.675528    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48eb73cb_231a_4462_9236_da01f4734826.slice/crio-8175216fcbb7e57cbe0715d86f490807c1dc7e9cf01d2c2e1144bd1d260e83c4 WatchSource:0}: Error finding container 8175216fcbb7e57cbe0715d86f490807c1dc7e9cf01d2c2e1144bd1d260e83c4: Status 404 returned error can't find the container with id 8175216fcbb7e57cbe0715d86f490807c1dc7e9cf01d2c2e1144bd1d260e83c4
Nov 02 19:49:20.678040 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:20.677983    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:49:20.678132 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:20.678073    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:49:20.678132 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:20.678109    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:49:21.235048 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:21.234991    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fjnvm" event={"ID":"48eb73cb-231a-4462-9236-da01f4734826","Type":"ContainerStarted","Data":"c4b358dd25656c7e93235a7be81307b24c00d8675f0add06277a95d120bfbfc0"}
Nov 02 19:49:21.235468 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:21.235063    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fjnvm" event={"ID":"48eb73cb-231a-4462-9236-da01f4734826","Type":"ContainerStarted","Data":"8175216fcbb7e57cbe0715d86f490807c1dc7e9cf01d2c2e1144bd1d260e83c4"}
Nov 02 19:49:21.254806 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:21.254778    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fjnvm"
Nov 02 19:49:21.256536 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:21.256504    4463 patch_prober.go:29] interesting pod/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fjnvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 02 19:49:21.256641 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:21.256566    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fjnvm" podUID="48eb73cb-231a-4462-9236-da01f4734826" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 02 19:49:21.428891 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:21.428838    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fjnvm" podStartSLOduration=2.428824209 podStartE2EDuration="2.428824209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:49:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:49:21.389725276 +0000 UTC m=+1267.996759056" watchObservedRunningTime="2026-11-02 19:49:21.428824209 +0000 UTC m=+1268.035857978"
Nov 02 19:49:22.274183 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:22.274135    4463 patch_prober.go:29] interesting pod/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fjnvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 02 19:49:22.274587 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:22.274206    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fjnvm" podUID="48eb73cb-231a-4462-9236-da01f4734826" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 02 19:49:23.271437 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:23.271398    4463 patch_prober.go:29] interesting pod/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fjnvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 02 19:49:23.271966 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:23.271930    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fjnvm" podUID="48eb73cb-231a-4462-9236-da01f4734826" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 02 19:49:24.260400 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:24.260364    4463 patch_prober.go:29] interesting pod/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fjnvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 02 19:49:24.260985 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:24.260955    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fjnvm" podUID="48eb73cb-231a-4462-9236-da01f4734826" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 02 19:49:25.274394 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:25.274359    4463 patch_prober.go:29] interesting pod/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fjnvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 02 19:49:25.274848 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:25.274424    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fjnvm" podUID="48eb73cb-231a-4462-9236-da01f4734826" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 02 19:49:26.270244 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:26.270210    4463 patch_prober.go:29] interesting pod/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fjnvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 02 19:49:26.270512 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:26.270483    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fjnvm" podUID="48eb73cb-231a-4462-9236-da01f4734826" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 02 19:49:27.255113 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:27.255074    4463 patch_prober.go:29] interesting pod/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fjnvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 02 19:49:27.255507 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:27.255138    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fjnvm" podUID="48eb73cb-231a-4462-9236-da01f4734826" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 02 19:49:28.259097 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:28.259066    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fjnvm"
Nov 02 19:49:28.259512 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:28.259484    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fjnvm"
Nov 02 19:49:28.260126 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:28.260105    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fjnvm"
Nov 02 19:49:30.757247 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:30.757214    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-d8ffab69-gw-openshift-default-79d965fd8f-dkcf5"]
Nov 02 19:49:30.761571 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:30.761546    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-d8ffab69-gw-openshift-default-79d965fd8f-dkcf5"
Nov 02 19:49:30.836174 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:30.836140    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-d8ffab69-gw-openshift-default-79d965fd8f-dkcf5"]
Nov 02 19:49:30.858848 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:30.858824    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-d8ffab69-gw-openshift-default-dockercfg-8glh6\""
Nov 02 19:49:30.885869 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:30.885841    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d1ebf76-5f16-412b-8784-8c29dd59ca56-istiod-ca-cert\") pod \"e2e-migrate-d8ffab69-gw-openshift-default-79d965fd8f-dkcf5\" (UID: \"1d1ebf76-5f16-412b-8784-8c29dd59ca56\") " pod="openshift-ingress/e2e-migrate-d8ffab69-gw-openshift-default-79d965fd8f-dkcf5"
Nov 02 19:49:30.885964 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:30.885887    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d1ebf76-5f16-412b-8784-8c29dd59ca56-credential-socket\") pod \"e2e-migrate-d8ffab69-gw-openshift-default-79d965fd8f-dkcf5\" (UID: \"1d1ebf76-5f16-412b-8784-8c29dd59ca56\") " pod="openshift-ingress/e2e-migrate-d8ffab69-gw-openshift-default-79d965fd8f-dkcf5"
Nov 02 19:49:30.885964 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:30.885955    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d1ebf76-5f16-412b-8784-8c29dd59ca56-istio-podinfo\") pod \"e2e-migrate-d8ffab69-gw-openshift-default-79d965fd8f-dkcf5\" (UID: \"1d1ebf76-5f16-412b-8784-8c29dd59ca56\") " pod="openshift-ingress/e2e-migrate-d8ffab69-gw-openshift-default-79d965fd8f-dkcf5"
Nov 02 19:49:30.886078 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:30.886062    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d1ebf76-5f16-412b-8784-8c29dd59ca56-istio-token\") pod \"e2e-migrate-d8ffab69-gw-openshift-default-79d965fd8f-dkcf5\" (UID: \"1d1ebf76-5f16-412b-8784-8c29dd59ca56\") " pod="openshift-ingress/e2e-migrate-d8ffab69-gw-openshift-default-79d965fd8f-dkcf5"
Nov 02 19:49:30.886111 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:30.886106    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d1ebf76-5f16-412b-8784-8c29dd59ca56-istio-data\") pod \"e2e-migrate-d8ffab69-gw-openshift-default-79d965fd8f-dkcf5\" (UID: \"1d1ebf76-5f16-412b-8784-8c29dd59ca56\") " pod="openshift-ingress/e2e-migrate-d8ffab69-gw-openshift-default-79d965fd8f-dkcf5"
Nov 02 19:49:30.886151 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:30.886127    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9tjq\" (UniqueName: \"kubernetes.io/projected/1d1ebf76-5f16-412b-8784-8c29dd59ca56-kube-api-access-c9tjq\") pod \"e2e-migrate-d8ffab69-gw-openshift-default-79d965fd8f-dkcf5\" (UID: \"1d1ebf76-5f16-412b-8784-8c29dd59ca56\") " pod="openshift-ingress/e2e-migrate-d8ffab69-gw-openshift-default-79d965fd8f-dkcf5"
Nov 02 19:49:30.886286 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:30.886268    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d1ebf76-5f16-412b-8784-8c29dd59ca56-istio-envoy\") pod \"e2e-migrate-d8ffab69-gw-openshift-default-79d965fd8f-dkcf5\" (UID: \"1d1ebf76-5f16-412b-8784-8c29dd59ca56\") " pod="openshift-ingress/e2e-migrate-d8ffab69-gw-openshift-default-79d965fd8f-dkcf5"
Nov 02 19:49:30.886322 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:30.886308    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d1ebf76-5f16-412b-8784-8c29dd59ca56-workload-certs\") pod \"e2e-migrate-d8ffab69-gw-openshift-default-79d965fd8f-dkcf5\" (UID: \"1d1ebf76-5f16-412b-8784-8c29dd59ca56\") " pod="openshift-ingress/e2e-migrate-d8ffab69-gw-openshift-default-79d965fd8f-dkcf5"
Nov 02 19:49:30.886423 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:30.886411    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d1ebf76-5f16-412b-8784-8c29dd59ca56-workload-socket\") pod \"e2e-migrate-d8ffab69-gw-openshift-default-79d965fd8f-dkcf5\" (UID: \"1d1ebf76-5f16-412b-8784-8c29dd59ca56\") " pod="openshift-ingress/e2e-migrate-d8ffab69-gw-openshift-default-79d965fd8f-dkcf5"
Nov 02 19:49:30.987074 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:30.987046    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d1ebf76-5f16-412b-8784-8c29dd59ca56-istio-token\") pod \"e2e-migrate-d8ffab69-gw-openshift-default-79d965fd8f-dkcf5\" (UID: \"1d1ebf76-5f16-412b-8784-8c29dd59ca56\") " pod="openshift-ingress/e2e-migrate-d8ffab69-gw-openshift-default-79d965fd8f-dkcf5"
Nov 02 19:49:30.987219 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:30.987112    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d1ebf76-5f16-412b-8784-8c29dd59ca56-istio-data\") pod \"e2e-migrate-d8ffab69-gw-openshift-default-79d965fd8f-dkcf5\" (UID: \"1d1ebf76-5f16-412b-8784-8c29dd59ca56\") " pod="openshift-ingress/e2e-migrate-d8ffab69-gw-openshift-default-79d965fd8f-dkcf5"
Nov 02 19:49:30.987219 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:30.987131    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9tjq\" (UniqueName: \"kubernetes.io/projected/1d1ebf76-5f16-412b-8784-8c29dd59ca56-kube-api-access-c9tjq\") pod \"e2e-migrate-d8ffab69-gw-openshift-default-79d965fd8f-dkcf5\" (UID: \"1d1ebf76-5f16-412b-8784-8c29dd59ca56\") " pod="openshift-ingress/e2e-migrate-d8ffab69-gw-openshift-default-79d965fd8f-dkcf5"
Nov 02 19:49:30.987219 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:30.987202    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d1ebf76-5f16-412b-8784-8c29dd59ca56-istio-envoy\") pod \"e2e-migrate-d8ffab69-gw-openshift-default-79d965fd8f-dkcf5\" (UID: \"1d1ebf76-5f16-412b-8784-8c29dd59ca56\") " pod="openshift-ingress/e2e-migrate-d8ffab69-gw-openshift-default-79d965fd8f-dkcf5"
Nov 02 19:49:30.987354 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:30.987234    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d1ebf76-5f16-412b-8784-8c29dd59ca56-workload-certs\") pod \"e2e-migrate-d8ffab69-gw-openshift-default-79d965fd8f-dkcf5\" (UID: \"1d1ebf76-5f16-412b-8784-8c29dd59ca56\") " pod="openshift-ingress/e2e-migrate-d8ffab69-gw-openshift-default-79d965fd8f-dkcf5"
Nov 02 19:49:30.987354 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:30.987315    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d1ebf76-5f16-412b-8784-8c29dd59ca56-workload-socket\") pod \"e2e-migrate-d8ffab69-gw-openshift-default-79d965fd8f-dkcf5\" (UID: \"1d1ebf76-5f16-412b-8784-8c29dd59ca56\") " pod="openshift-ingress/e2e-migrate-d8ffab69-gw-openshift-default-79d965fd8f-dkcf5"
Nov 02 19:49:30.987443 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:30.987353    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d1ebf76-5f16-412b-8784-8c29dd59ca56-istiod-ca-cert\") pod \"e2e-migrate-d8ffab69-gw-openshift-default-79d965fd8f-dkcf5\" (UID: \"1d1ebf76-5f16-412b-8784-8c29dd59ca56\") " pod="openshift-ingress/e2e-migrate-d8ffab69-gw-openshift-default-79d965fd8f-dkcf5"
Nov 02 19:49:30.987443 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:30.987397    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d1ebf76-5f16-412b-8784-8c29dd59ca56-credential-socket\") pod \"e2e-migrate-d8ffab69-gw-openshift-default-79d965fd8f-dkcf5\" (UID: \"1d1ebf76-5f16-412b-8784-8c29dd59ca56\") " pod="openshift-ingress/e2e-migrate-d8ffab69-gw-openshift-default-79d965fd8f-dkcf5"
Nov 02 19:49:30.987443 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:30.987437    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d1ebf76-5f16-412b-8784-8c29dd59ca56-istio-podinfo\") pod \"e2e-migrate-d8ffab69-gw-openshift-default-79d965fd8f-dkcf5\" (UID: \"1d1ebf76-5f16-412b-8784-8c29dd59ca56\") " pod="openshift-ingress/e2e-migrate-d8ffab69-gw-openshift-default-79d965fd8f-dkcf5"
Nov 02 19:49:30.987615 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:30.987532    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d1ebf76-5f16-412b-8784-8c29dd59ca56-istio-data\") pod \"e2e-migrate-d8ffab69-gw-openshift-default-79d965fd8f-dkcf5\" (UID: \"1d1ebf76-5f16-412b-8784-8c29dd59ca56\") " pod="openshift-ingress/e2e-migrate-d8ffab69-gw-openshift-default-79d965fd8f-dkcf5"
Nov 02 19:49:30.987734 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:30.987711    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d1ebf76-5f16-412b-8784-8c29dd59ca56-workload-certs\") pod \"e2e-migrate-d8ffab69-gw-openshift-default-79d965fd8f-dkcf5\" (UID: \"1d1ebf76-5f16-412b-8784-8c29dd59ca56\") " pod="openshift-ingress/e2e-migrate-d8ffab69-gw-openshift-default-79d965fd8f-dkcf5"
Nov 02 19:49:30.987844 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:30.987824    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d1ebf76-5f16-412b-8784-8c29dd59ca56-credential-socket\") pod \"e2e-migrate-d8ffab69-gw-openshift-default-79d965fd8f-dkcf5\" (UID: \"1d1ebf76-5f16-412b-8784-8c29dd59ca56\") " pod="openshift-ingress/e2e-migrate-d8ffab69-gw-openshift-default-79d965fd8f-dkcf5"
Nov 02 19:49:30.988042 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:30.988001    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d1ebf76-5f16-412b-8784-8c29dd59ca56-istiod-ca-cert\") pod \"e2e-migrate-d8ffab69-gw-openshift-default-79d965fd8f-dkcf5\" (UID: \"1d1ebf76-5f16-412b-8784-8c29dd59ca56\") " pod="openshift-ingress/e2e-migrate-d8ffab69-gw-openshift-default-79d965fd8f-dkcf5"
Nov 02 19:49:30.988276 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:30.988230    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d1ebf76-5f16-412b-8784-8c29dd59ca56-workload-socket\") pod \"e2e-migrate-d8ffab69-gw-openshift-default-79d965fd8f-dkcf5\" (UID: \"1d1ebf76-5f16-412b-8784-8c29dd59ca56\") " pod="openshift-ingress/e2e-migrate-d8ffab69-gw-openshift-default-79d965fd8f-dkcf5"
Nov 02 19:49:30.989854 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:30.989823    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d1ebf76-5f16-412b-8784-8c29dd59ca56-istio-envoy\") pod \"e2e-migrate-d8ffab69-gw-openshift-default-79d965fd8f-dkcf5\" (UID: \"1d1ebf76-5f16-412b-8784-8c29dd59ca56\") " pod="openshift-ingress/e2e-migrate-d8ffab69-gw-openshift-default-79d965fd8f-dkcf5"
Nov 02 19:49:30.990042 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:30.989991    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d1ebf76-5f16-412b-8784-8c29dd59ca56-istio-podinfo\") pod \"e2e-migrate-d8ffab69-gw-openshift-default-79d965fd8f-dkcf5\" (UID: \"1d1ebf76-5f16-412b-8784-8c29dd59ca56\") " pod="openshift-ingress/e2e-migrate-d8ffab69-gw-openshift-default-79d965fd8f-dkcf5"
Nov 02 19:49:31.041268 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:31.041183    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d1ebf76-5f16-412b-8784-8c29dd59ca56-istio-token\") pod \"e2e-migrate-d8ffab69-gw-openshift-default-79d965fd8f-dkcf5\" (UID: \"1d1ebf76-5f16-412b-8784-8c29dd59ca56\") " pod="openshift-ingress/e2e-migrate-d8ffab69-gw-openshift-default-79d965fd8f-dkcf5"
Nov 02 19:49:31.043987 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:31.043959    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9tjq\" (UniqueName: \"kubernetes.io/projected/1d1ebf76-5f16-412b-8784-8c29dd59ca56-kube-api-access-c9tjq\") pod \"e2e-migrate-d8ffab69-gw-openshift-default-79d965fd8f-dkcf5\" (UID: \"1d1ebf76-5f16-412b-8784-8c29dd59ca56\") " pod="openshift-ingress/e2e-migrate-d8ffab69-gw-openshift-default-79d965fd8f-dkcf5"
Nov 02 19:49:31.075632 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:31.075604    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-d8ffab69-gw-openshift-default-79d965fd8f-dkcf5"
Nov 02 19:49:31.518619 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:31.518586    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-d8ffab69-gw-openshift-default-79d965fd8f-dkcf5"]
Nov 02 19:49:31.529331 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:31.529299    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:49:31.529442 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:31.529377    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:49:31.529442 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:31.529407    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:49:32.413757 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:32.413718    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d8ffab69-gw-openshift-default-79d965fd8f-dkcf5" event={"ID":"1d1ebf76-5f16-412b-8784-8c29dd59ca56","Type":"ContainerStarted","Data":"3b51f80eed47519d070dc7ad7dd3f5092cbfcf2046d9def95d164c5cfa873a55"}
Nov 02 19:49:32.414171 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:32.413775    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d8ffab69-gw-openshift-default-79d965fd8f-dkcf5" event={"ID":"1d1ebf76-5f16-412b-8784-8c29dd59ca56","Type":"ContainerStarted","Data":"d999d584cf7d50901f321c9f9580adbad69e4c3a5e6929e4dab33563b546fcc7"}
Nov 02 19:49:32.525296 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:32.525243    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-d8ffab69-gw-openshift-default-79d965fd8f-dkcf5" podStartSLOduration=2.52522918 podStartE2EDuration="2.52522918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:49:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:49:32.521916746 +0000 UTC m=+1279.128950516" watchObservedRunningTime="2026-11-02 19:49:32.52522918 +0000 UTC m=+1279.132262949"
Nov 02 19:49:33.078390 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:33.077611    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-d8ffab69-gw-openshift-default-79d965fd8f-dkcf5"
Nov 02 19:49:33.096144 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:33.096111    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-d8ffab69-gw-openshift-default-79d965fd8f-dkcf5"
Nov 02 19:49:33.420134 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:33.420056    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-d8ffab69-gw-openshift-default-79d965fd8f-dkcf5"
Nov 02 19:49:33.421179 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:33.421156    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-d8ffab69-gw-openshift-default-79d965fd8f-dkcf5"
Nov 02 19:49:34.294194 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:49:34.294158    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 19:49:34.838362 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:34.838327    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-qdr9m"]
Nov 02 19:49:34.842695 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:34.842674    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-qdr9m"
Nov 02 19:49:34.970334 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:34.970303    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/115b980d-6906-4134-8657-9e02a2e6a478-workload-socket\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-qdr9m\" (UID: \"115b980d-6906-4134-8657-9e02a2e6a478\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-qdr9m"
Nov 02 19:49:34.970522 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:34.970361    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/115b980d-6906-4134-8657-9e02a2e6a478-istio-token\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-qdr9m\" (UID: \"115b980d-6906-4134-8657-9e02a2e6a478\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-qdr9m"
Nov 02 19:49:34.970589 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:34.970521    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/115b980d-6906-4134-8657-9e02a2e6a478-workload-certs\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-qdr9m\" (UID: \"115b980d-6906-4134-8657-9e02a2e6a478\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-qdr9m"
Nov 02 19:49:34.970589 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:34.970564    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2n82\" (UniqueName: \"kubernetes.io/projected/115b980d-6906-4134-8657-9e02a2e6a478-kube-api-access-h2n82\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-qdr9m\" (UID: \"115b980d-6906-4134-8657-9e02a2e6a478\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-qdr9m"
Nov 02 19:49:34.970697 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:34.970655    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/115b980d-6906-4134-8657-9e02a2e6a478-istio-envoy\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-qdr9m\" (UID: \"115b980d-6906-4134-8657-9e02a2e6a478\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-qdr9m"
Nov 02 19:49:34.970697 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:34.970687    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/115b980d-6906-4134-8657-9e02a2e6a478-istiod-ca-cert\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-qdr9m\" (UID: \"115b980d-6906-4134-8657-9e02a2e6a478\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-qdr9m"
Nov 02 19:49:34.970819 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:34.970800    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/115b980d-6906-4134-8657-9e02a2e6a478-credential-socket\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-qdr9m\" (UID: \"115b980d-6906-4134-8657-9e02a2e6a478\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-qdr9m"
Nov 02 19:49:34.970904 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:34.970861    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/115b980d-6906-4134-8657-9e02a2e6a478-istio-podinfo\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-qdr9m\" (UID: \"115b980d-6906-4134-8657-9e02a2e6a478\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-qdr9m"
Nov 02 19:49:34.970961 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:34.970939    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/115b980d-6906-4134-8657-9e02a2e6a478-istio-data\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-qdr9m\" (UID: \"115b980d-6906-4134-8657-9e02a2e6a478\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-qdr9m"
Nov 02 19:49:35.029521 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:35.029476    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-bbt8n"]
Nov 02 19:49:35.037927 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:35.035316    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-bbt8n"
Nov 02 19:49:35.065477 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:35.065448    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-bbt8n"]
Nov 02 19:49:35.072211 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:35.072178    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/115b980d-6906-4134-8657-9e02a2e6a478-workload-certs\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-qdr9m\" (UID: \"115b980d-6906-4134-8657-9e02a2e6a478\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-qdr9m"
Nov 02 19:49:35.072390 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:35.072237    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2n82\" (UniqueName: \"kubernetes.io/projected/115b980d-6906-4134-8657-9e02a2e6a478-kube-api-access-h2n82\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-qdr9m\" (UID: \"115b980d-6906-4134-8657-9e02a2e6a478\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-qdr9m"
Nov 02 19:49:35.072390 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:35.072327    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/115b980d-6906-4134-8657-9e02a2e6a478-istio-envoy\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-qdr9m\" (UID: \"115b980d-6906-4134-8657-9e02a2e6a478\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-qdr9m"
Nov 02 19:49:35.072390 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:35.072356    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/115b980d-6906-4134-8657-9e02a2e6a478-istiod-ca-cert\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-qdr9m\" (UID: \"115b980d-6906-4134-8657-9e02a2e6a478\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-qdr9m"
Nov 02 19:49:35.072549 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:35.072407    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/115b980d-6906-4134-8657-9e02a2e6a478-credential-socket\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-qdr9m\" (UID: \"115b980d-6906-4134-8657-9e02a2e6a478\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-qdr9m"
Nov 02 19:49:35.072549 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:35.072461    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/115b980d-6906-4134-8657-9e02a2e6a478-istio-podinfo\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-qdr9m\" (UID: \"115b980d-6906-4134-8657-9e02a2e6a478\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-qdr9m"
Nov 02 19:49:35.072549 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:35.072536    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/115b980d-6906-4134-8657-9e02a2e6a478-istio-data\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-qdr9m\" (UID: \"115b980d-6906-4134-8657-9e02a2e6a478\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-qdr9m"
Nov 02 19:49:35.072703 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:35.072608    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/115b980d-6906-4134-8657-9e02a2e6a478-workload-certs\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-qdr9m\" (UID: \"115b980d-6906-4134-8657-9e02a2e6a478\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-qdr9m"
Nov 02 19:49:35.072703 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:35.072628    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/115b980d-6906-4134-8657-9e02a2e6a478-workload-socket\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-qdr9m\" (UID: \"115b980d-6906-4134-8657-9e02a2e6a478\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-qdr9m"
Nov 02 19:49:35.072703 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:35.072696    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/115b980d-6906-4134-8657-9e02a2e6a478-istio-token\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-qdr9m\" (UID: \"115b980d-6906-4134-8657-9e02a2e6a478\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-qdr9m"
Nov 02 19:49:35.073439 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:35.073411    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/115b980d-6906-4134-8657-9e02a2e6a478-workload-socket\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-qdr9m\" (UID: \"115b980d-6906-4134-8657-9e02a2e6a478\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-qdr9m"
Nov 02 19:49:35.073721 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:35.073700    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/115b980d-6906-4134-8657-9e02a2e6a478-credential-socket\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-qdr9m\" (UID: \"115b980d-6906-4134-8657-9e02a2e6a478\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-qdr9m"
Nov 02 19:49:35.073990 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:35.073968    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/115b980d-6906-4134-8657-9e02a2e6a478-istiod-ca-cert\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-qdr9m\" (UID: \"115b980d-6906-4134-8657-9e02a2e6a478\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-qdr9m"
Nov 02 19:49:35.074435 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:35.074410    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/115b980d-6906-4134-8657-9e02a2e6a478-istio-data\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-qdr9m\" (UID: \"115b980d-6906-4134-8657-9e02a2e6a478\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-qdr9m"
Nov 02 19:49:35.077124 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:35.077102    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/115b980d-6906-4134-8657-9e02a2e6a478-istio-envoy\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-qdr9m\" (UID: \"115b980d-6906-4134-8657-9e02a2e6a478\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-qdr9m"
Nov 02 19:49:35.077938 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:35.077913    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/115b980d-6906-4134-8657-9e02a2e6a478-istio-podinfo\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-qdr9m\" (UID: \"115b980d-6906-4134-8657-9e02a2e6a478\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-qdr9m"
Nov 02 19:49:35.090665 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:35.090600    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-qdr9m"]
Nov 02 19:49:35.131940 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:35.131912    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/115b980d-6906-4134-8657-9e02a2e6a478-istio-token\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-qdr9m\" (UID: \"115b980d-6906-4134-8657-9e02a2e6a478\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-qdr9m"
Nov 02 19:49:35.132501 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:35.132476    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2n82\" (UniqueName: \"kubernetes.io/projected/115b980d-6906-4134-8657-9e02a2e6a478-kube-api-access-h2n82\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-qdr9m\" (UID: \"115b980d-6906-4134-8657-9e02a2e6a478\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-qdr9m"
Nov 02 19:49:35.156870 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:35.156843    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-qdr9m"
Nov 02 19:49:35.173978 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:35.173946    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0acfeac9-a741-45ec-afab-8fb4ecf5975f-istiod-ca-cert\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-bbt8n\" (UID: \"0acfeac9-a741-45ec-afab-8fb4ecf5975f\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-bbt8n"
Nov 02 19:49:35.174130 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:35.173996    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0acfeac9-a741-45ec-afab-8fb4ecf5975f-credential-socket\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-bbt8n\" (UID: \"0acfeac9-a741-45ec-afab-8fb4ecf5975f\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-bbt8n"
Nov 02 19:49:35.174202 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:35.174161    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0acfeac9-a741-45ec-afab-8fb4ecf5975f-workload-socket\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-bbt8n\" (UID: \"0acfeac9-a741-45ec-afab-8fb4ecf5975f\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-bbt8n"
Nov 02 19:49:35.174328 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:35.174276    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0acfeac9-a741-45ec-afab-8fb4ecf5975f-istio-token\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-bbt8n\" (UID: \"0acfeac9-a741-45ec-afab-8fb4ecf5975f\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-bbt8n"
Nov 02 19:49:35.174397 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:35.174349    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0acfeac9-a741-45ec-afab-8fb4ecf5975f-istio-envoy\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-bbt8n\" (UID: \"0acfeac9-a741-45ec-afab-8fb4ecf5975f\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-bbt8n"
Nov 02 19:49:35.174397 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:35.174391    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0acfeac9-a741-45ec-afab-8fb4ecf5975f-istio-data\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-bbt8n\" (UID: \"0acfeac9-a741-45ec-afab-8fb4ecf5975f\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-bbt8n"
Nov 02 19:49:35.174518 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:35.174503    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0acfeac9-a741-45ec-afab-8fb4ecf5975f-workload-certs\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-bbt8n\" (UID: \"0acfeac9-a741-45ec-afab-8fb4ecf5975f\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-bbt8n"
Nov 02 19:49:35.174578 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:35.174556    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkqst\" (UniqueName: \"kubernetes.io/projected/0acfeac9-a741-45ec-afab-8fb4ecf5975f-kube-api-access-vkqst\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-bbt8n\" (UID: \"0acfeac9-a741-45ec-afab-8fb4ecf5975f\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-bbt8n"
Nov 02 19:49:35.174765 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:35.174670    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0acfeac9-a741-45ec-afab-8fb4ecf5975f-istio-podinfo\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-bbt8n\" (UID: \"0acfeac9-a741-45ec-afab-8fb4ecf5975f\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-bbt8n"
Nov 02 19:49:35.276220 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:35.276174    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0acfeac9-a741-45ec-afab-8fb4ecf5975f-workload-certs\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-bbt8n\" (UID: \"0acfeac9-a741-45ec-afab-8fb4ecf5975f\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-bbt8n"
Nov 02 19:49:35.276389 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:35.276253    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkqst\" (UniqueName: \"kubernetes.io/projected/0acfeac9-a741-45ec-afab-8fb4ecf5975f-kube-api-access-vkqst\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-bbt8n\" (UID: \"0acfeac9-a741-45ec-afab-8fb4ecf5975f\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-bbt8n"
Nov 02 19:49:35.276389 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:35.276314    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0acfeac9-a741-45ec-afab-8fb4ecf5975f-istio-podinfo\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-bbt8n\" (UID: \"0acfeac9-a741-45ec-afab-8fb4ecf5975f\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-bbt8n"
Nov 02 19:49:35.276502 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:35.276443    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0acfeac9-a741-45ec-afab-8fb4ecf5975f-istiod-ca-cert\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-bbt8n\" (UID: \"0acfeac9-a741-45ec-afab-8fb4ecf5975f\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-bbt8n"
Nov 02 19:49:35.276502 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:35.276477    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0acfeac9-a741-45ec-afab-8fb4ecf5975f-credential-socket\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-bbt8n\" (UID: \"0acfeac9-a741-45ec-afab-8fb4ecf5975f\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-bbt8n"
Nov 02 19:49:35.276603 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:35.276535    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0acfeac9-a741-45ec-afab-8fb4ecf5975f-workload-socket\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-bbt8n\" (UID: \"0acfeac9-a741-45ec-afab-8fb4ecf5975f\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-bbt8n"
Nov 02 19:49:35.276680 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:35.276647    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0acfeac9-a741-45ec-afab-8fb4ecf5975f-istio-token\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-bbt8n\" (UID: \"0acfeac9-a741-45ec-afab-8fb4ecf5975f\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-bbt8n"
Nov 02 19:49:35.276680 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:35.276667    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0acfeac9-a741-45ec-afab-8fb4ecf5975f-workload-certs\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-bbt8n\" (UID: \"0acfeac9-a741-45ec-afab-8fb4ecf5975f\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-bbt8n"
Nov 02 19:49:35.276840 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:35.276705    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0acfeac9-a741-45ec-afab-8fb4ecf5975f-istio-envoy\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-bbt8n\" (UID: \"0acfeac9-a741-45ec-afab-8fb4ecf5975f\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-bbt8n"
Nov 02 19:49:35.276840 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:35.276753    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0acfeac9-a741-45ec-afab-8fb4ecf5975f-istio-data\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-bbt8n\" (UID: \"0acfeac9-a741-45ec-afab-8fb4ecf5975f\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-bbt8n"
Nov 02 19:49:35.277312 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:35.276948    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0acfeac9-a741-45ec-afab-8fb4ecf5975f-workload-socket\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-bbt8n\" (UID: \"0acfeac9-a741-45ec-afab-8fb4ecf5975f\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-bbt8n"
Nov 02 19:49:35.277312 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:35.277196    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0acfeac9-a741-45ec-afab-8fb4ecf5975f-istio-data\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-bbt8n\" (UID: \"0acfeac9-a741-45ec-afab-8fb4ecf5975f\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-bbt8n"
Nov 02 19:49:35.277312 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:35.277238    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0acfeac9-a741-45ec-afab-8fb4ecf5975f-credential-socket\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-bbt8n\" (UID: \"0acfeac9-a741-45ec-afab-8fb4ecf5975f\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-bbt8n"
Nov 02 19:49:35.277546 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:35.277377    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0acfeac9-a741-45ec-afab-8fb4ecf5975f-istiod-ca-cert\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-bbt8n\" (UID: \"0acfeac9-a741-45ec-afab-8fb4ecf5975f\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-bbt8n"
Nov 02 19:49:35.279115 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:35.279089    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0acfeac9-a741-45ec-afab-8fb4ecf5975f-istio-envoy\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-bbt8n\" (UID: \"0acfeac9-a741-45ec-afab-8fb4ecf5975f\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-bbt8n"
Nov 02 19:49:35.279666 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:35.279646    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0acfeac9-a741-45ec-afab-8fb4ecf5975f-istio-podinfo\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-bbt8n\" (UID: \"0acfeac9-a741-45ec-afab-8fb4ecf5975f\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-bbt8n"
Nov 02 19:49:35.329356 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:35.329335    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0acfeac9-a741-45ec-afab-8fb4ecf5975f-istio-token\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-bbt8n\" (UID: \"0acfeac9-a741-45ec-afab-8fb4ecf5975f\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-bbt8n"
Nov 02 19:49:35.329446 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:35.329422    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkqst\" (UniqueName: \"kubernetes.io/projected/0acfeac9-a741-45ec-afab-8fb4ecf5975f-kube-api-access-vkqst\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-bbt8n\" (UID: \"0acfeac9-a741-45ec-afab-8fb4ecf5975f\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-bbt8n"
Nov 02 19:49:35.363342 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:35.363291    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-bbt8n"
Nov 02 19:49:35.599896 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:35.599871    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-qdr9m"]
Nov 02 19:49:35.600603 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:49:35.600581    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod115b980d_6906_4134_8657_9e02a2e6a478.slice/crio-22cc4bfb2622b5d517a0c66f7e9553e58f45929c441787fd5b37aa9eae8da54f WatchSource:0}: Error finding container 22cc4bfb2622b5d517a0c66f7e9553e58f45929c441787fd5b37aa9eae8da54f: Status 404 returned error can't find the container with id 22cc4bfb2622b5d517a0c66f7e9553e58f45929c441787fd5b37aa9eae8da54f
Nov 02 19:49:35.602952 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:35.602923    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:49:35.603028 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:35.602984    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:49:35.603028 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:35.603023    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:49:35.736944 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:35.736911    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-bbt8n"]
Nov 02 19:49:35.738217 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:49:35.738181    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0acfeac9_a741_45ec_afab_8fb4ecf5975f.slice/crio-f86bde7b887e0d7ee739ddd45c6bdff4b405dfbb6ffca9e18aed1f7f8b993bec WatchSource:0}: Error finding container f86bde7b887e0d7ee739ddd45c6bdff4b405dfbb6ffca9e18aed1f7f8b993bec: Status 404 returned error can't find the container with id f86bde7b887e0d7ee739ddd45c6bdff4b405dfbb6ffca9e18aed1f7f8b993bec
Nov 02 19:49:35.741805 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:35.741763    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:49:35.741893 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:35.741877    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:49:35.741934 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:35.741919    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:49:36.496267 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:36.496199    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-qdr9m" event={"ID":"115b980d-6906-4134-8657-9e02a2e6a478","Type":"ContainerStarted","Data":"5788700e73dcedd7189aa5f250e4ae86c920062691f401a9c44d7cdd40a672aa"}
Nov 02 19:49:36.496763 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:36.496296    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-qdr9m" event={"ID":"115b980d-6906-4134-8657-9e02a2e6a478","Type":"ContainerStarted","Data":"22cc4bfb2622b5d517a0c66f7e9553e58f45929c441787fd5b37aa9eae8da54f"}
Nov 02 19:49:36.500909 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:36.500875    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-bbt8n" event={"ID":"0acfeac9-a741-45ec-afab-8fb4ecf5975f","Type":"ContainerStarted","Data":"1a3769a6f7367c62e1966d701f12fbf651e3b9dd1909b6e0b9f53ee37e507052"}
Nov 02 19:49:36.501240 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:36.501215    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-bbt8n" event={"ID":"0acfeac9-a741-45ec-afab-8fb4ecf5975f","Type":"ContainerStarted","Data":"f86bde7b887e0d7ee739ddd45c6bdff4b405dfbb6ffca9e18aed1f7f8b993bec"}
Nov 02 19:49:36.625863 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:36.625810    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-qdr9m" podStartSLOduration=2.625791671 podStartE2EDuration="2.625791671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:49:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:49:36.619550287 +0000 UTC m=+1283.226584066" watchObservedRunningTime="2026-11-02 19:49:36.625791671 +0000 UTC m=+1283.232825444"
Nov 02 19:49:36.775095 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:36.773697    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-bbt8n" podStartSLOduration=2.7736796679999998 podStartE2EDuration="2.773679668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:49:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:49:36.770370218 +0000 UTC m=+1283.377403986" watchObservedRunningTime="2026-11-02 19:49:36.773679668 +0000 UTC m=+1283.380713436"
Nov 02 19:49:37.163308 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:37.163209    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-qdr9m"
Nov 02 19:49:37.163683 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:37.163657    4463 patch_prober.go:29] interesting pod/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-qdr9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 02 19:49:37.163861 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:37.163826    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-qdr9m" podUID="115b980d-6906-4134-8657-9e02a2e6a478" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 02 19:49:37.370256 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:37.370224    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-bbt8n"
Nov 02 19:49:37.370596 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:37.370574    4463 patch_prober.go:29] interesting pod/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-bbt8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 02 19:49:37.370765 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:37.370741    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-bbt8n" podUID="0acfeac9-a741-45ec-afab-8fb4ecf5975f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 02 19:49:38.158036 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:38.157311    4463 patch_prober.go:29] interesting pod/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-qdr9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 02 19:49:38.158036 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:38.157368    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-qdr9m" podUID="115b980d-6906-4134-8657-9e02a2e6a478" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 02 19:49:38.367791 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:38.367595    4463 patch_prober.go:29] interesting pod/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-bbt8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 02 19:49:38.367791 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:38.367655    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-bbt8n" podUID="0acfeac9-a741-45ec-afab-8fb4ecf5975f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 02 19:49:39.166037 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:39.162518    4463 patch_prober.go:29] interesting pod/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-qdr9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 02 19:49:39.166037 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:39.162585    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-qdr9m" podUID="115b980d-6906-4134-8657-9e02a2e6a478" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 02 19:49:39.395059 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:39.384414    4463 patch_prober.go:29] interesting pod/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-bbt8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 02 19:49:39.395059 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:39.384481    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-bbt8n" podUID="0acfeac9-a741-45ec-afab-8fb4ecf5975f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 02 19:49:39.741678 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:39.741643    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d8ffab69-c7d7fc585-g5qln"]
Nov 02 19:49:39.747199 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:39.747174    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d8ffab69-c7d7fc585-g5qln"]
Nov 02 19:49:39.747313 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:39.747289    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-d8ffab69-c7d7fc585-g5qln"
Nov 02 19:49:39.839285 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:39.838343    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d99e7b7-63b9-4e1b-aa57-4ab098b789b4-plugins-config-volume\") pod \"payload-processing-e2e-migrate-d8ffab69-c7d7fc585-g5qln\" (UID: \"2d99e7b7-63b9-4e1b-aa57-4ab098b789b4\") " pod="openshift-ingress/payload-processing-e2e-migrate-d8ffab69-c7d7fc585-g5qln"
Nov 02 19:49:39.839285 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:39.838395    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdxx8\" (UniqueName: \"kubernetes.io/projected/2d99e7b7-63b9-4e1b-aa57-4ab098b789b4-kube-api-access-hdxx8\") pod \"payload-processing-e2e-migrate-d8ffab69-c7d7fc585-g5qln\" (UID: \"2d99e7b7-63b9-4e1b-aa57-4ab098b789b4\") " pod="openshift-ingress/payload-processing-e2e-migrate-d8ffab69-c7d7fc585-g5qln"
Nov 02 19:49:39.841800 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:39.841762    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-d8ffab69\""
Nov 02 19:49:39.843166 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:39.843145    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-d8ffab69-dockercfg-982tx\""
Nov 02 19:49:39.953038 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:39.949186    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d99e7b7-63b9-4e1b-aa57-4ab098b789b4-plugins-config-volume\") pod \"payload-processing-e2e-migrate-d8ffab69-c7d7fc585-g5qln\" (UID: \"2d99e7b7-63b9-4e1b-aa57-4ab098b789b4\") " pod="openshift-ingress/payload-processing-e2e-migrate-d8ffab69-c7d7fc585-g5qln"
Nov 02 19:49:39.954226 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:39.954200    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdxx8\" (UniqueName: \"kubernetes.io/projected/2d99e7b7-63b9-4e1b-aa57-4ab098b789b4-kube-api-access-hdxx8\") pod \"payload-processing-e2e-migrate-d8ffab69-c7d7fc585-g5qln\" (UID: \"2d99e7b7-63b9-4e1b-aa57-4ab098b789b4\") " pod="openshift-ingress/payload-processing-e2e-migrate-d8ffab69-c7d7fc585-g5qln"
Nov 02 19:49:39.954804 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:39.954133    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d99e7b7-63b9-4e1b-aa57-4ab098b789b4-plugins-config-volume\") pod \"payload-processing-e2e-migrate-d8ffab69-c7d7fc585-g5qln\" (UID: \"2d99e7b7-63b9-4e1b-aa57-4ab098b789b4\") " pod="openshift-ingress/payload-processing-e2e-migrate-d8ffab69-c7d7fc585-g5qln"
Nov 02 19:49:40.042799 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:40.042724    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdxx8\" (UniqueName: \"kubernetes.io/projected/2d99e7b7-63b9-4e1b-aa57-4ab098b789b4-kube-api-access-hdxx8\") pod \"payload-processing-e2e-migrate-d8ffab69-c7d7fc585-g5qln\" (UID: \"2d99e7b7-63b9-4e1b-aa57-4ab098b789b4\") " pod="openshift-ingress/payload-processing-e2e-migrate-d8ffab69-c7d7fc585-g5qln"
Nov 02 19:49:40.147786 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:40.147749    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-d8ffab69-c7d7fc585-g5qln"
Nov 02 19:49:40.190248 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:40.166120    4463 patch_prober.go:29] interesting pod/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-qdr9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 02 19:49:40.190248 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:40.166225    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-qdr9m" podUID="115b980d-6906-4134-8657-9e02a2e6a478" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 02 19:49:40.399243 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:40.399151    4463 patch_prober.go:29] interesting pod/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-bbt8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 02 19:49:40.399598 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:40.399563    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-bbt8n" podUID="0acfeac9-a741-45ec-afab-8fb4ecf5975f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 02 19:49:40.910632 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:40.910607    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d8ffab69-c7d7fc585-g5qln"]
Nov 02 19:49:41.178034 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:41.177336    4463 patch_prober.go:29] interesting pod/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-qdr9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 02 19:49:41.178034 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:41.177393    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-qdr9m" podUID="115b980d-6906-4134-8657-9e02a2e6a478" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 02 19:49:41.369070 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:41.364416    4463 patch_prober.go:29] interesting pod/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-bbt8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 02 19:49:41.369070 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:41.364474    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-bbt8n" podUID="0acfeac9-a741-45ec-afab-8fb4ecf5975f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 02 19:49:41.774037 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:41.771302    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-d8ffab69-c7d7fc585-g5qln" event={"ID":"2d99e7b7-63b9-4e1b-aa57-4ab098b789b4","Type":"ContainerStarted","Data":"0674c1400768fd94dfa36a257a4bad9cbbea921e648dac6e86760ff7708fb608"}
Nov 02 19:49:41.774037 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:41.771369    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-d8ffab69-c7d7fc585-g5qln" event={"ID":"2d99e7b7-63b9-4e1b-aa57-4ab098b789b4","Type":"ContainerStarted","Data":"71e19dcb6a5cd7cc51c63e82672eda5e8611ccdd588aaa77f58872ba1c60a75b"}
Nov 02 19:49:41.774037 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:41.771423    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-d8ffab69-c7d7fc585-g5qln"
Nov 02 19:49:41.887289 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:41.887230    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-d8ffab69-c7d7fc585-g5qln" podStartSLOduration=2.88721182 podStartE2EDuration="2.88721182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:49:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:49:41.881946127 +0000 UTC m=+1288.488979897" watchObservedRunningTime="2026-11-02 19:49:41.88721182 +0000 UTC m=+1288.494245591"
Nov 02 19:49:42.157618 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:42.157525    4463 patch_prober.go:29] interesting pod/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-qdr9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 02 19:49:42.157618 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:42.157581    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-qdr9m" podUID="115b980d-6906-4134-8657-9e02a2e6a478" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 02 19:49:42.365297 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:42.364572    4463 patch_prober.go:29] interesting pod/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-bbt8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 02 19:49:42.365297 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:42.364644    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-bbt8n" podUID="0acfeac9-a741-45ec-afab-8fb4ecf5975f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 02 19:49:43.473593 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:43.473560    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-qdr9m"
Nov 02 19:49:43.475754 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:43.475725    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-qdr9m"
Nov 02 19:49:43.477444 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:43.477427    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-qdr9m"
Nov 02 19:49:43.559064 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:43.559036    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-bbt8n"
Nov 02 19:49:43.559469 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:43.559444    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-bbt8n"
Nov 02 19:49:43.560199 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:43.560178    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-bbt8n"
Nov 02 19:49:43.718687 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:43.718660    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-66lh4"]
Nov 02 19:49:43.723107 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:43.723086    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-66lh4"
Nov 02 19:49:43.773535 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:43.773480    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 02 19:49:43.779179 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:43.779132    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 02 19:49:43.782257 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:43.782229    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-66lh4"]
Nov 02 19:49:43.822590 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:43.822556    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/307e0c79-e6b1-4751-b226-966f9b8f9283-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-66lh4\" (UID: \"307e0c79-e6b1-4751-b226-966f9b8f9283\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-66lh4"
Nov 02 19:49:43.822750 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:43.822611    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/307e0c79-e6b1-4751-b226-966f9b8f9283-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-66lh4\" (UID: \"307e0c79-e6b1-4751-b226-966f9b8f9283\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-66lh4"
Nov 02 19:49:43.822820 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:43.822770    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/307e0c79-e6b1-4751-b226-966f9b8f9283-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-66lh4\" (UID: \"307e0c79-e6b1-4751-b226-966f9b8f9283\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-66lh4"
Nov 02 19:49:43.822873 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:43.822856    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/307e0c79-e6b1-4751-b226-966f9b8f9283-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-66lh4\" (UID: \"307e0c79-e6b1-4751-b226-966f9b8f9283\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-66lh4"
Nov 02 19:49:43.822920 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:43.822885    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/307e0c79-e6b1-4751-b226-966f9b8f9283-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-66lh4\" (UID: \"307e0c79-e6b1-4751-b226-966f9b8f9283\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-66lh4"
Nov 02 19:49:43.823215 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:43.822992    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmvvk\" (UniqueName: \"kubernetes.io/projected/307e0c79-e6b1-4751-b226-966f9b8f9283-kube-api-access-rmvvk\") pod \"istiod-openshift-gateway-7ddd9cfdc8-66lh4\" (UID: \"307e0c79-e6b1-4751-b226-966f9b8f9283\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-66lh4"
Nov 02 19:49:43.823215 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:43.823108    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/307e0c79-e6b1-4751-b226-966f9b8f9283-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-66lh4\" (UID: \"307e0c79-e6b1-4751-b226-966f9b8f9283\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-66lh4"
Nov 02 19:49:43.873547 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:43.873518    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 02 19:49:43.873783 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:43.873759    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-gd6zm\""
Nov 02 19:49:43.924569 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:43.924532    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/307e0c79-e6b1-4751-b226-966f9b8f9283-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-66lh4\" (UID: \"307e0c79-e6b1-4751-b226-966f9b8f9283\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-66lh4"
Nov 02 19:49:43.924798 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:43.924773    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/307e0c79-e6b1-4751-b226-966f9b8f9283-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-66lh4\" (UID: \"307e0c79-e6b1-4751-b226-966f9b8f9283\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-66lh4"
Nov 02 19:49:43.924995 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:43.924952    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmvvk\" (UniqueName: \"kubernetes.io/projected/307e0c79-e6b1-4751-b226-966f9b8f9283-kube-api-access-rmvvk\") pod \"istiod-openshift-gateway-7ddd9cfdc8-66lh4\" (UID: \"307e0c79-e6b1-4751-b226-966f9b8f9283\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-66lh4"
Nov 02 19:49:43.925148 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:43.925089    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/307e0c79-e6b1-4751-b226-966f9b8f9283-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-66lh4\" (UID: \"307e0c79-e6b1-4751-b226-966f9b8f9283\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-66lh4"
Nov 02 19:49:43.925536 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:43.925347    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/307e0c79-e6b1-4751-b226-966f9b8f9283-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-66lh4\" (UID: \"307e0c79-e6b1-4751-b226-966f9b8f9283\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-66lh4"
Nov 02 19:49:43.925536 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:43.925380    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/307e0c79-e6b1-4751-b226-966f9b8f9283-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-66lh4\" (UID: \"307e0c79-e6b1-4751-b226-966f9b8f9283\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-66lh4"
Nov 02 19:49:43.925536 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:43.925423    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/307e0c79-e6b1-4751-b226-966f9b8f9283-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-66lh4\" (UID: \"307e0c79-e6b1-4751-b226-966f9b8f9283\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-66lh4"
Nov 02 19:49:43.925727 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:43.925544    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/307e0c79-e6b1-4751-b226-966f9b8f9283-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-66lh4\" (UID: \"307e0c79-e6b1-4751-b226-966f9b8f9283\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-66lh4"
Nov 02 19:49:43.927701 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:43.927683    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/307e0c79-e6b1-4751-b226-966f9b8f9283-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-66lh4\" (UID: \"307e0c79-e6b1-4751-b226-966f9b8f9283\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-66lh4"
Nov 02 19:49:43.927795 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:43.927720    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/307e0c79-e6b1-4751-b226-966f9b8f9283-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-66lh4\" (UID: \"307e0c79-e6b1-4751-b226-966f9b8f9283\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-66lh4"
Nov 02 19:49:43.927957 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:43.927938    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/307e0c79-e6b1-4751-b226-966f9b8f9283-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-66lh4\" (UID: \"307e0c79-e6b1-4751-b226-966f9b8f9283\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-66lh4"
Nov 02 19:49:43.928980 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:43.928962    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/307e0c79-e6b1-4751-b226-966f9b8f9283-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-66lh4\" (UID: \"307e0c79-e6b1-4751-b226-966f9b8f9283\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-66lh4"
Nov 02 19:49:43.979162 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:43.979130    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/307e0c79-e6b1-4751-b226-966f9b8f9283-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-66lh4\" (UID: \"307e0c79-e6b1-4751-b226-966f9b8f9283\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-66lh4"
Nov 02 19:49:43.979306 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:43.979218    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmvvk\" (UniqueName: \"kubernetes.io/projected/307e0c79-e6b1-4751-b226-966f9b8f9283-kube-api-access-rmvvk\") pod \"istiod-openshift-gateway-7ddd9cfdc8-66lh4\" (UID: \"307e0c79-e6b1-4751-b226-966f9b8f9283\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-66lh4"
Nov 02 19:49:44.035534 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:44.035459    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-66lh4"
Nov 02 19:49:44.453087 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:44.453064    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-66lh4"]
Nov 02 19:49:44.454731 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:49:44.454703    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod307e0c79_e6b1_4751_b226_966f9b8f9283.slice/crio-5abc6126f1f9ae6666b81fd46a990d994d32b293996a0683600f0d80dd909bec WatchSource:0}: Error finding container 5abc6126f1f9ae6666b81fd46a990d994d32b293996a0683600f0d80dd909bec: Status 404 returned error can't find the container with id 5abc6126f1f9ae6666b81fd46a990d994d32b293996a0683600f0d80dd909bec
Nov 02 19:49:44.877953 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:44.877867    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-66lh4" event={"ID":"307e0c79-e6b1-4751-b226-966f9b8f9283","Type":"ContainerStarted","Data":"5abc6126f1f9ae6666b81fd46a990d994d32b293996a0683600f0d80dd909bec"}
Nov 02 19:49:48.693875 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:48.693843    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:49:48.694201 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:48.693911    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:49:48.914201 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:48.914164    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-66lh4" event={"ID":"307e0c79-e6b1-4751-b226-966f9b8f9283","Type":"ContainerStarted","Data":"03510b69d335002bd8d59e325ccf69282f579d444c52e19dba82d6cb7f666884"}
Nov 02 19:49:48.914410 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:48.914318    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-66lh4"
Nov 02 19:49:49.116411 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:49.116277    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-66lh4" podStartSLOduration=1.880575178 podStartE2EDuration="6.116262756s" totalImagesPullingTime="4.235687578s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:49:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:49:49.067949153 +0000 UTC m=+1295.674982913" watchObservedRunningTime="2026-11-02 19:49:49.116262756 +0000 UTC m=+1295.723296521"
Nov 02 19:49:49.934919 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:49.934877    4463 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-66lh4 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 19:49:49.937384 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:49.934946    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-66lh4" podUID="307e0c79-e6b1-4751-b226-966f9b8f9283" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 19:49:52.853266 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:52.853236    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-d8ffab69-c7d7fc585-g5qln"
Nov 02 19:49:52.927648 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:52.927619    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-66lh4"
Nov 02 19:49:55.970093 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:55.969411    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d869e/distinct-2-w5-1d869e-kserve-574f55db46-4cv47" podUID="7e236108-328b-4170-be20-13f16b56c4fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Nov 02 19:49:56.047036 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:56.046147    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-26cfbe/simulator-w1-26cfbe-kserve-74c5578fbd-t79hq" podUID="50dc76f4-aebd-41f4-9f23-799ddc8c70f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:49:56.260098 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:56.216054    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8795bf/distinct-2-w4-8795bf-kserve-69c8b6b885-2chhn" podUID="9f430131-4938-47ca-ba86-b6a236f8cc3b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Nov 02 19:49:56.334150 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:56.334107    4463 patch_prober.go:29] interesting pod/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9ndrz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 19:49:56.334422 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:56.334391    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9ndrz" podUID="f5734864-6a35-46c3-80b8-bdac768e1de1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:49:56.462152 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:56.462105    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-26cfbe/distinct-w1-26cfbe-kserve-685985bfb-tkgn2" podUID="7415aa62-467c-43af-94c8-d604f57ec1c3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:49:56.788048 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:56.786100    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5c46af/distinct-w3-5c46af-kserve-68b6dff7f5-bp59f" podUID="8dd6362e-7dbd-489b-b6d8-0d9caab0fa69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:49:56.858040 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:56.857975    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d869e/premium-w5-1d869e-kserve-7745799ff8-pv2th" podUID="1d784b10-99e4-4561-bfb7-6614877d2b6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:49:56.893093 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:56.892117    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-43ceaa/premium-w0-43ceaa-kserve-67b55659b7-rckfd" podUID="7be32df1-5049-4f79-be23-a827f3f0a85f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:49:57.014038 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:56.985355    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d869e/distinct-2-w5-1d869e-kserve-574f55db46-4cv47" podUID="7e236108-328b-4170-be20-13f16b56c4fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:49:57.033647 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:57.020751    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8795bf/distinct-w4-8795bf-kserve-64b4796df4-8qgqh" podUID="ee7d28f4-e9c7-43e2-b983-1427589f1785" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:49:57.034184 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:57.034160    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-1d869e/distinct-2-w5-1d869e-kserve-574f55db46-4cv47"
Nov 02 19:49:57.060083 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:57.047097    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-26cfbe/simulator-w1-26cfbe-kserve-74c5578fbd-t79hq" podUID="50dc76f4-aebd-41f4-9f23-799ddc8c70f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:49:57.060083 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:57.047430    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-26cfbe/simulator-w1-26cfbe-kserve-74c5578fbd-t79hq"
Nov 02 19:49:57.514038 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:57.484418    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-26cfbe/distinct-w1-26cfbe-kserve-685985bfb-tkgn2" podUID="7415aa62-467c-43af-94c8-d604f57ec1c3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:49:57.514038 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:57.492130    4463 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-1d869e/premium-w5-1d869e-kserve-7745799ff8-pv2th" podUID="1d784b10-99e4-4561-bfb7-6614877d2b6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:49:57.514038 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:57.495054    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-26cfbe/distinct-w1-26cfbe-kserve-685985bfb-tkgn2"
Nov 02 19:49:57.635372 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:57.635318    4463 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-26cfbe/simulator-w1-26cfbe-kserve-74c5578fbd-t79hq" podUID="50dc76f4-aebd-41f4-9f23-799ddc8c70f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:49:57.786449 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:57.786321    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5c46af/distinct-w3-5c46af-kserve-68b6dff7f5-bp59f" podUID="8dd6362e-7dbd-489b-b6d8-0d9caab0fa69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:49:57.863363 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:57.863312    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d869e/premium-w5-1d869e-kserve-7745799ff8-pv2th" podUID="1d784b10-99e4-4561-bfb7-6614877d2b6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:49:57.879033 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:57.864211    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5c46af/distinct-w3-5c46af-kserve-68b6dff7f5-bp59f"
Nov 02 19:49:58.008261 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:58.008231    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-1d869e/premium-w5-1d869e-kserve-7745799ff8-pv2th"
Nov 02 19:49:58.009448 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:57.945356    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-43ceaa/premium-w0-43ceaa-kserve-67b55659b7-rckfd" podUID="7be32df1-5049-4f79-be23-a827f3f0a85f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:49:58.015256 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:58.015233    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-43ceaa/premium-w0-43ceaa-kserve-67b55659b7-rckfd"
Nov 02 19:49:58.034114 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:58.034075    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8795bf/distinct-w4-8795bf-kserve-64b4796df4-8qgqh" podUID="ee7d28f4-e9c7-43e2-b983-1427589f1785" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:49:58.040883 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:58.034392    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8795bf/distinct-w4-8795bf-kserve-64b4796df4-8qgqh"
Nov 02 19:49:58.040883 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:58.034237    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d869e/distinct-2-w5-1d869e-kserve-574f55db46-4cv47" podUID="7e236108-328b-4170-be20-13f16b56c4fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:49:58.059058 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:58.053230    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-26cfbe/simulator-w1-26cfbe-kserve-74c5578fbd-t79hq" podUID="50dc76f4-aebd-41f4-9f23-799ddc8c70f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:49:58.130993 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:58.130952    4463 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-1d869e/distinct-2-w5-1d869e-kserve-574f55db46-4cv47" podUID="7e236108-328b-4170-be20-13f16b56c4fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:49:58.146206 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:58.136276    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8795bf/distinct-2-w4-8795bf-kserve-69c8b6b885-2chhn" podUID="9f430131-4938-47ca-ba86-b6a236f8cc3b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:49:58.169833 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:58.168365    4463 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-43ceaa/premium-w0-43ceaa-kserve-67b55659b7-rckfd" podUID="7be32df1-5049-4f79-be23-a827f3f0a85f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:49:58.645040 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:58.635478    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-26cfbe/distinct-w1-26cfbe-kserve-685985bfb-tkgn2" podUID="7415aa62-467c-43af-94c8-d604f57ec1c3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:49:58.798702 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:58.703927    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5c46af/distinct-w3-5c46af-kserve-68b6dff7f5-bp59f"
Nov 02 19:49:58.798957 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:58.798944    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-1d869e/premium-w5-1d869e-kserve-7745799ff8-pv2th"
Nov 02 19:49:58.799082 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:58.799073    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-1d869e/distinct-2-w5-1d869e-kserve-574f55db46-4cv47"
Nov 02 19:49:58.799194 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:58.799185    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-26cfbe/simulator-w1-26cfbe-kserve-74c5578fbd-t79hq"
Nov 02 19:49:58.870097 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:58.870064    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8795bf/distinct-w4-8795bf-kserve-64b4796df4-8qgqh"
Nov 02 19:49:58.873723 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:58.873701    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-43ceaa/premium-w0-43ceaa-kserve-67b55659b7-rckfd"
Nov 02 19:49:58.875388 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:49:58.875370    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-26cfbe/distinct-w1-26cfbe-kserve-685985bfb-tkgn2"
Nov 02 19:50:04.294961 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:50:04.294923    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 19:50:34.294196 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:50:34.294149    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 19:50:39.425801 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:50:39.425771    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 19:50:49.462147 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:50:49.462113    4463 patch_prober.go:29] interesting pod/e2e-migrate-d8ffab69-gw-openshift-default-79d965fd8f-dkcf5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 19:50:49.462540 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:50:49.462172    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-d8ffab69-gw-openshift-default-79d965fd8f-dkcf5" podUID="1d1ebf76-5f16-412b-8784-8c29dd59ca56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:50:51.335724 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:50:51.335690    4463 generic.go:356] "Generic (PLEG): container finished" podID="1d1ebf76-5f16-412b-8784-8c29dd59ca56" containerID="3b51f80eed47519d070dc7ad7dd3f5092cbfcf2046d9def95d164c5cfa873a55" exitCode=0
Nov 02 19:50:51.336143 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:50:51.335759    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d8ffab69-gw-openshift-default-79d965fd8f-dkcf5" event={"ID":"1d1ebf76-5f16-412b-8784-8c29dd59ca56","Type":"ContainerDied","Data":"3b51f80eed47519d070dc7ad7dd3f5092cbfcf2046d9def95d164c5cfa873a55"}
Nov 02 19:50:51.336187 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:50:51.336141    4463 scope.go:122] "RemoveContainer" containerID="3b51f80eed47519d070dc7ad7dd3f5092cbfcf2046d9def95d164c5cfa873a55"
Nov 02 19:50:51.337564 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:50:51.337540    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:50:51.337631 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:50:51.337601    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:50:51.337666 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:50:51.337635    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:50:52.344524 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:50:52.344488    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d8ffab69-gw-openshift-default-79d965fd8f-dkcf5" event={"ID":"1d1ebf76-5f16-412b-8784-8c29dd59ca56","Type":"ContainerStarted","Data":"bdccaef7a17c8bcf1d59edfdfee775dab43b07a1d17897398be366f53294a59c"}
Nov 02 19:50:53.076749 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:50:53.076715    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-d8ffab69-gw-openshift-default-79d965fd8f-dkcf5"
Nov 02 19:50:53.076981 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:50:53.076956    4463 patch_prober.go:29] interesting pod/e2e-migrate-d8ffab69-gw-openshift-default-79d965fd8f-dkcf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 02 19:50:53.077097 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:50:53.077050    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-d8ffab69-gw-openshift-default-79d965fd8f-dkcf5" podUID="1d1ebf76-5f16-412b-8784-8c29dd59ca56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 02 19:50:54.076276 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:50:54.076239    4463 patch_prober.go:29] interesting pod/e2e-migrate-d8ffab69-gw-openshift-default-79d965fd8f-dkcf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 02 19:50:54.076721 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:50:54.076339    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-d8ffab69-gw-openshift-default-79d965fd8f-dkcf5" podUID="1d1ebf76-5f16-412b-8784-8c29dd59ca56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 02 19:50:55.076929 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:50:55.076893    4463 patch_prober.go:29] interesting pod/e2e-migrate-d8ffab69-gw-openshift-default-79d965fd8f-dkcf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 02 19:50:55.077443 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:50:55.076961    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-d8ffab69-gw-openshift-default-79d965fd8f-dkcf5" podUID="1d1ebf76-5f16-412b-8784-8c29dd59ca56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 02 19:50:56.076597 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:50:56.076560    4463 patch_prober.go:29] interesting pod/e2e-migrate-d8ffab69-gw-openshift-default-79d965fd8f-dkcf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 02 19:50:56.076788 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:50:56.076627    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-d8ffab69-gw-openshift-default-79d965fd8f-dkcf5" podUID="1d1ebf76-5f16-412b-8784-8c29dd59ca56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 02 19:50:57.076244 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:50:57.076203    4463 patch_prober.go:29] interesting pod/e2e-migrate-d8ffab69-gw-openshift-default-79d965fd8f-dkcf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 02 19:50:57.076773 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:50:57.076281    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-d8ffab69-gw-openshift-default-79d965fd8f-dkcf5" podUID="1d1ebf76-5f16-412b-8784-8c29dd59ca56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 02 19:50:58.080129 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:50:58.080102    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-d8ffab69-gw-openshift-default-79d965fd8f-dkcf5"
Nov 02 19:50:58.080624 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:50:58.080517    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-d8ffab69-gw-openshift-default-79d965fd8f-dkcf5"
Nov 02 19:50:58.081084 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:50:58.081063    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-d8ffab69-gw-openshift-default-79d965fd8f-dkcf5"
Nov 02 19:51:04.294121 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:51:04.294083    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 19:51:04.879515 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:51:04.879483    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h27fz"]
Nov 02 19:51:04.883678 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:51:04.883655    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h27fz"
Nov 02 19:51:04.903217 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:51:04.903193    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9f30411-f136-4711-9f5d-6f0ef7a24591-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-h27fz\" (UID: \"b9f30411-f136-4711-9f5d-6f0ef7a24591\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h27fz"
Nov 02 19:51:04.903313 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:51:04.903235    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9f30411-f136-4711-9f5d-6f0ef7a24591-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-h27fz\" (UID: \"b9f30411-f136-4711-9f5d-6f0ef7a24591\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h27fz"
Nov 02 19:51:04.903353 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:51:04.903307    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9f30411-f136-4711-9f5d-6f0ef7a24591-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-h27fz\" (UID: \"b9f30411-f136-4711-9f5d-6f0ef7a24591\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h27fz"
Nov 02 19:51:04.903386 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:51:04.903358    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9f30411-f136-4711-9f5d-6f0ef7a24591-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-h27fz\" (UID: \"b9f30411-f136-4711-9f5d-6f0ef7a24591\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h27fz"
Nov 02 19:51:04.903418 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:51:04.903382    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9f30411-f136-4711-9f5d-6f0ef7a24591-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-h27fz\" (UID: \"b9f30411-f136-4711-9f5d-6f0ef7a24591\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h27fz"
Nov 02 19:51:04.903489 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:51:04.903475    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9f30411-f136-4711-9f5d-6f0ef7a24591-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-h27fz\" (UID: \"b9f30411-f136-4711-9f5d-6f0ef7a24591\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h27fz"
Nov 02 19:51:04.903528 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:51:04.903516    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9f30411-f136-4711-9f5d-6f0ef7a24591-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-h27fz\" (UID: \"b9f30411-f136-4711-9f5d-6f0ef7a24591\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h27fz"
Nov 02 19:51:04.903583 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:51:04.903572    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfqfb\" (UniqueName: \"kubernetes.io/projected/b9f30411-f136-4711-9f5d-6f0ef7a24591-kube-api-access-hfqfb\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-h27fz\" (UID: \"b9f30411-f136-4711-9f5d-6f0ef7a24591\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h27fz"
Nov 02 19:51:04.903650 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:51:04.903638    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9f30411-f136-4711-9f5d-6f0ef7a24591-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-h27fz\" (UID: \"b9f30411-f136-4711-9f5d-6f0ef7a24591\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h27fz"
Nov 02 19:51:04.932024 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:51:04.931998    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-qmwl4\""
Nov 02 19:51:05.004749 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:51:05.004701    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9f30411-f136-4711-9f5d-6f0ef7a24591-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-h27fz\" (UID: \"b9f30411-f136-4711-9f5d-6f0ef7a24591\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h27fz"
Nov 02 19:51:05.004953 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:51:05.004770    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9f30411-f136-4711-9f5d-6f0ef7a24591-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-h27fz\" (UID: \"b9f30411-f136-4711-9f5d-6f0ef7a24591\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h27fz"
Nov 02 19:51:05.004953 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:51:05.004828    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9f30411-f136-4711-9f5d-6f0ef7a24591-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-h27fz\" (UID: \"b9f30411-f136-4711-9f5d-6f0ef7a24591\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h27fz"
Nov 02 19:51:05.004953 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:51:05.004863    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9f30411-f136-4711-9f5d-6f0ef7a24591-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-h27fz\" (UID: \"b9f30411-f136-4711-9f5d-6f0ef7a24591\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h27fz"
Nov 02 19:51:05.004953 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:51:05.004943    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9f30411-f136-4711-9f5d-6f0ef7a24591-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-h27fz\" (UID: \"b9f30411-f136-4711-9f5d-6f0ef7a24591\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h27fz"
Nov 02 19:51:05.005302 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:51:05.005001    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9f30411-f136-4711-9f5d-6f0ef7a24591-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-h27fz\" (UID: \"b9f30411-f136-4711-9f5d-6f0ef7a24591\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h27fz"
Nov 02 19:51:05.005302 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:51:05.005071    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfqfb\" (UniqueName: \"kubernetes.io/projected/b9f30411-f136-4711-9f5d-6f0ef7a24591-kube-api-access-hfqfb\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-h27fz\" (UID: \"b9f30411-f136-4711-9f5d-6f0ef7a24591\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h27fz"
Nov 02 19:51:05.005302 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:51:05.005146    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9f30411-f136-4711-9f5d-6f0ef7a24591-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-h27fz\" (UID: \"b9f30411-f136-4711-9f5d-6f0ef7a24591\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h27fz"
Nov 02 19:51:05.005302 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:51:05.005278    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9f30411-f136-4711-9f5d-6f0ef7a24591-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-h27fz\" (UID: \"b9f30411-f136-4711-9f5d-6f0ef7a24591\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h27fz"
Nov 02 19:51:05.005658 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:51:05.005633    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9f30411-f136-4711-9f5d-6f0ef7a24591-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-h27fz\" (UID: \"b9f30411-f136-4711-9f5d-6f0ef7a24591\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h27fz"
Nov 02 19:51:05.005827 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:51:05.005646    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9f30411-f136-4711-9f5d-6f0ef7a24591-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-h27fz\" (UID: \"b9f30411-f136-4711-9f5d-6f0ef7a24591\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h27fz"
Nov 02 19:51:05.006138 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:51:05.006084    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9f30411-f136-4711-9f5d-6f0ef7a24591-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-h27fz\" (UID: \"b9f30411-f136-4711-9f5d-6f0ef7a24591\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h27fz"
Nov 02 19:51:05.006480 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:51:05.006452    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9f30411-f136-4711-9f5d-6f0ef7a24591-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-h27fz\" (UID: \"b9f30411-f136-4711-9f5d-6f0ef7a24591\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h27fz"
Nov 02 19:51:05.006775 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:51:05.006755    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9f30411-f136-4711-9f5d-6f0ef7a24591-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-h27fz\" (UID: \"b9f30411-f136-4711-9f5d-6f0ef7a24591\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h27fz"
Nov 02 19:51:05.008042 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:51:05.007992    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9f30411-f136-4711-9f5d-6f0ef7a24591-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-h27fz\" (UID: \"b9f30411-f136-4711-9f5d-6f0ef7a24591\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h27fz"
Nov 02 19:51:05.009354 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:51:05.009334    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9f30411-f136-4711-9f5d-6f0ef7a24591-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-h27fz\" (UID: \"b9f30411-f136-4711-9f5d-6f0ef7a24591\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h27fz"
Nov 02 19:51:05.058531 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:51:05.058513    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9f30411-f136-4711-9f5d-6f0ef7a24591-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-h27fz\" (UID: \"b9f30411-f136-4711-9f5d-6f0ef7a24591\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h27fz"
Nov 02 19:51:05.058615 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:51:05.058570    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfqfb\" (UniqueName: \"kubernetes.io/projected/b9f30411-f136-4711-9f5d-6f0ef7a24591-kube-api-access-hfqfb\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-h27fz\" (UID: \"b9f30411-f136-4711-9f5d-6f0ef7a24591\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h27fz"
Nov 02 19:51:05.069225 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:51:05.069199    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h27fz"]
Nov 02 19:51:05.197093 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:51:05.197068    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h27fz"
Nov 02 19:51:05.613821 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:51:05.613793    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h27fz"]
Nov 02 19:51:05.614863 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:51:05.614839    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9f30411_f136_4711_9f5d_6f0ef7a24591.slice/crio-bb8f364f917c1984379902abffa7a419cf09af0191549f209c013962fc952e19 WatchSource:0}: Error finding container bb8f364f917c1984379902abffa7a419cf09af0191549f209c013962fc952e19: Status 404 returned error can't find the container with id bb8f364f917c1984379902abffa7a419cf09af0191549f209c013962fc952e19
Nov 02 19:51:05.616981 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:51:05.616951    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:51:05.617076 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:51:05.617028    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:51:05.617076 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:51:05.617061    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:51:06.480759 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:51:06.480712    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h27fz" event={"ID":"b9f30411-f136-4711-9f5d-6f0ef7a24591","Type":"ContainerStarted","Data":"bba87f664fb4209144c627e4124320f89fa64bebd2f4f86f72349778884f7942"}
Nov 02 19:51:06.480927 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:51:06.480776    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h27fz" event={"ID":"b9f30411-f136-4711-9f5d-6f0ef7a24591","Type":"ContainerStarted","Data":"bb8f364f917c1984379902abffa7a419cf09af0191549f209c013962fc952e19"}
Nov 02 19:51:06.684626 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:51:06.684573    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h27fz" podStartSLOduration=2.684558961 podStartE2EDuration="2.684558961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:51:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:51:06.634402282 +0000 UTC m=+1373.241436073" watchObservedRunningTime="2026-11-02 19:51:06.684558961 +0000 UTC m=+1373.291592729"
Nov 02 19:51:07.197850 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:51:07.197815    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h27fz"
Nov 02 19:51:07.199380 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:51:07.199359    4463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-h27fz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 02 19:51:07.199494 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:51:07.199417    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h27fz" podUID="b9f30411-f136-4711-9f5d-6f0ef7a24591" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 02 19:51:08.198313 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:51:08.198278    4463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-h27fz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 02 19:51:08.198716 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:51:08.198336    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h27fz" podUID="b9f30411-f136-4711-9f5d-6f0ef7a24591" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 02 19:51:09.198301 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:51:09.198264    4463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-h27fz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 02 19:51:09.198476 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:51:09.198325    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h27fz" podUID="b9f30411-f136-4711-9f5d-6f0ef7a24591" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 02 19:51:10.197661 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:51:10.197626    4463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-h27fz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 02 19:51:10.197941 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:51:10.197679    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h27fz" podUID="b9f30411-f136-4711-9f5d-6f0ef7a24591" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 02 19:51:11.197759 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:51:11.197721    4463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-h27fz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 02 19:51:11.198176 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:51:11.197771    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h27fz" podUID="b9f30411-f136-4711-9f5d-6f0ef7a24591" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 02 19:51:12.197652 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:51:12.197619    4463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-h27fz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 02 19:51:12.197829 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:51:12.197669    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h27fz" podUID="b9f30411-f136-4711-9f5d-6f0ef7a24591" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 02 19:51:13.957448 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:51:13.957409    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h27fz"
Nov 02 19:51:13.957858 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:51:13.957844    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h27fz"
Nov 02 19:51:13.958599 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:51:13.958578    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h27fz"
Nov 02 19:51:34.304907 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:51:34.304811    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 19:51:36.455500 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:51:36.455463    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d8ffab69-c7d7fc585-g5qln"]
Nov 02 19:51:41.456650 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:51:41.456594    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-d8ffab69-c7d7fc585-g5qln" podUID="2d99e7b7-63b9-4e1b-aa57-4ab098b789b4" containerName="payload-processing" containerID="cri-o://0674c1400768fd94dfa36a257a4bad9cbbea921e648dac6e86760ff7708fb608" gracePeriod=25
Nov 02 19:51:41.739356 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:51:41.739332    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-d8ffab69-c7d7fc585-g5qln"
Nov 02 19:51:41.759391 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:51:41.759361    4463 generic.go:356] "Generic (PLEG): container finished" podID="2d99e7b7-63b9-4e1b-aa57-4ab098b789b4" containerID="0674c1400768fd94dfa36a257a4bad9cbbea921e648dac6e86760ff7708fb608" exitCode=0
Nov 02 19:51:41.759515 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:51:41.759395    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-d8ffab69-c7d7fc585-g5qln"
Nov 02 19:51:41.759515 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:51:41.759441    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-d8ffab69-c7d7fc585-g5qln" event={"ID":"2d99e7b7-63b9-4e1b-aa57-4ab098b789b4","Type":"ContainerDied","Data":"0674c1400768fd94dfa36a257a4bad9cbbea921e648dac6e86760ff7708fb608"}
Nov 02 19:51:41.759515 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:51:41.759490    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-d8ffab69-c7d7fc585-g5qln" event={"ID":"2d99e7b7-63b9-4e1b-aa57-4ab098b789b4","Type":"ContainerDied","Data":"71e19dcb6a5cd7cc51c63e82672eda5e8611ccdd588aaa77f58872ba1c60a75b"}
Nov 02 19:51:41.759515 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:51:41.759513    4463 scope.go:122] "RemoveContainer" containerID="0674c1400768fd94dfa36a257a4bad9cbbea921e648dac6e86760ff7708fb608"
Nov 02 19:51:41.769299 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:51:41.769282    4463 scope.go:122] "RemoveContainer" containerID="0674c1400768fd94dfa36a257a4bad9cbbea921e648dac6e86760ff7708fb608"
Nov 02 19:51:41.769571 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:51:41.769542    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0674c1400768fd94dfa36a257a4bad9cbbea921e648dac6e86760ff7708fb608\": container with ID starting with 0674c1400768fd94dfa36a257a4bad9cbbea921e648dac6e86760ff7708fb608 not found: ID does not exist" containerID="0674c1400768fd94dfa36a257a4bad9cbbea921e648dac6e86760ff7708fb608"
Nov 02 19:51:41.769617 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:51:41.769577    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0674c1400768fd94dfa36a257a4bad9cbbea921e648dac6e86760ff7708fb608"} err="failed to get container status \"0674c1400768fd94dfa36a257a4bad9cbbea921e648dac6e86760ff7708fb608\": rpc error: code = NotFound desc = could not find container \"0674c1400768fd94dfa36a257a4bad9cbbea921e648dac6e86760ff7708fb608\": container with ID starting with 0674c1400768fd94dfa36a257a4bad9cbbea921e648dac6e86760ff7708fb608 not found: ID does not exist"
Nov 02 19:51:41.786142 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:51:41.786121    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d99e7b7-63b9-4e1b-aa57-4ab098b789b4-kube-api-access-hdxx8\" (UniqueName: \"kubernetes.io/projected/2d99e7b7-63b9-4e1b-aa57-4ab098b789b4-kube-api-access-hdxx8\") pod \"2d99e7b7-63b9-4e1b-aa57-4ab098b789b4\" (UID: \"2d99e7b7-63b9-4e1b-aa57-4ab098b789b4\") "
Nov 02 19:51:41.786282 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:51:41.786267    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d99e7b7-63b9-4e1b-aa57-4ab098b789b4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d99e7b7-63b9-4e1b-aa57-4ab098b789b4-plugins-config-volume\") pod \"2d99e7b7-63b9-4e1b-aa57-4ab098b789b4\" (UID: \"2d99e7b7-63b9-4e1b-aa57-4ab098b789b4\") "
Nov 02 19:51:41.786603 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:51:41.786584    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d99e7b7-63b9-4e1b-aa57-4ab098b789b4-plugins-config-volume" pod "2d99e7b7-63b9-4e1b-aa57-4ab098b789b4" (UID: "2d99e7b7-63b9-4e1b-aa57-4ab098b789b4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 19:51:41.786927 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:51:41.786910    4463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d99e7b7-63b9-4e1b-aa57-4ab098b789b4-plugins-config-volume\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:51:41.788178 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:51:41.788157    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d99e7b7-63b9-4e1b-aa57-4ab098b789b4-kube-api-access-hdxx8" pod "2d99e7b7-63b9-4e1b-aa57-4ab098b789b4" (UID: "2d99e7b7-63b9-4e1b-aa57-4ab098b789b4"). InnerVolumeSpecName "kube-api-access-hdxx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:51:41.888954 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:51:41.888925    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdxx8\" (UniqueName: \"kubernetes.io/projected/2d99e7b7-63b9-4e1b-aa57-4ab098b789b4-kube-api-access-hdxx8\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:51:42.176421 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:51:42.176390    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d8ffab69-c7d7fc585-g5qln"]
Nov 02 19:51:42.180842 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:51:42.180818    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d8ffab69-c7d7fc585-g5qln"]
Nov 02 19:51:43.587430 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:51:43.587400    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d99e7b7-63b9-4e1b-aa57-4ab098b789b4" path="/var/lib/kubelet/pods/2d99e7b7-63b9-4e1b-aa57-4ab098b789b4/volumes"
Nov 02 19:52:00.945141 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:00.945092    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-d8ffab69-gw-openshift-default-79d965fd8f-dkcf5"]
Nov 02 19:52:00.946037 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:00.945990    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-d8ffab69-gw-openshift-default-79d965fd8f-dkcf5" podUID="1d1ebf76-5f16-412b-8784-8c29dd59ca56" containerName="istio-proxy" containerID="cri-o://bdccaef7a17c8bcf1d59edfdfee775dab43b07a1d17897398be366f53294a59c" gracePeriod=30
Nov 02 19:52:04.301699 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:52:04.301649    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 19:52:06.244216 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:06.244190    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-d8ffab69-gw-openshift-default-79d965fd8f-dkcf5"
Nov 02 19:52:06.391437 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:06.391397    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d1ebf76-5f16-412b-8784-8c29dd59ca56-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d1ebf76-5f16-412b-8784-8c29dd59ca56-workload-socket\") pod \"1d1ebf76-5f16-412b-8784-8c29dd59ca56\" (UID: \"1d1ebf76-5f16-412b-8784-8c29dd59ca56\") "
Nov 02 19:52:06.391633 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:06.391460    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d1ebf76-5f16-412b-8784-8c29dd59ca56-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d1ebf76-5f16-412b-8784-8c29dd59ca56-credential-socket\") pod \"1d1ebf76-5f16-412b-8784-8c29dd59ca56\" (UID: \"1d1ebf76-5f16-412b-8784-8c29dd59ca56\") "
Nov 02 19:52:06.391633 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:06.391581    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1d1ebf76-5f16-412b-8784-8c29dd59ca56-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d1ebf76-5f16-412b-8784-8c29dd59ca56-istio-podinfo\") pod \"1d1ebf76-5f16-412b-8784-8c29dd59ca56\" (UID: \"1d1ebf76-5f16-412b-8784-8c29dd59ca56\") "
Nov 02 19:52:06.391764 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:06.391744    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d1ebf76-5f16-412b-8784-8c29dd59ca56-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d1ebf76-5f16-412b-8784-8c29dd59ca56-workload-certs\") pod \"1d1ebf76-5f16-412b-8784-8c29dd59ca56\" (UID: \"1d1ebf76-5f16-412b-8784-8c29dd59ca56\") "
Nov 02 19:52:06.391810 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:06.391757    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d1ebf76-5f16-412b-8784-8c29dd59ca56-workload-socket" pod "1d1ebf76-5f16-412b-8784-8c29dd59ca56" (UID: "1d1ebf76-5f16-412b-8784-8c29dd59ca56"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:52:06.391810 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:06.391769    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d1ebf76-5f16-412b-8784-8c29dd59ca56-credential-socket" pod "1d1ebf76-5f16-412b-8784-8c29dd59ca56" (UID: "1d1ebf76-5f16-412b-8784-8c29dd59ca56"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:52:06.391810 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:06.391795    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d1ebf76-5f16-412b-8784-8c29dd59ca56-istio-token\" (UniqueName: \"kubernetes.io/projected/1d1ebf76-5f16-412b-8784-8c29dd59ca56-istio-token\") pod \"1d1ebf76-5f16-412b-8784-8c29dd59ca56\" (UID: \"1d1ebf76-5f16-412b-8784-8c29dd59ca56\") "
Nov 02 19:52:06.391942 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:06.391861    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d1ebf76-5f16-412b-8784-8c29dd59ca56-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d1ebf76-5f16-412b-8784-8c29dd59ca56-istio-data\") pod \"1d1ebf76-5f16-412b-8784-8c29dd59ca56\" (UID: \"1d1ebf76-5f16-412b-8784-8c29dd59ca56\") "
Nov 02 19:52:06.391984 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:06.391932    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d1ebf76-5f16-412b-8784-8c29dd59ca56-workload-certs" pod "1d1ebf76-5f16-412b-8784-8c29dd59ca56" (UID: "1d1ebf76-5f16-412b-8784-8c29dd59ca56"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:52:06.391984 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:06.391937    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d1ebf76-5f16-412b-8784-8c29dd59ca56-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d1ebf76-5f16-412b-8784-8c29dd59ca56-istiod-ca-cert\") pod \"1d1ebf76-5f16-412b-8784-8c29dd59ca56\" (UID: \"1d1ebf76-5f16-412b-8784-8c29dd59ca56\") "
Nov 02 19:52:06.392180 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:06.392008    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d1ebf76-5f16-412b-8784-8c29dd59ca56-kube-api-access-c9tjq\" (UniqueName: \"kubernetes.io/projected/1d1ebf76-5f16-412b-8784-8c29dd59ca56-kube-api-access-c9tjq\") pod \"1d1ebf76-5f16-412b-8784-8c29dd59ca56\" (UID: \"1d1ebf76-5f16-412b-8784-8c29dd59ca56\") "
Nov 02 19:52:06.392306 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:06.392169    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d1ebf76-5f16-412b-8784-8c29dd59ca56-istio-data" pod "1d1ebf76-5f16-412b-8784-8c29dd59ca56" (UID: "1d1ebf76-5f16-412b-8784-8c29dd59ca56"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:52:06.392306 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:06.392270    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d1ebf76-5f16-412b-8784-8c29dd59ca56-istiod-ca-cert" pod "1d1ebf76-5f16-412b-8784-8c29dd59ca56" (UID: "1d1ebf76-5f16-412b-8784-8c29dd59ca56"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 19:52:06.392306 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:06.392276    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d1ebf76-5f16-412b-8784-8c29dd59ca56-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d1ebf76-5f16-412b-8784-8c29dd59ca56-istio-envoy\") pod \"1d1ebf76-5f16-412b-8784-8c29dd59ca56\" (UID: \"1d1ebf76-5f16-412b-8784-8c29dd59ca56\") "
Nov 02 19:52:06.393061 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:06.393036    4463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d1ebf76-5f16-412b-8784-8c29dd59ca56-istio-data\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:52:06.393179 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:06.393064    4463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d1ebf76-5f16-412b-8784-8c29dd59ca56-istiod-ca-cert\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:52:06.393179 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:06.393079    4463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d1ebf76-5f16-412b-8784-8c29dd59ca56-workload-socket\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:52:06.393179 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:06.393094    4463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d1ebf76-5f16-412b-8784-8c29dd59ca56-credential-socket\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:52:06.393179 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:06.393108    4463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d1ebf76-5f16-412b-8784-8c29dd59ca56-workload-certs\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:52:06.394229 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:06.394199    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d1ebf76-5f16-412b-8784-8c29dd59ca56-kube-api-access-c9tjq" pod "1d1ebf76-5f16-412b-8784-8c29dd59ca56" (UID: "1d1ebf76-5f16-412b-8784-8c29dd59ca56"). InnerVolumeSpecName "kube-api-access-c9tjq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:52:06.394229 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:06.394213    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d1ebf76-5f16-412b-8784-8c29dd59ca56-istio-token" pod "1d1ebf76-5f16-412b-8784-8c29dd59ca56" (UID: "1d1ebf76-5f16-412b-8784-8c29dd59ca56"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:52:06.394343 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:06.394215    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1d1ebf76-5f16-412b-8784-8c29dd59ca56-istio-podinfo" pod "1d1ebf76-5f16-412b-8784-8c29dd59ca56" (UID: "1d1ebf76-5f16-412b-8784-8c29dd59ca56"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 19:52:06.394343 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:06.394234    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d1ebf76-5f16-412b-8784-8c29dd59ca56-istio-envoy" pod "1d1ebf76-5f16-412b-8784-8c29dd59ca56" (UID: "1d1ebf76-5f16-412b-8784-8c29dd59ca56"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:52:06.494595 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:06.494517    4463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d1ebf76-5f16-412b-8784-8c29dd59ca56-istio-podinfo\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:52:06.494595 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:06.494544    4463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d1ebf76-5f16-412b-8784-8c29dd59ca56-istio-token\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:52:06.494595 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:06.494554    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9tjq\" (UniqueName: \"kubernetes.io/projected/1d1ebf76-5f16-412b-8784-8c29dd59ca56-kube-api-access-c9tjq\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:52:06.494595 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:06.494563    4463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d1ebf76-5f16-412b-8784-8c29dd59ca56-istio-envoy\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:52:06.983090 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:06.983054    4463 generic.go:356] "Generic (PLEG): container finished" podID="1d1ebf76-5f16-412b-8784-8c29dd59ca56" containerID="bdccaef7a17c8bcf1d59edfdfee775dab43b07a1d17897398be366f53294a59c" exitCode=0
Nov 02 19:52:06.983090 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:06.983089    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-d8ffab69-gw-openshift-default-79d965fd8f-dkcf5"
Nov 02 19:52:06.983331 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:06.983109    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d8ffab69-gw-openshift-default-79d965fd8f-dkcf5" event={"ID":"1d1ebf76-5f16-412b-8784-8c29dd59ca56","Type":"ContainerDied","Data":"bdccaef7a17c8bcf1d59edfdfee775dab43b07a1d17897398be366f53294a59c"}
Nov 02 19:52:06.983331 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:06.983164    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d8ffab69-gw-openshift-default-79d965fd8f-dkcf5" event={"ID":"1d1ebf76-5f16-412b-8784-8c29dd59ca56","Type":"ContainerDied","Data":"d999d584cf7d50901f321c9f9580adbad69e4c3a5e6929e4dab33563b546fcc7"}
Nov 02 19:52:06.983331 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:06.983188    4463 scope.go:122] "RemoveContainer" containerID="bdccaef7a17c8bcf1d59edfdfee775dab43b07a1d17897398be366f53294a59c"
Nov 02 19:52:06.994348 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:06.994323    4463 scope.go:122] "RemoveContainer" containerID="3b51f80eed47519d070dc7ad7dd3f5092cbfcf2046d9def95d164c5cfa873a55"
Nov 02 19:52:07.004503 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:07.004485    4463 scope.go:122] "RemoveContainer" containerID="bdccaef7a17c8bcf1d59edfdfee775dab43b07a1d17897398be366f53294a59c"
Nov 02 19:52:07.004782 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:52:07.004763    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdccaef7a17c8bcf1d59edfdfee775dab43b07a1d17897398be366f53294a59c\": container with ID starting with bdccaef7a17c8bcf1d59edfdfee775dab43b07a1d17897398be366f53294a59c not found: ID does not exist" containerID="bdccaef7a17c8bcf1d59edfdfee775dab43b07a1d17897398be366f53294a59c"
Nov 02 19:52:07.004829 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:07.004790    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdccaef7a17c8bcf1d59edfdfee775dab43b07a1d17897398be366f53294a59c"} err="failed to get container status \"bdccaef7a17c8bcf1d59edfdfee775dab43b07a1d17897398be366f53294a59c\": rpc error: code = NotFound desc = could not find container \"bdccaef7a17c8bcf1d59edfdfee775dab43b07a1d17897398be366f53294a59c\": container with ID starting with bdccaef7a17c8bcf1d59edfdfee775dab43b07a1d17897398be366f53294a59c not found: ID does not exist"
Nov 02 19:52:07.004829 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:07.004813    4463 scope.go:122] "RemoveContainer" containerID="3b51f80eed47519d070dc7ad7dd3f5092cbfcf2046d9def95d164c5cfa873a55"
Nov 02 19:52:07.005094 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:52:07.005072    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b51f80eed47519d070dc7ad7dd3f5092cbfcf2046d9def95d164c5cfa873a55\": container with ID starting with 3b51f80eed47519d070dc7ad7dd3f5092cbfcf2046d9def95d164c5cfa873a55 not found: ID does not exist" containerID="3b51f80eed47519d070dc7ad7dd3f5092cbfcf2046d9def95d164c5cfa873a55"
Nov 02 19:52:07.005147 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:07.005102    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b51f80eed47519d070dc7ad7dd3f5092cbfcf2046d9def95d164c5cfa873a55"} err="failed to get container status \"3b51f80eed47519d070dc7ad7dd3f5092cbfcf2046d9def95d164c5cfa873a55\": rpc error: code = NotFound desc = could not find container \"3b51f80eed47519d070dc7ad7dd3f5092cbfcf2046d9def95d164c5cfa873a55\": container with ID starting with 3b51f80eed47519d070dc7ad7dd3f5092cbfcf2046d9def95d164c5cfa873a55 not found: ID does not exist"
Nov 02 19:52:07.188861 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:07.188825    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-d8ffab69-gw-openshift-default-79d965fd8f-dkcf5"]
Nov 02 19:52:07.235952 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:07.235878    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-d8ffab69-gw-openshift-default-79d965fd8f-dkcf5"]
Nov 02 19:52:07.597096 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:07.596993    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d1ebf76-5f16-412b-8784-8c29dd59ca56" path="/var/lib/kubelet/pods/1d1ebf76-5f16-412b-8784-8c29dd59ca56/volumes"
Nov 02 19:52:13.486288 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:13.486255    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 19:52:34.294558 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:52:34.294520    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 19:52:34.952189 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:34.952156    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-pgq8d"]
Nov 02 19:52:34.952777 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:34.952764    4463 cpu_manager.go:414] "containerMap: removing container" podUID="1d1ebf76-5f16-412b-8784-8c29dd59ca56" containerName="istio-proxy"
Nov 02 19:52:34.952823 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:34.952779    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d1ebf76-5f16-412b-8784-8c29dd59ca56" containerName="istio-proxy"
Nov 02 19:52:34.952823 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:34.952807    4463 cpu_manager.go:414] "containerMap: removing container" podUID="2d99e7b7-63b9-4e1b-aa57-4ab098b789b4" containerName="payload-processing"
Nov 02 19:52:34.952823 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:34.952813    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d99e7b7-63b9-4e1b-aa57-4ab098b789b4" containerName="payload-processing"
Nov 02 19:52:34.952912 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:34.952907    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d1ebf76-5f16-412b-8784-8c29dd59ca56" containerName="istio-proxy"
Nov 02 19:52:34.952950 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:34.952917    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d1ebf76-5f16-412b-8784-8c29dd59ca56" containerName="istio-proxy"
Nov 02 19:52:34.952950 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:34.952925    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d99e7b7-63b9-4e1b-aa57-4ab098b789b4" containerName="payload-processing"
Nov 02 19:52:34.955327 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:34.955310    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-pgq8d"
Nov 02 19:52:34.998913 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:34.998885    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5bbdh"]
Nov 02 19:52:34.999594 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:34.999566    4463 cpu_manager.go:414] "containerMap: removing container" podUID="1d1ebf76-5f16-412b-8784-8c29dd59ca56" containerName="istio-proxy"
Nov 02 19:52:34.999594 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:34.999593    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d1ebf76-5f16-412b-8784-8c29dd59ca56" containerName="istio-proxy"
Nov 02 19:52:35.003984 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:35.003951    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5bbdh"
Nov 02 19:52:35.044777 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:35.044747    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-lkfn9"]
Nov 02 19:52:35.044976 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:35.044955    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-lkfn9" podUID="b33f5f29-4f1a-4e70-a821-df5018260ad7" containerName="istio-proxy" containerID="cri-o://7133490961b3a61bd67e108fd1984df415ac2839fef434550c4cbc2f85b9dc86" gracePeriod=30
Nov 02 19:52:35.057866 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:35.057840    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kpr8z"]
Nov 02 19:52:35.058048 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:35.058007    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kpr8z" podUID="c725b384-6c96-444e-a657-d4ebeb0e5685" containerName="istio-proxy" containerID="cri-o://253434d791e65f1934bb94b67c4e2f77546f9f6376672c2b019d7c97861ba244" gracePeriod=30
Nov 02 19:52:35.061568 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:35.061543    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kglqn"]
Nov 02 19:52:35.061757 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:35.061738    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kglqn" podUID="8c25aace-5449-4ed4-a07a-dd08f8eb4a4e" containerName="istio-proxy" containerID="cri-o://77e675d907ae53155138da5ad3f3ffdd046434507979aff1ec567d11b4da7706" gracePeriod=30
Nov 02 19:52:35.084887 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:35.084858    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-k7k7k"]
Nov 02 19:52:35.085101 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:35.085074    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-k7k7k" podUID="bb05ff45-67c9-44fb-995d-5f9e0b4059e9" containerName="istio-proxy" containerID="cri-o://9e77367018fb447387d1a4222f5204d4bc156185e6f0ffec132c0a35da61c123" gracePeriod=30
Nov 02 19:52:35.097999 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:35.097973    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5bbdh"]
Nov 02 19:52:35.098969 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:35.098939    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b600454-98b7-49b2-a129-6b65406c4d9c-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5bbdh\" (UID: \"6b600454-98b7-49b2-a129-6b65406c4d9c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5bbdh"
Nov 02 19:52:35.099184 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:35.099008    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ff8e962-f7e6-4d82-a818-261d21bdba26-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-pgq8d\" (UID: \"6ff8e962-f7e6-4d82-a818-261d21bdba26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-pgq8d"
Nov 02 19:52:35.099184 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:35.099071    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ff8e962-f7e6-4d82-a818-261d21bdba26-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-pgq8d\" (UID: \"6ff8e962-f7e6-4d82-a818-261d21bdba26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-pgq8d"
Nov 02 19:52:35.099184 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:35.099088    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b600454-98b7-49b2-a129-6b65406c4d9c-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5bbdh\" (UID: \"6b600454-98b7-49b2-a129-6b65406c4d9c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5bbdh"
Nov 02 19:52:35.099184 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:35.099109    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ff8e962-f7e6-4d82-a818-261d21bdba26-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-pgq8d\" (UID: \"6ff8e962-f7e6-4d82-a818-261d21bdba26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-pgq8d"
Nov 02 19:52:35.099184 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:35.099172    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ff8e962-f7e6-4d82-a818-261d21bdba26-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-pgq8d\" (UID: \"6ff8e962-f7e6-4d82-a818-261d21bdba26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-pgq8d"
Nov 02 19:52:35.099468 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:35.099210    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npr8j\" (UniqueName: \"kubernetes.io/projected/6b600454-98b7-49b2-a129-6b65406c4d9c-kube-api-access-npr8j\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5bbdh\" (UID: \"6b600454-98b7-49b2-a129-6b65406c4d9c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5bbdh"
Nov 02 19:52:35.099468 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:35.099274    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ff8e962-f7e6-4d82-a818-261d21bdba26-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-pgq8d\" (UID: \"6ff8e962-f7e6-4d82-a818-261d21bdba26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-pgq8d"
Nov 02 19:52:35.099468 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:35.099333    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b600454-98b7-49b2-a129-6b65406c4d9c-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5bbdh\" (UID: \"6b600454-98b7-49b2-a129-6b65406c4d9c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5bbdh"
Nov 02 19:52:35.099468 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:35.099359    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ff8e962-f7e6-4d82-a818-261d21bdba26-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-pgq8d\" (UID: \"6ff8e962-f7e6-4d82-a818-261d21bdba26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-pgq8d"
Nov 02 19:52:35.099468 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:35.099383    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b600454-98b7-49b2-a129-6b65406c4d9c-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5bbdh\" (UID: \"6b600454-98b7-49b2-a129-6b65406c4d9c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5bbdh"
Nov 02 19:52:35.099468 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:35.099447    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b600454-98b7-49b2-a129-6b65406c4d9c-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5bbdh\" (UID: \"6b600454-98b7-49b2-a129-6b65406c4d9c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5bbdh"
Nov 02 19:52:35.099468 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:35.099469    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b600454-98b7-49b2-a129-6b65406c4d9c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5bbdh\" (UID: \"6b600454-98b7-49b2-a129-6b65406c4d9c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5bbdh"
Nov 02 19:52:35.099831 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:35.099501    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ff8e962-f7e6-4d82-a818-261d21bdba26-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-pgq8d\" (UID: \"6ff8e962-f7e6-4d82-a818-261d21bdba26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-pgq8d"
Nov 02 19:52:35.099831 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:35.099527    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b600454-98b7-49b2-a129-6b65406c4d9c-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5bbdh\" (UID: \"6b600454-98b7-49b2-a129-6b65406c4d9c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5bbdh"
Nov 02 19:52:35.099831 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:35.099569    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrmps\" (UniqueName: \"kubernetes.io/projected/6ff8e962-f7e6-4d82-a818-261d21bdba26-kube-api-access-mrmps\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-pgq8d\" (UID: \"6ff8e962-f7e6-4d82-a818-261d21bdba26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-pgq8d"
Nov 02 19:52:35.099831 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:35.099600    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b600454-98b7-49b2-a129-6b65406c4d9c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5bbdh\" (UID: \"6b600454-98b7-49b2-a129-6b65406c4d9c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5bbdh"
Nov 02 19:52:35.099831 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:35.099679    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ff8e962-f7e6-4d82-a818-261d21bdba26-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-pgq8d\" (UID: \"6ff8e962-f7e6-4d82-a818-261d21bdba26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-pgq8d"
Nov 02 19:52:35.153764 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:35.153733    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-pgq8d"]
Nov 02 19:52:35.201491 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:35.201455    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ff8e962-f7e6-4d82-a818-261d21bdba26-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-pgq8d\" (UID: \"6ff8e962-f7e6-4d82-a818-261d21bdba26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-pgq8d"
Nov 02 19:52:35.201691 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:35.201514    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npr8j\" (UniqueName: \"kubernetes.io/projected/6b600454-98b7-49b2-a129-6b65406c4d9c-kube-api-access-npr8j\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5bbdh\" (UID: \"6b600454-98b7-49b2-a129-6b65406c4d9c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5bbdh"
Nov 02 19:52:35.201691 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:35.201564    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ff8e962-f7e6-4d82-a818-261d21bdba26-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-pgq8d\" (UID: \"6ff8e962-f7e6-4d82-a818-261d21bdba26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-pgq8d"
Nov 02 19:52:35.201691 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:35.201611    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b600454-98b7-49b2-a129-6b65406c4d9c-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5bbdh\" (UID: \"6b600454-98b7-49b2-a129-6b65406c4d9c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5bbdh"
Nov 02 19:52:35.201691 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:35.201639    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ff8e962-f7e6-4d82-a818-261d21bdba26-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-pgq8d\" (UID: \"6ff8e962-f7e6-4d82-a818-261d21bdba26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-pgq8d"
Nov 02 19:52:35.201691 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:35.201661    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b600454-98b7-49b2-a129-6b65406c4d9c-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5bbdh\" (UID: \"6b600454-98b7-49b2-a129-6b65406c4d9c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5bbdh"
Nov 02 19:52:35.201975 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:35.201724    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b600454-98b7-49b2-a129-6b65406c4d9c-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5bbdh\" (UID: \"6b600454-98b7-49b2-a129-6b65406c4d9c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5bbdh"
Nov 02 19:52:35.201975 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:35.201748    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b600454-98b7-49b2-a129-6b65406c4d9c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5bbdh\" (UID: \"6b600454-98b7-49b2-a129-6b65406c4d9c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5bbdh"
Nov 02 19:52:35.201975 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:35.201780    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ff8e962-f7e6-4d82-a818-261d21bdba26-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-pgq8d\" (UID: \"6ff8e962-f7e6-4d82-a818-261d21bdba26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-pgq8d"
Nov 02 19:52:35.201975 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:35.201806    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b600454-98b7-49b2-a129-6b65406c4d9c-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5bbdh\" (UID: \"6b600454-98b7-49b2-a129-6b65406c4d9c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5bbdh"
Nov 02 19:52:35.201975 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:35.201848    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrmps\" (UniqueName: \"kubernetes.io/projected/6ff8e962-f7e6-4d82-a818-261d21bdba26-kube-api-access-mrmps\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-pgq8d\" (UID: \"6ff8e962-f7e6-4d82-a818-261d21bdba26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-pgq8d"
Nov 02 19:52:35.201975 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:35.201884    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b600454-98b7-49b2-a129-6b65406c4d9c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5bbdh\" (UID: \"6b600454-98b7-49b2-a129-6b65406c4d9c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5bbdh"
Nov 02 19:52:35.201975 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:35.201960    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ff8e962-f7e6-4d82-a818-261d21bdba26-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-pgq8d\" (UID: \"6ff8e962-f7e6-4d82-a818-261d21bdba26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-pgq8d"
Nov 02 19:52:35.202385 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:35.202056    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b600454-98b7-49b2-a129-6b65406c4d9c-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5bbdh\" (UID: \"6b600454-98b7-49b2-a129-6b65406c4d9c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5bbdh"
Nov 02 19:52:35.202385 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:35.202120    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ff8e962-f7e6-4d82-a818-261d21bdba26-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-pgq8d\" (UID: \"6ff8e962-f7e6-4d82-a818-261d21bdba26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-pgq8d"
Nov 02 19:52:35.202385 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:35.202134    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b600454-98b7-49b2-a129-6b65406c4d9c-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5bbdh\" (UID: \"6b600454-98b7-49b2-a129-6b65406c4d9c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5bbdh"
Nov 02 19:52:35.202385 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:35.202125    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ff8e962-f7e6-4d82-a818-261d21bdba26-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-pgq8d\" (UID: \"6ff8e962-f7e6-4d82-a818-261d21bdba26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-pgq8d"
Nov 02 19:52:35.202385 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:35.202192    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ff8e962-f7e6-4d82-a818-261d21bdba26-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-pgq8d\" (UID: \"6ff8e962-f7e6-4d82-a818-261d21bdba26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-pgq8d"
Nov 02 19:52:35.202385 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:35.202218    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b600454-98b7-49b2-a129-6b65406c4d9c-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5bbdh\" (UID: \"6b600454-98b7-49b2-a129-6b65406c4d9c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5bbdh"
Nov 02 19:52:35.202385 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:35.202264    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ff8e962-f7e6-4d82-a818-261d21bdba26-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-pgq8d\" (UID: \"6ff8e962-f7e6-4d82-a818-261d21bdba26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-pgq8d"
Nov 02 19:52:35.202385 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:35.202342    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b600454-98b7-49b2-a129-6b65406c4d9c-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5bbdh\" (UID: \"6b600454-98b7-49b2-a129-6b65406c4d9c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5bbdh"
Nov 02 19:52:35.202804 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:35.202674    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b600454-98b7-49b2-a129-6b65406c4d9c-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5bbdh\" (UID: \"6b600454-98b7-49b2-a129-6b65406c4d9c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5bbdh"
Nov 02 19:52:35.202923 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:35.202827    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ff8e962-f7e6-4d82-a818-261d21bdba26-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-pgq8d\" (UID: \"6ff8e962-f7e6-4d82-a818-261d21bdba26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-pgq8d"
Nov 02 19:52:35.203826 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:35.203139    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ff8e962-f7e6-4d82-a818-261d21bdba26-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-pgq8d\" (UID: \"6ff8e962-f7e6-4d82-a818-261d21bdba26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-pgq8d"
Nov 02 19:52:35.203826 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:35.203388    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b600454-98b7-49b2-a129-6b65406c4d9c-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5bbdh\" (UID: \"6b600454-98b7-49b2-a129-6b65406c4d9c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5bbdh"
Nov 02 19:52:35.203826 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:35.203578    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ff8e962-f7e6-4d82-a818-261d21bdba26-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-pgq8d\" (UID: \"6ff8e962-f7e6-4d82-a818-261d21bdba26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-pgq8d"
Nov 02 19:52:35.203826 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:35.203786    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b600454-98b7-49b2-a129-6b65406c4d9c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5bbdh\" (UID: \"6b600454-98b7-49b2-a129-6b65406c4d9c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5bbdh"
Nov 02 19:52:35.204474 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:35.204444    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ff8e962-f7e6-4d82-a818-261d21bdba26-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-pgq8d\" (UID: \"6ff8e962-f7e6-4d82-a818-261d21bdba26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-pgq8d"
Nov 02 19:52:35.206175 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:35.206149    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ff8e962-f7e6-4d82-a818-261d21bdba26-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-pgq8d\" (UID: \"6ff8e962-f7e6-4d82-a818-261d21bdba26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-pgq8d"
Nov 02 19:52:35.208550 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:35.208524    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b600454-98b7-49b2-a129-6b65406c4d9c-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5bbdh\" (UID: \"6b600454-98b7-49b2-a129-6b65406c4d9c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5bbdh"
Nov 02 19:52:35.208689 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:35.208666    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ff8e962-f7e6-4d82-a818-261d21bdba26-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-pgq8d\" (UID: \"6ff8e962-f7e6-4d82-a818-261d21bdba26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-pgq8d"
Nov 02 19:52:35.209925 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:35.209878    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b600454-98b7-49b2-a129-6b65406c4d9c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5bbdh\" (UID: \"6b600454-98b7-49b2-a129-6b65406c4d9c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5bbdh"
Nov 02 19:52:35.254164 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:35.254136    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npr8j\" (UniqueName: \"kubernetes.io/projected/6b600454-98b7-49b2-a129-6b65406c4d9c-kube-api-access-npr8j\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5bbdh\" (UID: \"6b600454-98b7-49b2-a129-6b65406c4d9c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5bbdh"
Nov 02 19:52:35.254570 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:35.254549    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ff8e962-f7e6-4d82-a818-261d21bdba26-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-pgq8d\" (UID: \"6ff8e962-f7e6-4d82-a818-261d21bdba26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-pgq8d"
Nov 02 19:52:35.254933 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:35.254916    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b600454-98b7-49b2-a129-6b65406c4d9c-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5bbdh\" (UID: \"6b600454-98b7-49b2-a129-6b65406c4d9c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5bbdh"
Nov 02 19:52:35.255194 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:35.255173    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrmps\" (UniqueName: \"kubernetes.io/projected/6ff8e962-f7e6-4d82-a818-261d21bdba26-kube-api-access-mrmps\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-pgq8d\" (UID: \"6ff8e962-f7e6-4d82-a818-261d21bdba26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-pgq8d"
Nov 02 19:52:35.267976 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:35.267957    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-pgq8d"
Nov 02 19:52:35.322324 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:35.322234    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5bbdh"
Nov 02 19:52:35.640128 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:35.640093    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-pgq8d"]
Nov 02 19:52:35.640702 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:52:35.640676    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ff8e962_f7e6_4d82_a818_261d21bdba26.slice/crio-b8831688646578f918979c4f90d63f089bc601477c286e8feadb2fc90bb46215 WatchSource:0}: Error finding container b8831688646578f918979c4f90d63f089bc601477c286e8feadb2fc90bb46215: Status 404 returned error can't find the container with id b8831688646578f918979c4f90d63f089bc601477c286e8feadb2fc90bb46215
Nov 02 19:52:35.643179 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:35.643146    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:52:35.643272 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:35.643208    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:52:35.643272 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:35.643235    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:52:35.708240 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:35.708210    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5bbdh"]
Nov 02 19:52:35.711804 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:52:35.711773    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b600454_98b7_49b2_a129_6b65406c4d9c.slice/crio-7a476d4de6ea333903df129c65af4badc21e1e047e6ea4871267fafcd8efe223 WatchSource:0}: Error finding container 7a476d4de6ea333903df129c65af4badc21e1e047e6ea4871267fafcd8efe223: Status 404 returned error can't find the container with id 7a476d4de6ea333903df129c65af4badc21e1e047e6ea4871267fafcd8efe223
Nov 02 19:52:35.714623 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:35.714578    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:52:35.714737 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:35.714661    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:52:35.714795 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:35.714733    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:52:36.210037 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:36.207889    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-pgq8d" event={"ID":"6ff8e962-f7e6-4d82-a818-261d21bdba26","Type":"ContainerStarted","Data":"afb37a4cfedbff6c20f0827d8ae976f43b3857e0291ab2f02989bf9ea5f90b3c"}
Nov 02 19:52:36.210246 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:36.210179    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-pgq8d" event={"ID":"6ff8e962-f7e6-4d82-a818-261d21bdba26","Type":"ContainerStarted","Data":"b8831688646578f918979c4f90d63f089bc601477c286e8feadb2fc90bb46215"}
Nov 02 19:52:36.210246 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:36.210219    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5bbdh" event={"ID":"6b600454-98b7-49b2-a129-6b65406c4d9c","Type":"ContainerStarted","Data":"0da393673fc95d9a09c11a20ca0ebf71d39f7d1ff7395aef4ff48346a0a341f1"}
Nov 02 19:52:36.210246 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:36.210244    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5bbdh" event={"ID":"6b600454-98b7-49b2-a129-6b65406c4d9c","Type":"ContainerStarted","Data":"7a476d4de6ea333903df129c65af4badc21e1e047e6ea4871267fafcd8efe223"}
Nov 02 19:52:36.271485 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:36.271455    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-pgq8d"
Nov 02 19:52:36.273331 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:36.273302    4463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-pgq8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 02 19:52:36.273514 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:36.273490    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-pgq8d" podUID="6ff8e962-f7e6-4d82-a818-261d21bdba26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 02 19:52:36.323457 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:36.323315    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5bbdh"
Nov 02 19:52:36.326136 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:36.326087    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-pgq8d" podStartSLOduration=2.3260729700000002 podStartE2EDuration="2.32607297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:52:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:52:36.321720434 +0000 UTC m=+1462.928754222" watchObservedRunningTime="2026-11-02 19:52:36.32607297 +0000 UTC m=+1462.933106739"
Nov 02 19:52:36.326336 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:36.326314    4463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-5bbdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Nov 02 19:52:36.326383 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:36.326356    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5bbdh" podUID="6b600454-98b7-49b2-a129-6b65406c4d9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Nov 02 19:52:36.434465 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:36.434359    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5bbdh" podStartSLOduration=2.43434519 podStartE2EDuration="2.43434519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:52:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:52:36.43374388 +0000 UTC m=+1463.040777665" watchObservedRunningTime="2026-11-02 19:52:36.43434519 +0000 UTC m=+1463.041378959"
Nov 02 19:52:37.275041 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:37.272541    4463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-pgq8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 02 19:52:37.275041 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:37.272598    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-pgq8d" podUID="6ff8e962-f7e6-4d82-a818-261d21bdba26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 02 19:52:37.323674 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:37.323634    4463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-5bbdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Nov 02 19:52:37.324483 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:37.323692    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5bbdh" podUID="6b600454-98b7-49b2-a129-6b65406c4d9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Nov 02 19:52:38.283035 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:38.279139    4463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-pgq8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 02 19:52:38.283035 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:38.279199    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-pgq8d" podUID="6ff8e962-f7e6-4d82-a818-261d21bdba26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 02 19:52:38.323582 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:38.323550    4463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-5bbdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Nov 02 19:52:38.323746 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:38.323602    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5bbdh" podUID="6b600454-98b7-49b2-a129-6b65406c4d9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Nov 02 19:52:39.271872 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:39.271128    4463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-pgq8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 02 19:52:39.271872 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:39.271228    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-pgq8d" podUID="6ff8e962-f7e6-4d82-a818-261d21bdba26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 02 19:52:39.326038 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:39.324956    4463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-5bbdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Nov 02 19:52:39.326038 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:39.325117    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5bbdh" podUID="6b600454-98b7-49b2-a129-6b65406c4d9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Nov 02 19:52:40.302039 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:40.294882    4463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-pgq8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 02 19:52:40.302039 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:40.294958    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-pgq8d" podUID="6ff8e962-f7e6-4d82-a818-261d21bdba26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 02 19:52:40.310776 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:40.310734    4463 generic.go:356] "Generic (PLEG): container finished" podID="b33f5f29-4f1a-4e70-a821-df5018260ad7" containerID="7133490961b3a61bd67e108fd1984df415ac2839fef434550c4cbc2f85b9dc86" exitCode=0
Nov 02 19:52:40.310941 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:40.310905    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-lkfn9" event={"ID":"b33f5f29-4f1a-4e70-a821-df5018260ad7","Type":"ContainerDied","Data":"7133490961b3a61bd67e108fd1984df415ac2839fef434550c4cbc2f85b9dc86"}
Nov 02 19:52:40.316930 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:40.312990    4463 generic.go:356] "Generic (PLEG): container finished" podID="c725b384-6c96-444e-a657-d4ebeb0e5685" containerID="253434d791e65f1934bb94b67c4e2f77546f9f6376672c2b019d7c97861ba244" exitCode=0
Nov 02 19:52:40.316930 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:40.313075    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kpr8z" event={"ID":"c725b384-6c96-444e-a657-d4ebeb0e5685","Type":"ContainerDied","Data":"253434d791e65f1934bb94b67c4e2f77546f9f6376672c2b019d7c97861ba244"}
Nov 02 19:52:40.316930 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:40.314648    4463 generic.go:356] "Generic (PLEG): container finished" podID="bb05ff45-67c9-44fb-995d-5f9e0b4059e9" containerID="9e77367018fb447387d1a4222f5204d4bc156185e6f0ffec132c0a35da61c123" exitCode=0
Nov 02 19:52:40.316930 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:40.314823    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-k7k7k" event={"ID":"bb05ff45-67c9-44fb-995d-5f9e0b4059e9","Type":"ContainerDied","Data":"9e77367018fb447387d1a4222f5204d4bc156185e6f0ffec132c0a35da61c123"}
Nov 02 19:52:40.316930 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:40.316797    4463 generic.go:356] "Generic (PLEG): container finished" podID="8c25aace-5449-4ed4-a07a-dd08f8eb4a4e" containerID="77e675d907ae53155138da5ad3f3ffdd046434507979aff1ec567d11b4da7706" exitCode=0
Nov 02 19:52:40.316930 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:40.316832    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kglqn" event={"ID":"8c25aace-5449-4ed4-a07a-dd08f8eb4a4e","Type":"ContainerDied","Data":"77e675d907ae53155138da5ad3f3ffdd046434507979aff1ec567d11b4da7706"}
Nov 02 19:52:40.324066 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:40.323435    4463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-5bbdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Nov 02 19:52:40.324066 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:40.323476    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5bbdh" podUID="6b600454-98b7-49b2-a129-6b65406c4d9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Nov 02 19:52:40.416096 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:40.416065    4463 patch_prober.go:29] interesting pod/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kpr8z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Nov 02 19:52:40.416576 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:40.416130    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kpr8z" podUID="c725b384-6c96-444e-a657-d4ebeb0e5685" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Nov 02 19:52:40.626355 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:40.626327    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kglqn"
Nov 02 19:52:40.723973 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:40.723935    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c25aace-5449-4ed4-a07a-dd08f8eb4a4e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c25aace-5449-4ed4-a07a-dd08f8eb4a4e-workload-socket\") pod \"8c25aace-5449-4ed4-a07a-dd08f8eb4a4e\" (UID: \"8c25aace-5449-4ed4-a07a-dd08f8eb4a4e\") "
Nov 02 19:52:40.724216 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:40.724050    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c25aace-5449-4ed4-a07a-dd08f8eb4a4e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c25aace-5449-4ed4-a07a-dd08f8eb4a4e-credential-socket\") pod \"8c25aace-5449-4ed4-a07a-dd08f8eb4a4e\" (UID: \"8c25aace-5449-4ed4-a07a-dd08f8eb4a4e\") "
Nov 02 19:52:40.724216 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:40.724094    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c25aace-5449-4ed4-a07a-dd08f8eb4a4e-istio-token\" (UniqueName: \"kubernetes.io/projected/8c25aace-5449-4ed4-a07a-dd08f8eb4a4e-istio-token\") pod \"8c25aace-5449-4ed4-a07a-dd08f8eb4a4e\" (UID: \"8c25aace-5449-4ed4-a07a-dd08f8eb4a4e\") "
Nov 02 19:52:40.724216 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:40.724157    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c25aace-5449-4ed4-a07a-dd08f8eb4a4e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c25aace-5449-4ed4-a07a-dd08f8eb4a4e-istiod-ca-cert\") pod \"8c25aace-5449-4ed4-a07a-dd08f8eb4a4e\" (UID: \"8c25aace-5449-4ed4-a07a-dd08f8eb4a4e\") "
Nov 02 19:52:40.724467 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:40.724351    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c25aace-5449-4ed4-a07a-dd08f8eb4a4e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c25aace-5449-4ed4-a07a-dd08f8eb4a4e-istio-envoy\") pod \"8c25aace-5449-4ed4-a07a-dd08f8eb4a4e\" (UID: \"8c25aace-5449-4ed4-a07a-dd08f8eb4a4e\") "
Nov 02 19:52:40.724467 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:40.724385    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c25aace-5449-4ed4-a07a-dd08f8eb4a4e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c25aace-5449-4ed4-a07a-dd08f8eb4a4e-workload-certs\") pod \"8c25aace-5449-4ed4-a07a-dd08f8eb4a4e\" (UID: \"8c25aace-5449-4ed4-a07a-dd08f8eb4a4e\") "
Nov 02 19:52:40.724467 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:40.724442    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c25aace-5449-4ed4-a07a-dd08f8eb4a4e-kube-api-access-6wd7r\" (UniqueName: \"kubernetes.io/projected/8c25aace-5449-4ed4-a07a-dd08f8eb4a4e-kube-api-access-6wd7r\") pod \"8c25aace-5449-4ed4-a07a-dd08f8eb4a4e\" (UID: \"8c25aace-5449-4ed4-a07a-dd08f8eb4a4e\") "
Nov 02 19:52:40.724617 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:40.724521    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8c25aace-5449-4ed4-a07a-dd08f8eb4a4e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c25aace-5449-4ed4-a07a-dd08f8eb4a4e-istio-podinfo\") pod \"8c25aace-5449-4ed4-a07a-dd08f8eb4a4e\" (UID: \"8c25aace-5449-4ed4-a07a-dd08f8eb4a4e\") "
Nov 02 19:52:40.724667 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:40.724614    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c25aace-5449-4ed4-a07a-dd08f8eb4a4e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c25aace-5449-4ed4-a07a-dd08f8eb4a4e-istio-data\") pod \"8c25aace-5449-4ed4-a07a-dd08f8eb4a4e\" (UID: \"8c25aace-5449-4ed4-a07a-dd08f8eb4a4e\") "
Nov 02 19:52:40.725938 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:40.725266    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c25aace-5449-4ed4-a07a-dd08f8eb4a4e-istiod-ca-cert" pod "8c25aace-5449-4ed4-a07a-dd08f8eb4a4e" (UID: "8c25aace-5449-4ed4-a07a-dd08f8eb4a4e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 19:52:40.725938 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:40.725559    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c25aace-5449-4ed4-a07a-dd08f8eb4a4e-workload-certs" pod "8c25aace-5449-4ed4-a07a-dd08f8eb4a4e" (UID: "8c25aace-5449-4ed4-a07a-dd08f8eb4a4e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:52:40.725938 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:40.725792    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c25aace-5449-4ed4-a07a-dd08f8eb4a4e-istio-data" pod "8c25aace-5449-4ed4-a07a-dd08f8eb4a4e" (UID: "8c25aace-5449-4ed4-a07a-dd08f8eb4a4e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:52:40.725938 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:40.725913    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c25aace-5449-4ed4-a07a-dd08f8eb4a4e-credential-socket" pod "8c25aace-5449-4ed4-a07a-dd08f8eb4a4e" (UID: "8c25aace-5449-4ed4-a07a-dd08f8eb4a4e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:52:40.726350 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:40.726322    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c25aace-5449-4ed4-a07a-dd08f8eb4a4e-workload-socket" pod "8c25aace-5449-4ed4-a07a-dd08f8eb4a4e" (UID: "8c25aace-5449-4ed4-a07a-dd08f8eb4a4e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:52:40.730538 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:40.730451    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kpr8z"
Nov 02 19:52:40.732970 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:40.732937    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8c25aace-5449-4ed4-a07a-dd08f8eb4a4e-istio-podinfo" pod "8c25aace-5449-4ed4-a07a-dd08f8eb4a4e" (UID: "8c25aace-5449-4ed4-a07a-dd08f8eb4a4e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 19:52:40.733209 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:40.733189    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c25aace-5449-4ed4-a07a-dd08f8eb4a4e-istio-envoy" pod "8c25aace-5449-4ed4-a07a-dd08f8eb4a4e" (UID: "8c25aace-5449-4ed4-a07a-dd08f8eb4a4e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:52:40.740714 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:40.740685    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c25aace-5449-4ed4-a07a-dd08f8eb4a4e-kube-api-access-6wd7r" pod "8c25aace-5449-4ed4-a07a-dd08f8eb4a4e" (UID: "8c25aace-5449-4ed4-a07a-dd08f8eb4a4e"). InnerVolumeSpecName "kube-api-access-6wd7r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:52:40.741575 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:40.741540    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c25aace-5449-4ed4-a07a-dd08f8eb4a4e-istio-token" pod "8c25aace-5449-4ed4-a07a-dd08f8eb4a4e" (UID: "8c25aace-5449-4ed4-a07a-dd08f8eb4a4e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:52:40.833567 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:40.833529    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c725b384-6c96-444e-a657-d4ebeb0e5685-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c725b384-6c96-444e-a657-d4ebeb0e5685-istio-podinfo\") pod \"c725b384-6c96-444e-a657-d4ebeb0e5685\" (UID: \"c725b384-6c96-444e-a657-d4ebeb0e5685\") "
Nov 02 19:52:40.841048 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:40.840109    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c725b384-6c96-444e-a657-d4ebeb0e5685-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c725b384-6c96-444e-a657-d4ebeb0e5685-workload-socket\") pod \"c725b384-6c96-444e-a657-d4ebeb0e5685\" (UID: \"c725b384-6c96-444e-a657-d4ebeb0e5685\") "
Nov 02 19:52:40.841048 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:40.840265    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c725b384-6c96-444e-a657-d4ebeb0e5685-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c725b384-6c96-444e-a657-d4ebeb0e5685-workload-certs\") pod \"c725b384-6c96-444e-a657-d4ebeb0e5685\" (UID: \"c725b384-6c96-444e-a657-d4ebeb0e5685\") "
Nov 02 19:52:40.841048 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:40.840386    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c725b384-6c96-444e-a657-d4ebeb0e5685-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c725b384-6c96-444e-a657-d4ebeb0e5685-istio-envoy\") pod \"c725b384-6c96-444e-a657-d4ebeb0e5685\" (UID: \"c725b384-6c96-444e-a657-d4ebeb0e5685\") "
Nov 02 19:52:40.841048 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:40.840502    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c725b384-6c96-444e-a657-d4ebeb0e5685-istio-token\" (UniqueName: \"kubernetes.io/projected/c725b384-6c96-444e-a657-d4ebeb0e5685-istio-token\") pod \"c725b384-6c96-444e-a657-d4ebeb0e5685\" (UID: \"c725b384-6c96-444e-a657-d4ebeb0e5685\") "
Nov 02 19:52:40.841048 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:40.840545    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c725b384-6c96-444e-a657-d4ebeb0e5685-kube-api-access-lfp7t\" (UniqueName: \"kubernetes.io/projected/c725b384-6c96-444e-a657-d4ebeb0e5685-kube-api-access-lfp7t\") pod \"c725b384-6c96-444e-a657-d4ebeb0e5685\" (UID: \"c725b384-6c96-444e-a657-d4ebeb0e5685\") "
Nov 02 19:52:40.841048 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:40.840642    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c725b384-6c96-444e-a657-d4ebeb0e5685-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c725b384-6c96-444e-a657-d4ebeb0e5685-credential-socket\") pod \"c725b384-6c96-444e-a657-d4ebeb0e5685\" (UID: \"c725b384-6c96-444e-a657-d4ebeb0e5685\") "
Nov 02 19:52:40.841048 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:40.840703    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c725b384-6c96-444e-a657-d4ebeb0e5685-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c725b384-6c96-444e-a657-d4ebeb0e5685-istiod-ca-cert\") pod \"c725b384-6c96-444e-a657-d4ebeb0e5685\" (UID: \"c725b384-6c96-444e-a657-d4ebeb0e5685\") "
Nov 02 19:52:40.841048 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:40.840823    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c725b384-6c96-444e-a657-d4ebeb0e5685-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c725b384-6c96-444e-a657-d4ebeb0e5685-istio-data\") pod \"c725b384-6c96-444e-a657-d4ebeb0e5685\" (UID: \"c725b384-6c96-444e-a657-d4ebeb0e5685\") "
Nov 02 19:52:40.841642 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:40.838734    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c725b384-6c96-444e-a657-d4ebeb0e5685-istio-podinfo" pod "c725b384-6c96-444e-a657-d4ebeb0e5685" (UID: "c725b384-6c96-444e-a657-d4ebeb0e5685"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 19:52:40.841731 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:40.841688    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c725b384-6c96-444e-a657-d4ebeb0e5685-istio-data" pod "c725b384-6c96-444e-a657-d4ebeb0e5685" (UID: "c725b384-6c96-444e-a657-d4ebeb0e5685"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:52:40.842251 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:40.841622    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c725b384-6c96-444e-a657-d4ebeb0e5685-workload-socket" pod "c725b384-6c96-444e-a657-d4ebeb0e5685" (UID: "c725b384-6c96-444e-a657-d4ebeb0e5685"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:52:40.843321 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:40.843100    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c725b384-6c96-444e-a657-d4ebeb0e5685-workload-certs" pod "c725b384-6c96-444e-a657-d4ebeb0e5685" (UID: "c725b384-6c96-444e-a657-d4ebeb0e5685"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:52:40.843583 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:40.843521    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c725b384-6c96-444e-a657-d4ebeb0e5685-istiod-ca-cert" pod "c725b384-6c96-444e-a657-d4ebeb0e5685" (UID: "c725b384-6c96-444e-a657-d4ebeb0e5685"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 19:52:40.843693 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:40.843670    4463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c25aace-5449-4ed4-a07a-dd08f8eb4a4e-istio-data\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:52:40.843759 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:40.843703    4463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c25aace-5449-4ed4-a07a-dd08f8eb4a4e-workload-socket\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:52:40.847716 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:40.845726    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c725b384-6c96-444e-a657-d4ebeb0e5685-istio-envoy" pod "c725b384-6c96-444e-a657-d4ebeb0e5685" (UID: "c725b384-6c96-444e-a657-d4ebeb0e5685"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:52:40.847716 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:40.843714    4463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c25aace-5449-4ed4-a07a-dd08f8eb4a4e-credential-socket\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:52:40.847716 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:40.845907    4463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c725b384-6c96-444e-a657-d4ebeb0e5685-istio-data\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:52:40.847716 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:40.845932    4463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c25aace-5449-4ed4-a07a-dd08f8eb4a4e-istio-token\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:52:40.847716 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:40.845944    4463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c25aace-5449-4ed4-a07a-dd08f8eb4a4e-istiod-ca-cert\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:52:40.847716 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:40.845954    4463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c725b384-6c96-444e-a657-d4ebeb0e5685-istio-podinfo\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:52:40.847716 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:40.845963    4463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c725b384-6c96-444e-a657-d4ebeb0e5685-workload-socket\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:52:40.847716 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:40.845971    4463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c25aace-5449-4ed4-a07a-dd08f8eb4a4e-istio-envoy\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:52:40.847716 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:40.845980    4463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c25aace-5449-4ed4-a07a-dd08f8eb4a4e-workload-certs\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:52:40.847716 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:40.845990    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wd7r\" (UniqueName: \"kubernetes.io/projected/8c25aace-5449-4ed4-a07a-dd08f8eb4a4e-kube-api-access-6wd7r\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:52:40.847716 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:40.846031    4463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c25aace-5449-4ed4-a07a-dd08f8eb4a4e-istio-podinfo\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:52:40.848602 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:40.848312    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c725b384-6c96-444e-a657-d4ebeb0e5685-credential-socket" pod "c725b384-6c96-444e-a657-d4ebeb0e5685" (UID: "c725b384-6c96-444e-a657-d4ebeb0e5685"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:52:40.926949 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:40.926911    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-lkfn9"
Nov 02 19:52:40.949236 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:40.933367    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c725b384-6c96-444e-a657-d4ebeb0e5685-kube-api-access-lfp7t" pod "c725b384-6c96-444e-a657-d4ebeb0e5685" (UID: "c725b384-6c96-444e-a657-d4ebeb0e5685"). InnerVolumeSpecName "kube-api-access-lfp7t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:52:40.949236 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:40.935765    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c725b384-6c96-444e-a657-d4ebeb0e5685-istio-token" pod "c725b384-6c96-444e-a657-d4ebeb0e5685" (UID: "c725b384-6c96-444e-a657-d4ebeb0e5685"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:52:40.949236 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:40.940560    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-k7k7k"
Nov 02 19:52:40.949236 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:40.946641    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b33f5f29-4f1a-4e70-a821-df5018260ad7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b33f5f29-4f1a-4e70-a821-df5018260ad7-istiod-ca-cert\") pod \"b33f5f29-4f1a-4e70-a821-df5018260ad7\" (UID: \"b33f5f29-4f1a-4e70-a821-df5018260ad7\") "
Nov 02 19:52:40.949236 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:40.946711    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b33f5f29-4f1a-4e70-a821-df5018260ad7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b33f5f29-4f1a-4e70-a821-df5018260ad7-istio-data\") pod \"b33f5f29-4f1a-4e70-a821-df5018260ad7\" (UID: \"b33f5f29-4f1a-4e70-a821-df5018260ad7\") "
Nov 02 19:52:40.949236 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:40.946827    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b33f5f29-4f1a-4e70-a821-df5018260ad7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b33f5f29-4f1a-4e70-a821-df5018260ad7-workload-certs\") pod \"b33f5f29-4f1a-4e70-a821-df5018260ad7\" (UID: \"b33f5f29-4f1a-4e70-a821-df5018260ad7\") "
Nov 02 19:52:40.949236 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:40.946876    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b33f5f29-4f1a-4e70-a821-df5018260ad7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b33f5f29-4f1a-4e70-a821-df5018260ad7-istio-envoy\") pod \"b33f5f29-4f1a-4e70-a821-df5018260ad7\" (UID: \"b33f5f29-4f1a-4e70-a821-df5018260ad7\") "
Nov 02 19:52:40.949236 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:40.947068    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b33f5f29-4f1a-4e70-a821-df5018260ad7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b33f5f29-4f1a-4e70-a821-df5018260ad7-istio-podinfo\") pod \"b33f5f29-4f1a-4e70-a821-df5018260ad7\" (UID: \"b33f5f29-4f1a-4e70-a821-df5018260ad7\") "
Nov 02 19:52:40.949236 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:40.947153    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b33f5f29-4f1a-4e70-a821-df5018260ad7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b33f5f29-4f1a-4e70-a821-df5018260ad7-workload-socket\") pod \"b33f5f29-4f1a-4e70-a821-df5018260ad7\" (UID: \"b33f5f29-4f1a-4e70-a821-df5018260ad7\") "
Nov 02 19:52:40.949236 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:40.947193    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b33f5f29-4f1a-4e70-a821-df5018260ad7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b33f5f29-4f1a-4e70-a821-df5018260ad7-credential-socket\") pod \"b33f5f29-4f1a-4e70-a821-df5018260ad7\" (UID: \"b33f5f29-4f1a-4e70-a821-df5018260ad7\") "
Nov 02 19:52:40.949236 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:40.947228    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b33f5f29-4f1a-4e70-a821-df5018260ad7-kube-api-access-5x72b\" (UniqueName: \"kubernetes.io/projected/b33f5f29-4f1a-4e70-a821-df5018260ad7-kube-api-access-5x72b\") pod \"b33f5f29-4f1a-4e70-a821-df5018260ad7\" (UID: \"b33f5f29-4f1a-4e70-a821-df5018260ad7\") "
Nov 02 19:52:40.949236 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:40.947274    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b33f5f29-4f1a-4e70-a821-df5018260ad7-istio-token\" (UniqueName: \"kubernetes.io/projected/b33f5f29-4f1a-4e70-a821-df5018260ad7-istio-token\") pod \"b33f5f29-4f1a-4e70-a821-df5018260ad7\" (UID: \"b33f5f29-4f1a-4e70-a821-df5018260ad7\") "
Nov 02 19:52:40.949236 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:40.947748    4463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c725b384-6c96-444e-a657-d4ebeb0e5685-workload-certs\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:52:40.949236 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:40.947760    4463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c725b384-6c96-444e-a657-d4ebeb0e5685-istio-envoy\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:52:40.949236 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:40.947770    4463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c725b384-6c96-444e-a657-d4ebeb0e5685-istio-token\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:52:40.949236 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:40.947780    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfp7t\" (UniqueName: \"kubernetes.io/projected/c725b384-6c96-444e-a657-d4ebeb0e5685-kube-api-access-lfp7t\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:52:40.949236 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:40.947788    4463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c725b384-6c96-444e-a657-d4ebeb0e5685-credential-socket\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:52:40.949236 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:40.947797    4463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c725b384-6c96-444e-a657-d4ebeb0e5685-istiod-ca-cert\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:52:41.020417 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:41.020378    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b33f5f29-4f1a-4e70-a821-df5018260ad7-istio-data" pod "b33f5f29-4f1a-4e70-a821-df5018260ad7" (UID: "b33f5f29-4f1a-4e70-a821-df5018260ad7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:52:41.027347 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:41.027310    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b33f5f29-4f1a-4e70-a821-df5018260ad7-istiod-ca-cert" pod "b33f5f29-4f1a-4e70-a821-df5018260ad7" (UID: "b33f5f29-4f1a-4e70-a821-df5018260ad7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 19:52:41.028472 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:41.028422    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b33f5f29-4f1a-4e70-a821-df5018260ad7-workload-certs" pod "b33f5f29-4f1a-4e70-a821-df5018260ad7" (UID: "b33f5f29-4f1a-4e70-a821-df5018260ad7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:52:41.029110 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:41.029084    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b33f5f29-4f1a-4e70-a821-df5018260ad7-credential-socket" pod "b33f5f29-4f1a-4e70-a821-df5018260ad7" (UID: "b33f5f29-4f1a-4e70-a821-df5018260ad7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:52:41.029281 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:41.029262    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b33f5f29-4f1a-4e70-a821-df5018260ad7-workload-socket" pod "b33f5f29-4f1a-4e70-a821-df5018260ad7" (UID: "b33f5f29-4f1a-4e70-a821-df5018260ad7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:52:41.039420 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:41.039387    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b33f5f29-4f1a-4e70-a821-df5018260ad7-istio-token" pod "b33f5f29-4f1a-4e70-a821-df5018260ad7" (UID: "b33f5f29-4f1a-4e70-a821-df5018260ad7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:52:41.039567 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:41.039549    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b33f5f29-4f1a-4e70-a821-df5018260ad7-istio-envoy" pod "b33f5f29-4f1a-4e70-a821-df5018260ad7" (UID: "b33f5f29-4f1a-4e70-a821-df5018260ad7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:52:41.039871 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:41.039850    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b33f5f29-4f1a-4e70-a821-df5018260ad7-istio-podinfo" pod "b33f5f29-4f1a-4e70-a821-df5018260ad7" (UID: "b33f5f29-4f1a-4e70-a821-df5018260ad7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 19:52:41.041888 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:41.041867    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b33f5f29-4f1a-4e70-a821-df5018260ad7-kube-api-access-5x72b" pod "b33f5f29-4f1a-4e70-a821-df5018260ad7" (UID: "b33f5f29-4f1a-4e70-a821-df5018260ad7"). InnerVolumeSpecName "kube-api-access-5x72b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:52:41.048877 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:41.048849    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb05ff45-67c9-44fb-995d-5f9e0b4059e9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb05ff45-67c9-44fb-995d-5f9e0b4059e9-workload-socket\") pod \"bb05ff45-67c9-44fb-995d-5f9e0b4059e9\" (UID: \"bb05ff45-67c9-44fb-995d-5f9e0b4059e9\") "
Nov 02 19:52:41.057204 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:41.057178    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb05ff45-67c9-44fb-995d-5f9e0b4059e9-istio-token\" (UniqueName: \"kubernetes.io/projected/bb05ff45-67c9-44fb-995d-5f9e0b4059e9-istio-token\") pod \"bb05ff45-67c9-44fb-995d-5f9e0b4059e9\" (UID: \"bb05ff45-67c9-44fb-995d-5f9e0b4059e9\") "
Nov 02 19:52:41.114279 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:41.114241    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bb05ff45-67c9-44fb-995d-5f9e0b4059e9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb05ff45-67c9-44fb-995d-5f9e0b4059e9-istio-podinfo\") pod \"bb05ff45-67c9-44fb-995d-5f9e0b4059e9\" (UID: \"bb05ff45-67c9-44fb-995d-5f9e0b4059e9\") "
Nov 02 19:52:41.115522 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:41.115435    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb05ff45-67c9-44fb-995d-5f9e0b4059e9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb05ff45-67c9-44fb-995d-5f9e0b4059e9-istio-data\") pod \"bb05ff45-67c9-44fb-995d-5f9e0b4059e9\" (UID: \"bb05ff45-67c9-44fb-995d-5f9e0b4059e9\") "
Nov 02 19:52:41.116526 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:41.057098    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb05ff45-67c9-44fb-995d-5f9e0b4059e9-workload-socket" pod "bb05ff45-67c9-44fb-995d-5f9e0b4059e9" (UID: "bb05ff45-67c9-44fb-995d-5f9e0b4059e9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:52:41.116770 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:41.115876    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb05ff45-67c9-44fb-995d-5f9e0b4059e9-istio-data" pod "bb05ff45-67c9-44fb-995d-5f9e0b4059e9" (UID: "bb05ff45-67c9-44fb-995d-5f9e0b4059e9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:52:41.116889 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:41.116683    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb05ff45-67c9-44fb-995d-5f9e0b4059e9-kube-api-access-k5vnk\" (UniqueName: \"kubernetes.io/projected/bb05ff45-67c9-44fb-995d-5f9e0b4059e9-kube-api-access-k5vnk\") pod \"bb05ff45-67c9-44fb-995d-5f9e0b4059e9\" (UID: \"bb05ff45-67c9-44fb-995d-5f9e0b4059e9\") "
Nov 02 19:52:41.117090 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:41.117068    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb05ff45-67c9-44fb-995d-5f9e0b4059e9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb05ff45-67c9-44fb-995d-5f9e0b4059e9-istiod-ca-cert\") pod \"bb05ff45-67c9-44fb-995d-5f9e0b4059e9\" (UID: \"bb05ff45-67c9-44fb-995d-5f9e0b4059e9\") "
Nov 02 19:52:41.117254 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:41.117237    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb05ff45-67c9-44fb-995d-5f9e0b4059e9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb05ff45-67c9-44fb-995d-5f9e0b4059e9-credential-socket\") pod \"bb05ff45-67c9-44fb-995d-5f9e0b4059e9\" (UID: \"bb05ff45-67c9-44fb-995d-5f9e0b4059e9\") "
Nov 02 19:52:41.118143 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:41.117958    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb05ff45-67c9-44fb-995d-5f9e0b4059e9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb05ff45-67c9-44fb-995d-5f9e0b4059e9-workload-certs\") pod \"bb05ff45-67c9-44fb-995d-5f9e0b4059e9\" (UID: \"bb05ff45-67c9-44fb-995d-5f9e0b4059e9\") "
Nov 02 19:52:41.118327 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:41.118307    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb05ff45-67c9-44fb-995d-5f9e0b4059e9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb05ff45-67c9-44fb-995d-5f9e0b4059e9-istio-envoy\") pod \"bb05ff45-67c9-44fb-995d-5f9e0b4059e9\" (UID: \"bb05ff45-67c9-44fb-995d-5f9e0b4059e9\") "
Nov 02 19:52:41.119235 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:41.119172    4463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b33f5f29-4f1a-4e70-a821-df5018260ad7-credential-socket\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:52:41.119379 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:41.119363    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5x72b\" (UniqueName: \"kubernetes.io/projected/b33f5f29-4f1a-4e70-a821-df5018260ad7-kube-api-access-5x72b\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:52:41.119469 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:41.119457    4463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b33f5f29-4f1a-4e70-a821-df5018260ad7-istio-token\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:52:41.119566 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:41.119553    4463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb05ff45-67c9-44fb-995d-5f9e0b4059e9-workload-socket\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:52:41.119655 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:41.119643    4463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b33f5f29-4f1a-4e70-a821-df5018260ad7-istiod-ca-cert\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:52:41.119745 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:41.119733    4463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b33f5f29-4f1a-4e70-a821-df5018260ad7-istio-data\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:52:41.119835 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:41.119822    4463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb05ff45-67c9-44fb-995d-5f9e0b4059e9-istio-data\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:52:41.119921 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:41.119909    4463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b33f5f29-4f1a-4e70-a821-df5018260ad7-workload-certs\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:52:41.120029 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:41.120001    4463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b33f5f29-4f1a-4e70-a821-df5018260ad7-istio-envoy\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:52:41.120141 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:41.120127    4463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b33f5f29-4f1a-4e70-a821-df5018260ad7-istio-podinfo\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:52:41.120240 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:41.120228    4463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b33f5f29-4f1a-4e70-a821-df5018260ad7-workload-socket\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:52:41.120342 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:41.117590    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb05ff45-67c9-44fb-995d-5f9e0b4059e9-credential-socket" pod "bb05ff45-67c9-44fb-995d-5f9e0b4059e9" (UID: "bb05ff45-67c9-44fb-995d-5f9e0b4059e9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:52:41.120491 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:41.117891    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb05ff45-67c9-44fb-995d-5f9e0b4059e9-istiod-ca-cert" pod "bb05ff45-67c9-44fb-995d-5f9e0b4059e9" (UID: "bb05ff45-67c9-44fb-995d-5f9e0b4059e9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 19:52:41.120598 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:41.119693    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bb05ff45-67c9-44fb-995d-5f9e0b4059e9-istio-podinfo" pod "bb05ff45-67c9-44fb-995d-5f9e0b4059e9" (UID: "bb05ff45-67c9-44fb-995d-5f9e0b4059e9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 19:52:41.120685 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:41.119979    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb05ff45-67c9-44fb-995d-5f9e0b4059e9-workload-certs" pod "bb05ff45-67c9-44fb-995d-5f9e0b4059e9" (UID: "bb05ff45-67c9-44fb-995d-5f9e0b4059e9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:52:41.120766 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:41.120096    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb05ff45-67c9-44fb-995d-5f9e0b4059e9-istio-token" pod "bb05ff45-67c9-44fb-995d-5f9e0b4059e9" (UID: "bb05ff45-67c9-44fb-995d-5f9e0b4059e9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:52:41.121577 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:41.121545    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb05ff45-67c9-44fb-995d-5f9e0b4059e9-kube-api-access-k5vnk" pod "bb05ff45-67c9-44fb-995d-5f9e0b4059e9" (UID: "bb05ff45-67c9-44fb-995d-5f9e0b4059e9"). InnerVolumeSpecName "kube-api-access-k5vnk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:52:41.121949 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:41.121921    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb05ff45-67c9-44fb-995d-5f9e0b4059e9-istio-envoy" pod "bb05ff45-67c9-44fb-995d-5f9e0b4059e9" (UID: "bb05ff45-67c9-44fb-995d-5f9e0b4059e9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:52:41.223292 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:41.223194    4463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb05ff45-67c9-44fb-995d-5f9e0b4059e9-workload-certs\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:52:41.223292 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:41.223236    4463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb05ff45-67c9-44fb-995d-5f9e0b4059e9-istio-envoy\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:52:41.223292 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:41.223252    4463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb05ff45-67c9-44fb-995d-5f9e0b4059e9-istio-token\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:52:41.223292 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:41.223267    4463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb05ff45-67c9-44fb-995d-5f9e0b4059e9-istio-podinfo\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:52:41.223292 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:41.223283    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5vnk\" (UniqueName: \"kubernetes.io/projected/bb05ff45-67c9-44fb-995d-5f9e0b4059e9-kube-api-access-k5vnk\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:52:41.223686 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:41.223298    4463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb05ff45-67c9-44fb-995d-5f9e0b4059e9-istiod-ca-cert\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:52:41.223686 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:41.223312    4463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb05ff45-67c9-44fb-995d-5f9e0b4059e9-credential-socket\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:52:41.298040 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:41.268615    4463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-pgq8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 02 19:52:41.298040 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:41.268706    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-pgq8d" podUID="6ff8e962-f7e6-4d82-a818-261d21bdba26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 02 19:52:41.323434 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:41.323398    4463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-5bbdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Nov 02 19:52:41.323609 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:41.323454    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5bbdh" podUID="6b600454-98b7-49b2-a129-6b65406c4d9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Nov 02 19:52:41.325291 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:41.325265    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kglqn" event={"ID":"8c25aace-5449-4ed4-a07a-dd08f8eb4a4e","Type":"ContainerDied","Data":"ebd9fe405d5ef0e7ea06d91b0cf55bceb342f53ecd825f02d0a4aa22c789bfc9"}
Nov 02 19:52:41.325364 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:41.325285    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kglqn"
Nov 02 19:52:41.325398 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:41.325364    4463 scope.go:122] "RemoveContainer" containerID="77e675d907ae53155138da5ad3f3ffdd046434507979aff1ec567d11b4da7706"
Nov 02 19:52:41.328440 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:41.328412    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-lkfn9" event={"ID":"b33f5f29-4f1a-4e70-a821-df5018260ad7","Type":"ContainerDied","Data":"f15aab927713ea3c47fd5b59d6a96e9930426ccfc9af3923a46c54a4ff628c51"}
Nov 02 19:52:41.328559 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:41.328497    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-lkfn9"
Nov 02 19:52:41.330403 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:41.330378    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kpr8z"
Nov 02 19:52:41.330512 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:41.330410    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kpr8z" event={"ID":"c725b384-6c96-444e-a657-d4ebeb0e5685","Type":"ContainerDied","Data":"e506e5cf94db1dcddb9ee1cdd5fa3d60c0823c1e2b3392fbc2ee221e317b943d"}
Nov 02 19:52:41.340513 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:41.340472    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-k7k7k" event={"ID":"bb05ff45-67c9-44fb-995d-5f9e0b4059e9","Type":"ContainerDied","Data":"1325e35f2e4c41fa53b13b2736f9f549ea6534a4775f20426df4c33de68139f1"}
Nov 02 19:52:41.340689 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:41.340598    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-k7k7k"
Nov 02 19:52:41.346089 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:41.344502    4463 scope.go:122] "RemoveContainer" containerID="7133490961b3a61bd67e108fd1984df415ac2839fef434550c4cbc2f85b9dc86"
Nov 02 19:52:41.434433 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:41.434405    4463 scope.go:122] "RemoveContainer" containerID="253434d791e65f1934bb94b67c4e2f77546f9f6376672c2b019d7c97861ba244"
Nov 02 19:52:41.504039 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:41.503238    4463 scope.go:122] "RemoveContainer" containerID="9e77367018fb447387d1a4222f5204d4bc156185e6f0ffec132c0a35da61c123"
Nov 02 19:52:41.531046 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:41.530057    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-lkfn9"]
Nov 02 19:52:41.534048 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:41.534006    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-lkfn9"]
Nov 02 19:52:41.616340 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:41.616303    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b33f5f29-4f1a-4e70-a821-df5018260ad7" path="/var/lib/kubelet/pods/b33f5f29-4f1a-4e70-a821-df5018260ad7/volumes"
Nov 02 19:52:41.632903 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:41.632870    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kglqn"]
Nov 02 19:52:41.638929 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:41.638899    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kglqn"]
Nov 02 19:52:41.812447 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:41.812335    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kpr8z"]
Nov 02 19:52:41.817320 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:41.817288    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-kpr8z"]
Nov 02 19:52:41.933261 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:41.918502    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-k7k7k"]
Nov 02 19:52:41.933261 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:41.927504    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-k7k7k"]
Nov 02 19:52:42.268657 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:42.268617    4463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-pgq8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 02 19:52:42.268837 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:42.268695    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-pgq8d" podUID="6ff8e962-f7e6-4d82-a818-261d21bdba26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 02 19:52:42.323238 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:42.323201    4463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-5bbdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Nov 02 19:52:42.323402 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:42.323281    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5bbdh" podUID="6b600454-98b7-49b2-a129-6b65406c4d9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Nov 02 19:52:43.268416 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:43.268377    4463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-pgq8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 02 19:52:43.268911 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:43.268461    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-pgq8d" podUID="6ff8e962-f7e6-4d82-a818-261d21bdba26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 02 19:52:43.323272 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:43.323226    4463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-5bbdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Nov 02 19:52:43.323474 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:43.323292    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5bbdh" podUID="6b600454-98b7-49b2-a129-6b65406c4d9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Nov 02 19:52:43.592166 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:43.592070    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c25aace-5449-4ed4-a07a-dd08f8eb4a4e" path="/var/lib/kubelet/pods/8c25aace-5449-4ed4-a07a-dd08f8eb4a4e/volumes"
Nov 02 19:52:43.593040 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:43.592999    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb05ff45-67c9-44fb-995d-5f9e0b4059e9" path="/var/lib/kubelet/pods/bb05ff45-67c9-44fb-995d-5f9e0b4059e9/volumes"
Nov 02 19:52:43.595954 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:43.593988    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c725b384-6c96-444e-a657-d4ebeb0e5685" path="/var/lib/kubelet/pods/c725b384-6c96-444e-a657-d4ebeb0e5685/volumes"
Nov 02 19:52:44.416853 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:44.416812    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-pgq8d"
Nov 02 19:52:44.417314 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:44.417238    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-pgq8d"
Nov 02 19:52:44.418303 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:44.418277    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-pgq8d"
Nov 02 19:52:44.480860 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:44.480831    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5bbdh"
Nov 02 19:52:44.481284 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:44.481257    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5bbdh"
Nov 02 19:52:44.482243 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:44.482221    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5bbdh"
Nov 02 19:52:47.892894 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:47.892857    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-46f86d84-56b6698b4b-shr75"]
Nov 02 19:52:47.893527 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:47.893511    4463 cpu_manager.go:414] "containerMap: removing container" podUID="c725b384-6c96-444e-a657-d4ebeb0e5685" containerName="istio-proxy"
Nov 02 19:52:47.893566 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:47.893528    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c725b384-6c96-444e-a657-d4ebeb0e5685" containerName="istio-proxy"
Nov 02 19:52:47.893566 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:47.893543    4463 cpu_manager.go:414] "containerMap: removing container" podUID="8c25aace-5449-4ed4-a07a-dd08f8eb4a4e" containerName="istio-proxy"
Nov 02 19:52:47.893566 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:47.893548    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c25aace-5449-4ed4-a07a-dd08f8eb4a4e" containerName="istio-proxy"
Nov 02 19:52:47.893566 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:47.893558    4463 cpu_manager.go:414] "containerMap: removing container" podUID="bb05ff45-67c9-44fb-995d-5f9e0b4059e9" containerName="istio-proxy"
Nov 02 19:52:47.893566 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:47.893563    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb05ff45-67c9-44fb-995d-5f9e0b4059e9" containerName="istio-proxy"
Nov 02 19:52:47.893738 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:47.893582    4463 cpu_manager.go:414] "containerMap: removing container" podUID="b33f5f29-4f1a-4e70-a821-df5018260ad7" containerName="istio-proxy"
Nov 02 19:52:47.893738 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:47.893587    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b33f5f29-4f1a-4e70-a821-df5018260ad7" containerName="istio-proxy"
Nov 02 19:52:47.893738 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:47.893685    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c25aace-5449-4ed4-a07a-dd08f8eb4a4e" containerName="istio-proxy"
Nov 02 19:52:47.893738 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:47.893694    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb05ff45-67c9-44fb-995d-5f9e0b4059e9" containerName="istio-proxy"
Nov 02 19:52:47.893738 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:47.893704    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="b33f5f29-4f1a-4e70-a821-df5018260ad7" containerName="istio-proxy"
Nov 02 19:52:47.893738 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:47.893715    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="c725b384-6c96-444e-a657-d4ebeb0e5685" containerName="istio-proxy"
Nov 02 19:52:47.897378 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:47.897358    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-46f86d84-56b6698b4b-shr75"
Nov 02 19:52:47.992170 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:47.992132    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-46f86d84-serving-cert\""
Nov 02 19:52:47.995904 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:47.995858    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-46f86d84-56b6698b4b-shr75"]
Nov 02 19:52:48.042420 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:48.042371    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4n5dd\" (UniqueName: \"kubernetes.io/projected/cfc50f82-a168-48e3-b508-f67232982f1f-kube-api-access-4n5dd\") pod \"maas-api-e2e-derive-46f86d84-56b6698b4b-shr75\" (UID: \"cfc50f82-a168-48e3-b508-f67232982f1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-46f86d84-56b6698b4b-shr75"
Nov 02 19:52:48.042612 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:48.042425    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfc50f82-a168-48e3-b508-f67232982f1f-openshift-service-ca\") pod \"maas-api-e2e-derive-46f86d84-56b6698b4b-shr75\" (UID: \"cfc50f82-a168-48e3-b508-f67232982f1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-46f86d84-56b6698b4b-shr75"
Nov 02 19:52:48.042612 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:48.042550    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfc50f82-a168-48e3-b508-f67232982f1f-maas-api-tls\") pod \"maas-api-e2e-derive-46f86d84-56b6698b4b-shr75\" (UID: \"cfc50f82-a168-48e3-b508-f67232982f1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-46f86d84-56b6698b4b-shr75"
Nov 02 19:52:48.042739 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:48.042624    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfc50f82-a168-48e3-b508-f67232982f1f-metrics-certs\") pod \"maas-api-e2e-derive-46f86d84-56b6698b4b-shr75\" (UID: \"cfc50f82-a168-48e3-b508-f67232982f1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-46f86d84-56b6698b4b-shr75"
Nov 02 19:52:48.144057 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:48.143941    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4n5dd\" (UniqueName: \"kubernetes.io/projected/cfc50f82-a168-48e3-b508-f67232982f1f-kube-api-access-4n5dd\") pod \"maas-api-e2e-derive-46f86d84-56b6698b4b-shr75\" (UID: \"cfc50f82-a168-48e3-b508-f67232982f1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-46f86d84-56b6698b4b-shr75"
Nov 02 19:52:48.144057 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:48.144000    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfc50f82-a168-48e3-b508-f67232982f1f-openshift-service-ca\") pod \"maas-api-e2e-derive-46f86d84-56b6698b4b-shr75\" (UID: \"cfc50f82-a168-48e3-b508-f67232982f1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-46f86d84-56b6698b4b-shr75"
Nov 02 19:52:48.144264 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:48.144152    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfc50f82-a168-48e3-b508-f67232982f1f-maas-api-tls\") pod \"maas-api-e2e-derive-46f86d84-56b6698b4b-shr75\" (UID: \"cfc50f82-a168-48e3-b508-f67232982f1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-46f86d84-56b6698b4b-shr75"
Nov 02 19:52:48.144264 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:48.144230    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfc50f82-a168-48e3-b508-f67232982f1f-metrics-certs\") pod \"maas-api-e2e-derive-46f86d84-56b6698b4b-shr75\" (UID: \"cfc50f82-a168-48e3-b508-f67232982f1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-46f86d84-56b6698b4b-shr75"
Nov 02 19:52:48.144386 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:52:48.144292    4463 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-46f86d84-serving-cert: secret "maas-api-e2e-derive-46f86d84-serving-cert" not found
Nov 02 19:52:48.144386 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:52:48.144368    4463 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfc50f82-a168-48e3-b508-f67232982f1f-maas-api-tls podName:cfc50f82-a168-48e3-b508-f67232982f1f nodeName:}" failed. No retries permitted until 2026-11-02 19:52:48.644346332 +0000 UTC m=+1475.251380094 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cfc50f82-a168-48e3-b508-f67232982f1f-maas-api-tls") pod "maas-api-e2e-derive-46f86d84-56b6698b4b-shr75" (UID: "cfc50f82-a168-48e3-b508-f67232982f1f") : secret "maas-api-e2e-derive-46f86d84-serving-cert" not found
Nov 02 19:52:48.145689 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:48.144915    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfc50f82-a168-48e3-b508-f67232982f1f-openshift-service-ca\") pod \"maas-api-e2e-derive-46f86d84-56b6698b4b-shr75\" (UID: \"cfc50f82-a168-48e3-b508-f67232982f1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-46f86d84-56b6698b4b-shr75"
Nov 02 19:52:48.147140 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:48.147107    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfc50f82-a168-48e3-b508-f67232982f1f-metrics-certs\") pod \"maas-api-e2e-derive-46f86d84-56b6698b4b-shr75\" (UID: \"cfc50f82-a168-48e3-b508-f67232982f1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-46f86d84-56b6698b4b-shr75"
Nov 02 19:52:48.201046 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:48.200999    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4n5dd\" (UniqueName: \"kubernetes.io/projected/cfc50f82-a168-48e3-b508-f67232982f1f-kube-api-access-4n5dd\") pod \"maas-api-e2e-derive-46f86d84-56b6698b4b-shr75\" (UID: \"cfc50f82-a168-48e3-b508-f67232982f1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-46f86d84-56b6698b4b-shr75"
Nov 02 19:52:48.651948 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:48.651913    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfc50f82-a168-48e3-b508-f67232982f1f-maas-api-tls\") pod \"maas-api-e2e-derive-46f86d84-56b6698b4b-shr75\" (UID: \"cfc50f82-a168-48e3-b508-f67232982f1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-46f86d84-56b6698b4b-shr75"
Nov 02 19:52:48.654366 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:48.654343    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfc50f82-a168-48e3-b508-f67232982f1f-maas-api-tls\") pod \"maas-api-e2e-derive-46f86d84-56b6698b4b-shr75\" (UID: \"cfc50f82-a168-48e3-b508-f67232982f1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-46f86d84-56b6698b4b-shr75"
Nov 02 19:52:48.810460 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:48.810428    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-46f86d84-56b6698b4b-shr75"
Nov 02 19:52:49.182746 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:49.182715    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-46f86d84-56b6698b4b-shr75"]
Nov 02 19:52:49.183584 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:52:49.183554    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfc50f82_a168_48e3_b508_f67232982f1f.slice/crio-3d74f4a93f682a14212b525a561dac62ae5e0dfac0b1728f0a88a928671aab4e WatchSource:0}: Error finding container 3d74f4a93f682a14212b525a561dac62ae5e0dfac0b1728f0a88a928671aab4e: Status 404 returned error can't find the container with id 3d74f4a93f682a14212b525a561dac62ae5e0dfac0b1728f0a88a928671aab4e
Nov 02 19:52:49.185241 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:49.185221    4463 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 19:52:49.406849 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:49.406811    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-46f86d84-56b6698b4b-shr75" event={"ID":"cfc50f82-a168-48e3-b508-f67232982f1f","Type":"ContainerStarted","Data":"3d74f4a93f682a14212b525a561dac62ae5e0dfac0b1728f0a88a928671aab4e"}
Nov 02 19:52:50.069036 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:50.068996    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jzjj4"]
Nov 02 19:52:50.072867 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:50.072844    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jzjj4"
Nov 02 19:52:50.185063 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:50.178311    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jzjj4"]
Nov 02 19:52:50.185063 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:50.179172    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73818a40-7ba5-410c-aa9f-f6833bd95623-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jzjj4\" (UID: \"73818a40-7ba5-410c-aa9f-f6833bd95623\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jzjj4"
Nov 02 19:52:50.185063 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:50.179243    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73818a40-7ba5-410c-aa9f-f6833bd95623-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jzjj4\" (UID: \"73818a40-7ba5-410c-aa9f-f6833bd95623\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jzjj4"
Nov 02 19:52:50.185063 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:50.179302    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w22c9\" (UniqueName: \"kubernetes.io/projected/73818a40-7ba5-410c-aa9f-f6833bd95623-kube-api-access-w22c9\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jzjj4\" (UID: \"73818a40-7ba5-410c-aa9f-f6833bd95623\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jzjj4"
Nov 02 19:52:50.185063 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:50.179371    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73818a40-7ba5-410c-aa9f-f6833bd95623-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jzjj4\" (UID: \"73818a40-7ba5-410c-aa9f-f6833bd95623\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jzjj4"
Nov 02 19:52:50.185063 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:50.179413    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73818a40-7ba5-410c-aa9f-f6833bd95623-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jzjj4\" (UID: \"73818a40-7ba5-410c-aa9f-f6833bd95623\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jzjj4"
Nov 02 19:52:50.185063 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:50.179479    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73818a40-7ba5-410c-aa9f-f6833bd95623-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jzjj4\" (UID: \"73818a40-7ba5-410c-aa9f-f6833bd95623\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jzjj4"
Nov 02 19:52:50.185063 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:50.179551    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73818a40-7ba5-410c-aa9f-f6833bd95623-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jzjj4\" (UID: \"73818a40-7ba5-410c-aa9f-f6833bd95623\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jzjj4"
Nov 02 19:52:50.185063 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:50.179621    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73818a40-7ba5-410c-aa9f-f6833bd95623-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jzjj4\" (UID: \"73818a40-7ba5-410c-aa9f-f6833bd95623\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jzjj4"
Nov 02 19:52:50.185063 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:50.179672    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73818a40-7ba5-410c-aa9f-f6833bd95623-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jzjj4\" (UID: \"73818a40-7ba5-410c-aa9f-f6833bd95623\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jzjj4"
Nov 02 19:52:50.280737 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:50.280701    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73818a40-7ba5-410c-aa9f-f6833bd95623-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jzjj4\" (UID: \"73818a40-7ba5-410c-aa9f-f6833bd95623\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jzjj4"
Nov 02 19:52:50.280943 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:50.280801    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73818a40-7ba5-410c-aa9f-f6833bd95623-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jzjj4\" (UID: \"73818a40-7ba5-410c-aa9f-f6833bd95623\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jzjj4"
Nov 02 19:52:50.280943 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:50.280834    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73818a40-7ba5-410c-aa9f-f6833bd95623-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jzjj4\" (UID: \"73818a40-7ba5-410c-aa9f-f6833bd95623\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jzjj4"
Nov 02 19:52:50.280943 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:50.280869    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w22c9\" (UniqueName: \"kubernetes.io/projected/73818a40-7ba5-410c-aa9f-f6833bd95623-kube-api-access-w22c9\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jzjj4\" (UID: \"73818a40-7ba5-410c-aa9f-f6833bd95623\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jzjj4"
Nov 02 19:52:50.280943 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:50.280916    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73818a40-7ba5-410c-aa9f-f6833bd95623-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jzjj4\" (UID: \"73818a40-7ba5-410c-aa9f-f6833bd95623\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jzjj4"
Nov 02 19:52:50.280943 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:50.280939    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73818a40-7ba5-410c-aa9f-f6833bd95623-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jzjj4\" (UID: \"73818a40-7ba5-410c-aa9f-f6833bd95623\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jzjj4"
Nov 02 19:52:50.281288 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:50.280979    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73818a40-7ba5-410c-aa9f-f6833bd95623-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jzjj4\" (UID: \"73818a40-7ba5-410c-aa9f-f6833bd95623\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jzjj4"
Nov 02 19:52:50.281288 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:50.281056    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73818a40-7ba5-410c-aa9f-f6833bd95623-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jzjj4\" (UID: \"73818a40-7ba5-410c-aa9f-f6833bd95623\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jzjj4"
Nov 02 19:52:50.281288 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:50.281125    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73818a40-7ba5-410c-aa9f-f6833bd95623-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jzjj4\" (UID: \"73818a40-7ba5-410c-aa9f-f6833bd95623\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jzjj4"
Nov 02 19:52:50.281447 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:50.281384    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73818a40-7ba5-410c-aa9f-f6833bd95623-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jzjj4\" (UID: \"73818a40-7ba5-410c-aa9f-f6833bd95623\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jzjj4"
Nov 02 19:52:50.281447 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:50.281414    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73818a40-7ba5-410c-aa9f-f6833bd95623-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jzjj4\" (UID: \"73818a40-7ba5-410c-aa9f-f6833bd95623\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jzjj4"
Nov 02 19:52:50.281552 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:50.281448    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73818a40-7ba5-410c-aa9f-f6833bd95623-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jzjj4\" (UID: \"73818a40-7ba5-410c-aa9f-f6833bd95623\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jzjj4"
Nov 02 19:52:50.281747 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:50.281725    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73818a40-7ba5-410c-aa9f-f6833bd95623-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jzjj4\" (UID: \"73818a40-7ba5-410c-aa9f-f6833bd95623\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jzjj4"
Nov 02 19:52:50.281797 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:50.281730    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73818a40-7ba5-410c-aa9f-f6833bd95623-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jzjj4\" (UID: \"73818a40-7ba5-410c-aa9f-f6833bd95623\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jzjj4"
Nov 02 19:52:50.283397 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:50.283377    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73818a40-7ba5-410c-aa9f-f6833bd95623-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jzjj4\" (UID: \"73818a40-7ba5-410c-aa9f-f6833bd95623\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jzjj4"
Nov 02 19:52:50.283563 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:50.283544    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73818a40-7ba5-410c-aa9f-f6833bd95623-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jzjj4\" (UID: \"73818a40-7ba5-410c-aa9f-f6833bd95623\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jzjj4"
Nov 02 19:52:50.334172 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:50.334094    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73818a40-7ba5-410c-aa9f-f6833bd95623-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jzjj4\" (UID: \"73818a40-7ba5-410c-aa9f-f6833bd95623\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jzjj4"
Nov 02 19:52:50.334294 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:50.334209    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w22c9\" (UniqueName: \"kubernetes.io/projected/73818a40-7ba5-410c-aa9f-f6833bd95623-kube-api-access-w22c9\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jzjj4\" (UID: \"73818a40-7ba5-410c-aa9f-f6833bd95623\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jzjj4"
Nov 02 19:52:50.395424 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:50.395388    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jzjj4"
Nov 02 19:52:50.416699 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:50.416660    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-46f86d84-56b6698b4b-shr75" event={"ID":"cfc50f82-a168-48e3-b508-f67232982f1f","Type":"ContainerStarted","Data":"c13ab46c1fb0f99ccfcdd25b19e232c86ca0769dc0c2b1936f38160247742b13"}
Nov 02 19:52:50.416838 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:50.416726    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-46f86d84-56b6698b4b-shr75"
Nov 02 19:52:50.527969 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:50.527899    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-46f86d84-56b6698b4b-shr75" podStartSLOduration=2.676777178 podStartE2EDuration="3.527880166s" totalImagesPullingTime="851.102988ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:52:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:52:50.523031344 +0000 UTC m=+1477.130065116" watchObservedRunningTime="2026-11-02 19:52:50.527880166 +0000 UTC m=+1477.134913936"
Nov 02 19:52:50.817195 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:50.817160    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jzjj4"]
Nov 02 19:52:50.819177 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:52:50.819137    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73818a40_7ba5_410c_aa9f_f6833bd95623.slice/crio-730d71192490c729996907aa748d0f4727256fd69814847c6f0afb8ee73ecb3e WatchSource:0}: Error finding container 730d71192490c729996907aa748d0f4727256fd69814847c6f0afb8ee73ecb3e: Status 404 returned error can't find the container with id 730d71192490c729996907aa748d0f4727256fd69814847c6f0afb8ee73ecb3e
Nov 02 19:52:50.821719 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:50.821673    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:52:50.821790 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:50.821765    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:52:50.821832 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:50.821808    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:52:51.425494 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:51.425461    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jzjj4" event={"ID":"73818a40-7ba5-410c-aa9f-f6833bd95623","Type":"ContainerStarted","Data":"2cd00b7707e73cf906c4db1983024a52404fd53081a00c64c4f7bd1e1872679c"}
Nov 02 19:52:51.425889 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:51.425512    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jzjj4" event={"ID":"73818a40-7ba5-410c-aa9f-f6833bd95623","Type":"ContainerStarted","Data":"730d71192490c729996907aa748d0f4727256fd69814847c6f0afb8ee73ecb3e"}
Nov 02 19:52:51.542337 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:51.542285    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jzjj4" podStartSLOduration=1.542270365 podStartE2EDuration="1.542270365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:52:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:52:51.538350647 +0000 UTC m=+1478.145384444" watchObservedRunningTime="2026-11-02 19:52:51.542270365 +0000 UTC m=+1478.149304134"
Nov 02 19:52:52.396528 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:52.396491    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jzjj4"
Nov 02 19:52:52.398575 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:52.398546    4463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-jzjj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 02 19:52:52.398707 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:52.398602    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jzjj4" podUID="73818a40-7ba5-410c-aa9f-f6833bd95623" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 02 19:52:53.396164 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:53.396128    4463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-jzjj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 02 19:52:53.396573 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:53.396184    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jzjj4" podUID="73818a40-7ba5-410c-aa9f-f6833bd95623" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 02 19:52:54.395964 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:54.395918    4463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-jzjj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 02 19:52:54.396166 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:54.395969    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jzjj4" podUID="73818a40-7ba5-410c-aa9f-f6833bd95623" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 02 19:52:55.405053 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:55.397386    4463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-jzjj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 02 19:52:55.405053 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:55.397446    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jzjj4" podUID="73818a40-7ba5-410c-aa9f-f6833bd95623" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 02 19:52:56.404492 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:56.404400    4463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-jzjj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 02 19:52:56.404757 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:56.404724    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jzjj4" podUID="73818a40-7ba5-410c-aa9f-f6833bd95623" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 02 19:52:56.542038 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:56.540995    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-46f86d84-56b6698b4b-shr75"
Nov 02 19:52:56.747036 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:56.734101    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5c46af/distinct-w3-5c46af-kserve-68b6dff7f5-bp59f" podUID="8dd6362e-7dbd-489b-b6d8-0d9caab0fa69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:52:56.756136 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:56.756093    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d869e/distinct-2-w5-1d869e-kserve-574f55db46-4cv47" podUID="7e236108-328b-4170-be20-13f16b56c4fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:52:56.756682 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:56.756653    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d869e/premium-w5-1d869e-kserve-7745799ff8-pv2th" podUID="1d784b10-99e4-4561-bfb7-6614877d2b6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 02 19:52:56.909205 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:56.909158    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-43ceaa/premium-w0-43ceaa-kserve-67b55659b7-rckfd" podUID="7be32df1-5049-4f79-be23-a827f3f0a85f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 02 19:52:56.926819 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:56.915467    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-26cfbe/distinct-w1-26cfbe-kserve-685985bfb-tkgn2" podUID="7415aa62-467c-43af-94c8-d604f57ec1c3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:52:56.929147 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:56.929111    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8795bf/distinct-w4-8795bf-kserve-64b4796df4-8qgqh" podUID="ee7d28f4-e9c7-43e2-b983-1427589f1785" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:52:57.091985 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:57.091078    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-26cfbe/simulator-w1-26cfbe-kserve-74c5578fbd-t79hq" podUID="50dc76f4-aebd-41f4-9f23-799ddc8c70f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:52:57.111706 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:57.105074    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4848d2-565fb9d78c-pd59p"]
Nov 02 19:52:57.111706 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:57.109280    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4848d2-565fb9d78c-pd59p"
Nov 02 19:52:57.174053 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:57.173985    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-4848d2-serving-cert\""
Nov 02 19:52:57.184241 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:57.184215    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4848d2-565fb9d78c-pd59p"]
Nov 02 19:52:57.192679 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:57.192655    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chq9t\" (UniqueName: \"kubernetes.io/projected/aa56df3b-fd52-46b8-a626-0dd7e827ab8c-kube-api-access-chq9t\") pod \"maas-api-e2e-shared-a-w1-4848d2-565fb9d78c-pd59p\" (UID: \"aa56df3b-fd52-46b8-a626-0dd7e827ab8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4848d2-565fb9d78c-pd59p"
Nov 02 19:52:57.192763 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:57.192691    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa56df3b-fd52-46b8-a626-0dd7e827ab8c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-4848d2-565fb9d78c-pd59p\" (UID: \"aa56df3b-fd52-46b8-a626-0dd7e827ab8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4848d2-565fb9d78c-pd59p"
Nov 02 19:52:57.192835 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:57.192819    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa56df3b-fd52-46b8-a626-0dd7e827ab8c-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-4848d2-565fb9d78c-pd59p\" (UID: \"aa56df3b-fd52-46b8-a626-0dd7e827ab8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4848d2-565fb9d78c-pd59p"
Nov 02 19:52:57.192901 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:57.192887    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa56df3b-fd52-46b8-a626-0dd7e827ab8c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-4848d2-565fb9d78c-pd59p\" (UID: \"aa56df3b-fd52-46b8-a626-0dd7e827ab8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4848d2-565fb9d78c-pd59p"
Nov 02 19:52:57.294784 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:57.294737    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chq9t\" (UniqueName: \"kubernetes.io/projected/aa56df3b-fd52-46b8-a626-0dd7e827ab8c-kube-api-access-chq9t\") pod \"maas-api-e2e-shared-a-w1-4848d2-565fb9d78c-pd59p\" (UID: \"aa56df3b-fd52-46b8-a626-0dd7e827ab8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4848d2-565fb9d78c-pd59p"
Nov 02 19:52:57.294959 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:57.294793    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa56df3b-fd52-46b8-a626-0dd7e827ab8c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-4848d2-565fb9d78c-pd59p\" (UID: \"aa56df3b-fd52-46b8-a626-0dd7e827ab8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4848d2-565fb9d78c-pd59p"
Nov 02 19:52:57.294959 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:57.294896    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa56df3b-fd52-46b8-a626-0dd7e827ab8c-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-4848d2-565fb9d78c-pd59p\" (UID: \"aa56df3b-fd52-46b8-a626-0dd7e827ab8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4848d2-565fb9d78c-pd59p"
Nov 02 19:52:57.295109 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:57.294984    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa56df3b-fd52-46b8-a626-0dd7e827ab8c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-4848d2-565fb9d78c-pd59p\" (UID: \"aa56df3b-fd52-46b8-a626-0dd7e827ab8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4848d2-565fb9d78c-pd59p"
Nov 02 19:52:57.295176 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:52:57.295149    4463 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4848d2-serving-cert: secret "maas-api-e2e-shared-a-w1-4848d2-serving-cert" not found
Nov 02 19:52:57.295258 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:52:57.295246    4463 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa56df3b-fd52-46b8-a626-0dd7e827ab8c-maas-api-tls podName:aa56df3b-fd52-46b8-a626-0dd7e827ab8c nodeName:}" failed. No retries permitted until 2026-11-02 19:52:57.795223317 +0000 UTC m=+1484.402257081 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/aa56df3b-fd52-46b8-a626-0dd7e827ab8c-maas-api-tls") pod "maas-api-e2e-shared-a-w1-4848d2-565fb9d78c-pd59p" (UID: "aa56df3b-fd52-46b8-a626-0dd7e827ab8c") : secret "maas-api-e2e-shared-a-w1-4848d2-serving-cert" not found
Nov 02 19:52:57.295625 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:57.295601    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa56df3b-fd52-46b8-a626-0dd7e827ab8c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-4848d2-565fb9d78c-pd59p\" (UID: \"aa56df3b-fd52-46b8-a626-0dd7e827ab8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4848d2-565fb9d78c-pd59p"
Nov 02 19:52:57.297532 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:57.297505    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa56df3b-fd52-46b8-a626-0dd7e827ab8c-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-4848d2-565fb9d78c-pd59p\" (UID: \"aa56df3b-fd52-46b8-a626-0dd7e827ab8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4848d2-565fb9d78c-pd59p"
Nov 02 19:52:57.367516 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:57.367239    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chq9t\" (UniqueName: \"kubernetes.io/projected/aa56df3b-fd52-46b8-a626-0dd7e827ab8c-kube-api-access-chq9t\") pod \"maas-api-e2e-shared-a-w1-4848d2-565fb9d78c-pd59p\" (UID: \"aa56df3b-fd52-46b8-a626-0dd7e827ab8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4848d2-565fb9d78c-pd59p"
Nov 02 19:52:57.398461 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:57.398412    4463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-jzjj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 02 19:52:57.398660 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:57.398482    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jzjj4" podUID="73818a40-7ba5-410c-aa9f-f6833bd95623" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 02 19:52:57.808026 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:57.807990    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa56df3b-fd52-46b8-a626-0dd7e827ab8c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-4848d2-565fb9d78c-pd59p\" (UID: \"aa56df3b-fd52-46b8-a626-0dd7e827ab8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4848d2-565fb9d78c-pd59p"
Nov 02 19:52:57.810409 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:57.810382    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa56df3b-fd52-46b8-a626-0dd7e827ab8c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-4848d2-565fb9d78c-pd59p\" (UID: \"aa56df3b-fd52-46b8-a626-0dd7e827ab8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4848d2-565fb9d78c-pd59p"
Nov 02 19:52:58.039303 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:58.039267    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4848d2-565fb9d78c-pd59p"
Nov 02 19:52:58.395951 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:58.395862    4463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-jzjj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 02 19:52:58.395951 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:58.395910    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jzjj4" podUID="73818a40-7ba5-410c-aa9f-f6833bd95623" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 02 19:52:58.412782 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:58.412760    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4848d2-565fb9d78c-pd59p"]
Nov 02 19:52:58.414690 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:52:58.414666    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa56df3b_fd52_46b8_a626_0dd7e827ab8c.slice/crio-e45235736ec8ccffd5eac053f651e922702b3f70850baac7e082a61b204d3424 WatchSource:0}: Error finding container e45235736ec8ccffd5eac053f651e922702b3f70850baac7e082a61b204d3424: Status 404 returned error can't find the container with id e45235736ec8ccffd5eac053f651e922702b3f70850baac7e082a61b204d3424
Nov 02 19:52:58.594853 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:58.594816    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4848d2-565fb9d78c-pd59p" event={"ID":"aa56df3b-fd52-46b8-a626-0dd7e827ab8c","Type":"ContainerStarted","Data":"e45235736ec8ccffd5eac053f651e922702b3f70850baac7e082a61b204d3424"}
Nov 02 19:52:59.604279 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:59.604185    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4848d2-565fb9d78c-pd59p" event={"ID":"aa56df3b-fd52-46b8-a626-0dd7e827ab8c","Type":"ContainerStarted","Data":"34aa0a30d4ba98e7b72463224c419a7cad22a36effdbc70329f675d137d678d3"}
Nov 02 19:52:59.604279 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:59.604264    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4848d2-565fb9d78c-pd59p"
Nov 02 19:52:59.717947 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:52:59.717879    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4848d2-565fb9d78c-pd59p" podStartSLOduration=1.846024745 podStartE2EDuration="2.717860088s" totalImagesPullingTime="871.835343ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:52:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:52:59.711396002 +0000 UTC m=+1486.318429774" watchObservedRunningTime="2026-11-02 19:52:59.717860088 +0000 UTC m=+1486.324893858"
Nov 02 19:53:00.396215 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:00.396181    4463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-jzjj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 19:53:00.396401 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:00.396240    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jzjj4" podUID="73818a40-7ba5-410c-aa9f-f6833bd95623" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:53:01.437153 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:01.437115    4463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-jzjj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 19:53:01.437537 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:01.437171    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jzjj4" podUID="73818a40-7ba5-410c-aa9f-f6833bd95623" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:53:01.870263 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:01.870171    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jzjj4"
Nov 02 19:53:01.870592 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:01.870555    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jzjj4"
Nov 02 19:53:01.888049 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:01.887982    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jzjj4"
Nov 02 19:53:04.294803 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:53:04.294764    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 19:53:05.242390 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:05.242355    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-26hn5"]
Nov 02 19:53:05.242577 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:05.242553    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-26hn5" podUID="1d535f89-ff88-4a74-af33-99c15a1b05d4" containerName="istio-proxy" containerID="cri-o://a2f9386caab420175aec3a114b6daaecffbf1c42c609af1c0721a2186b1542de" gracePeriod=30
Nov 02 19:53:05.621151 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:05.621075    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4848d2-565fb9d78c-pd59p"
Nov 02 19:53:10.508583 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:10.508527    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-26hn5"
Nov 02 19:53:10.612915 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:10.612882    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d535f89-ff88-4a74-af33-99c15a1b05d4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d535f89-ff88-4a74-af33-99c15a1b05d4-istio-envoy\") pod \"1d535f89-ff88-4a74-af33-99c15a1b05d4\" (UID: \"1d535f89-ff88-4a74-af33-99c15a1b05d4\") "
Nov 02 19:53:10.613090 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:10.612928    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1d535f89-ff88-4a74-af33-99c15a1b05d4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d535f89-ff88-4a74-af33-99c15a1b05d4-istio-podinfo\") pod \"1d535f89-ff88-4a74-af33-99c15a1b05d4\" (UID: \"1d535f89-ff88-4a74-af33-99c15a1b05d4\") "
Nov 02 19:53:10.613090 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:10.612963    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d535f89-ff88-4a74-af33-99c15a1b05d4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d535f89-ff88-4a74-af33-99c15a1b05d4-workload-socket\") pod \"1d535f89-ff88-4a74-af33-99c15a1b05d4\" (UID: \"1d535f89-ff88-4a74-af33-99c15a1b05d4\") "
Nov 02 19:53:10.613090 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:10.613070    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d535f89-ff88-4a74-af33-99c15a1b05d4-kube-api-access-d7t4t\" (UniqueName: \"kubernetes.io/projected/1d535f89-ff88-4a74-af33-99c15a1b05d4-kube-api-access-d7t4t\") pod \"1d535f89-ff88-4a74-af33-99c15a1b05d4\" (UID: \"1d535f89-ff88-4a74-af33-99c15a1b05d4\") "
Nov 02 19:53:10.613289 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:10.613266    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d535f89-ff88-4a74-af33-99c15a1b05d4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d535f89-ff88-4a74-af33-99c15a1b05d4-istio-data\") pod \"1d535f89-ff88-4a74-af33-99c15a1b05d4\" (UID: \"1d535f89-ff88-4a74-af33-99c15a1b05d4\") "
Nov 02 19:53:10.613354 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:10.613277    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d535f89-ff88-4a74-af33-99c15a1b05d4-workload-socket" pod "1d535f89-ff88-4a74-af33-99c15a1b05d4" (UID: "1d535f89-ff88-4a74-af33-99c15a1b05d4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:53:10.613354 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:10.613326    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d535f89-ff88-4a74-af33-99c15a1b05d4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d535f89-ff88-4a74-af33-99c15a1b05d4-istiod-ca-cert\") pod \"1d535f89-ff88-4a74-af33-99c15a1b05d4\" (UID: \"1d535f89-ff88-4a74-af33-99c15a1b05d4\") "
Nov 02 19:53:10.613475 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:10.613363    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d535f89-ff88-4a74-af33-99c15a1b05d4-istio-token\" (UniqueName: \"kubernetes.io/projected/1d535f89-ff88-4a74-af33-99c15a1b05d4-istio-token\") pod \"1d535f89-ff88-4a74-af33-99c15a1b05d4\" (UID: \"1d535f89-ff88-4a74-af33-99c15a1b05d4\") "
Nov 02 19:53:10.613475 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:10.613447    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d535f89-ff88-4a74-af33-99c15a1b05d4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d535f89-ff88-4a74-af33-99c15a1b05d4-credential-socket\") pod \"1d535f89-ff88-4a74-af33-99c15a1b05d4\" (UID: \"1d535f89-ff88-4a74-af33-99c15a1b05d4\") "
Nov 02 19:53:10.613577 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:10.613500    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d535f89-ff88-4a74-af33-99c15a1b05d4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d535f89-ff88-4a74-af33-99c15a1b05d4-workload-certs\") pod \"1d535f89-ff88-4a74-af33-99c15a1b05d4\" (UID: \"1d535f89-ff88-4a74-af33-99c15a1b05d4\") "
Nov 02 19:53:10.613748 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:10.613723    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d535f89-ff88-4a74-af33-99c15a1b05d4-istiod-ca-cert" pod "1d535f89-ff88-4a74-af33-99c15a1b05d4" (UID: "1d535f89-ff88-4a74-af33-99c15a1b05d4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 19:53:10.614005 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:10.613953    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d535f89-ff88-4a74-af33-99c15a1b05d4-workload-certs" pod "1d535f89-ff88-4a74-af33-99c15a1b05d4" (UID: "1d535f89-ff88-4a74-af33-99c15a1b05d4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:53:10.614005 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:10.614000    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d535f89-ff88-4a74-af33-99c15a1b05d4-credential-socket" pod "1d535f89-ff88-4a74-af33-99c15a1b05d4" (UID: "1d535f89-ff88-4a74-af33-99c15a1b05d4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:53:10.614200 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:10.614121    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d535f89-ff88-4a74-af33-99c15a1b05d4-istio-data" pod "1d535f89-ff88-4a74-af33-99c15a1b05d4" (UID: "1d535f89-ff88-4a74-af33-99c15a1b05d4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:53:10.614264 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:10.614232    4463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d535f89-ff88-4a74-af33-99c15a1b05d4-workload-socket\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:53:10.614264 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:10.614245    4463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d535f89-ff88-4a74-af33-99c15a1b05d4-istio-data\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:53:10.614264 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:10.614254    4463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d535f89-ff88-4a74-af33-99c15a1b05d4-istiod-ca-cert\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:53:10.614264 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:10.614264    4463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d535f89-ff88-4a74-af33-99c15a1b05d4-credential-socket\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:53:10.614474 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:10.614272    4463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d535f89-ff88-4a74-af33-99c15a1b05d4-workload-certs\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:53:10.615189 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:10.615157    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1d535f89-ff88-4a74-af33-99c15a1b05d4-istio-podinfo" pod "1d535f89-ff88-4a74-af33-99c15a1b05d4" (UID: "1d535f89-ff88-4a74-af33-99c15a1b05d4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 19:53:10.615419 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:10.615387    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d535f89-ff88-4a74-af33-99c15a1b05d4-istio-envoy" pod "1d535f89-ff88-4a74-af33-99c15a1b05d4" (UID: "1d535f89-ff88-4a74-af33-99c15a1b05d4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:53:10.615513 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:10.615399    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d535f89-ff88-4a74-af33-99c15a1b05d4-kube-api-access-d7t4t" pod "1d535f89-ff88-4a74-af33-99c15a1b05d4" (UID: "1d535f89-ff88-4a74-af33-99c15a1b05d4"). InnerVolumeSpecName "kube-api-access-d7t4t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:53:10.615680 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:10.615657    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d535f89-ff88-4a74-af33-99c15a1b05d4-istio-token" pod "1d535f89-ff88-4a74-af33-99c15a1b05d4" (UID: "1d535f89-ff88-4a74-af33-99c15a1b05d4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:53:10.681636 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:10.681597    4463 generic.go:356] "Generic (PLEG): container finished" podID="1d535f89-ff88-4a74-af33-99c15a1b05d4" containerID="a2f9386caab420175aec3a114b6daaecffbf1c42c609af1c0721a2186b1542de" exitCode=0
Nov 02 19:53:10.681784 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:10.681771    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-26hn5" event={"ID":"1d535f89-ff88-4a74-af33-99c15a1b05d4","Type":"ContainerDied","Data":"a2f9386caab420175aec3a114b6daaecffbf1c42c609af1c0721a2186b1542de"}
Nov 02 19:53:10.681858 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:10.681825    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-26hn5" event={"ID":"1d535f89-ff88-4a74-af33-99c15a1b05d4","Type":"ContainerDied","Data":"e54e4a0cb29c06165d7f1dafc25d44a0674b6d723c845deaa1992b04db81b6aa"}
Nov 02 19:53:10.681923 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:10.681855    4463 scope.go:122] "RemoveContainer" containerID="a2f9386caab420175aec3a114b6daaecffbf1c42c609af1c0721a2186b1542de"
Nov 02 19:53:10.682004 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:10.681984    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-26hn5"
Nov 02 19:53:10.707161 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:10.707136    4463 scope.go:122] "RemoveContainer" containerID="a2f9386caab420175aec3a114b6daaecffbf1c42c609af1c0721a2186b1542de"
Nov 02 19:53:10.716174 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:53:10.716142    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2f9386caab420175aec3a114b6daaecffbf1c42c609af1c0721a2186b1542de\": container with ID starting with a2f9386caab420175aec3a114b6daaecffbf1c42c609af1c0721a2186b1542de not found: ID does not exist" containerID="a2f9386caab420175aec3a114b6daaecffbf1c42c609af1c0721a2186b1542de"
Nov 02 19:53:10.716331 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:10.716303    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2f9386caab420175aec3a114b6daaecffbf1c42c609af1c0721a2186b1542de"} err="failed to get container status \"a2f9386caab420175aec3a114b6daaecffbf1c42c609af1c0721a2186b1542de\": rpc error: code = NotFound desc = could not find container \"a2f9386caab420175aec3a114b6daaecffbf1c42c609af1c0721a2186b1542de\": container with ID starting with a2f9386caab420175aec3a114b6daaecffbf1c42c609af1c0721a2186b1542de not found: ID does not exist"
Nov 02 19:53:10.717859 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:10.717840    4463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d535f89-ff88-4a74-af33-99c15a1b05d4-istio-envoy\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:53:10.718314 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:10.718291    4463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d535f89-ff88-4a74-af33-99c15a1b05d4-istio-podinfo\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:53:10.718416 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:10.718321    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7t4t\" (UniqueName: \"kubernetes.io/projected/1d535f89-ff88-4a74-af33-99c15a1b05d4-kube-api-access-d7t4t\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:53:10.718416 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:10.718338    4463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d535f89-ff88-4a74-af33-99c15a1b05d4-istio-token\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:53:10.836376 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:10.836278    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-26hn5"]
Nov 02 19:53:10.839765 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:10.839733    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-26hn5"]
Nov 02 19:53:11.587071 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:11.587039    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d535f89-ff88-4a74-af33-99c15a1b05d4" path="/var/lib/kubelet/pods/1d535f89-ff88-4a74-af33-99c15a1b05d4/volumes"
Nov 02 19:53:11.721628 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:11.721588    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-4099cbd3-openshift-default-698ddb8d77-lrzxg"]
Nov 02 19:53:11.723076 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:11.722993    4463 cpu_manager.go:414] "containerMap: removing container" podUID="1d535f89-ff88-4a74-af33-99c15a1b05d4" containerName="istio-proxy"
Nov 02 19:53:11.723076 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:11.723076    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d535f89-ff88-4a74-af33-99c15a1b05d4" containerName="istio-proxy"
Nov 02 19:53:11.723278 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:11.723267    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d535f89-ff88-4a74-af33-99c15a1b05d4" containerName="istio-proxy"
Nov 02 19:53:11.729277 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:11.729252    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4099cbd3-openshift-default-698ddb8d77-lrzxg"
Nov 02 19:53:11.776920 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:11.776872    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-4099cbd3-openshift-default-698ddb8d77-lrzxg"]
Nov 02 19:53:11.851420 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:11.851338    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-4099cbd3-openshift-default-dockercfg-kd76c\""
Nov 02 19:53:11.917865 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:11.898557    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44a12c0e-e3df-4368-a5f8-61c3b8f88da6-istiod-ca-cert\") pod \"e2e-mt-4099cbd3-openshift-default-698ddb8d77-lrzxg\" (UID: \"44a12c0e-e3df-4368-a5f8-61c3b8f88da6\") " pod="openshift-ingress/e2e-mt-4099cbd3-openshift-default-698ddb8d77-lrzxg"
Nov 02 19:53:11.917865 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:11.898621    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44a12c0e-e3df-4368-a5f8-61c3b8f88da6-istio-data\") pod \"e2e-mt-4099cbd3-openshift-default-698ddb8d77-lrzxg\" (UID: \"44a12c0e-e3df-4368-a5f8-61c3b8f88da6\") " pod="openshift-ingress/e2e-mt-4099cbd3-openshift-default-698ddb8d77-lrzxg"
Nov 02 19:53:11.917865 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:11.898684    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44a12c0e-e3df-4368-a5f8-61c3b8f88da6-workload-certs\") pod \"e2e-mt-4099cbd3-openshift-default-698ddb8d77-lrzxg\" (UID: \"44a12c0e-e3df-4368-a5f8-61c3b8f88da6\") " pod="openshift-ingress/e2e-mt-4099cbd3-openshift-default-698ddb8d77-lrzxg"
Nov 02 19:53:11.917865 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:11.898827    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44a12c0e-e3df-4368-a5f8-61c3b8f88da6-istio-podinfo\") pod \"e2e-mt-4099cbd3-openshift-default-698ddb8d77-lrzxg\" (UID: \"44a12c0e-e3df-4368-a5f8-61c3b8f88da6\") " pod="openshift-ingress/e2e-mt-4099cbd3-openshift-default-698ddb8d77-lrzxg"
Nov 02 19:53:11.917865 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:11.898865    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44a12c0e-e3df-4368-a5f8-61c3b8f88da6-istio-token\") pod \"e2e-mt-4099cbd3-openshift-default-698ddb8d77-lrzxg\" (UID: \"44a12c0e-e3df-4368-a5f8-61c3b8f88da6\") " pod="openshift-ingress/e2e-mt-4099cbd3-openshift-default-698ddb8d77-lrzxg"
Nov 02 19:53:11.917865 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:11.898888    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w48zt\" (UniqueName: \"kubernetes.io/projected/44a12c0e-e3df-4368-a5f8-61c3b8f88da6-kube-api-access-w48zt\") pod \"e2e-mt-4099cbd3-openshift-default-698ddb8d77-lrzxg\" (UID: \"44a12c0e-e3df-4368-a5f8-61c3b8f88da6\") " pod="openshift-ingress/e2e-mt-4099cbd3-openshift-default-698ddb8d77-lrzxg"
Nov 02 19:53:11.917865 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:11.898939    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a12c0e-e3df-4368-a5f8-61c3b8f88da6-workload-socket\") pod \"e2e-mt-4099cbd3-openshift-default-698ddb8d77-lrzxg\" (UID: \"44a12c0e-e3df-4368-a5f8-61c3b8f88da6\") " pod="openshift-ingress/e2e-mt-4099cbd3-openshift-default-698ddb8d77-lrzxg"
Nov 02 19:53:11.917865 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:11.898970    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44a12c0e-e3df-4368-a5f8-61c3b8f88da6-istio-envoy\") pod \"e2e-mt-4099cbd3-openshift-default-698ddb8d77-lrzxg\" (UID: \"44a12c0e-e3df-4368-a5f8-61c3b8f88da6\") " pod="openshift-ingress/e2e-mt-4099cbd3-openshift-default-698ddb8d77-lrzxg"
Nov 02 19:53:11.917865 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:11.899038    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a12c0e-e3df-4368-a5f8-61c3b8f88da6-credential-socket\") pod \"e2e-mt-4099cbd3-openshift-default-698ddb8d77-lrzxg\" (UID: \"44a12c0e-e3df-4368-a5f8-61c3b8f88da6\") " pod="openshift-ingress/e2e-mt-4099cbd3-openshift-default-698ddb8d77-lrzxg"
Nov 02 19:53:12.000105 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:12.000070    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44a12c0e-e3df-4368-a5f8-61c3b8f88da6-istio-token\") pod \"e2e-mt-4099cbd3-openshift-default-698ddb8d77-lrzxg\" (UID: \"44a12c0e-e3df-4368-a5f8-61c3b8f88da6\") " pod="openshift-ingress/e2e-mt-4099cbd3-openshift-default-698ddb8d77-lrzxg"
Nov 02 19:53:12.000316 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:12.000117    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w48zt\" (UniqueName: \"kubernetes.io/projected/44a12c0e-e3df-4368-a5f8-61c3b8f88da6-kube-api-access-w48zt\") pod \"e2e-mt-4099cbd3-openshift-default-698ddb8d77-lrzxg\" (UID: \"44a12c0e-e3df-4368-a5f8-61c3b8f88da6\") " pod="openshift-ingress/e2e-mt-4099cbd3-openshift-default-698ddb8d77-lrzxg"
Nov 02 19:53:12.000316 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:12.000174    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a12c0e-e3df-4368-a5f8-61c3b8f88da6-workload-socket\") pod \"e2e-mt-4099cbd3-openshift-default-698ddb8d77-lrzxg\" (UID: \"44a12c0e-e3df-4368-a5f8-61c3b8f88da6\") " pod="openshift-ingress/e2e-mt-4099cbd3-openshift-default-698ddb8d77-lrzxg"
Nov 02 19:53:12.000316 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:12.000213    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44a12c0e-e3df-4368-a5f8-61c3b8f88da6-istio-envoy\") pod \"e2e-mt-4099cbd3-openshift-default-698ddb8d77-lrzxg\" (UID: \"44a12c0e-e3df-4368-a5f8-61c3b8f88da6\") " pod="openshift-ingress/e2e-mt-4099cbd3-openshift-default-698ddb8d77-lrzxg"
Nov 02 19:53:12.000316 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:12.000268    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a12c0e-e3df-4368-a5f8-61c3b8f88da6-credential-socket\") pod \"e2e-mt-4099cbd3-openshift-default-698ddb8d77-lrzxg\" (UID: \"44a12c0e-e3df-4368-a5f8-61c3b8f88da6\") " pod="openshift-ingress/e2e-mt-4099cbd3-openshift-default-698ddb8d77-lrzxg"
Nov 02 19:53:12.000547 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:12.000346    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44a12c0e-e3df-4368-a5f8-61c3b8f88da6-istiod-ca-cert\") pod \"e2e-mt-4099cbd3-openshift-default-698ddb8d77-lrzxg\" (UID: \"44a12c0e-e3df-4368-a5f8-61c3b8f88da6\") " pod="openshift-ingress/e2e-mt-4099cbd3-openshift-default-698ddb8d77-lrzxg"
Nov 02 19:53:12.000547 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:12.000377    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44a12c0e-e3df-4368-a5f8-61c3b8f88da6-istio-data\") pod \"e2e-mt-4099cbd3-openshift-default-698ddb8d77-lrzxg\" (UID: \"44a12c0e-e3df-4368-a5f8-61c3b8f88da6\") " pod="openshift-ingress/e2e-mt-4099cbd3-openshift-default-698ddb8d77-lrzxg"
Nov 02 19:53:12.000547 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:12.000440    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44a12c0e-e3df-4368-a5f8-61c3b8f88da6-workload-certs\") pod \"e2e-mt-4099cbd3-openshift-default-698ddb8d77-lrzxg\" (UID: \"44a12c0e-e3df-4368-a5f8-61c3b8f88da6\") " pod="openshift-ingress/e2e-mt-4099cbd3-openshift-default-698ddb8d77-lrzxg"
Nov 02 19:53:12.000706 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:12.000635    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44a12c0e-e3df-4368-a5f8-61c3b8f88da6-istio-podinfo\") pod \"e2e-mt-4099cbd3-openshift-default-698ddb8d77-lrzxg\" (UID: \"44a12c0e-e3df-4368-a5f8-61c3b8f88da6\") " pod="openshift-ingress/e2e-mt-4099cbd3-openshift-default-698ddb8d77-lrzxg"
Nov 02 19:53:12.000706 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:12.000650    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a12c0e-e3df-4368-a5f8-61c3b8f88da6-workload-socket\") pod \"e2e-mt-4099cbd3-openshift-default-698ddb8d77-lrzxg\" (UID: \"44a12c0e-e3df-4368-a5f8-61c3b8f88da6\") " pod="openshift-ingress/e2e-mt-4099cbd3-openshift-default-698ddb8d77-lrzxg"
Nov 02 19:53:12.000812 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:12.000777    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a12c0e-e3df-4368-a5f8-61c3b8f88da6-credential-socket\") pod \"e2e-mt-4099cbd3-openshift-default-698ddb8d77-lrzxg\" (UID: \"44a12c0e-e3df-4368-a5f8-61c3b8f88da6\") " pod="openshift-ingress/e2e-mt-4099cbd3-openshift-default-698ddb8d77-lrzxg"
Nov 02 19:53:12.000971 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:12.000948    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44a12c0e-e3df-4368-a5f8-61c3b8f88da6-workload-certs\") pod \"e2e-mt-4099cbd3-openshift-default-698ddb8d77-lrzxg\" (UID: \"44a12c0e-e3df-4368-a5f8-61c3b8f88da6\") " pod="openshift-ingress/e2e-mt-4099cbd3-openshift-default-698ddb8d77-lrzxg"
Nov 02 19:53:12.001142 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:12.000965    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44a12c0e-e3df-4368-a5f8-61c3b8f88da6-istio-data\") pod \"e2e-mt-4099cbd3-openshift-default-698ddb8d77-lrzxg\" (UID: \"44a12c0e-e3df-4368-a5f8-61c3b8f88da6\") " pod="openshift-ingress/e2e-mt-4099cbd3-openshift-default-698ddb8d77-lrzxg"
Nov 02 19:53:12.001252 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:12.001233    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44a12c0e-e3df-4368-a5f8-61c3b8f88da6-istiod-ca-cert\") pod \"e2e-mt-4099cbd3-openshift-default-698ddb8d77-lrzxg\" (UID: \"44a12c0e-e3df-4368-a5f8-61c3b8f88da6\") " pod="openshift-ingress/e2e-mt-4099cbd3-openshift-default-698ddb8d77-lrzxg"
Nov 02 19:53:12.003751 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:12.003722    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44a12c0e-e3df-4368-a5f8-61c3b8f88da6-istio-envoy\") pod \"e2e-mt-4099cbd3-openshift-default-698ddb8d77-lrzxg\" (UID: \"44a12c0e-e3df-4368-a5f8-61c3b8f88da6\") " pod="openshift-ingress/e2e-mt-4099cbd3-openshift-default-698ddb8d77-lrzxg"
Nov 02 19:53:12.003981 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:12.003958    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44a12c0e-e3df-4368-a5f8-61c3b8f88da6-istio-podinfo\") pod \"e2e-mt-4099cbd3-openshift-default-698ddb8d77-lrzxg\" (UID: \"44a12c0e-e3df-4368-a5f8-61c3b8f88da6\") " pod="openshift-ingress/e2e-mt-4099cbd3-openshift-default-698ddb8d77-lrzxg"
Nov 02 19:53:12.053483 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:12.053449    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44a12c0e-e3df-4368-a5f8-61c3b8f88da6-istio-token\") pod \"e2e-mt-4099cbd3-openshift-default-698ddb8d77-lrzxg\" (UID: \"44a12c0e-e3df-4368-a5f8-61c3b8f88da6\") " pod="openshift-ingress/e2e-mt-4099cbd3-openshift-default-698ddb8d77-lrzxg"
Nov 02 19:53:12.054072 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:12.054047    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w48zt\" (UniqueName: \"kubernetes.io/projected/44a12c0e-e3df-4368-a5f8-61c3b8f88da6-kube-api-access-w48zt\") pod \"e2e-mt-4099cbd3-openshift-default-698ddb8d77-lrzxg\" (UID: \"44a12c0e-e3df-4368-a5f8-61c3b8f88da6\") " pod="openshift-ingress/e2e-mt-4099cbd3-openshift-default-698ddb8d77-lrzxg"
Nov 02 19:53:12.100169 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:12.100136    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4099cbd3-openshift-default-698ddb8d77-lrzxg"
Nov 02 19:53:12.548260 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:12.548227    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-4099cbd3-openshift-default-698ddb8d77-lrzxg"]
Nov 02 19:53:12.549269 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:53:12.549225    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44a12c0e_e3df_4368_a5f8_61c3b8f88da6.slice/crio-e2470c47b6172db7e87d773e783eee5bb56acddd310a506a92536acb83bf9dc4 WatchSource:0}: Error finding container e2470c47b6172db7e87d773e783eee5bb56acddd310a506a92536acb83bf9dc4: Status 404 returned error can't find the container with id e2470c47b6172db7e87d773e783eee5bb56acddd310a506a92536acb83bf9dc4
Nov 02 19:53:12.551834 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:12.551804    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:53:12.551901 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:12.551869    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:53:12.551901 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:12.551897    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:53:12.698741 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:12.698703    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4099cbd3-openshift-default-698ddb8d77-lrzxg" event={"ID":"44a12c0e-e3df-4368-a5f8-61c3b8f88da6","Type":"ContainerStarted","Data":"ad3d9c3c6ac7afe5b3d137580b7014e71578c849b53231bb9f5635ff39853c38"}
Nov 02 19:53:12.699136 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:12.698769    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4099cbd3-openshift-default-698ddb8d77-lrzxg" event={"ID":"44a12c0e-e3df-4368-a5f8-61c3b8f88da6","Type":"ContainerStarted","Data":"e2470c47b6172db7e87d773e783eee5bb56acddd310a506a92536acb83bf9dc4"}
Nov 02 19:53:12.808979 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:12.808867    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-4099cbd3-openshift-default-698ddb8d77-lrzxg" podStartSLOduration=1.808849664 podStartE2EDuration="1.808849664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:53:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:53:12.806422286 +0000 UTC m=+1499.413456054" watchObservedRunningTime="2026-11-02 19:53:12.808849664 +0000 UTC m=+1499.415883433"
Nov 02 19:53:13.107127 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:13.107032    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-4099cbd3-openshift-default-698ddb8d77-lrzxg"
Nov 02 19:53:13.114451 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:13.114229    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-4099cbd3-openshift-default-698ddb8d77-lrzxg"
Nov 02 19:53:13.249225 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:13.249192    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 19:53:13.712527 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:13.712489    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-4099cbd3-openshift-default-698ddb8d77-lrzxg"
Nov 02 19:53:13.713904 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:13.713873    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-4099cbd3-openshift-default-698ddb8d77-lrzxg"
Nov 02 19:53:14.043264 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:14.043203    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-4fd0d5-openshift-default-8566cd5547-tvdjk"]
Nov 02 19:53:14.048706 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:14.048691    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-4fd0d5-openshift-default-8566cd5547-tvdjk"
Nov 02 19:53:14.080673 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:14.077103    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6d73c91-e211-41f4-a929-d3e45bc82366-istiod-ca-cert\") pod \"e2e-shared-a-w5-4fd0d5-openshift-default-8566cd5547-tvdjk\" (UID: \"c6d73c91-e211-41f4-a929-d3e45bc82366\") " pod="openshift-ingress/e2e-shared-a-w5-4fd0d5-openshift-default-8566cd5547-tvdjk"
Nov 02 19:53:14.095582 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:14.088129    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6d73c91-e211-41f4-a929-d3e45bc82366-workload-socket\") pod \"e2e-shared-a-w5-4fd0d5-openshift-default-8566cd5547-tvdjk\" (UID: \"c6d73c91-e211-41f4-a929-d3e45bc82366\") " pod="openshift-ingress/e2e-shared-a-w5-4fd0d5-openshift-default-8566cd5547-tvdjk"
Nov 02 19:53:14.095582 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:14.088228    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6znv\" (UniqueName: \"kubernetes.io/projected/c6d73c91-e211-41f4-a929-d3e45bc82366-kube-api-access-r6znv\") pod \"e2e-shared-a-w5-4fd0d5-openshift-default-8566cd5547-tvdjk\" (UID: \"c6d73c91-e211-41f4-a929-d3e45bc82366\") " pod="openshift-ingress/e2e-shared-a-w5-4fd0d5-openshift-default-8566cd5547-tvdjk"
Nov 02 19:53:14.095582 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:14.088349    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6d73c91-e211-41f4-a929-d3e45bc82366-istio-podinfo\") pod \"e2e-shared-a-w5-4fd0d5-openshift-default-8566cd5547-tvdjk\" (UID: \"c6d73c91-e211-41f4-a929-d3e45bc82366\") " pod="openshift-ingress/e2e-shared-a-w5-4fd0d5-openshift-default-8566cd5547-tvdjk"
Nov 02 19:53:14.095582 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:14.088388    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6d73c91-e211-41f4-a929-d3e45bc82366-workload-certs\") pod \"e2e-shared-a-w5-4fd0d5-openshift-default-8566cd5547-tvdjk\" (UID: \"c6d73c91-e211-41f4-a929-d3e45bc82366\") " pod="openshift-ingress/e2e-shared-a-w5-4fd0d5-openshift-default-8566cd5547-tvdjk"
Nov 02 19:53:14.095582 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:14.088485    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6d73c91-e211-41f4-a929-d3e45bc82366-istio-envoy\") pod \"e2e-shared-a-w5-4fd0d5-openshift-default-8566cd5547-tvdjk\" (UID: \"c6d73c91-e211-41f4-a929-d3e45bc82366\") " pod="openshift-ingress/e2e-shared-a-w5-4fd0d5-openshift-default-8566cd5547-tvdjk"
Nov 02 19:53:14.095582 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:14.088553    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6d73c91-e211-41f4-a929-d3e45bc82366-credential-socket\") pod \"e2e-shared-a-w5-4fd0d5-openshift-default-8566cd5547-tvdjk\" (UID: \"c6d73c91-e211-41f4-a929-d3e45bc82366\") " pod="openshift-ingress/e2e-shared-a-w5-4fd0d5-openshift-default-8566cd5547-tvdjk"
Nov 02 19:53:14.095582 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:14.088686    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6d73c91-e211-41f4-a929-d3e45bc82366-istio-data\") pod \"e2e-shared-a-w5-4fd0d5-openshift-default-8566cd5547-tvdjk\" (UID: \"c6d73c91-e211-41f4-a929-d3e45bc82366\") " pod="openshift-ingress/e2e-shared-a-w5-4fd0d5-openshift-default-8566cd5547-tvdjk"
Nov 02 19:53:14.095582 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:14.088736    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6d73c91-e211-41f4-a929-d3e45bc82366-istio-token\") pod \"e2e-shared-a-w5-4fd0d5-openshift-default-8566cd5547-tvdjk\" (UID: \"c6d73c91-e211-41f4-a929-d3e45bc82366\") " pod="openshift-ingress/e2e-shared-a-w5-4fd0d5-openshift-default-8566cd5547-tvdjk"
Nov 02 19:53:14.173037 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:14.164336    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w5-4fd0d5-openshift-default-dockercfg-sntnl\""
Nov 02 19:53:14.234103 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:14.202656    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6d73c91-e211-41f4-a929-d3e45bc82366-istio-data\") pod \"e2e-shared-a-w5-4fd0d5-openshift-default-8566cd5547-tvdjk\" (UID: \"c6d73c91-e211-41f4-a929-d3e45bc82366\") " pod="openshift-ingress/e2e-shared-a-w5-4fd0d5-openshift-default-8566cd5547-tvdjk"
Nov 02 19:53:14.234103 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:14.202715    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6d73c91-e211-41f4-a929-d3e45bc82366-istio-token\") pod \"e2e-shared-a-w5-4fd0d5-openshift-default-8566cd5547-tvdjk\" (UID: \"c6d73c91-e211-41f4-a929-d3e45bc82366\") " pod="openshift-ingress/e2e-shared-a-w5-4fd0d5-openshift-default-8566cd5547-tvdjk"
Nov 02 19:53:14.234103 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:14.202791    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6d73c91-e211-41f4-a929-d3e45bc82366-istiod-ca-cert\") pod \"e2e-shared-a-w5-4fd0d5-openshift-default-8566cd5547-tvdjk\" (UID: \"c6d73c91-e211-41f4-a929-d3e45bc82366\") " pod="openshift-ingress/e2e-shared-a-w5-4fd0d5-openshift-default-8566cd5547-tvdjk"
Nov 02 19:53:14.234103 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:14.202849    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6d73c91-e211-41f4-a929-d3e45bc82366-workload-socket\") pod \"e2e-shared-a-w5-4fd0d5-openshift-default-8566cd5547-tvdjk\" (UID: \"c6d73c91-e211-41f4-a929-d3e45bc82366\") " pod="openshift-ingress/e2e-shared-a-w5-4fd0d5-openshift-default-8566cd5547-tvdjk"
Nov 02 19:53:14.234103 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:14.202908    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6znv\" (UniqueName: \"kubernetes.io/projected/c6d73c91-e211-41f4-a929-d3e45bc82366-kube-api-access-r6znv\") pod \"e2e-shared-a-w5-4fd0d5-openshift-default-8566cd5547-tvdjk\" (UID: \"c6d73c91-e211-41f4-a929-d3e45bc82366\") " pod="openshift-ingress/e2e-shared-a-w5-4fd0d5-openshift-default-8566cd5547-tvdjk"
Nov 02 19:53:14.234103 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:14.202989    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6d73c91-e211-41f4-a929-d3e45bc82366-istio-podinfo\") pod \"e2e-shared-a-w5-4fd0d5-openshift-default-8566cd5547-tvdjk\" (UID: \"c6d73c91-e211-41f4-a929-d3e45bc82366\") " pod="openshift-ingress/e2e-shared-a-w5-4fd0d5-openshift-default-8566cd5547-tvdjk"
Nov 02 19:53:14.234103 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:14.203034    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6d73c91-e211-41f4-a929-d3e45bc82366-workload-certs\") pod \"e2e-shared-a-w5-4fd0d5-openshift-default-8566cd5547-tvdjk\" (UID: \"c6d73c91-e211-41f4-a929-d3e45bc82366\") " pod="openshift-ingress/e2e-shared-a-w5-4fd0d5-openshift-default-8566cd5547-tvdjk"
Nov 02 19:53:14.234103 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:14.203121    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6d73c91-e211-41f4-a929-d3e45bc82366-istio-envoy\") pod \"e2e-shared-a-w5-4fd0d5-openshift-default-8566cd5547-tvdjk\" (UID: \"c6d73c91-e211-41f4-a929-d3e45bc82366\") " pod="openshift-ingress/e2e-shared-a-w5-4fd0d5-openshift-default-8566cd5547-tvdjk"
Nov 02 19:53:14.234103 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:14.203168    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6d73c91-e211-41f4-a929-d3e45bc82366-credential-socket\") pod \"e2e-shared-a-w5-4fd0d5-openshift-default-8566cd5547-tvdjk\" (UID: \"c6d73c91-e211-41f4-a929-d3e45bc82366\") " pod="openshift-ingress/e2e-shared-a-w5-4fd0d5-openshift-default-8566cd5547-tvdjk"
Nov 02 19:53:14.234103 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:14.203606    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6d73c91-e211-41f4-a929-d3e45bc82366-credential-socket\") pod \"e2e-shared-a-w5-4fd0d5-openshift-default-8566cd5547-tvdjk\" (UID: \"c6d73c91-e211-41f4-a929-d3e45bc82366\") " pod="openshift-ingress/e2e-shared-a-w5-4fd0d5-openshift-default-8566cd5547-tvdjk"
Nov 02 19:53:14.234103 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:14.223543    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6d73c91-e211-41f4-a929-d3e45bc82366-istio-data\") pod \"e2e-shared-a-w5-4fd0d5-openshift-default-8566cd5547-tvdjk\" (UID: \"c6d73c91-e211-41f4-a929-d3e45bc82366\") " pod="openshift-ingress/e2e-shared-a-w5-4fd0d5-openshift-default-8566cd5547-tvdjk"
Nov 02 19:53:14.234103 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:14.224536    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6d73c91-e211-41f4-a929-d3e45bc82366-istiod-ca-cert\") pod \"e2e-shared-a-w5-4fd0d5-openshift-default-8566cd5547-tvdjk\" (UID: \"c6d73c91-e211-41f4-a929-d3e45bc82366\") " pod="openshift-ingress/e2e-shared-a-w5-4fd0d5-openshift-default-8566cd5547-tvdjk"
Nov 02 19:53:14.234103 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:14.224804    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6d73c91-e211-41f4-a929-d3e45bc82366-workload-socket\") pod \"e2e-shared-a-w5-4fd0d5-openshift-default-8566cd5547-tvdjk\" (UID: \"c6d73c91-e211-41f4-a929-d3e45bc82366\") " pod="openshift-ingress/e2e-shared-a-w5-4fd0d5-openshift-default-8566cd5547-tvdjk"
Nov 02 19:53:14.269791 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:14.243186    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6d73c91-e211-41f4-a929-d3e45bc82366-workload-certs\") pod \"e2e-shared-a-w5-4fd0d5-openshift-default-8566cd5547-tvdjk\" (UID: \"c6d73c91-e211-41f4-a929-d3e45bc82366\") " pod="openshift-ingress/e2e-shared-a-w5-4fd0d5-openshift-default-8566cd5547-tvdjk"
Nov 02 19:53:14.318931 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:14.318806    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6d73c91-e211-41f4-a929-d3e45bc82366-istio-token\") pod \"e2e-shared-a-w5-4fd0d5-openshift-default-8566cd5547-tvdjk\" (UID: \"c6d73c91-e211-41f4-a929-d3e45bc82366\") " pod="openshift-ingress/e2e-shared-a-w5-4fd0d5-openshift-default-8566cd5547-tvdjk"
Nov 02 19:53:14.332701 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:14.331898    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6znv\" (UniqueName: \"kubernetes.io/projected/c6d73c91-e211-41f4-a929-d3e45bc82366-kube-api-access-r6znv\") pod \"e2e-shared-a-w5-4fd0d5-openshift-default-8566cd5547-tvdjk\" (UID: \"c6d73c91-e211-41f4-a929-d3e45bc82366\") " pod="openshift-ingress/e2e-shared-a-w5-4fd0d5-openshift-default-8566cd5547-tvdjk"
Nov 02 19:53:14.336704 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:14.336341    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6d73c91-e211-41f4-a929-d3e45bc82366-istio-podinfo\") pod \"e2e-shared-a-w5-4fd0d5-openshift-default-8566cd5547-tvdjk\" (UID: \"c6d73c91-e211-41f4-a929-d3e45bc82366\") " pod="openshift-ingress/e2e-shared-a-w5-4fd0d5-openshift-default-8566cd5547-tvdjk"
Nov 02 19:53:14.347037 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:14.345996    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-4fd0d5-openshift-default-8566cd5547-tvdjk"]
Nov 02 19:53:14.351289 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:14.349589    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6d73c91-e211-41f4-a929-d3e45bc82366-istio-envoy\") pod \"e2e-shared-a-w5-4fd0d5-openshift-default-8566cd5547-tvdjk\" (UID: \"c6d73c91-e211-41f4-a929-d3e45bc82366\") " pod="openshift-ingress/e2e-shared-a-w5-4fd0d5-openshift-default-8566cd5547-tvdjk"
Nov 02 19:53:14.361873 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:14.361855    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-4fd0d5-openshift-default-8566cd5547-tvdjk"
Nov 02 19:53:14.818873 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:14.818847    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-4fd0d5-openshift-default-8566cd5547-tvdjk"]
Nov 02 19:53:14.820056 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:53:14.820026    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6d73c91_e211_41f4_a929_d3e45bc82366.slice/crio-55d20456aecc3eeac092778416a08cad7af3e5d8affa43e9943fa93b8a5bb8cd WatchSource:0}: Error finding container 55d20456aecc3eeac092778416a08cad7af3e5d8affa43e9943fa93b8a5bb8cd: Status 404 returned error can't find the container with id 55d20456aecc3eeac092778416a08cad7af3e5d8affa43e9943fa93b8a5bb8cd
Nov 02 19:53:14.822468 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:14.822435    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:53:14.822574 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:14.822510    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:53:14.822574 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:14.822553    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:53:15.732337 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:15.732281    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-4fd0d5-openshift-default-8566cd5547-tvdjk" event={"ID":"c6d73c91-e211-41f4-a929-d3e45bc82366","Type":"ContainerStarted","Data":"1bee209fcbd131dde8b21eea904bd5824cfd810296e9501ca02ac775c223f888"}
Nov 02 19:53:15.732541 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:15.732399    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-4fd0d5-openshift-default-8566cd5547-tvdjk" event={"ID":"c6d73c91-e211-41f4-a929-d3e45bc82366","Type":"ContainerStarted","Data":"55d20456aecc3eeac092778416a08cad7af3e5d8affa43e9943fa93b8a5bb8cd"}
Nov 02 19:53:15.890388 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:15.890337    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-4fd0d5-openshift-default-8566cd5547-tvdjk" podStartSLOduration=1.890322586 podStartE2EDuration="1.890322586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:53:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:53:15.888384107 +0000 UTC m=+1502.495417901" watchObservedRunningTime="2026-11-02 19:53:15.890322586 +0000 UTC m=+1502.497356354"
Nov 02 19:53:16.362913 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:16.362883    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-4fd0d5-openshift-default-8566cd5547-tvdjk"
Nov 02 19:53:16.368753 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:16.368727    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-4fd0d5-openshift-default-8566cd5547-tvdjk"
Nov 02 19:53:16.739356 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:16.739323    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-4fd0d5-openshift-default-8566cd5547-tvdjk"
Nov 02 19:53:16.740192 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:16.740167    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-4fd0d5-openshift-default-8566cd5547-tvdjk"
Nov 02 19:53:20.152613 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:20.152576    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4099cbd3-84f88b85c5-hdbml"]
Nov 02 19:53:20.157958 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:20.157935    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-4099cbd3-84f88b85c5-hdbml"
Nov 02 19:53:20.207241 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:20.203129    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq7nn\" (UniqueName: \"kubernetes.io/projected/1c4ee35e-073e-4e22-9b22-955fa8c3bc94-kube-api-access-jq7nn\") pod \"payload-processing-e2e-mt-4099cbd3-84f88b85c5-hdbml\" (UID: \"1c4ee35e-073e-4e22-9b22-955fa8c3bc94\") " pod="openshift-ingress/payload-processing-e2e-mt-4099cbd3-84f88b85c5-hdbml"
Nov 02 19:53:20.207241 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:20.203645    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c4ee35e-073e-4e22-9b22-955fa8c3bc94-plugins-config-volume\") pod \"payload-processing-e2e-mt-4099cbd3-84f88b85c5-hdbml\" (UID: \"1c4ee35e-073e-4e22-9b22-955fa8c3bc94\") " pod="openshift-ingress/payload-processing-e2e-mt-4099cbd3-84f88b85c5-hdbml"
Nov 02 19:53:20.219948 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:20.219912    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4099cbd3-84f88b85c5-hdbml"]
Nov 02 19:53:20.255035 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:20.254304    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-4099cbd3\""
Nov 02 19:53:20.259036 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:20.255474    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-4099cbd3-dockercfg-s5rk6\""
Nov 02 19:53:20.315604 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:20.315560    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c4ee35e-073e-4e22-9b22-955fa8c3bc94-plugins-config-volume\") pod \"payload-processing-e2e-mt-4099cbd3-84f88b85c5-hdbml\" (UID: \"1c4ee35e-073e-4e22-9b22-955fa8c3bc94\") " pod="openshift-ingress/payload-processing-e2e-mt-4099cbd3-84f88b85c5-hdbml"
Nov 02 19:53:20.316541 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:20.316169    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq7nn\" (UniqueName: \"kubernetes.io/projected/1c4ee35e-073e-4e22-9b22-955fa8c3bc94-kube-api-access-jq7nn\") pod \"payload-processing-e2e-mt-4099cbd3-84f88b85c5-hdbml\" (UID: \"1c4ee35e-073e-4e22-9b22-955fa8c3bc94\") " pod="openshift-ingress/payload-processing-e2e-mt-4099cbd3-84f88b85c5-hdbml"
Nov 02 19:53:20.330196 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:20.321619    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c4ee35e-073e-4e22-9b22-955fa8c3bc94-plugins-config-volume\") pod \"payload-processing-e2e-mt-4099cbd3-84f88b85c5-hdbml\" (UID: \"1c4ee35e-073e-4e22-9b22-955fa8c3bc94\") " pod="openshift-ingress/payload-processing-e2e-mt-4099cbd3-84f88b85c5-hdbml"
Nov 02 19:53:20.401081 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:20.401050    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq7nn\" (UniqueName: \"kubernetes.io/projected/1c4ee35e-073e-4e22-9b22-955fa8c3bc94-kube-api-access-jq7nn\") pod \"payload-processing-e2e-mt-4099cbd3-84f88b85c5-hdbml\" (UID: \"1c4ee35e-073e-4e22-9b22-955fa8c3bc94\") " pod="openshift-ingress/payload-processing-e2e-mt-4099cbd3-84f88b85c5-hdbml"
Nov 02 19:53:20.474261 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:20.474225    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-4099cbd3-84f88b85c5-hdbml"
Nov 02 19:53:20.852194 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:53:20.852158    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c4ee35e_073e_4e22_9b22_955fa8c3bc94.slice/crio-e5142585b9d84bd980ba255ae3ace7f6c39cb1ae28aa47473aef299cb25e74f7 WatchSource:0}: Error finding container e5142585b9d84bd980ba255ae3ace7f6c39cb1ae28aa47473aef299cb25e74f7: Status 404 returned error can't find the container with id e5142585b9d84bd980ba255ae3ace7f6c39cb1ae28aa47473aef299cb25e74f7
Nov 02 19:53:20.854469 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:20.854434    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4099cbd3-84f88b85c5-hdbml"]
Nov 02 19:53:21.815725 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:21.815695    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-4099cbd3-84f88b85c5-hdbml" event={"ID":"1c4ee35e-073e-4e22-9b22-955fa8c3bc94","Type":"ContainerStarted","Data":"db86449d26bf1b61f42879180d671d13ce98a5b2bfd5e15b5f73e489048dcfe4"}
Nov 02 19:53:21.816230 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:21.815753    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-4099cbd3-84f88b85c5-hdbml" event={"ID":"1c4ee35e-073e-4e22-9b22-955fa8c3bc94","Type":"ContainerStarted","Data":"e5142585b9d84bd980ba255ae3ace7f6c39cb1ae28aa47473aef299cb25e74f7"}
Nov 02 19:53:21.816230 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:21.815930    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-4099cbd3-84f88b85c5-hdbml"
Nov 02 19:53:21.924387 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:21.924339    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-4099cbd3-84f88b85c5-hdbml" podStartSLOduration=1.924326403 podStartE2EDuration="1.924326403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:53:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:53:21.919929469 +0000 UTC m=+1508.526963237" watchObservedRunningTime="2026-11-02 19:53:21.924326403 +0000 UTC m=+1508.531360172"
Nov 02 19:53:25.852971 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:25.852939    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-4fd0d5-5b9fb6dd49-cp9lc"]
Nov 02 19:53:25.857617 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:25.857598    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-4fd0d5-5b9fb6dd49-cp9lc"
Nov 02 19:53:25.932068 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:25.929381    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-4fd0d5-5b9fb6dd49-cp9lc"]
Nov 02 19:53:25.933394 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:25.933361    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb9cc982-8fba-4a27-a0c4-71504d971705-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-4fd0d5-5b9fb6dd49-cp9lc\" (UID: \"fb9cc982-8fba-4a27-a0c4-71504d971705\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-4fd0d5-5b9fb6dd49-cp9lc"
Nov 02 19:53:25.933809 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:25.933779    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sj47m\" (UniqueName: \"kubernetes.io/projected/fb9cc982-8fba-4a27-a0c4-71504d971705-kube-api-access-sj47m\") pod \"payload-processing-e2e-shared-a-w5-4fd0d5-5b9fb6dd49-cp9lc\" (UID: \"fb9cc982-8fba-4a27-a0c4-71504d971705\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-4fd0d5-5b9fb6dd49-cp9lc"
Nov 02 19:53:25.959279 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:25.959242    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w5-4fd0d5-dockercfg-dc9zd\""
Nov 02 19:53:25.959448 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:25.959405    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w5-4fd0d5\""
Nov 02 19:53:26.058097 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:26.044329    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb9cc982-8fba-4a27-a0c4-71504d971705-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-4fd0d5-5b9fb6dd49-cp9lc\" (UID: \"fb9cc982-8fba-4a27-a0c4-71504d971705\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-4fd0d5-5b9fb6dd49-cp9lc"
Nov 02 19:53:26.058097 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:26.049156    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb9cc982-8fba-4a27-a0c4-71504d971705-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-4fd0d5-5b9fb6dd49-cp9lc\" (UID: \"fb9cc982-8fba-4a27-a0c4-71504d971705\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-4fd0d5-5b9fb6dd49-cp9lc"
Nov 02 19:53:26.058097 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:26.049477    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj47m\" (UniqueName: \"kubernetes.io/projected/fb9cc982-8fba-4a27-a0c4-71504d971705-kube-api-access-sj47m\") pod \"payload-processing-e2e-shared-a-w5-4fd0d5-5b9fb6dd49-cp9lc\" (UID: \"fb9cc982-8fba-4a27-a0c4-71504d971705\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-4fd0d5-5b9fb6dd49-cp9lc"
Nov 02 19:53:26.103673 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:26.103561    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj47m\" (UniqueName: \"kubernetes.io/projected/fb9cc982-8fba-4a27-a0c4-71504d971705-kube-api-access-sj47m\") pod \"payload-processing-e2e-shared-a-w5-4fd0d5-5b9fb6dd49-cp9lc\" (UID: \"fb9cc982-8fba-4a27-a0c4-71504d971705\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-4fd0d5-5b9fb6dd49-cp9lc"
Nov 02 19:53:26.170709 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:26.170670    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-4fd0d5-5b9fb6dd49-cp9lc"
Nov 02 19:53:26.569086 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:26.569057    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-4fd0d5-5b9fb6dd49-cp9lc"]
Nov 02 19:53:26.569627 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:53:26.569605    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb9cc982_8fba_4a27_a0c4_71504d971705.slice/crio-985f2f9e64be592c74f6e4e83990af302563cc2cd801e59986e6524d371f9862 WatchSource:0}: Error finding container 985f2f9e64be592c74f6e4e83990af302563cc2cd801e59986e6524d371f9862: Status 404 returned error can't find the container with id 985f2f9e64be592c74f6e4e83990af302563cc2cd801e59986e6524d371f9862
Nov 02 19:53:26.850751 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:26.850665    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-4fd0d5-5b9fb6dd49-cp9lc" event={"ID":"fb9cc982-8fba-4a27-a0c4-71504d971705","Type":"ContainerStarted","Data":"882a9118b1def71280d9eefac68a4ec97868197d521bed5497d193d296651d9a"}
Nov 02 19:53:26.850751 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:26.850720    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-4fd0d5-5b9fb6dd49-cp9lc" event={"ID":"fb9cc982-8fba-4a27-a0c4-71504d971705","Type":"ContainerStarted","Data":"985f2f9e64be592c74f6e4e83990af302563cc2cd801e59986e6524d371f9862"}
Nov 02 19:53:26.850751 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:26.850744    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-4fd0d5-5b9fb6dd49-cp9lc"
Nov 02 19:53:26.980501 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:26.980452    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-4fd0d5-5b9fb6dd49-cp9lc" podStartSLOduration=1.980438603 podStartE2EDuration="1.980438603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:53:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:53:26.97646502 +0000 UTC m=+1513.583498788" watchObservedRunningTime="2026-11-02 19:53:26.980438603 +0000 UTC m=+1513.587472372"
Nov 02 19:53:32.826079 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:32.826052    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-4099cbd3-84f88b85c5-hdbml"
Nov 02 19:53:34.296160 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:53:34.296121    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 19:53:37.860955 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:53:37.860928    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-4fd0d5-5b9fb6dd49-cp9lc"
Nov 02 19:54:04.296003 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:54:04.295965    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 19:54:34.295586 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:54:34.295486    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 19:54:35.841377 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:35.841342    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-46f86d84-56b6698b4b-shr75"]
Nov 02 19:54:35.841848 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:35.841537    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-46f86d84-56b6698b4b-shr75" podUID="cfc50f82-a168-48e3-b508-f67232982f1f" containerName="maas-api" containerID="cri-o://c13ab46c1fb0f99ccfcdd25b19e232c86ca0769dc0c2b1936f38160247742b13" gracePeriod=30
Nov 02 19:54:36.155904 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:36.155877    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-46f86d84-56b6698b4b-shr75"
Nov 02 19:54:36.171788 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:36.171753    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cfc50f82-a168-48e3-b508-f67232982f1f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfc50f82-a168-48e3-b508-f67232982f1f-maas-api-tls\") pod \"cfc50f82-a168-48e3-b508-f67232982f1f\" (UID: \"cfc50f82-a168-48e3-b508-f67232982f1f\") "
Nov 02 19:54:36.172084 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:36.171836    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cfc50f82-a168-48e3-b508-f67232982f1f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfc50f82-a168-48e3-b508-f67232982f1f-metrics-certs\") pod \"cfc50f82-a168-48e3-b508-f67232982f1f\" (UID: \"cfc50f82-a168-48e3-b508-f67232982f1f\") "
Nov 02 19:54:36.172084 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:36.171953    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfc50f82-a168-48e3-b508-f67232982f1f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfc50f82-a168-48e3-b508-f67232982f1f-openshift-service-ca\") pod \"cfc50f82-a168-48e3-b508-f67232982f1f\" (UID: \"cfc50f82-a168-48e3-b508-f67232982f1f\") "
Nov 02 19:54:36.172084 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:36.172051    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfc50f82-a168-48e3-b508-f67232982f1f-kube-api-access-4n5dd\" (UniqueName: \"kubernetes.io/projected/cfc50f82-a168-48e3-b508-f67232982f1f-kube-api-access-4n5dd\") pod \"cfc50f82-a168-48e3-b508-f67232982f1f\" (UID: \"cfc50f82-a168-48e3-b508-f67232982f1f\") "
Nov 02 19:54:36.172790 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:36.172759    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfc50f82-a168-48e3-b508-f67232982f1f-openshift-service-ca" pod "cfc50f82-a168-48e3-b508-f67232982f1f" (UID: "cfc50f82-a168-48e3-b508-f67232982f1f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 19:54:36.174353 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:36.174327    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfc50f82-a168-48e3-b508-f67232982f1f-maas-api-tls" pod "cfc50f82-a168-48e3-b508-f67232982f1f" (UID: "cfc50f82-a168-48e3-b508-f67232982f1f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 19:54:36.174734 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:36.174712    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfc50f82-a168-48e3-b508-f67232982f1f-metrics-certs" pod "cfc50f82-a168-48e3-b508-f67232982f1f" (UID: "cfc50f82-a168-48e3-b508-f67232982f1f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 19:54:36.174803 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:36.174760    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfc50f82-a168-48e3-b508-f67232982f1f-kube-api-access-4n5dd" pod "cfc50f82-a168-48e3-b508-f67232982f1f" (UID: "cfc50f82-a168-48e3-b508-f67232982f1f"). InnerVolumeSpecName "kube-api-access-4n5dd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:54:36.283971 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:36.283937    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4n5dd\" (UniqueName: \"kubernetes.io/projected/cfc50f82-a168-48e3-b508-f67232982f1f-kube-api-access-4n5dd\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:54:36.283971 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:36.283967    4463 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfc50f82-a168-48e3-b508-f67232982f1f-maas-api-tls\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:54:36.283971 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:36.283977    4463 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfc50f82-a168-48e3-b508-f67232982f1f-metrics-certs\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:54:36.284283 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:36.283986    4463 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfc50f82-a168-48e3-b508-f67232982f1f-openshift-service-ca\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:54:36.407898 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:36.407751    4463 generic.go:356] "Generic (PLEG): container finished" podID="cfc50f82-a168-48e3-b508-f67232982f1f" containerID="c13ab46c1fb0f99ccfcdd25b19e232c86ca0769dc0c2b1936f38160247742b13" exitCode=0
Nov 02 19:54:36.408144 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:36.407902    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-46f86d84-56b6698b4b-shr75" event={"ID":"cfc50f82-a168-48e3-b508-f67232982f1f","Type":"ContainerDied","Data":"c13ab46c1fb0f99ccfcdd25b19e232c86ca0769dc0c2b1936f38160247742b13"}
Nov 02 19:54:36.408284 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:36.408192    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-46f86d84-56b6698b4b-shr75"
Nov 02 19:54:36.409037 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:36.408412    4463 scope.go:122] "RemoveContainer" containerID="c13ab46c1fb0f99ccfcdd25b19e232c86ca0769dc0c2b1936f38160247742b13"
Nov 02 19:54:36.409245 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:36.407994    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-46f86d84-56b6698b4b-shr75" event={"ID":"cfc50f82-a168-48e3-b508-f67232982f1f","Type":"ContainerDied","Data":"3d74f4a93f682a14212b525a561dac62ae5e0dfac0b1728f0a88a928671aab4e"}
Nov 02 19:54:36.434212 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:36.434180    4463 scope.go:122] "RemoveContainer" containerID="c13ab46c1fb0f99ccfcdd25b19e232c86ca0769dc0c2b1936f38160247742b13"
Nov 02 19:54:36.436583 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:54:36.434593    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c13ab46c1fb0f99ccfcdd25b19e232c86ca0769dc0c2b1936f38160247742b13\": container with ID starting with c13ab46c1fb0f99ccfcdd25b19e232c86ca0769dc0c2b1936f38160247742b13 not found: ID does not exist" containerID="c13ab46c1fb0f99ccfcdd25b19e232c86ca0769dc0c2b1936f38160247742b13"
Nov 02 19:54:36.436583 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:36.434632    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c13ab46c1fb0f99ccfcdd25b19e232c86ca0769dc0c2b1936f38160247742b13"} err="failed to get container status \"c13ab46c1fb0f99ccfcdd25b19e232c86ca0769dc0c2b1936f38160247742b13\": rpc error: code = NotFound desc = could not find container \"c13ab46c1fb0f99ccfcdd25b19e232c86ca0769dc0c2b1936f38160247742b13\": container with ID starting with c13ab46c1fb0f99ccfcdd25b19e232c86ca0769dc0c2b1936f38160247742b13 not found: ID does not exist"
Nov 02 19:54:36.574871 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:36.574828    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-46f86d84-56b6698b4b-shr75"]
Nov 02 19:54:36.578749 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:36.578709    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-46f86d84-56b6698b4b-shr75"]
Nov 02 19:54:37.585860 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:37.585829    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfc50f82-a168-48e3-b508-f67232982f1f" path="/var/lib/kubelet/pods/cfc50f82-a168-48e3-b508-f67232982f1f/volumes"
Nov 02 19:54:40.063979 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:40.063942    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 19:54:43.845956 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:43.845922    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-d1aaed-786fbbccc-zth6x"]
Nov 02 19:54:43.847577 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:43.847552    4463 cpu_manager.go:414] "containerMap: removing container" podUID="cfc50f82-a168-48e3-b508-f67232982f1f" containerName="maas-api"
Nov 02 19:54:43.847713 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:43.847700    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfc50f82-a168-48e3-b508-f67232982f1f" containerName="maas-api"
Nov 02 19:54:43.847990 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:43.847975    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfc50f82-a168-48e3-b508-f67232982f1f" containerName="maas-api"
Nov 02 19:54:43.928275 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:43.926952    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-d1aaed-786fbbccc-zth6x"]
Nov 02 19:54:43.928275 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:43.928232    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-d1aaed-786fbbccc-zth6x"
Nov 02 19:54:43.935645 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:43.932389    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66mft\" (UniqueName: \"kubernetes.io/projected/86b84714-1e20-47b0-9a7b-f289c2589ffc-kube-api-access-66mft\") pod \"payload-processing-e2e-shared-a-w6-d1aaed-786fbbccc-zth6x\" (UID: \"86b84714-1e20-47b0-9a7b-f289c2589ffc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-d1aaed-786fbbccc-zth6x"
Nov 02 19:54:43.935645 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:43.932518    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86b84714-1e20-47b0-9a7b-f289c2589ffc-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-d1aaed-786fbbccc-zth6x\" (UID: \"86b84714-1e20-47b0-9a7b-f289c2589ffc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-d1aaed-786fbbccc-zth6x"
Nov 02 19:54:43.942402 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:43.942372    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-d1aaed-79985b544c-c8zc9"]
Nov 02 19:54:43.948193 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:43.948171    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-d1aaed-79985b544c-c8zc9"]
Nov 02 19:54:43.948446 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:43.948429    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-d1aaed-79985b544c-c8zc9"
Nov 02 19:54:43.982037 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:43.981569    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w6-d1aaed-dockercfg-b56qp\""
Nov 02 19:54:43.982037 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:43.981676    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w6-d1aaed\""
Nov 02 19:54:44.049667 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:44.043402    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66mft\" (UniqueName: \"kubernetes.io/projected/86b84714-1e20-47b0-9a7b-f289c2589ffc-kube-api-access-66mft\") pod \"payload-processing-e2e-shared-a-w6-d1aaed-786fbbccc-zth6x\" (UID: \"86b84714-1e20-47b0-9a7b-f289c2589ffc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-d1aaed-786fbbccc-zth6x"
Nov 02 19:54:44.049667 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:44.043518    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86b84714-1e20-47b0-9a7b-f289c2589ffc-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-d1aaed-786fbbccc-zth6x\" (UID: \"86b84714-1e20-47b0-9a7b-f289c2589ffc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-d1aaed-786fbbccc-zth6x"
Nov 02 19:54:44.049667 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:44.043730    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfkfj\" (UniqueName: \"kubernetes.io/projected/9fbc861b-be1c-460c-833d-478346a9b840-kube-api-access-gfkfj\") pod \"payload-pre-processing-e2e-shared-a-w6-d1aaed-79985b544c-c8zc9\" (UID: \"9fbc861b-be1c-460c-833d-478346a9b840\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-d1aaed-79985b544c-c8zc9"
Nov 02 19:54:44.049667 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:44.043768    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fbc861b-be1c-460c-833d-478346a9b840-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-d1aaed-79985b544c-c8zc9\" (UID: \"9fbc861b-be1c-460c-833d-478346a9b840\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-d1aaed-79985b544c-c8zc9"
Nov 02 19:54:44.049667 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:44.045085    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86b84714-1e20-47b0-9a7b-f289c2589ffc-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-d1aaed-786fbbccc-zth6x\" (UID: \"86b84714-1e20-47b0-9a7b-f289c2589ffc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-d1aaed-786fbbccc-zth6x"
Nov 02 19:54:44.127813 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:44.127728    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66mft\" (UniqueName: \"kubernetes.io/projected/86b84714-1e20-47b0-9a7b-f289c2589ffc-kube-api-access-66mft\") pod \"payload-processing-e2e-shared-a-w6-d1aaed-786fbbccc-zth6x\" (UID: \"86b84714-1e20-47b0-9a7b-f289c2589ffc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-d1aaed-786fbbccc-zth6x"
Nov 02 19:54:44.145827 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:44.145797    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfkfj\" (UniqueName: \"kubernetes.io/projected/9fbc861b-be1c-460c-833d-478346a9b840-kube-api-access-gfkfj\") pod \"payload-pre-processing-e2e-shared-a-w6-d1aaed-79985b544c-c8zc9\" (UID: \"9fbc861b-be1c-460c-833d-478346a9b840\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-d1aaed-79985b544c-c8zc9"
Nov 02 19:54:44.145958 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:44.145840    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fbc861b-be1c-460c-833d-478346a9b840-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-d1aaed-79985b544c-c8zc9\" (UID: \"9fbc861b-be1c-460c-833d-478346a9b840\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-d1aaed-79985b544c-c8zc9"
Nov 02 19:54:44.146698 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:44.146674    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fbc861b-be1c-460c-833d-478346a9b840-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-d1aaed-79985b544c-c8zc9\" (UID: \"9fbc861b-be1c-460c-833d-478346a9b840\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-d1aaed-79985b544c-c8zc9"
Nov 02 19:54:44.208273 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:44.208214    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfkfj\" (UniqueName: \"kubernetes.io/projected/9fbc861b-be1c-460c-833d-478346a9b840-kube-api-access-gfkfj\") pod \"payload-pre-processing-e2e-shared-a-w6-d1aaed-79985b544c-c8zc9\" (UID: \"9fbc861b-be1c-460c-833d-478346a9b840\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-d1aaed-79985b544c-c8zc9"
Nov 02 19:54:44.371300 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:44.370080    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-d1aaed-786fbbccc-zth6x"
Nov 02 19:54:44.431594 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:44.431569    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-d1aaed-79985b544c-c8zc9"
Nov 02 19:54:44.834901 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:44.834875    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-d1aaed-786fbbccc-zth6x"]
Nov 02 19:54:44.835993 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:54:44.835965    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86b84714_1e20_47b0_9a7b_f289c2589ffc.slice/crio-1994751112eb6e39a559f0ba6b6c28483c6b9eb3bbf23e227ac329b1a94b6ad1 WatchSource:0}: Error finding container 1994751112eb6e39a559f0ba6b6c28483c6b9eb3bbf23e227ac329b1a94b6ad1: Status 404 returned error can't find the container with id 1994751112eb6e39a559f0ba6b6c28483c6b9eb3bbf23e227ac329b1a94b6ad1
Nov 02 19:54:44.915620 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:54:44.915589    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fbc861b_be1c_460c_833d_478346a9b840.slice/crio-441503211c7add3123959b3b80c44c8e699a642071e63f5b255c90c7e1355837 WatchSource:0}: Error finding container 441503211c7add3123959b3b80c44c8e699a642071e63f5b255c90c7e1355837: Status 404 returned error can't find the container with id 441503211c7add3123959b3b80c44c8e699a642071e63f5b255c90c7e1355837
Nov 02 19:54:44.917236 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:44.917210    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-d1aaed-79985b544c-c8zc9"]
Nov 02 19:54:45.821374 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:45.821337    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-d1aaed-79985b544c-c8zc9" event={"ID":"9fbc861b-be1c-460c-833d-478346a9b840","Type":"ContainerStarted","Data":"c1b4faf6d65ba71bc0e1393d32414f0ad9fb529abad71d9c238f73cd168a67c9"}
Nov 02 19:54:45.821588 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:45.821395    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-d1aaed-79985b544c-c8zc9" event={"ID":"9fbc861b-be1c-460c-833d-478346a9b840","Type":"ContainerStarted","Data":"441503211c7add3123959b3b80c44c8e699a642071e63f5b255c90c7e1355837"}
Nov 02 19:54:45.823065 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:45.823036    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-d1aaed-786fbbccc-zth6x" event={"ID":"86b84714-1e20-47b0-9a7b-f289c2589ffc","Type":"ContainerStarted","Data":"face98f556aa2e473144f6751cb1769deeffe4b5c50c1eab576f0303fc1956d6"}
Nov 02 19:54:45.823194 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:45.823097    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-d1aaed-786fbbccc-zth6x" event={"ID":"86b84714-1e20-47b0-9a7b-f289c2589ffc","Type":"ContainerStarted","Data":"1994751112eb6e39a559f0ba6b6c28483c6b9eb3bbf23e227ac329b1a94b6ad1"}
Nov 02 19:54:45.823194 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:45.823118    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-d1aaed-786fbbccc-zth6x"
Nov 02 19:54:45.946329 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:45.945648    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-d1aaed-79985b544c-c8zc9" podStartSLOduration=2.945632119 podStartE2EDuration="2.945632119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:54:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:54:45.943168274 +0000 UTC m=+1592.550202042" watchObservedRunningTime="2026-11-02 19:54:45.945632119 +0000 UTC m=+1592.552665935"
Nov 02 19:54:46.101155 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:46.101039    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-d1aaed-786fbbccc-zth6x" podStartSLOduration=3.101003978 podStartE2EDuration="3.101003978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:54:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:54:46.068794868 +0000 UTC m=+1592.675828674" watchObservedRunningTime="2026-11-02 19:54:46.101003978 +0000 UTC m=+1592.708037748"
Nov 02 19:54:46.829475 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:46.829438    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-d1aaed-79985b544c-c8zc9"
Nov 02 19:54:51.867249 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:51.867166    4463 generic.go:356] "Generic (PLEG): container finished" podID="44a12c0e-e3df-4368-a5f8-61c3b8f88da6" containerID="ad3d9c3c6ac7afe5b3d137580b7014e71578c849b53231bb9f5635ff39853c38" exitCode=0
Nov 02 19:54:51.867249 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:51.867237    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4099cbd3-openshift-default-698ddb8d77-lrzxg" event={"ID":"44a12c0e-e3df-4368-a5f8-61c3b8f88da6","Type":"ContainerDied","Data":"ad3d9c3c6ac7afe5b3d137580b7014e71578c849b53231bb9f5635ff39853c38"}
Nov 02 19:54:51.867772 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:51.867754    4463 scope.go:122] "RemoveContainer" containerID="ad3d9c3c6ac7afe5b3d137580b7014e71578c849b53231bb9f5635ff39853c38"
Nov 02 19:54:51.869145 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:51.869121    4463 generic.go:356] "Generic (PLEG): container finished" podID="c6d73c91-e211-41f4-a929-d3e45bc82366" containerID="1bee209fcbd131dde8b21eea904bd5824cfd810296e9501ca02ac775c223f888" exitCode=0
Nov 02 19:54:51.869250 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:51.869147    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-4fd0d5-openshift-default-8566cd5547-tvdjk" event={"ID":"c6d73c91-e211-41f4-a929-d3e45bc82366","Type":"ContainerDied","Data":"1bee209fcbd131dde8b21eea904bd5824cfd810296e9501ca02ac775c223f888"}
Nov 02 19:54:51.869419 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:51.869393    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:54:51.869474 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:51.869461    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:54:51.869515 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:51.869495    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:54:51.869515 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:51.869503    4463 scope.go:122] "RemoveContainer" containerID="1bee209fcbd131dde8b21eea904bd5824cfd810296e9501ca02ac775c223f888"
Nov 02 19:54:51.871171 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:51.871134    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:54:51.871278 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:51.871217    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:54:51.871278 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:51.871259    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:54:52.895703 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:52.895657    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4099cbd3-openshift-default-698ddb8d77-lrzxg" event={"ID":"44a12c0e-e3df-4368-a5f8-61c3b8f88da6","Type":"ContainerStarted","Data":"db0cdeba200bc7d15dc63faa761ccde5dc2e4d6dc3421b76262678b424d893c6"}
Nov 02 19:54:52.901586 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:52.901553    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-4fd0d5-openshift-default-8566cd5547-tvdjk" event={"ID":"c6d73c91-e211-41f4-a929-d3e45bc82366","Type":"ContainerStarted","Data":"40104457ee4e5fee6490909a12c8470adb9202518cc38a948753eb994ebdf514"}
Nov 02 19:54:53.100749 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:53.100709    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-4099cbd3-openshift-default-698ddb8d77-lrzxg"
Nov 02 19:54:53.101238 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:53.101208    4463 patch_prober.go:29] interesting pod/e2e-mt-4099cbd3-openshift-default-698ddb8d77-lrzxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 02 19:54:53.101447 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:53.101421    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4099cbd3-openshift-default-698ddb8d77-lrzxg" podUID="44a12c0e-e3df-4368-a5f8-61c3b8f88da6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 02 19:54:53.369453 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:53.369417    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-4fd0d5-openshift-default-8566cd5547-tvdjk"
Nov 02 19:54:53.369841 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:53.369746    4463 patch_prober.go:29] interesting pod/e2e-shared-a-w5-4fd0d5-openshift-default-8566cd5547-tvdjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Nov 02 19:54:53.370086 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:53.370060    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-4fd0d5-openshift-default-8566cd5547-tvdjk" podUID="c6d73c91-e211-41f4-a929-d3e45bc82366" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Nov 02 19:54:54.102040 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:54.101394    4463 patch_prober.go:29] interesting pod/e2e-mt-4099cbd3-openshift-default-698ddb8d77-lrzxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 02 19:54:54.102040 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:54.101481    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4099cbd3-openshift-default-698ddb8d77-lrzxg" podUID="44a12c0e-e3df-4368-a5f8-61c3b8f88da6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 02 19:54:54.364193 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:54.364106    4463 patch_prober.go:29] interesting pod/e2e-shared-a-w5-4fd0d5-openshift-default-8566cd5547-tvdjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Nov 02 19:54:54.365057 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:54.365004    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-4fd0d5-openshift-default-8566cd5547-tvdjk" podUID="c6d73c91-e211-41f4-a929-d3e45bc82366" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Nov 02 19:54:55.103036 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:55.102330    4463 patch_prober.go:29] interesting pod/e2e-mt-4099cbd3-openshift-default-698ddb8d77-lrzxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 02 19:54:55.103036 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:55.102387    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4099cbd3-openshift-default-698ddb8d77-lrzxg" podUID="44a12c0e-e3df-4368-a5f8-61c3b8f88da6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 02 19:54:55.370484 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:55.370381    4463 patch_prober.go:29] interesting pod/e2e-shared-a-w5-4fd0d5-openshift-default-8566cd5547-tvdjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Nov 02 19:54:55.370484 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:55.370463    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-4fd0d5-openshift-default-8566cd5547-tvdjk" podUID="c6d73c91-e211-41f4-a929-d3e45bc82366" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Nov 02 19:54:56.100565 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:56.100520    4463 patch_prober.go:29] interesting pod/e2e-mt-4099cbd3-openshift-default-698ddb8d77-lrzxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 02 19:54:56.100753 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:56.100592    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4099cbd3-openshift-default-698ddb8d77-lrzxg" podUID="44a12c0e-e3df-4368-a5f8-61c3b8f88da6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 02 19:54:56.375093 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:56.372419    4463 patch_prober.go:29] interesting pod/e2e-shared-a-w5-4fd0d5-openshift-default-8566cd5547-tvdjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Nov 02 19:54:56.375093 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:56.372496    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-4fd0d5-openshift-default-8566cd5547-tvdjk" podUID="c6d73c91-e211-41f4-a929-d3e45bc82366" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Nov 02 19:54:56.831815 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:56.831782    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-d1aaed-786fbbccc-zth6x"
Nov 02 19:54:57.111394 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:57.108845    4463 patch_prober.go:29] interesting pod/e2e-mt-4099cbd3-openshift-default-698ddb8d77-lrzxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 02 19:54:57.111394 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:57.108921    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4099cbd3-openshift-default-698ddb8d77-lrzxg" podUID="44a12c0e-e3df-4368-a5f8-61c3b8f88da6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 02 19:54:57.376094 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:57.374411    4463 patch_prober.go:29] interesting pod/e2e-shared-a-w5-4fd0d5-openshift-default-8566cd5547-tvdjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Nov 02 19:54:57.376094 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:57.374471    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-4fd0d5-openshift-default-8566cd5547-tvdjk" podUID="c6d73c91-e211-41f4-a929-d3e45bc82366" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Nov 02 19:54:57.840878 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:57.840848    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-d1aaed-79985b544c-c8zc9"
Nov 02 19:54:58.110107 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:58.109520    4463 patch_prober.go:29] interesting pod/e2e-mt-4099cbd3-openshift-default-698ddb8d77-lrzxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 02 19:54:58.110107 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:58.109597    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4099cbd3-openshift-default-698ddb8d77-lrzxg" podUID="44a12c0e-e3df-4368-a5f8-61c3b8f88da6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 02 19:54:58.363057 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:58.362955    4463 patch_prober.go:29] interesting pod/e2e-shared-a-w5-4fd0d5-openshift-default-8566cd5547-tvdjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Nov 02 19:54:58.363057 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:58.363007    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-4fd0d5-openshift-default-8566cd5547-tvdjk" podUID="c6d73c91-e211-41f4-a929-d3e45bc82366" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Nov 02 19:54:58.712931 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:58.712900    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-4099cbd3-openshift-default-698ddb8d77-lrzxg"
Nov 02 19:54:59.105344 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:59.105261    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-4099cbd3-openshift-default-698ddb8d77-lrzxg"
Nov 02 19:54:59.106241 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:59.106219    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-4099cbd3-openshift-default-698ddb8d77-lrzxg"
Nov 02 19:54:59.366935 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:59.366865    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-4fd0d5-openshift-default-8566cd5547-tvdjk"
Nov 02 19:54:59.367251 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:59.367232    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-4fd0d5-openshift-default-8566cd5547-tvdjk"
Nov 02 19:54:59.367842 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:54:59.367825    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-4fd0d5-openshift-default-8566cd5547-tvdjk"
Nov 02 19:55:04.294601 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:55:04.294565    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 19:55:15.968448 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:15.968407    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 19:55:19.878695 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:19.877090    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8795bf/distinct-w4-8795bf-kserve-64b4796df4-8qgqh" podUID="ee7d28f4-e9c7-43e2-b983-1427589f1785" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:55:21.117407 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:21.117371    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-q9v28"]
Nov 02 19:55:21.117859 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:21.117573    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-q9v28" podUID="b9bfbdea-9214-414e-84af-e5c9ea6406ff" containerName="istio-proxy" containerID="cri-o://65a2df16c2f73565703bcf2da8604827789c6fe32c195fd1064ff05c05221c32" gracePeriod=30
Nov 02 19:55:21.163488 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:21.163464    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-8mkqq"]
Nov 02 19:55:21.163670 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:21.163650    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-8mkqq" podUID="dca3f84e-1191-48bb-a59c-8456088614ac" containerName="istio-proxy" containerID="cri-o://3b618c0735a004eb627ce924fb547cfcf4ed904ae3d9732bf021b460a04c2d8b" gracePeriod=30
Nov 02 19:55:21.167760 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:21.167735    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-w5gw2"]
Nov 02 19:55:21.167952 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:21.167931    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-w5gw2" podUID="d09ad81c-4422-4b1a-b845-a616f5c9e93c" containerName="istio-proxy" containerID="cri-o://a02110035b528ab4aad6fd5a33abe1f2af48f93bd083a52279beddd7e1521b01" gracePeriod=30
Nov 02 19:55:21.170772 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:21.170750    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9ndrz"]
Nov 02 19:55:21.170950 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:21.170931    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9ndrz" podUID="f5734864-6a35-46c3-80b8-bdac768e1de1" containerName="istio-proxy" containerID="cri-o://916aecd458917f4668c09a0921566e17197367f37bcc3dd208817f4d230a20c4" gracePeriod=30
Nov 02 19:55:25.279834 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:25.279789    4463 patch_prober.go:29] interesting pod/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9ndrz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 19:55:25.280331 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:25.279855    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9ndrz" podUID="f5734864-6a35-46c3-80b8-bdac768e1de1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 19:55:26.470120 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.470089    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-q9v28"
Nov 02 19:55:26.508376 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.508297    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-8mkqq"
Nov 02 19:55:26.545075 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.545005    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9bfbdea-9214-414e-84af-e5c9ea6406ff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9bfbdea-9214-414e-84af-e5c9ea6406ff-istio-data\") pod \"b9bfbdea-9214-414e-84af-e5c9ea6406ff\" (UID: \"b9bfbdea-9214-414e-84af-e5c9ea6406ff\") "
Nov 02 19:55:26.545266 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.545209    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9bfbdea-9214-414e-84af-e5c9ea6406ff-istio-token\" (UniqueName: \"kubernetes.io/projected/b9bfbdea-9214-414e-84af-e5c9ea6406ff-istio-token\") pod \"b9bfbdea-9214-414e-84af-e5c9ea6406ff\" (UID: \"b9bfbdea-9214-414e-84af-e5c9ea6406ff\") "
Nov 02 19:55:26.545329 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.545266    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9bfbdea-9214-414e-84af-e5c9ea6406ff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9bfbdea-9214-414e-84af-e5c9ea6406ff-istiod-ca-cert\") pod \"b9bfbdea-9214-414e-84af-e5c9ea6406ff\" (UID: \"b9bfbdea-9214-414e-84af-e5c9ea6406ff\") "
Nov 02 19:55:26.545329 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.545288    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9bfbdea-9214-414e-84af-e5c9ea6406ff-istio-data" pod "b9bfbdea-9214-414e-84af-e5c9ea6406ff" (UID: "b9bfbdea-9214-414e-84af-e5c9ea6406ff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:55:26.545329 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.545316    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b9bfbdea-9214-414e-84af-e5c9ea6406ff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9bfbdea-9214-414e-84af-e5c9ea6406ff-istio-podinfo\") pod \"b9bfbdea-9214-414e-84af-e5c9ea6406ff\" (UID: \"b9bfbdea-9214-414e-84af-e5c9ea6406ff\") "
Nov 02 19:55:26.545512 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.545494    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9bfbdea-9214-414e-84af-e5c9ea6406ff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9bfbdea-9214-414e-84af-e5c9ea6406ff-istio-envoy\") pod \"b9bfbdea-9214-414e-84af-e5c9ea6406ff\" (UID: \"b9bfbdea-9214-414e-84af-e5c9ea6406ff\") "
Nov 02 19:55:26.545621 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.545594    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9bfbdea-9214-414e-84af-e5c9ea6406ff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9bfbdea-9214-414e-84af-e5c9ea6406ff-credential-socket\") pod \"b9bfbdea-9214-414e-84af-e5c9ea6406ff\" (UID: \"b9bfbdea-9214-414e-84af-e5c9ea6406ff\") "
Nov 02 19:55:26.545697 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.545666    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9bfbdea-9214-414e-84af-e5c9ea6406ff-kube-api-access-t4xhh\" (UniqueName: \"kubernetes.io/projected/b9bfbdea-9214-414e-84af-e5c9ea6406ff-kube-api-access-t4xhh\") pod \"b9bfbdea-9214-414e-84af-e5c9ea6406ff\" (UID: \"b9bfbdea-9214-414e-84af-e5c9ea6406ff\") "
Nov 02 19:55:26.545753 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.545735    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9bfbdea-9214-414e-84af-e5c9ea6406ff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9bfbdea-9214-414e-84af-e5c9ea6406ff-workload-certs\") pod \"b9bfbdea-9214-414e-84af-e5c9ea6406ff\" (UID: \"b9bfbdea-9214-414e-84af-e5c9ea6406ff\") "
Nov 02 19:55:26.545807 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.545775    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9bfbdea-9214-414e-84af-e5c9ea6406ff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9bfbdea-9214-414e-84af-e5c9ea6406ff-workload-socket\") pod \"b9bfbdea-9214-414e-84af-e5c9ea6406ff\" (UID: \"b9bfbdea-9214-414e-84af-e5c9ea6406ff\") "
Nov 02 19:55:26.546685 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.546532    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9bfbdea-9214-414e-84af-e5c9ea6406ff-credential-socket" pod "b9bfbdea-9214-414e-84af-e5c9ea6406ff" (UID: "b9bfbdea-9214-414e-84af-e5c9ea6406ff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:55:26.546685 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.546547    4463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9bfbdea-9214-414e-84af-e5c9ea6406ff-istio-data\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:55:26.546685 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.546640    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9bfbdea-9214-414e-84af-e5c9ea6406ff-workload-certs" pod "b9bfbdea-9214-414e-84af-e5c9ea6406ff" (UID: "b9bfbdea-9214-414e-84af-e5c9ea6406ff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:55:26.546957 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.546838    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9bfbdea-9214-414e-84af-e5c9ea6406ff-workload-socket" pod "b9bfbdea-9214-414e-84af-e5c9ea6406ff" (UID: "b9bfbdea-9214-414e-84af-e5c9ea6406ff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:55:26.547482 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.547456    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9bfbdea-9214-414e-84af-e5c9ea6406ff-istiod-ca-cert" pod "b9bfbdea-9214-414e-84af-e5c9ea6406ff" (UID: "b9bfbdea-9214-414e-84af-e5c9ea6406ff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 19:55:26.548158 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.548132    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9bfbdea-9214-414e-84af-e5c9ea6406ff-istio-token" pod "b9bfbdea-9214-414e-84af-e5c9ea6406ff" (UID: "b9bfbdea-9214-414e-84af-e5c9ea6406ff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:55:26.548280 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.548242    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9bfbdea-9214-414e-84af-e5c9ea6406ff-kube-api-access-t4xhh" pod "b9bfbdea-9214-414e-84af-e5c9ea6406ff" (UID: "b9bfbdea-9214-414e-84af-e5c9ea6406ff"). InnerVolumeSpecName "kube-api-access-t4xhh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:55:26.549371 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.549350    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b9bfbdea-9214-414e-84af-e5c9ea6406ff-istio-podinfo" pod "b9bfbdea-9214-414e-84af-e5c9ea6406ff" (UID: "b9bfbdea-9214-414e-84af-e5c9ea6406ff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 19:55:26.549764 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.549740    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9bfbdea-9214-414e-84af-e5c9ea6406ff-istio-envoy" pod "b9bfbdea-9214-414e-84af-e5c9ea6406ff" (UID: "b9bfbdea-9214-414e-84af-e5c9ea6406ff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:55:26.571064 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.571039    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-w5gw2"
Nov 02 19:55:26.578440 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.578415    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9ndrz"
Nov 02 19:55:26.648097 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.647985    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dca3f84e-1191-48bb-a59c-8456088614ac-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dca3f84e-1191-48bb-a59c-8456088614ac-istio-podinfo\") pod \"dca3f84e-1191-48bb-a59c-8456088614ac\" (UID: \"dca3f84e-1191-48bb-a59c-8456088614ac\") "
Nov 02 19:55:26.648097 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.648069    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d09ad81c-4422-4b1a-b845-a616f5c9e93c-istio-token\" (UniqueName: \"kubernetes.io/projected/d09ad81c-4422-4b1a-b845-a616f5c9e93c-istio-token\") pod \"d09ad81c-4422-4b1a-b845-a616f5c9e93c\" (UID: \"d09ad81c-4422-4b1a-b845-a616f5c9e93c\") "
Nov 02 19:55:26.648334 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.648118    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dca3f84e-1191-48bb-a59c-8456088614ac-kube-api-access-hllx4\" (UniqueName: \"kubernetes.io/projected/dca3f84e-1191-48bb-a59c-8456088614ac-kube-api-access-hllx4\") pod \"dca3f84e-1191-48bb-a59c-8456088614ac\" (UID: \"dca3f84e-1191-48bb-a59c-8456088614ac\") "
Nov 02 19:55:26.648334 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.648164    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d09ad81c-4422-4b1a-b845-a616f5c9e93c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d09ad81c-4422-4b1a-b845-a616f5c9e93c-istio-data\") pod \"d09ad81c-4422-4b1a-b845-a616f5c9e93c\" (UID: \"d09ad81c-4422-4b1a-b845-a616f5c9e93c\") "
Nov 02 19:55:26.648334 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.648259    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dca3f84e-1191-48bb-a59c-8456088614ac-istio-token\" (UniqueName: \"kubernetes.io/projected/dca3f84e-1191-48bb-a59c-8456088614ac-istio-token\") pod \"dca3f84e-1191-48bb-a59c-8456088614ac\" (UID: \"dca3f84e-1191-48bb-a59c-8456088614ac\") "
Nov 02 19:55:26.648508 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.648382    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5734864-6a35-46c3-80b8-bdac768e1de1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5734864-6a35-46c3-80b8-bdac768e1de1-workload-socket\") pod \"f5734864-6a35-46c3-80b8-bdac768e1de1\" (UID: \"f5734864-6a35-46c3-80b8-bdac768e1de1\") "
Nov 02 19:55:26.648508 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.648428    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dca3f84e-1191-48bb-a59c-8456088614ac-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dca3f84e-1191-48bb-a59c-8456088614ac-istio-data\") pod \"dca3f84e-1191-48bb-a59c-8456088614ac\" (UID: \"dca3f84e-1191-48bb-a59c-8456088614ac\") "
Nov 02 19:55:26.648508 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.648458    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d09ad81c-4422-4b1a-b845-a616f5c9e93c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d09ad81c-4422-4b1a-b845-a616f5c9e93c-istio-podinfo\") pod \"d09ad81c-4422-4b1a-b845-a616f5c9e93c\" (UID: \"d09ad81c-4422-4b1a-b845-a616f5c9e93c\") "
Nov 02 19:55:26.648508 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.648496    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d09ad81c-4422-4b1a-b845-a616f5c9e93c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d09ad81c-4422-4b1a-b845-a616f5c9e93c-credential-socket\") pod \"d09ad81c-4422-4b1a-b845-a616f5c9e93c\" (UID: \"d09ad81c-4422-4b1a-b845-a616f5c9e93c\") "
Nov 02 19:55:26.648710 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.648533    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dca3f84e-1191-48bb-a59c-8456088614ac-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dca3f84e-1191-48bb-a59c-8456088614ac-istio-envoy\") pod \"dca3f84e-1191-48bb-a59c-8456088614ac\" (UID: \"dca3f84e-1191-48bb-a59c-8456088614ac\") "
Nov 02 19:55:26.648710 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.648572    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dca3f84e-1191-48bb-a59c-8456088614ac-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dca3f84e-1191-48bb-a59c-8456088614ac-credential-socket\") pod \"dca3f84e-1191-48bb-a59c-8456088614ac\" (UID: \"dca3f84e-1191-48bb-a59c-8456088614ac\") "
Nov 02 19:55:26.648710 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.648606    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5734864-6a35-46c3-80b8-bdac768e1de1-istio-token\" (UniqueName: \"kubernetes.io/projected/f5734864-6a35-46c3-80b8-bdac768e1de1-istio-token\") pod \"f5734864-6a35-46c3-80b8-bdac768e1de1\" (UID: \"f5734864-6a35-46c3-80b8-bdac768e1de1\") "
Nov 02 19:55:26.648710 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.648618    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d09ad81c-4422-4b1a-b845-a616f5c9e93c-istio-data" pod "d09ad81c-4422-4b1a-b845-a616f5c9e93c" (UID: "d09ad81c-4422-4b1a-b845-a616f5c9e93c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:55:26.648710 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.648644    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5734864-6a35-46c3-80b8-bdac768e1de1-kube-api-access-tsfwd\" (UniqueName: \"kubernetes.io/projected/f5734864-6a35-46c3-80b8-bdac768e1de1-kube-api-access-tsfwd\") pod \"f5734864-6a35-46c3-80b8-bdac768e1de1\" (UID: \"f5734864-6a35-46c3-80b8-bdac768e1de1\") "
Nov 02 19:55:26.648710 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.648672    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d09ad81c-4422-4b1a-b845-a616f5c9e93c-kube-api-access-hphl6\" (UniqueName: \"kubernetes.io/projected/d09ad81c-4422-4b1a-b845-a616f5c9e93c-kube-api-access-hphl6\") pod \"d09ad81c-4422-4b1a-b845-a616f5c9e93c\" (UID: \"d09ad81c-4422-4b1a-b845-a616f5c9e93c\") "
Nov 02 19:55:26.649084 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.648718    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d09ad81c-4422-4b1a-b845-a616f5c9e93c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d09ad81c-4422-4b1a-b845-a616f5c9e93c-istiod-ca-cert\") pod \"d09ad81c-4422-4b1a-b845-a616f5c9e93c\" (UID: \"d09ad81c-4422-4b1a-b845-a616f5c9e93c\") "
Nov 02 19:55:26.649084 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.648792    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5734864-6a35-46c3-80b8-bdac768e1de1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5734864-6a35-46c3-80b8-bdac768e1de1-istio-envoy\") pod \"f5734864-6a35-46c3-80b8-bdac768e1de1\" (UID: \"f5734864-6a35-46c3-80b8-bdac768e1de1\") "
Nov 02 19:55:26.649084 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.648818    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dca3f84e-1191-48bb-a59c-8456088614ac-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dca3f84e-1191-48bb-a59c-8456088614ac-istiod-ca-cert\") pod \"dca3f84e-1191-48bb-a59c-8456088614ac\" (UID: \"dca3f84e-1191-48bb-a59c-8456088614ac\") "
Nov 02 19:55:26.649084 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.648861    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f5734864-6a35-46c3-80b8-bdac768e1de1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5734864-6a35-46c3-80b8-bdac768e1de1-istio-podinfo\") pod \"f5734864-6a35-46c3-80b8-bdac768e1de1\" (UID: \"f5734864-6a35-46c3-80b8-bdac768e1de1\") "
Nov 02 19:55:26.649084 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.648911    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5734864-6a35-46c3-80b8-bdac768e1de1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5734864-6a35-46c3-80b8-bdac768e1de1-credential-socket\") pod \"f5734864-6a35-46c3-80b8-bdac768e1de1\" (UID: \"f5734864-6a35-46c3-80b8-bdac768e1de1\") "
Nov 02 19:55:26.649084 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.648945    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5734864-6a35-46c3-80b8-bdac768e1de1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5734864-6a35-46c3-80b8-bdac768e1de1-istiod-ca-cert\") pod \"f5734864-6a35-46c3-80b8-bdac768e1de1\" (UID: \"f5734864-6a35-46c3-80b8-bdac768e1de1\") "
Nov 02 19:55:26.649084 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.648979    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5734864-6a35-46c3-80b8-bdac768e1de1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5734864-6a35-46c3-80b8-bdac768e1de1-workload-certs\") pod \"f5734864-6a35-46c3-80b8-bdac768e1de1\" (UID: \"f5734864-6a35-46c3-80b8-bdac768e1de1\") "
Nov 02 19:55:26.649084 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.649070    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5734864-6a35-46c3-80b8-bdac768e1de1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5734864-6a35-46c3-80b8-bdac768e1de1-istio-data\") pod \"f5734864-6a35-46c3-80b8-bdac768e1de1\" (UID: \"f5734864-6a35-46c3-80b8-bdac768e1de1\") "
Nov 02 19:55:26.649505 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.649131    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dca3f84e-1191-48bb-a59c-8456088614ac-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dca3f84e-1191-48bb-a59c-8456088614ac-workload-certs\") pod \"dca3f84e-1191-48bb-a59c-8456088614ac\" (UID: \"dca3f84e-1191-48bb-a59c-8456088614ac\") "
Nov 02 19:55:26.649505 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.649210    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d09ad81c-4422-4b1a-b845-a616f5c9e93c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d09ad81c-4422-4b1a-b845-a616f5c9e93c-workload-certs\") pod \"d09ad81c-4422-4b1a-b845-a616f5c9e93c\" (UID: \"d09ad81c-4422-4b1a-b845-a616f5c9e93c\") "
Nov 02 19:55:26.649505 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.649290    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d09ad81c-4422-4b1a-b845-a616f5c9e93c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d09ad81c-4422-4b1a-b845-a616f5c9e93c-istio-envoy\") pod \"d09ad81c-4422-4b1a-b845-a616f5c9e93c\" (UID: \"d09ad81c-4422-4b1a-b845-a616f5c9e93c\") "
Nov 02 19:55:26.649505 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.649349    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d09ad81c-4422-4b1a-b845-a616f5c9e93c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d09ad81c-4422-4b1a-b845-a616f5c9e93c-workload-socket\") pod \"d09ad81c-4422-4b1a-b845-a616f5c9e93c\" (UID: \"d09ad81c-4422-4b1a-b845-a616f5c9e93c\") "
Nov 02 19:55:26.649505 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.649390    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dca3f84e-1191-48bb-a59c-8456088614ac-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dca3f84e-1191-48bb-a59c-8456088614ac-workload-socket\") pod \"dca3f84e-1191-48bb-a59c-8456088614ac\" (UID: \"dca3f84e-1191-48bb-a59c-8456088614ac\") "
Nov 02 19:55:26.649765 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.649636    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5734864-6a35-46c3-80b8-bdac768e1de1-workload-certs" pod "f5734864-6a35-46c3-80b8-bdac768e1de1" (UID: "f5734864-6a35-46c3-80b8-bdac768e1de1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:55:26.649881 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.649857    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dca3f84e-1191-48bb-a59c-8456088614ac-workload-socket" pod "dca3f84e-1191-48bb-a59c-8456088614ac" (UID: "dca3f84e-1191-48bb-a59c-8456088614ac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:55:26.650194 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.650168    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5734864-6a35-46c3-80b8-bdac768e1de1-istio-data" pod "f5734864-6a35-46c3-80b8-bdac768e1de1" (UID: "f5734864-6a35-46c3-80b8-bdac768e1de1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:55:26.650522 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.650336    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d09ad81c-4422-4b1a-b845-a616f5c9e93c-workload-certs" pod "d09ad81c-4422-4b1a-b845-a616f5c9e93c" (UID: "d09ad81c-4422-4b1a-b845-a616f5c9e93c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:55:26.650857 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.650115    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5734864-6a35-46c3-80b8-bdac768e1de1-workload-socket" pod "f5734864-6a35-46c3-80b8-bdac768e1de1" (UID: "f5734864-6a35-46c3-80b8-bdac768e1de1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:55:26.652103 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.651074    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dca3f84e-1191-48bb-a59c-8456088614ac-workload-certs" pod "dca3f84e-1191-48bb-a59c-8456088614ac" (UID: "dca3f84e-1191-48bb-a59c-8456088614ac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:55:26.652103 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.651412    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dca3f84e-1191-48bb-a59c-8456088614ac-credential-socket" pod "dca3f84e-1191-48bb-a59c-8456088614ac" (UID: "dca3f84e-1191-48bb-a59c-8456088614ac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:55:26.652103 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.651445    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5734864-6a35-46c3-80b8-bdac768e1de1-istiod-ca-cert" pod "f5734864-6a35-46c3-80b8-bdac768e1de1" (UID: "f5734864-6a35-46c3-80b8-bdac768e1de1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 19:55:26.653291 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.652421    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d09ad81c-4422-4b1a-b845-a616f5c9e93c-credential-socket" pod "d09ad81c-4422-4b1a-b845-a616f5c9e93c" (UID: "d09ad81c-4422-4b1a-b845-a616f5c9e93c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:55:26.653291 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.652520    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dca3f84e-1191-48bb-a59c-8456088614ac-istiod-ca-cert" pod "dca3f84e-1191-48bb-a59c-8456088614ac" (UID: "dca3f84e-1191-48bb-a59c-8456088614ac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 19:55:26.653291 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.652730    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d09ad81c-4422-4b1a-b845-a616f5c9e93c-workload-socket" pod "d09ad81c-4422-4b1a-b845-a616f5c9e93c" (UID: "d09ad81c-4422-4b1a-b845-a616f5c9e93c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:55:26.653291 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.652973    4463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9bfbdea-9214-414e-84af-e5c9ea6406ff-workload-certs\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:55:26.653291 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.653075    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d09ad81c-4422-4b1a-b845-a616f5c9e93c-istio-token" pod "d09ad81c-4422-4b1a-b845-a616f5c9e93c" (UID: "d09ad81c-4422-4b1a-b845-a616f5c9e93c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:55:26.653291 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.653253    4463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9bfbdea-9214-414e-84af-e5c9ea6406ff-workload-socket\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:55:26.653291 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.653277    4463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dca3f84e-1191-48bb-a59c-8456088614ac-credential-socket\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:55:26.653291 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.653293    4463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5734864-6a35-46c3-80b8-bdac768e1de1-istiod-ca-cert\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:55:26.653676 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.653307    4463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5734864-6a35-46c3-80b8-bdac768e1de1-workload-certs\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:55:26.653676 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.653322    4463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5734864-6a35-46c3-80b8-bdac768e1de1-istio-data\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:55:26.653676 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.653337    4463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dca3f84e-1191-48bb-a59c-8456088614ac-workload-certs\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:55:26.653676 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.653350    4463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d09ad81c-4422-4b1a-b845-a616f5c9e93c-workload-certs\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:55:26.653676 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.653365    4463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9bfbdea-9214-414e-84af-e5c9ea6406ff-istio-token\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:55:26.653676 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.653380    4463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dca3f84e-1191-48bb-a59c-8456088614ac-workload-socket\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:55:26.653676 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.653398    4463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9bfbdea-9214-414e-84af-e5c9ea6406ff-istiod-ca-cert\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:55:26.653676 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.653425    4463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9bfbdea-9214-414e-84af-e5c9ea6406ff-istio-podinfo\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:55:26.653676 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.653458    4463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9bfbdea-9214-414e-84af-e5c9ea6406ff-istio-envoy\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:55:26.653676 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.653474    4463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d09ad81c-4422-4b1a-b845-a616f5c9e93c-istio-data\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:55:26.653676 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.653489    4463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9bfbdea-9214-414e-84af-e5c9ea6406ff-credential-socket\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:55:26.653676 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.653504    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4xhh\" (UniqueName: \"kubernetes.io/projected/b9bfbdea-9214-414e-84af-e5c9ea6406ff-kube-api-access-t4xhh\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:55:26.653676 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.653519    4463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5734864-6a35-46c3-80b8-bdac768e1de1-workload-socket\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:55:26.654169 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.654086    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dca3f84e-1191-48bb-a59c-8456088614ac-istio-envoy" pod "dca3f84e-1191-48bb-a59c-8456088614ac" (UID: "dca3f84e-1191-48bb-a59c-8456088614ac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:55:26.654364 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.654344    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d09ad81c-4422-4b1a-b845-a616f5c9e93c-istiod-ca-cert" pod "d09ad81c-4422-4b1a-b845-a616f5c9e93c" (UID: "d09ad81c-4422-4b1a-b845-a616f5c9e93c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 19:55:26.654454 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.654403    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5734864-6a35-46c3-80b8-bdac768e1de1-credential-socket" pod "f5734864-6a35-46c3-80b8-bdac768e1de1" (UID: "f5734864-6a35-46c3-80b8-bdac768e1de1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:55:26.654529 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.654450    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dca3f84e-1191-48bb-a59c-8456088614ac-istio-data" pod "dca3f84e-1191-48bb-a59c-8456088614ac" (UID: "dca3f84e-1191-48bb-a59c-8456088614ac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:55:26.656139 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.656117    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d09ad81c-4422-4b1a-b845-a616f5c9e93c-kube-api-access-hphl6" pod "d09ad81c-4422-4b1a-b845-a616f5c9e93c" (UID: "d09ad81c-4422-4b1a-b845-a616f5c9e93c"). InnerVolumeSpecName "kube-api-access-hphl6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:55:26.656730 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.656664    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5734864-6a35-46c3-80b8-bdac768e1de1-istio-envoy" pod "f5734864-6a35-46c3-80b8-bdac768e1de1" (UID: "f5734864-6a35-46c3-80b8-bdac768e1de1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:55:26.656730 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.656676    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5734864-6a35-46c3-80b8-bdac768e1de1-kube-api-access-tsfwd" pod "f5734864-6a35-46c3-80b8-bdac768e1de1" (UID: "f5734864-6a35-46c3-80b8-bdac768e1de1"). InnerVolumeSpecName "kube-api-access-tsfwd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:55:26.656730 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.656682    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d09ad81c-4422-4b1a-b845-a616f5c9e93c-istio-envoy" pod "d09ad81c-4422-4b1a-b845-a616f5c9e93c" (UID: "d09ad81c-4422-4b1a-b845-a616f5c9e93c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:55:26.656730 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.656691    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f5734864-6a35-46c3-80b8-bdac768e1de1-istio-podinfo" pod "f5734864-6a35-46c3-80b8-bdac768e1de1" (UID: "f5734864-6a35-46c3-80b8-bdac768e1de1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 19:55:26.656730 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.656700    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dca3f84e-1191-48bb-a59c-8456088614ac-istio-token" pod "dca3f84e-1191-48bb-a59c-8456088614ac" (UID: "dca3f84e-1191-48bb-a59c-8456088614ac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:55:26.656730 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.656708    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dca3f84e-1191-48bb-a59c-8456088614ac-kube-api-access-hllx4" pod "dca3f84e-1191-48bb-a59c-8456088614ac" (UID: "dca3f84e-1191-48bb-a59c-8456088614ac"). InnerVolumeSpecName "kube-api-access-hllx4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:55:26.656985 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.656784    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d09ad81c-4422-4b1a-b845-a616f5c9e93c-istio-podinfo" pod "d09ad81c-4422-4b1a-b845-a616f5c9e93c" (UID: "d09ad81c-4422-4b1a-b845-a616f5c9e93c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 19:55:26.657176 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.657153    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dca3f84e-1191-48bb-a59c-8456088614ac-istio-podinfo" pod "dca3f84e-1191-48bb-a59c-8456088614ac" (UID: "dca3f84e-1191-48bb-a59c-8456088614ac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 19:55:26.658301 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.658282    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5734864-6a35-46c3-80b8-bdac768e1de1-istio-token" pod "f5734864-6a35-46c3-80b8-bdac768e1de1" (UID: "f5734864-6a35-46c3-80b8-bdac768e1de1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:55:26.755464 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.755421    4463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d09ad81c-4422-4b1a-b845-a616f5c9e93c-credential-socket\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:55:26.755464 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.755461    4463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dca3f84e-1191-48bb-a59c-8456088614ac-istio-envoy\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:55:26.755696 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.755475    4463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5734864-6a35-46c3-80b8-bdac768e1de1-istio-token\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:55:26.755696 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.755489    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsfwd\" (UniqueName: \"kubernetes.io/projected/f5734864-6a35-46c3-80b8-bdac768e1de1-kube-api-access-tsfwd\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:55:26.755696 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.755504    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hphl6\" (UniqueName: \"kubernetes.io/projected/d09ad81c-4422-4b1a-b845-a616f5c9e93c-kube-api-access-hphl6\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:55:26.755696 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.755517    4463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d09ad81c-4422-4b1a-b845-a616f5c9e93c-istiod-ca-cert\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:55:26.755696 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.755530    4463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5734864-6a35-46c3-80b8-bdac768e1de1-istio-envoy\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:55:26.755696 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.755542    4463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dca3f84e-1191-48bb-a59c-8456088614ac-istiod-ca-cert\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:55:26.755696 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.755555    4463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5734864-6a35-46c3-80b8-bdac768e1de1-istio-podinfo\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:55:26.755696 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.755568    4463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5734864-6a35-46c3-80b8-bdac768e1de1-credential-socket\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:55:26.755696 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.755583    4463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d09ad81c-4422-4b1a-b845-a616f5c9e93c-istio-envoy\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:55:26.755696 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.755596    4463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d09ad81c-4422-4b1a-b845-a616f5c9e93c-workload-socket\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:55:26.755696 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.755608    4463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dca3f84e-1191-48bb-a59c-8456088614ac-istio-podinfo\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:55:26.755696 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.755621    4463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d09ad81c-4422-4b1a-b845-a616f5c9e93c-istio-token\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:55:26.755696 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.755635    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hllx4\" (UniqueName: \"kubernetes.io/projected/dca3f84e-1191-48bb-a59c-8456088614ac-kube-api-access-hllx4\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:55:26.755696 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.755649    4463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dca3f84e-1191-48bb-a59c-8456088614ac-istio-token\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:55:26.755696 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.755662    4463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dca3f84e-1191-48bb-a59c-8456088614ac-istio-data\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:55:26.755696 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:26.755675    4463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d09ad81c-4422-4b1a-b845-a616f5c9e93c-istio-podinfo\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:55:27.218057 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:27.216483    4463 generic.go:356] "Generic (PLEG): container finished" podID="d09ad81c-4422-4b1a-b845-a616f5c9e93c" containerID="a02110035b528ab4aad6fd5a33abe1f2af48f93bd083a52279beddd7e1521b01" exitCode=0
Nov 02 19:55:27.218057 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:27.216599    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-w5gw2" event={"ID":"d09ad81c-4422-4b1a-b845-a616f5c9e93c","Type":"ContainerDied","Data":"a02110035b528ab4aad6fd5a33abe1f2af48f93bd083a52279beddd7e1521b01"}
Nov 02 19:55:27.218057 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:27.216650    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-w5gw2" event={"ID":"d09ad81c-4422-4b1a-b845-a616f5c9e93c","Type":"ContainerDied","Data":"61c34e5c1adb7cca8a2c36281bd73cc1db60899d300d132a434ee1a8f2b26f09"}
Nov 02 19:55:27.218057 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:27.216679    4463 scope.go:122] "RemoveContainer" containerID="a02110035b528ab4aad6fd5a33abe1f2af48f93bd083a52279beddd7e1521b01"
Nov 02 19:55:27.218057 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:27.216820    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-w5gw2"
Nov 02 19:55:27.220880 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:27.220857    4463 generic.go:356] "Generic (PLEG): container finished" podID="dca3f84e-1191-48bb-a59c-8456088614ac" containerID="3b618c0735a004eb627ce924fb547cfcf4ed904ae3d9732bf021b460a04c2d8b" exitCode=0
Nov 02 19:55:27.220979 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:27.220903    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-8mkqq"
Nov 02 19:55:27.220979 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:27.220948    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-8mkqq" event={"ID":"dca3f84e-1191-48bb-a59c-8456088614ac","Type":"ContainerDied","Data":"3b618c0735a004eb627ce924fb547cfcf4ed904ae3d9732bf021b460a04c2d8b"}
Nov 02 19:55:27.221126 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:27.220990    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-8mkqq" event={"ID":"dca3f84e-1191-48bb-a59c-8456088614ac","Type":"ContainerDied","Data":"0cd25feb44969d243509cf3925b8a424df5c0f646016103a017b5e2b9da71666"}
Nov 02 19:55:27.222527 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:27.222502    4463 generic.go:356] "Generic (PLEG): container finished" podID="b9bfbdea-9214-414e-84af-e5c9ea6406ff" containerID="65a2df16c2f73565703bcf2da8604827789c6fe32c195fd1064ff05c05221c32" exitCode=0
Nov 02 19:55:27.222621 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:27.222552    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-q9v28"
Nov 02 19:55:27.222621 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:27.222583    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-q9v28" event={"ID":"b9bfbdea-9214-414e-84af-e5c9ea6406ff","Type":"ContainerDied","Data":"65a2df16c2f73565703bcf2da8604827789c6fe32c195fd1064ff05c05221c32"}
Nov 02 19:55:27.222737 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:27.222629    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-q9v28" event={"ID":"b9bfbdea-9214-414e-84af-e5c9ea6406ff","Type":"ContainerDied","Data":"b49c60078e8ee9145658abc4d4717a06efa3ce2404157b5943791e8283207fb9"}
Nov 02 19:55:27.224564 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:27.224543    4463 generic.go:356] "Generic (PLEG): container finished" podID="f5734864-6a35-46c3-80b8-bdac768e1de1" containerID="916aecd458917f4668c09a0921566e17197367f37bcc3dd208817f4d230a20c4" exitCode=0
Nov 02 19:55:27.224639 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:27.224597    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9ndrz"
Nov 02 19:55:27.224739 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:27.224601    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9ndrz" event={"ID":"f5734864-6a35-46c3-80b8-bdac768e1de1","Type":"ContainerDied","Data":"916aecd458917f4668c09a0921566e17197367f37bcc3dd208817f4d230a20c4"}
Nov 02 19:55:27.224739 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:27.224708    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9ndrz" event={"ID":"f5734864-6a35-46c3-80b8-bdac768e1de1","Type":"ContainerDied","Data":"916c8d53bb355408adcb9a051f33408077e90d26fccd696eea82acee250bbbf9"}
Nov 02 19:55:27.232383 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:27.232333    4463 scope.go:122] "RemoveContainer" containerID="a02110035b528ab4aad6fd5a33abe1f2af48f93bd083a52279beddd7e1521b01"
Nov 02 19:55:27.232669 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:55:27.232649    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a02110035b528ab4aad6fd5a33abe1f2af48f93bd083a52279beddd7e1521b01\": container with ID starting with a02110035b528ab4aad6fd5a33abe1f2af48f93bd083a52279beddd7e1521b01 not found: ID does not exist" containerID="a02110035b528ab4aad6fd5a33abe1f2af48f93bd083a52279beddd7e1521b01"
Nov 02 19:55:27.232769 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:27.232675    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a02110035b528ab4aad6fd5a33abe1f2af48f93bd083a52279beddd7e1521b01"} err="failed to get container status \"a02110035b528ab4aad6fd5a33abe1f2af48f93bd083a52279beddd7e1521b01\": rpc error: code = NotFound desc = could not find container \"a02110035b528ab4aad6fd5a33abe1f2af48f93bd083a52279beddd7e1521b01\": container with ID starting with a02110035b528ab4aad6fd5a33abe1f2af48f93bd083a52279beddd7e1521b01 not found: ID does not exist"
Nov 02 19:55:27.232769 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:27.232699    4463 scope.go:122] "RemoveContainer" containerID="3b618c0735a004eb627ce924fb547cfcf4ed904ae3d9732bf021b460a04c2d8b"
Nov 02 19:55:27.243087 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:27.243071    4463 scope.go:122] "RemoveContainer" containerID="3b618c0735a004eb627ce924fb547cfcf4ed904ae3d9732bf021b460a04c2d8b"
Nov 02 19:55:27.243331 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:55:27.243313    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b618c0735a004eb627ce924fb547cfcf4ed904ae3d9732bf021b460a04c2d8b\": container with ID starting with 3b618c0735a004eb627ce924fb547cfcf4ed904ae3d9732bf021b460a04c2d8b not found: ID does not exist" containerID="3b618c0735a004eb627ce924fb547cfcf4ed904ae3d9732bf021b460a04c2d8b"
Nov 02 19:55:27.243404 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:27.243340    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b618c0735a004eb627ce924fb547cfcf4ed904ae3d9732bf021b460a04c2d8b"} err="failed to get container status \"3b618c0735a004eb627ce924fb547cfcf4ed904ae3d9732bf021b460a04c2d8b\": rpc error: code = NotFound desc = could not find container \"3b618c0735a004eb627ce924fb547cfcf4ed904ae3d9732bf021b460a04c2d8b\": container with ID starting with 3b618c0735a004eb627ce924fb547cfcf4ed904ae3d9732bf021b460a04c2d8b not found: ID does not exist"
Nov 02 19:55:27.243404 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:27.243364    4463 scope.go:122] "RemoveContainer" containerID="65a2df16c2f73565703bcf2da8604827789c6fe32c195fd1064ff05c05221c32"
Nov 02 19:55:27.252235 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:27.252213    4463 scope.go:122] "RemoveContainer" containerID="65a2df16c2f73565703bcf2da8604827789c6fe32c195fd1064ff05c05221c32"
Nov 02 19:55:27.252553 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:55:27.252526    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65a2df16c2f73565703bcf2da8604827789c6fe32c195fd1064ff05c05221c32\": container with ID starting with 65a2df16c2f73565703bcf2da8604827789c6fe32c195fd1064ff05c05221c32 not found: ID does not exist" containerID="65a2df16c2f73565703bcf2da8604827789c6fe32c195fd1064ff05c05221c32"
Nov 02 19:55:27.252648 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:27.252562    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65a2df16c2f73565703bcf2da8604827789c6fe32c195fd1064ff05c05221c32"} err="failed to get container status \"65a2df16c2f73565703bcf2da8604827789c6fe32c195fd1064ff05c05221c32\": rpc error: code = NotFound desc = could not find container \"65a2df16c2f73565703bcf2da8604827789c6fe32c195fd1064ff05c05221c32\": container with ID starting with 65a2df16c2f73565703bcf2da8604827789c6fe32c195fd1064ff05c05221c32 not found: ID does not exist"
Nov 02 19:55:27.252648 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:27.252584    4463 scope.go:122] "RemoveContainer" containerID="916aecd458917f4668c09a0921566e17197367f37bcc3dd208817f4d230a20c4"
Nov 02 19:55:27.261180 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:27.261159    4463 scope.go:122] "RemoveContainer" containerID="23387a024c12a6210c34a1f28ef800c18033948f5ff83a61e84ac54b4597fd4b"
Nov 02 19:55:27.270035 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:27.270005    4463 scope.go:122] "RemoveContainer" containerID="916aecd458917f4668c09a0921566e17197367f37bcc3dd208817f4d230a20c4"
Nov 02 19:55:27.270281 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:55:27.270263    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"916aecd458917f4668c09a0921566e17197367f37bcc3dd208817f4d230a20c4\": container with ID starting with 916aecd458917f4668c09a0921566e17197367f37bcc3dd208817f4d230a20c4 not found: ID does not exist" containerID="916aecd458917f4668c09a0921566e17197367f37bcc3dd208817f4d230a20c4"
Nov 02 19:55:27.270339 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:27.270285    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"916aecd458917f4668c09a0921566e17197367f37bcc3dd208817f4d230a20c4"} err="failed to get container status \"916aecd458917f4668c09a0921566e17197367f37bcc3dd208817f4d230a20c4\": rpc error: code = NotFound desc = could not find container \"916aecd458917f4668c09a0921566e17197367f37bcc3dd208817f4d230a20c4\": container with ID starting with 916aecd458917f4668c09a0921566e17197367f37bcc3dd208817f4d230a20c4 not found: ID does not exist"
Nov 02 19:55:27.270339 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:27.270298    4463 scope.go:122] "RemoveContainer" containerID="23387a024c12a6210c34a1f28ef800c18033948f5ff83a61e84ac54b4597fd4b"
Nov 02 19:55:27.270545 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:55:27.270531    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23387a024c12a6210c34a1f28ef800c18033948f5ff83a61e84ac54b4597fd4b\": container with ID starting with 23387a024c12a6210c34a1f28ef800c18033948f5ff83a61e84ac54b4597fd4b not found: ID does not exist" containerID="23387a024c12a6210c34a1f28ef800c18033948f5ff83a61e84ac54b4597fd4b"
Nov 02 19:55:27.270585 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:27.270549    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23387a024c12a6210c34a1f28ef800c18033948f5ff83a61e84ac54b4597fd4b"} err="failed to get container status \"23387a024c12a6210c34a1f28ef800c18033948f5ff83a61e84ac54b4597fd4b\": rpc error: code = NotFound desc = could not find container \"23387a024c12a6210c34a1f28ef800c18033948f5ff83a61e84ac54b4597fd4b\": container with ID starting with 23387a024c12a6210c34a1f28ef800c18033948f5ff83a61e84ac54b4597fd4b not found: ID does not exist"
Nov 02 19:55:27.375559 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:27.375523    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-q9v28"]
Nov 02 19:55:27.419832 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:27.419754    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-q9v28"]
Nov 02 19:55:27.478288 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:27.478247    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-8mkqq"]
Nov 02 19:55:27.481961 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:27.481935    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-8mkqq"]
Nov 02 19:55:27.588491 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:27.588462    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9bfbdea-9214-414e-84af-e5c9ea6406ff" path="/var/lib/kubelet/pods/b9bfbdea-9214-414e-84af-e5c9ea6406ff/volumes"
Nov 02 19:55:27.588983 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:27.588969    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dca3f84e-1191-48bb-a59c-8456088614ac" path="/var/lib/kubelet/pods/dca3f84e-1191-48bb-a59c-8456088614ac/volumes"
Nov 02 19:55:27.589331 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:27.589305    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-w5gw2"]
Nov 02 19:55:27.589438 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:27.589419    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-w5gw2"]
Nov 02 19:55:27.688837 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:27.688806    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9ndrz"]
Nov 02 19:55:27.692181 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:27.692159    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9ndrz"]
Nov 02 19:55:29.586448 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:29.586414    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d09ad81c-4422-4b1a-b845-a616f5c9e93c" path="/var/lib/kubelet/pods/d09ad81c-4422-4b1a-b845-a616f5c9e93c/volumes"
Nov 02 19:55:29.586984 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:29.586967    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5734864-6a35-46c3-80b8-bdac768e1de1" path="/var/lib/kubelet/pods/f5734864-6a35-46c3-80b8-bdac768e1de1/volumes"
Nov 02 19:55:32.381922 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:32.381885    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-32abda-64b746699-qq2dz"]
Nov 02 19:55:32.382569 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:32.382551    4463 cpu_manager.go:414] "containerMap: removing container" podUID="f5734864-6a35-46c3-80b8-bdac768e1de1" containerName="istio-proxy"
Nov 02 19:55:32.382618 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:32.382570    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5734864-6a35-46c3-80b8-bdac768e1de1" containerName="istio-proxy"
Nov 02 19:55:32.382618 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:32.382585    4463 cpu_manager.go:414] "containerMap: removing container" podUID="dca3f84e-1191-48bb-a59c-8456088614ac" containerName="istio-proxy"
Nov 02 19:55:32.382618 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:32.382590    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dca3f84e-1191-48bb-a59c-8456088614ac" containerName="istio-proxy"
Nov 02 19:55:32.382618 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:32.382613    4463 cpu_manager.go:414] "containerMap: removing container" podUID="b9bfbdea-9214-414e-84af-e5c9ea6406ff" containerName="istio-proxy"
Nov 02 19:55:32.382618 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:32.382618    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9bfbdea-9214-414e-84af-e5c9ea6406ff" containerName="istio-proxy"
Nov 02 19:55:32.382765 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:32.382636    4463 cpu_manager.go:414] "containerMap: removing container" podUID="d09ad81c-4422-4b1a-b845-a616f5c9e93c" containerName="istio-proxy"
Nov 02 19:55:32.382765 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:32.382640    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d09ad81c-4422-4b1a-b845-a616f5c9e93c" containerName="istio-proxy"
Nov 02 19:55:32.382765 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:32.382745    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5734864-6a35-46c3-80b8-bdac768e1de1" containerName="istio-proxy"
Nov 02 19:55:32.382765 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:32.382754    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="dca3f84e-1191-48bb-a59c-8456088614ac" containerName="istio-proxy"
Nov 02 19:55:32.382765 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:32.382762    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5734864-6a35-46c3-80b8-bdac768e1de1" containerName="istio-proxy"
Nov 02 19:55:32.382912 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:32.382772    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9bfbdea-9214-414e-84af-e5c9ea6406ff" containerName="istio-proxy"
Nov 02 19:55:32.382912 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:32.382780    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="d09ad81c-4422-4b1a-b845-a616f5c9e93c" containerName="istio-proxy"
Nov 02 19:55:32.386724 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:32.386704    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-32abda-64b746699-qq2dz"
Nov 02 19:55:32.440203 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:32.440170    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-32abda-64b746699-qq2dz"]
Nov 02 19:55:32.480271 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:32.480242    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w5-32abda-dockercfg-g9vqd\""
Nov 02 19:55:32.480271 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:32.480247    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w5-32abda\""
Nov 02 19:55:32.539917 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:32.539253    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt6fd\" (UniqueName: \"kubernetes.io/projected/38b5c5e1-3522-458d-a7c4-950105f94a3e-kube-api-access-gt6fd\") pod \"payload-processing-e2e-shared-b-w5-32abda-64b746699-qq2dz\" (UID: \"38b5c5e1-3522-458d-a7c4-950105f94a3e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-32abda-64b746699-qq2dz"
Nov 02 19:55:32.539917 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:32.539477    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38b5c5e1-3522-458d-a7c4-950105f94a3e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-32abda-64b746699-qq2dz\" (UID: \"38b5c5e1-3522-458d-a7c4-950105f94a3e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-32abda-64b746699-qq2dz"
Nov 02 19:55:32.554078 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:32.554046    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-32abda-9b855d4f9-qb8kc"]
Nov 02 19:55:32.554673 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:32.554658    4463 cpu_manager.go:414] "containerMap: removing container" podUID="f5734864-6a35-46c3-80b8-bdac768e1de1" containerName="istio-proxy"
Nov 02 19:55:32.554673 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:32.554672    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5734864-6a35-46c3-80b8-bdac768e1de1" containerName="istio-proxy"
Nov 02 19:55:32.557424 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:32.557402    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-32abda-9b855d4f9-qb8kc"
Nov 02 19:55:32.621061 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:32.618258    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-32abda-9b855d4f9-qb8kc"]
Nov 02 19:55:32.641390 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:32.641286    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38b5c5e1-3522-458d-a7c4-950105f94a3e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-32abda-64b746699-qq2dz\" (UID: \"38b5c5e1-3522-458d-a7c4-950105f94a3e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-32abda-64b746699-qq2dz"
Nov 02 19:55:32.641791 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:32.641761    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt6fd\" (UniqueName: \"kubernetes.io/projected/38b5c5e1-3522-458d-a7c4-950105f94a3e-kube-api-access-gt6fd\") pod \"payload-processing-e2e-shared-b-w5-32abda-64b746699-qq2dz\" (UID: \"38b5c5e1-3522-458d-a7c4-950105f94a3e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-32abda-64b746699-qq2dz"
Nov 02 19:55:32.644250 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:32.644226    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38b5c5e1-3522-458d-a7c4-950105f94a3e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-32abda-64b746699-qq2dz\" (UID: \"38b5c5e1-3522-458d-a7c4-950105f94a3e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-32abda-64b746699-qq2dz"
Nov 02 19:55:32.697922 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:32.697888    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt6fd\" (UniqueName: \"kubernetes.io/projected/38b5c5e1-3522-458d-a7c4-950105f94a3e-kube-api-access-gt6fd\") pod \"payload-processing-e2e-shared-b-w5-32abda-64b746699-qq2dz\" (UID: \"38b5c5e1-3522-458d-a7c4-950105f94a3e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-32abda-64b746699-qq2dz"
Nov 02 19:55:32.700767 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:32.700744    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-32abda-64b746699-qq2dz"
Nov 02 19:55:32.748056 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:32.743631    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9917b249-a6ac-4b35-babb-fc0ca2ae6a83-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-32abda-9b855d4f9-qb8kc\" (UID: \"9917b249-a6ac-4b35-babb-fc0ca2ae6a83\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-32abda-9b855d4f9-qb8kc"
Nov 02 19:55:32.748056 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:32.743713    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwzrz\" (UniqueName: \"kubernetes.io/projected/9917b249-a6ac-4b35-babb-fc0ca2ae6a83-kube-api-access-jwzrz\") pod \"payload-pre-processing-e2e-shared-b-w5-32abda-9b855d4f9-qb8kc\" (UID: \"9917b249-a6ac-4b35-babb-fc0ca2ae6a83\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-32abda-9b855d4f9-qb8kc"
Nov 02 19:55:32.849316 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:32.849204    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9917b249-a6ac-4b35-babb-fc0ca2ae6a83-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-32abda-9b855d4f9-qb8kc\" (UID: \"9917b249-a6ac-4b35-babb-fc0ca2ae6a83\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-32abda-9b855d4f9-qb8kc"
Nov 02 19:55:32.852977 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:32.852041    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwzrz\" (UniqueName: \"kubernetes.io/projected/9917b249-a6ac-4b35-babb-fc0ca2ae6a83-kube-api-access-jwzrz\") pod \"payload-pre-processing-e2e-shared-b-w5-32abda-9b855d4f9-qb8kc\" (UID: \"9917b249-a6ac-4b35-babb-fc0ca2ae6a83\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-32abda-9b855d4f9-qb8kc"
Nov 02 19:55:32.859040 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:32.857972    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9917b249-a6ac-4b35-babb-fc0ca2ae6a83-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-32abda-9b855d4f9-qb8kc\" (UID: \"9917b249-a6ac-4b35-babb-fc0ca2ae6a83\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-32abda-9b855d4f9-qb8kc"
Nov 02 19:55:32.904890 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:32.904829    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwzrz\" (UniqueName: \"kubernetes.io/projected/9917b249-a6ac-4b35-babb-fc0ca2ae6a83-kube-api-access-jwzrz\") pod \"payload-pre-processing-e2e-shared-b-w5-32abda-9b855d4f9-qb8kc\" (UID: \"9917b249-a6ac-4b35-babb-fc0ca2ae6a83\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-32abda-9b855d4f9-qb8kc"
Nov 02 19:55:33.078042 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:33.077986    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-32abda-64b746699-qq2dz"]
Nov 02 19:55:33.078704 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:55:33.078677    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38b5c5e1_3522_458d_a7c4_950105f94a3e.slice/crio-098525b4a88f88acc6d39355c49b7d4906a20d0c587bd034b65ce66ea20ff737 WatchSource:0}: Error finding container 098525b4a88f88acc6d39355c49b7d4906a20d0c587bd034b65ce66ea20ff737: Status 404 returned error can't find the container with id 098525b4a88f88acc6d39355c49b7d4906a20d0c587bd034b65ce66ea20ff737
Nov 02 19:55:33.130494 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:33.130469    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4099cbd3-84f88b85c5-hdbml"]
Nov 02 19:55:33.171272 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:33.171216    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-32abda-9b855d4f9-qb8kc"
Nov 02 19:55:33.270058 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:33.269612    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-32abda-64b746699-qq2dz" event={"ID":"38b5c5e1-3522-458d-a7c4-950105f94a3e","Type":"ContainerStarted","Data":"bfaf5d1476b8343df33b39d5bddd056880acbdb825e1e0e40b3ee30a35015ab1"}
Nov 02 19:55:33.270058 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:33.269671    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-32abda-64b746699-qq2dz" event={"ID":"38b5c5e1-3522-458d-a7c4-950105f94a3e","Type":"ContainerStarted","Data":"098525b4a88f88acc6d39355c49b7d4906a20d0c587bd034b65ce66ea20ff737"}
Nov 02 19:55:33.270058 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:33.269702    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-32abda-64b746699-qq2dz"
Nov 02 19:55:33.376050 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:33.375964    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-32abda-64b746699-qq2dz" podStartSLOduration=1.375951018 podStartE2EDuration="1.375951018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:55:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:55:33.374197733 +0000 UTC m=+1639.981231515" watchObservedRunningTime="2026-11-02 19:55:33.375951018 +0000 UTC m=+1639.982984783"
Nov 02 19:55:33.538169 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:33.538138    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-32abda-9b855d4f9-qb8kc"]
Nov 02 19:55:33.538937 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:55:33.538904    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9917b249_a6ac_4b35_babb_fc0ca2ae6a83.slice/crio-f0351acae1dcac8a6098ea9b28083ab97558c043ddf03b6ed7452ff2712f1f77 WatchSource:0}: Error finding container f0351acae1dcac8a6098ea9b28083ab97558c043ddf03b6ed7452ff2712f1f77: Status 404 returned error can't find the container with id f0351acae1dcac8a6098ea9b28083ab97558c043ddf03b6ed7452ff2712f1f77
Nov 02 19:55:33.727142 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:33.726957    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-d1aaed-5449d767fd-tcwj8"]
Nov 02 19:55:33.730886 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:33.730865    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-d1aaed-5449d767fd-tcwj8"
Nov 02 19:55:33.772242 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:33.772213    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4de3a3a-cbd8-4488-9d04-4ec03e4d53ce-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-d1aaed-5449d767fd-tcwj8\" (UID: \"b4de3a3a-cbd8-4488-9d04-4ec03e4d53ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-d1aaed-5449d767fd-tcwj8"
Nov 02 19:55:33.772479 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:33.772449    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4de3a3a-cbd8-4488-9d04-4ec03e4d53ce-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-d1aaed-5449d767fd-tcwj8\" (UID: \"b4de3a3a-cbd8-4488-9d04-4ec03e4d53ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-d1aaed-5449d767fd-tcwj8"
Nov 02 19:55:33.772571 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:33.772530    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4de3a3a-cbd8-4488-9d04-4ec03e4d53ce-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-d1aaed-5449d767fd-tcwj8\" (UID: \"b4de3a3a-cbd8-4488-9d04-4ec03e4d53ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-d1aaed-5449d767fd-tcwj8"
Nov 02 19:55:33.772641 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:33.772615    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7q6m\" (UniqueName: \"kubernetes.io/projected/b4de3a3a-cbd8-4488-9d04-4ec03e4d53ce-kube-api-access-p7q6m\") pod \"maas-api-e2e-shared-a-w6-d1aaed-5449d767fd-tcwj8\" (UID: \"b4de3a3a-cbd8-4488-9d04-4ec03e4d53ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-d1aaed-5449d767fd-tcwj8"
Nov 02 19:55:33.779534 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:33.779505    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w6-d1aaed-serving-cert\""
Nov 02 19:55:33.782027 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:33.781986    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-d1aaed-5449d767fd-tcwj8"]
Nov 02 19:55:33.892108 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:33.876730    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4de3a3a-cbd8-4488-9d04-4ec03e4d53ce-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-d1aaed-5449d767fd-tcwj8\" (UID: \"b4de3a3a-cbd8-4488-9d04-4ec03e4d53ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-d1aaed-5449d767fd-tcwj8"
Nov 02 19:55:33.892108 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:33.878029    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4de3a3a-cbd8-4488-9d04-4ec03e4d53ce-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-d1aaed-5449d767fd-tcwj8\" (UID: \"b4de3a3a-cbd8-4488-9d04-4ec03e4d53ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-d1aaed-5449d767fd-tcwj8"
Nov 02 19:55:33.892108 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:33.882298    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4de3a3a-cbd8-4488-9d04-4ec03e4d53ce-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-d1aaed-5449d767fd-tcwj8\" (UID: \"b4de3a3a-cbd8-4488-9d04-4ec03e4d53ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-d1aaed-5449d767fd-tcwj8"
Nov 02 19:55:33.892108 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:33.882354    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4de3a3a-cbd8-4488-9d04-4ec03e4d53ce-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-d1aaed-5449d767fd-tcwj8\" (UID: \"b4de3a3a-cbd8-4488-9d04-4ec03e4d53ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-d1aaed-5449d767fd-tcwj8"
Nov 02 19:55:33.892108 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:33.882380    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7q6m\" (UniqueName: \"kubernetes.io/projected/b4de3a3a-cbd8-4488-9d04-4ec03e4d53ce-kube-api-access-p7q6m\") pod \"maas-api-e2e-shared-a-w6-d1aaed-5449d767fd-tcwj8\" (UID: \"b4de3a3a-cbd8-4488-9d04-4ec03e4d53ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-d1aaed-5449d767fd-tcwj8"
Nov 02 19:55:33.892108 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:55:33.882932    4463 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-d1aaed-serving-cert: secret "maas-api-e2e-shared-a-w6-d1aaed-serving-cert" not found
Nov 02 19:55:33.892108 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:55:33.883025    4463 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4de3a3a-cbd8-4488-9d04-4ec03e4d53ce-maas-api-tls podName:b4de3a3a-cbd8-4488-9d04-4ec03e4d53ce nodeName:}" failed. No retries permitted until 2026-11-02 19:55:34.382988073 +0000 UTC m=+1640.990021833 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b4de3a3a-cbd8-4488-9d04-4ec03e4d53ce-maas-api-tls") pod "maas-api-e2e-shared-a-w6-d1aaed-5449d767fd-tcwj8" (UID: "b4de3a3a-cbd8-4488-9d04-4ec03e4d53ce") : secret "maas-api-e2e-shared-a-w6-d1aaed-serving-cert" not found
Nov 02 19:55:33.892108 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:33.891272    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4de3a3a-cbd8-4488-9d04-4ec03e4d53ce-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-d1aaed-5449d767fd-tcwj8\" (UID: \"b4de3a3a-cbd8-4488-9d04-4ec03e4d53ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-d1aaed-5449d767fd-tcwj8"
Nov 02 19:55:33.935802 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:33.935769    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7q6m\" (UniqueName: \"kubernetes.io/projected/b4de3a3a-cbd8-4488-9d04-4ec03e4d53ce-kube-api-access-p7q6m\") pod \"maas-api-e2e-shared-a-w6-d1aaed-5449d767fd-tcwj8\" (UID: \"b4de3a3a-cbd8-4488-9d04-4ec03e4d53ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-d1aaed-5449d767fd-tcwj8"
Nov 02 19:55:34.283644 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:34.282918    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-32abda-9b855d4f9-qb8kc" event={"ID":"9917b249-a6ac-4b35-babb-fc0ca2ae6a83","Type":"ContainerStarted","Data":"e00434e74a1f24ab8d0e99840da58e904e897fa2d14e63dd959348b952990810"}
Nov 02 19:55:34.283644 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:34.283001    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-32abda-9b855d4f9-qb8kc" event={"ID":"9917b249-a6ac-4b35-babb-fc0ca2ae6a83","Type":"ContainerStarted","Data":"f0351acae1dcac8a6098ea9b28083ab97558c043ddf03b6ed7452ff2712f1f77"}
Nov 02 19:55:34.283644 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:34.283207    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-32abda-9b855d4f9-qb8kc"
Nov 02 19:55:34.300960 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:55:34.300925    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 19:55:34.391265 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:34.390849    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4de3a3a-cbd8-4488-9d04-4ec03e4d53ce-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-d1aaed-5449d767fd-tcwj8\" (UID: \"b4de3a3a-cbd8-4488-9d04-4ec03e4d53ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-d1aaed-5449d767fd-tcwj8"
Nov 02 19:55:34.391971 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:34.391915    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-32abda-9b855d4f9-qb8kc" podStartSLOduration=2.391902838 podStartE2EDuration="2.391902838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:55:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:55:34.388592272 +0000 UTC m=+1640.995626040" watchObservedRunningTime="2026-11-02 19:55:34.391902838 +0000 UTC m=+1640.998936606"
Nov 02 19:55:34.393868 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:34.393838    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4de3a3a-cbd8-4488-9d04-4ec03e4d53ce-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-d1aaed-5449d767fd-tcwj8\" (UID: \"b4de3a3a-cbd8-4488-9d04-4ec03e4d53ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-d1aaed-5449d767fd-tcwj8"
Nov 02 19:55:34.645420 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:34.645339    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-d1aaed-5449d767fd-tcwj8"
Nov 02 19:55:35.063287 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:35.063254    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-d1aaed-5449d767fd-tcwj8"]
Nov 02 19:55:35.064470 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:55:35.064442    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4de3a3a_cbd8_4488_9d04_4ec03e4d53ce.slice/crio-dc918c9284d4e64654e23b2dd455cfb60c764011084f5a1cb8f0aa14d6c5f5b6 WatchSource:0}: Error finding container dc918c9284d4e64654e23b2dd455cfb60c764011084f5a1cb8f0aa14d6c5f5b6: Status 404 returned error can't find the container with id dc918c9284d4e64654e23b2dd455cfb60c764011084f5a1cb8f0aa14d6c5f5b6
Nov 02 19:55:35.302716 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:35.302672    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-d1aaed-5449d767fd-tcwj8" event={"ID":"b4de3a3a-cbd8-4488-9d04-4ec03e4d53ce","Type":"ContainerStarted","Data":"dc918c9284d4e64654e23b2dd455cfb60c764011084f5a1cb8f0aa14d6c5f5b6"}
Nov 02 19:55:36.310727 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:36.310638    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-d1aaed-5449d767fd-tcwj8" event={"ID":"b4de3a3a-cbd8-4488-9d04-4ec03e4d53ce","Type":"ContainerStarted","Data":"4f0a21b797b70b88bd750b4d7f055f3da4426454fc2c97e814b4d43de72c0000"}
Nov 02 19:55:36.310727 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:36.310713    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-d1aaed-5449d767fd-tcwj8"
Nov 02 19:55:36.417209 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:36.417161    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-d1aaed-5449d767fd-tcwj8" podStartSLOduration=2.527873371 podStartE2EDuration="3.417149392s" totalImagesPullingTime="889.276021ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:55:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:55:36.416719826 +0000 UTC m=+1643.023753594" watchObservedRunningTime="2026-11-02 19:55:36.417149392 +0000 UTC m=+1643.024183204"
Nov 02 19:55:38.130773 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:38.130731    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-4099cbd3-84f88b85c5-hdbml" podUID="1c4ee35e-073e-4e22-9b22-955fa8c3bc94" containerName="payload-processing" containerID="cri-o://db86449d26bf1b61f42879180d671d13ce98a5b2bfd5e15b5f73e489048dcfe4" gracePeriod=25
Nov 02 19:55:38.327635 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:38.327590    4463 generic.go:356] "Generic (PLEG): container finished" podID="1c4ee35e-073e-4e22-9b22-955fa8c3bc94" containerID="db86449d26bf1b61f42879180d671d13ce98a5b2bfd5e15b5f73e489048dcfe4" exitCode=0
Nov 02 19:55:38.327775 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:38.327665    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-4099cbd3-84f88b85c5-hdbml" event={"ID":"1c4ee35e-073e-4e22-9b22-955fa8c3bc94","Type":"ContainerDied","Data":"db86449d26bf1b61f42879180d671d13ce98a5b2bfd5e15b5f73e489048dcfe4"}
Nov 02 19:55:38.397366 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:38.397306    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-4099cbd3-84f88b85c5-hdbml"
Nov 02 19:55:38.473501 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:38.473475    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c4ee35e-073e-4e22-9b22-955fa8c3bc94-kube-api-access-jq7nn\" (UniqueName: \"kubernetes.io/projected/1c4ee35e-073e-4e22-9b22-955fa8c3bc94-kube-api-access-jq7nn\") pod \"1c4ee35e-073e-4e22-9b22-955fa8c3bc94\" (UID: \"1c4ee35e-073e-4e22-9b22-955fa8c3bc94\") "
Nov 02 19:55:38.473644 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:38.473534    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c4ee35e-073e-4e22-9b22-955fa8c3bc94-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c4ee35e-073e-4e22-9b22-955fa8c3bc94-plugins-config-volume\") pod \"1c4ee35e-073e-4e22-9b22-955fa8c3bc94\" (UID: \"1c4ee35e-073e-4e22-9b22-955fa8c3bc94\") "
Nov 02 19:55:38.473905 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:38.473886    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c4ee35e-073e-4e22-9b22-955fa8c3bc94-plugins-config-volume" pod "1c4ee35e-073e-4e22-9b22-955fa8c3bc94" (UID: "1c4ee35e-073e-4e22-9b22-955fa8c3bc94"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 19:55:38.474135 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:38.474122    4463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c4ee35e-073e-4e22-9b22-955fa8c3bc94-plugins-config-volume\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:55:38.475551 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:38.475530    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c4ee35e-073e-4e22-9b22-955fa8c3bc94-kube-api-access-jq7nn" pod "1c4ee35e-073e-4e22-9b22-955fa8c3bc94" (UID: "1c4ee35e-073e-4e22-9b22-955fa8c3bc94"). InnerVolumeSpecName "kube-api-access-jq7nn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:55:38.580111 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:38.580081    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jq7nn\" (UniqueName: \"kubernetes.io/projected/1c4ee35e-073e-4e22-9b22-955fa8c3bc94-kube-api-access-jq7nn\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:55:39.346543 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:39.346513    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-4099cbd3-84f88b85c5-hdbml"
Nov 02 19:55:39.346994 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:39.346540    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-4099cbd3-84f88b85c5-hdbml" event={"ID":"1c4ee35e-073e-4e22-9b22-955fa8c3bc94","Type":"ContainerDied","Data":"e5142585b9d84bd980ba255ae3ace7f6c39cb1ae28aa47473aef299cb25e74f7"}
Nov 02 19:55:39.346994 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:39.346627    4463 scope.go:122] "RemoveContainer" containerID="db86449d26bf1b61f42879180d671d13ce98a5b2bfd5e15b5f73e489048dcfe4"
Nov 02 19:55:39.498604 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:39.498573    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4099cbd3-84f88b85c5-hdbml"]
Nov 02 19:55:39.544338 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:39.544299    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4099cbd3-84f88b85c5-hdbml"]
Nov 02 19:55:39.585976 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:39.585948    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c4ee35e-073e-4e22-9b22-955fa8c3bc94" path="/var/lib/kubelet/pods/1c4ee35e-073e-4e22-9b22-955fa8c3bc94/volumes"
Nov 02 19:55:42.323592 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:42.323561    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-d1aaed-5449d767fd-tcwj8"
Nov 02 19:55:44.286819 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:44.286788    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-32abda-64b746699-qq2dz"
Nov 02 19:55:45.306103 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:55:45.306078    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-32abda-9b855d4f9-qb8kc"
Nov 02 19:56:04.301974 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:56:04.301940    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 19:56:05.092124 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:56:05.092089    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 19:56:16.832889 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:56:16.832849    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-4099cbd3-openshift-default-698ddb8d77-lrzxg"]
Nov 02 19:56:16.833468 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:56:16.833123    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-4099cbd3-openshift-default-698ddb8d77-lrzxg" podUID="44a12c0e-e3df-4368-a5f8-61c3b8f88da6" containerName="istio-proxy" containerID="cri-o://db0cdeba200bc7d15dc63faa761ccde5dc2e4d6dc3421b76262678b424d893c6" gracePeriod=30
Nov 02 19:56:22.116391 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:56:22.116300    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4099cbd3-openshift-default-698ddb8d77-lrzxg"
Nov 02 19:56:22.160726 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:56:22.160701    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44a12c0e-e3df-4368-a5f8-61c3b8f88da6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44a12c0e-e3df-4368-a5f8-61c3b8f88da6-istio-data\") pod \"44a12c0e-e3df-4368-a5f8-61c3b8f88da6\" (UID: \"44a12c0e-e3df-4368-a5f8-61c3b8f88da6\") "
Nov 02 19:56:22.160845 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:56:22.160784    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44a12c0e-e3df-4368-a5f8-61c3b8f88da6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a12c0e-e3df-4368-a5f8-61c3b8f88da6-credential-socket\") pod \"44a12c0e-e3df-4368-a5f8-61c3b8f88da6\" (UID: \"44a12c0e-e3df-4368-a5f8-61c3b8f88da6\") "
Nov 02 19:56:22.160845 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:56:22.160832    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44a12c0e-e3df-4368-a5f8-61c3b8f88da6-kube-api-access-w48zt\" (UniqueName: \"kubernetes.io/projected/44a12c0e-e3df-4368-a5f8-61c3b8f88da6-kube-api-access-w48zt\") pod \"44a12c0e-e3df-4368-a5f8-61c3b8f88da6\" (UID: \"44a12c0e-e3df-4368-a5f8-61c3b8f88da6\") "
Nov 02 19:56:22.160920 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:56:22.160862    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44a12c0e-e3df-4368-a5f8-61c3b8f88da6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44a12c0e-e3df-4368-a5f8-61c3b8f88da6-istio-envoy\") pod \"44a12c0e-e3df-4368-a5f8-61c3b8f88da6\" (UID: \"44a12c0e-e3df-4368-a5f8-61c3b8f88da6\") "
Nov 02 19:56:22.160975 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:56:22.160964    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44a12c0e-e3df-4368-a5f8-61c3b8f88da6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44a12c0e-e3df-4368-a5f8-61c3b8f88da6-workload-certs\") pod \"44a12c0e-e3df-4368-a5f8-61c3b8f88da6\" (UID: \"44a12c0e-e3df-4368-a5f8-61c3b8f88da6\") "
Nov 02 19:56:22.161053 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:56:22.160986    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44a12c0e-e3df-4368-a5f8-61c3b8f88da6-istio-data" pod "44a12c0e-e3df-4368-a5f8-61c3b8f88da6" (UID: "44a12c0e-e3df-4368-a5f8-61c3b8f88da6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:56:22.161053 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:56:22.161040    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44a12c0e-e3df-4368-a5f8-61c3b8f88da6-credential-socket" pod "44a12c0e-e3df-4368-a5f8-61c3b8f88da6" (UID: "44a12c0e-e3df-4368-a5f8-61c3b8f88da6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:56:22.161176 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:56:22.161072    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/44a12c0e-e3df-4368-a5f8-61c3b8f88da6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44a12c0e-e3df-4368-a5f8-61c3b8f88da6-istio-podinfo\") pod \"44a12c0e-e3df-4368-a5f8-61c3b8f88da6\" (UID: \"44a12c0e-e3df-4368-a5f8-61c3b8f88da6\") "
Nov 02 19:56:22.161176 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:56:22.161101    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44a12c0e-e3df-4368-a5f8-61c3b8f88da6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44a12c0e-e3df-4368-a5f8-61c3b8f88da6-istiod-ca-cert\") pod \"44a12c0e-e3df-4368-a5f8-61c3b8f88da6\" (UID: \"44a12c0e-e3df-4368-a5f8-61c3b8f88da6\") "
Nov 02 19:56:22.161176 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:56:22.161122    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44a12c0e-e3df-4368-a5f8-61c3b8f88da6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a12c0e-e3df-4368-a5f8-61c3b8f88da6-workload-socket\") pod \"44a12c0e-e3df-4368-a5f8-61c3b8f88da6\" (UID: \"44a12c0e-e3df-4368-a5f8-61c3b8f88da6\") "
Nov 02 19:56:22.161326 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:56:22.161178    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44a12c0e-e3df-4368-a5f8-61c3b8f88da6-istio-token\" (UniqueName: \"kubernetes.io/projected/44a12c0e-e3df-4368-a5f8-61c3b8f88da6-istio-token\") pod \"44a12c0e-e3df-4368-a5f8-61c3b8f88da6\" (UID: \"44a12c0e-e3df-4368-a5f8-61c3b8f88da6\") "
Nov 02 19:56:22.161426 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:56:22.161326    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44a12c0e-e3df-4368-a5f8-61c3b8f88da6-workload-certs" pod "44a12c0e-e3df-4368-a5f8-61c3b8f88da6" (UID: "44a12c0e-e3df-4368-a5f8-61c3b8f88da6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:56:22.161517 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:56:22.161486    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44a12c0e-e3df-4368-a5f8-61c3b8f88da6-workload-socket" pod "44a12c0e-e3df-4368-a5f8-61c3b8f88da6" (UID: "44a12c0e-e3df-4368-a5f8-61c3b8f88da6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:56:22.161517 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:56:22.161506    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44a12c0e-e3df-4368-a5f8-61c3b8f88da6-istiod-ca-cert" pod "44a12c0e-e3df-4368-a5f8-61c3b8f88da6" (UID: "44a12c0e-e3df-4368-a5f8-61c3b8f88da6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 19:56:22.161869 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:56:22.161840    4463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a12c0e-e3df-4368-a5f8-61c3b8f88da6-credential-socket\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:56:22.161869 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:56:22.161861    4463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44a12c0e-e3df-4368-a5f8-61c3b8f88da6-workload-certs\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:56:22.162008 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:56:22.161872    4463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44a12c0e-e3df-4368-a5f8-61c3b8f88da6-istiod-ca-cert\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:56:22.162008 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:56:22.161885    4463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a12c0e-e3df-4368-a5f8-61c3b8f88da6-workload-socket\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:56:22.162008 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:56:22.161894    4463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44a12c0e-e3df-4368-a5f8-61c3b8f88da6-istio-data\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:56:22.163178 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:56:22.163149    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44a12c0e-e3df-4368-a5f8-61c3b8f88da6-istio-envoy" pod "44a12c0e-e3df-4368-a5f8-61c3b8f88da6" (UID: "44a12c0e-e3df-4368-a5f8-61c3b8f88da6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:56:22.163289 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:56:22.163242    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/44a12c0e-e3df-4368-a5f8-61c3b8f88da6-istio-podinfo" pod "44a12c0e-e3df-4368-a5f8-61c3b8f88da6" (UID: "44a12c0e-e3df-4368-a5f8-61c3b8f88da6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 19:56:22.163495 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:56:22.163475    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44a12c0e-e3df-4368-a5f8-61c3b8f88da6-kube-api-access-w48zt" pod "44a12c0e-e3df-4368-a5f8-61c3b8f88da6" (UID: "44a12c0e-e3df-4368-a5f8-61c3b8f88da6"). InnerVolumeSpecName "kube-api-access-w48zt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:56:22.163579 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:56:22.163513    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44a12c0e-e3df-4368-a5f8-61c3b8f88da6-istio-token" pod "44a12c0e-e3df-4368-a5f8-61c3b8f88da6" (UID: "44a12c0e-e3df-4368-a5f8-61c3b8f88da6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:56:22.262828 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:56:22.262797    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w48zt\" (UniqueName: \"kubernetes.io/projected/44a12c0e-e3df-4368-a5f8-61c3b8f88da6-kube-api-access-w48zt\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:56:22.262828 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:56:22.262823    4463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44a12c0e-e3df-4368-a5f8-61c3b8f88da6-istio-envoy\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:56:22.262828 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:56:22.262833    4463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44a12c0e-e3df-4368-a5f8-61c3b8f88da6-istio-podinfo\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:56:22.263076 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:56:22.262841    4463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44a12c0e-e3df-4368-a5f8-61c3b8f88da6-istio-token\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:56:22.772726 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:56:22.772694    4463 generic.go:356] "Generic (PLEG): container finished" podID="44a12c0e-e3df-4368-a5f8-61c3b8f88da6" containerID="db0cdeba200bc7d15dc63faa761ccde5dc2e4d6dc3421b76262678b424d893c6" exitCode=0
Nov 02 19:56:22.772894 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:56:22.772745    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4099cbd3-openshift-default-698ddb8d77-lrzxg"
Nov 02 19:56:22.772894 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:56:22.772776    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4099cbd3-openshift-default-698ddb8d77-lrzxg" event={"ID":"44a12c0e-e3df-4368-a5f8-61c3b8f88da6","Type":"ContainerDied","Data":"db0cdeba200bc7d15dc63faa761ccde5dc2e4d6dc3421b76262678b424d893c6"}
Nov 02 19:56:22.772894 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:56:22.772835    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4099cbd3-openshift-default-698ddb8d77-lrzxg" event={"ID":"44a12c0e-e3df-4368-a5f8-61c3b8f88da6","Type":"ContainerDied","Data":"e2470c47b6172db7e87d773e783eee5bb56acddd310a506a92536acb83bf9dc4"}
Nov 02 19:56:22.772894 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:56:22.772863    4463 scope.go:122] "RemoveContainer" containerID="db0cdeba200bc7d15dc63faa761ccde5dc2e4d6dc3421b76262678b424d893c6"
Nov 02 19:56:22.784006 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:56:22.783990    4463 scope.go:122] "RemoveContainer" containerID="ad3d9c3c6ac7afe5b3d137580b7014e71578c849b53231bb9f5635ff39853c38"
Nov 02 19:56:22.792857 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:56:22.792832    4463 scope.go:122] "RemoveContainer" containerID="db0cdeba200bc7d15dc63faa761ccde5dc2e4d6dc3421b76262678b424d893c6"
Nov 02 19:56:22.793167 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:56:22.793149    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db0cdeba200bc7d15dc63faa761ccde5dc2e4d6dc3421b76262678b424d893c6\": container with ID starting with db0cdeba200bc7d15dc63faa761ccde5dc2e4d6dc3421b76262678b424d893c6 not found: ID does not exist" containerID="db0cdeba200bc7d15dc63faa761ccde5dc2e4d6dc3421b76262678b424d893c6"
Nov 02 19:56:22.793222 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:56:22.793175    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db0cdeba200bc7d15dc63faa761ccde5dc2e4d6dc3421b76262678b424d893c6"} err="failed to get container status \"db0cdeba200bc7d15dc63faa761ccde5dc2e4d6dc3421b76262678b424d893c6\": rpc error: code = NotFound desc = could not find container \"db0cdeba200bc7d15dc63faa761ccde5dc2e4d6dc3421b76262678b424d893c6\": container with ID starting with db0cdeba200bc7d15dc63faa761ccde5dc2e4d6dc3421b76262678b424d893c6 not found: ID does not exist"
Nov 02 19:56:22.793222 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:56:22.793191    4463 scope.go:122] "RemoveContainer" containerID="ad3d9c3c6ac7afe5b3d137580b7014e71578c849b53231bb9f5635ff39853c38"
Nov 02 19:56:22.793429 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:56:22.793412    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad3d9c3c6ac7afe5b3d137580b7014e71578c849b53231bb9f5635ff39853c38\": container with ID starting with ad3d9c3c6ac7afe5b3d137580b7014e71578c849b53231bb9f5635ff39853c38 not found: ID does not exist" containerID="ad3d9c3c6ac7afe5b3d137580b7014e71578c849b53231bb9f5635ff39853c38"
Nov 02 19:56:22.793486 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:56:22.793466    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad3d9c3c6ac7afe5b3d137580b7014e71578c849b53231bb9f5635ff39853c38"} err="failed to get container status \"ad3d9c3c6ac7afe5b3d137580b7014e71578c849b53231bb9f5635ff39853c38\": rpc error: code = NotFound desc = could not find container \"ad3d9c3c6ac7afe5b3d137580b7014e71578c849b53231bb9f5635ff39853c38\": container with ID starting with ad3d9c3c6ac7afe5b3d137580b7014e71578c849b53231bb9f5635ff39853c38 not found: ID does not exist"
Nov 02 19:56:22.929629 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:56:22.929592    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-4099cbd3-openshift-default-698ddb8d77-lrzxg"]
Nov 02 19:56:22.975352 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:56:22.975322    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-4099cbd3-openshift-default-698ddb8d77-lrzxg"]
Nov 02 19:56:23.587326 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:56:23.587297    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44a12c0e-e3df-4368-a5f8-61c3b8f88da6" path="/var/lib/kubelet/pods/44a12c0e-e3df-4368-a5f8-61c3b8f88da6/volumes"
Nov 02 19:56:31.917041 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:56:31.916544    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-43ceaa/premium-w0-43ceaa-kserve-67b55659b7-rckfd" podUID="7be32df1-5049-4f79-be23-a827f3f0a85f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:56:34.295153 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:56:34.295115    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 19:56:43.090823 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:56:43.090793    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-24ef8b-openshift-default-685d675f47-76xb9"]
Nov 02 19:56:43.091463 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:56:43.091446    4463 cpu_manager.go:414] "containerMap: removing container" podUID="44a12c0e-e3df-4368-a5f8-61c3b8f88da6" containerName="istio-proxy"
Nov 02 19:56:43.091506 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:56:43.091464    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44a12c0e-e3df-4368-a5f8-61c3b8f88da6" containerName="istio-proxy"
Nov 02 19:56:43.091506 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:56:43.091479    4463 cpu_manager.go:414] "containerMap: removing container" podUID="1c4ee35e-073e-4e22-9b22-955fa8c3bc94" containerName="payload-processing"
Nov 02 19:56:43.091506 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:56:43.091484    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c4ee35e-073e-4e22-9b22-955fa8c3bc94" containerName="payload-processing"
Nov 02 19:56:43.091614 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:56:43.091605    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="44a12c0e-e3df-4368-a5f8-61c3b8f88da6" containerName="istio-proxy"
Nov 02 19:56:43.091647 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:56:43.091618    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c4ee35e-073e-4e22-9b22-955fa8c3bc94" containerName="payload-processing"
Nov 02 19:56:43.094214 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:56:43.094195    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-24ef8b-openshift-default-685d675f47-76xb9"
Nov 02 19:56:43.162035 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:56:43.158552    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-24ef8b-openshift-default-dockercfg-5zklv\""
Nov 02 19:56:43.283094 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:56:43.277301    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2c0f86f-3e9e-4996-8710-664c445e9113-credential-socket\") pod \"e2e-shared-b-w6-24ef8b-openshift-default-685d675f47-76xb9\" (UID: \"b2c0f86f-3e9e-4996-8710-664c445e9113\") " pod="openshift-ingress/e2e-shared-b-w6-24ef8b-openshift-default-685d675f47-76xb9"
Nov 02 19:56:43.283094 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:56:43.277397    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqw4l\" (UniqueName: \"kubernetes.io/projected/b2c0f86f-3e9e-4996-8710-664c445e9113-kube-api-access-pqw4l\") pod \"e2e-shared-b-w6-24ef8b-openshift-default-685d675f47-76xb9\" (UID: \"b2c0f86f-3e9e-4996-8710-664c445e9113\") " pod="openshift-ingress/e2e-shared-b-w6-24ef8b-openshift-default-685d675f47-76xb9"
Nov 02 19:56:43.283094 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:56:43.277436    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2c0f86f-3e9e-4996-8710-664c445e9113-istio-envoy\") pod \"e2e-shared-b-w6-24ef8b-openshift-default-685d675f47-76xb9\" (UID: \"b2c0f86f-3e9e-4996-8710-664c445e9113\") " pod="openshift-ingress/e2e-shared-b-w6-24ef8b-openshift-default-685d675f47-76xb9"
Nov 02 19:56:43.283094 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:56:43.277511    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2c0f86f-3e9e-4996-8710-664c445e9113-istio-data\") pod \"e2e-shared-b-w6-24ef8b-openshift-default-685d675f47-76xb9\" (UID: \"b2c0f86f-3e9e-4996-8710-664c445e9113\") " pod="openshift-ingress/e2e-shared-b-w6-24ef8b-openshift-default-685d675f47-76xb9"
Nov 02 19:56:43.283094 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:56:43.277627    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2c0f86f-3e9e-4996-8710-664c445e9113-workload-certs\") pod \"e2e-shared-b-w6-24ef8b-openshift-default-685d675f47-76xb9\" (UID: \"b2c0f86f-3e9e-4996-8710-664c445e9113\") " pod="openshift-ingress/e2e-shared-b-w6-24ef8b-openshift-default-685d675f47-76xb9"
Nov 02 19:56:43.283094 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:56:43.277685    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2c0f86f-3e9e-4996-8710-664c445e9113-istiod-ca-cert\") pod \"e2e-shared-b-w6-24ef8b-openshift-default-685d675f47-76xb9\" (UID: \"b2c0f86f-3e9e-4996-8710-664c445e9113\") " pod="openshift-ingress/e2e-shared-b-w6-24ef8b-openshift-default-685d675f47-76xb9"
Nov 02 19:56:43.283094 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:56:43.277718    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2c0f86f-3e9e-4996-8710-664c445e9113-istio-podinfo\") pod \"e2e-shared-b-w6-24ef8b-openshift-default-685d675f47-76xb9\" (UID: \"b2c0f86f-3e9e-4996-8710-664c445e9113\") " pod="openshift-ingress/e2e-shared-b-w6-24ef8b-openshift-default-685d675f47-76xb9"
Nov 02 19:56:43.283094 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:56:43.277846    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2c0f86f-3e9e-4996-8710-664c445e9113-istio-token\") pod \"e2e-shared-b-w6-24ef8b-openshift-default-685d675f47-76xb9\" (UID: \"b2c0f86f-3e9e-4996-8710-664c445e9113\") " pod="openshift-ingress/e2e-shared-b-w6-24ef8b-openshift-default-685d675f47-76xb9"
Nov 02 19:56:43.283094 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:56:43.277930    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2c0f86f-3e9e-4996-8710-664c445e9113-workload-socket\") pod \"e2e-shared-b-w6-24ef8b-openshift-default-685d675f47-76xb9\" (UID: \"b2c0f86f-3e9e-4996-8710-664c445e9113\") " pod="openshift-ingress/e2e-shared-b-w6-24ef8b-openshift-default-685d675f47-76xb9"
Nov 02 19:56:43.301120 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:56:43.301075    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-24ef8b-openshift-default-685d675f47-76xb9"]
Nov 02 19:56:43.380172 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:56:43.380078    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2c0f86f-3e9e-4996-8710-664c445e9113-workload-certs\") pod \"e2e-shared-b-w6-24ef8b-openshift-default-685d675f47-76xb9\" (UID: \"b2c0f86f-3e9e-4996-8710-664c445e9113\") " pod="openshift-ingress/e2e-shared-b-w6-24ef8b-openshift-default-685d675f47-76xb9"
Nov 02 19:56:43.380172 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:56:43.380134    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2c0f86f-3e9e-4996-8710-664c445e9113-istiod-ca-cert\") pod \"e2e-shared-b-w6-24ef8b-openshift-default-685d675f47-76xb9\" (UID: \"b2c0f86f-3e9e-4996-8710-664c445e9113\") " pod="openshift-ingress/e2e-shared-b-w6-24ef8b-openshift-default-685d675f47-76xb9"
Nov 02 19:56:43.380172 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:56:43.380163    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2c0f86f-3e9e-4996-8710-664c445e9113-istio-podinfo\") pod \"e2e-shared-b-w6-24ef8b-openshift-default-685d675f47-76xb9\" (UID: \"b2c0f86f-3e9e-4996-8710-664c445e9113\") " pod="openshift-ingress/e2e-shared-b-w6-24ef8b-openshift-default-685d675f47-76xb9"
Nov 02 19:56:43.380478 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:56:43.380258    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2c0f86f-3e9e-4996-8710-664c445e9113-istio-token\") pod \"e2e-shared-b-w6-24ef8b-openshift-default-685d675f47-76xb9\" (UID: \"b2c0f86f-3e9e-4996-8710-664c445e9113\") " pod="openshift-ingress/e2e-shared-b-w6-24ef8b-openshift-default-685d675f47-76xb9"
Nov 02 19:56:43.380538 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:56:43.380494    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2c0f86f-3e9e-4996-8710-664c445e9113-workload-certs\") pod \"e2e-shared-b-w6-24ef8b-openshift-default-685d675f47-76xb9\" (UID: \"b2c0f86f-3e9e-4996-8710-664c445e9113\") " pod="openshift-ingress/e2e-shared-b-w6-24ef8b-openshift-default-685d675f47-76xb9"
Nov 02 19:56:43.380538 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:56:43.380512    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2c0f86f-3e9e-4996-8710-664c445e9113-workload-socket\") pod \"e2e-shared-b-w6-24ef8b-openshift-default-685d675f47-76xb9\" (UID: \"b2c0f86f-3e9e-4996-8710-664c445e9113\") " pod="openshift-ingress/e2e-shared-b-w6-24ef8b-openshift-default-685d675f47-76xb9"
Nov 02 19:56:43.380637 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:56:43.380595    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2c0f86f-3e9e-4996-8710-664c445e9113-credential-socket\") pod \"e2e-shared-b-w6-24ef8b-openshift-default-685d675f47-76xb9\" (UID: \"b2c0f86f-3e9e-4996-8710-664c445e9113\") " pod="openshift-ingress/e2e-shared-b-w6-24ef8b-openshift-default-685d675f47-76xb9"
Nov 02 19:56:43.380693 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:56:43.380651    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqw4l\" (UniqueName: \"kubernetes.io/projected/b2c0f86f-3e9e-4996-8710-664c445e9113-kube-api-access-pqw4l\") pod \"e2e-shared-b-w6-24ef8b-openshift-default-685d675f47-76xb9\" (UID: \"b2c0f86f-3e9e-4996-8710-664c445e9113\") " pod="openshift-ingress/e2e-shared-b-w6-24ef8b-openshift-default-685d675f47-76xb9"
Nov 02 19:56:43.380693 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:56:43.380686    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2c0f86f-3e9e-4996-8710-664c445e9113-istio-envoy\") pod \"e2e-shared-b-w6-24ef8b-openshift-default-685d675f47-76xb9\" (UID: \"b2c0f86f-3e9e-4996-8710-664c445e9113\") " pod="openshift-ingress/e2e-shared-b-w6-24ef8b-openshift-default-685d675f47-76xb9"
Nov 02 19:56:43.380796 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:56:43.380750    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2c0f86f-3e9e-4996-8710-664c445e9113-istio-data\") pod \"e2e-shared-b-w6-24ef8b-openshift-default-685d675f47-76xb9\" (UID: \"b2c0f86f-3e9e-4996-8710-664c445e9113\") " pod="openshift-ingress/e2e-shared-b-w6-24ef8b-openshift-default-685d675f47-76xb9"
Nov 02 19:56:43.380848 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:56:43.380821    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2c0f86f-3e9e-4996-8710-664c445e9113-workload-socket\") pod \"e2e-shared-b-w6-24ef8b-openshift-default-685d675f47-76xb9\" (UID: \"b2c0f86f-3e9e-4996-8710-664c445e9113\") " pod="openshift-ingress/e2e-shared-b-w6-24ef8b-openshift-default-685d675f47-76xb9"
Nov 02 19:56:43.380927 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:56:43.380904    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2c0f86f-3e9e-4996-8710-664c445e9113-istiod-ca-cert\") pod \"e2e-shared-b-w6-24ef8b-openshift-default-685d675f47-76xb9\" (UID: \"b2c0f86f-3e9e-4996-8710-664c445e9113\") " pod="openshift-ingress/e2e-shared-b-w6-24ef8b-openshift-default-685d675f47-76xb9"
Nov 02 19:56:43.380985 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:56:43.380905    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2c0f86f-3e9e-4996-8710-664c445e9113-credential-socket\") pod \"e2e-shared-b-w6-24ef8b-openshift-default-685d675f47-76xb9\" (UID: \"b2c0f86f-3e9e-4996-8710-664c445e9113\") " pod="openshift-ingress/e2e-shared-b-w6-24ef8b-openshift-default-685d675f47-76xb9"
Nov 02 19:56:43.381181 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:56:43.381161    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2c0f86f-3e9e-4996-8710-664c445e9113-istio-data\") pod \"e2e-shared-b-w6-24ef8b-openshift-default-685d675f47-76xb9\" (UID: \"b2c0f86f-3e9e-4996-8710-664c445e9113\") " pod="openshift-ingress/e2e-shared-b-w6-24ef8b-openshift-default-685d675f47-76xb9"
Nov 02 19:56:43.382948 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:56:43.382928    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2c0f86f-3e9e-4996-8710-664c445e9113-istio-envoy\") pod \"e2e-shared-b-w6-24ef8b-openshift-default-685d675f47-76xb9\" (UID: \"b2c0f86f-3e9e-4996-8710-664c445e9113\") " pod="openshift-ingress/e2e-shared-b-w6-24ef8b-openshift-default-685d675f47-76xb9"
Nov 02 19:56:43.383267 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:56:43.383249    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2c0f86f-3e9e-4996-8710-664c445e9113-istio-podinfo\") pod \"e2e-shared-b-w6-24ef8b-openshift-default-685d675f47-76xb9\" (UID: \"b2c0f86f-3e9e-4996-8710-664c445e9113\") " pod="openshift-ingress/e2e-shared-b-w6-24ef8b-openshift-default-685d675f47-76xb9"
Nov 02 19:56:43.433631 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:56:43.433604    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2c0f86f-3e9e-4996-8710-664c445e9113-istio-token\") pod \"e2e-shared-b-w6-24ef8b-openshift-default-685d675f47-76xb9\" (UID: \"b2c0f86f-3e9e-4996-8710-664c445e9113\") " pod="openshift-ingress/e2e-shared-b-w6-24ef8b-openshift-default-685d675f47-76xb9"
Nov 02 19:56:43.433770 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:56:43.433753    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqw4l\" (UniqueName: \"kubernetes.io/projected/b2c0f86f-3e9e-4996-8710-664c445e9113-kube-api-access-pqw4l\") pod \"e2e-shared-b-w6-24ef8b-openshift-default-685d675f47-76xb9\" (UID: \"b2c0f86f-3e9e-4996-8710-664c445e9113\") " pod="openshift-ingress/e2e-shared-b-w6-24ef8b-openshift-default-685d675f47-76xb9"
Nov 02 19:56:43.600514 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:56:43.600473    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-24ef8b-openshift-default-685d675f47-76xb9"
Nov 02 19:56:44.030933 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:56:44.030895    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-24ef8b-openshift-default-685d675f47-76xb9"]
Nov 02 19:56:44.031270 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:56:44.031247    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2c0f86f_3e9e_4996_8710_664c445e9113.slice/crio-02bd90f6b755cb4e6ccef328c61c999b80f551f74e182a625f178384a33515ba WatchSource:0}: Error finding container 02bd90f6b755cb4e6ccef328c61c999b80f551f74e182a625f178384a33515ba: Status 404 returned error can't find the container with id 02bd90f6b755cb4e6ccef328c61c999b80f551f74e182a625f178384a33515ba
Nov 02 19:56:44.033871 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:56:44.033841    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:56:44.033967 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:56:44.033902    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:56:44.033967 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:56:44.033938    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:56:45.030661 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:56:45.030624    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-24ef8b-openshift-default-685d675f47-76xb9" event={"ID":"b2c0f86f-3e9e-4996-8710-664c445e9113","Type":"ContainerStarted","Data":"36d39bfe6697d6d7172d4e0ead0d7f1b422687a8126f90f98969a12155c00d60"}
Nov 02 19:56:45.031042 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:56:45.030699    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-24ef8b-openshift-default-685d675f47-76xb9" event={"ID":"b2c0f86f-3e9e-4996-8710-664c445e9113","Type":"ContainerStarted","Data":"02bd90f6b755cb4e6ccef328c61c999b80f551f74e182a625f178384a33515ba"}
Nov 02 19:56:45.231915 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:56:45.231864    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-24ef8b-openshift-default-685d675f47-76xb9" podStartSLOduration=2.231850989 podStartE2EDuration="2.231850989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:56:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:56:45.183635047 +0000 UTC m=+1711.790668813" watchObservedRunningTime="2026-11-02 19:56:45.231850989 +0000 UTC m=+1711.838884756"
Nov 02 19:56:45.601185 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:56:45.601157    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-24ef8b-openshift-default-685d675f47-76xb9"
Nov 02 19:56:45.606959 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:56:45.606936    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-24ef8b-openshift-default-685d675f47-76xb9"
Nov 02 19:56:46.037181 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:56:46.037151    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-24ef8b-openshift-default-685d675f47-76xb9"
Nov 02 19:56:46.037945 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:56:46.037924    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-24ef8b-openshift-default-685d675f47-76xb9"
Nov 02 19:56:59.122157 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:56:59.122122    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 19:57:04.294665 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:57:04.294627    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 19:57:13.181977 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:13.181944    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-89a22a-7587d7b466-6c9dq"]
Nov 02 19:57:13.182580 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:13.182565    4463 cpu_manager.go:414] "containerMap: removing container" podUID="44a12c0e-e3df-4368-a5f8-61c3b8f88da6" containerName="istio-proxy"
Nov 02 19:57:13.182634 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:13.182582    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44a12c0e-e3df-4368-a5f8-61c3b8f88da6" containerName="istio-proxy"
Nov 02 19:57:13.182721 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:13.182713    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="44a12c0e-e3df-4368-a5f8-61c3b8f88da6" containerName="istio-proxy"
Nov 02 19:57:13.185195 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:13.185179    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-89a22a-7587d7b466-6c9dq"
Nov 02 19:57:13.280414 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:13.280381    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-89a22a-serving-cert\""
Nov 02 19:57:13.284052 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:13.284030    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-89a22a-7587d7b466-6c9dq"]
Nov 02 19:57:13.312993 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:13.312950    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/787ed8bc-d29c-4e3b-b661-87d0712a506f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-89a22a-7587d7b466-6c9dq\" (UID: \"787ed8bc-d29c-4e3b-b661-87d0712a506f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-89a22a-7587d7b466-6c9dq"
Nov 02 19:57:13.312993 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:13.312994    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/787ed8bc-d29c-4e3b-b661-87d0712a506f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-89a22a-7587d7b466-6c9dq\" (UID: \"787ed8bc-d29c-4e3b-b661-87d0712a506f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-89a22a-7587d7b466-6c9dq"
Nov 02 19:57:13.313234 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:13.313087    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/787ed8bc-d29c-4e3b-b661-87d0712a506f-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-89a22a-7587d7b466-6c9dq\" (UID: \"787ed8bc-d29c-4e3b-b661-87d0712a506f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-89a22a-7587d7b466-6c9dq"
Nov 02 19:57:13.313234 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:13.313108    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9ksk\" (UniqueName: \"kubernetes.io/projected/787ed8bc-d29c-4e3b-b661-87d0712a506f-kube-api-access-j9ksk\") pod \"maas-api-e2e-shared-a-w3-89a22a-7587d7b466-6c9dq\" (UID: \"787ed8bc-d29c-4e3b-b661-87d0712a506f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-89a22a-7587d7b466-6c9dq"
Nov 02 19:57:13.414189 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:13.414156    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/787ed8bc-d29c-4e3b-b661-87d0712a506f-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-89a22a-7587d7b466-6c9dq\" (UID: \"787ed8bc-d29c-4e3b-b661-87d0712a506f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-89a22a-7587d7b466-6c9dq"
Nov 02 19:57:13.414862 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:13.414837    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9ksk\" (UniqueName: \"kubernetes.io/projected/787ed8bc-d29c-4e3b-b661-87d0712a506f-kube-api-access-j9ksk\") pod \"maas-api-e2e-shared-a-w3-89a22a-7587d7b466-6c9dq\" (UID: \"787ed8bc-d29c-4e3b-b661-87d0712a506f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-89a22a-7587d7b466-6c9dq"
Nov 02 19:57:13.415766 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:13.415740    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/787ed8bc-d29c-4e3b-b661-87d0712a506f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-89a22a-7587d7b466-6c9dq\" (UID: \"787ed8bc-d29c-4e3b-b661-87d0712a506f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-89a22a-7587d7b466-6c9dq"
Nov 02 19:57:13.423659 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:13.423635    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/787ed8bc-d29c-4e3b-b661-87d0712a506f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-89a22a-7587d7b466-6c9dq\" (UID: \"787ed8bc-d29c-4e3b-b661-87d0712a506f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-89a22a-7587d7b466-6c9dq"
Nov 02 19:57:13.424071 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:13.423542    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/787ed8bc-d29c-4e3b-b661-87d0712a506f-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-89a22a-7587d7b466-6c9dq\" (UID: \"787ed8bc-d29c-4e3b-b661-87d0712a506f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-89a22a-7587d7b466-6c9dq"
Nov 02 19:57:13.424202 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:57:13.416071    4463 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-89a22a-serving-cert: secret "maas-api-e2e-shared-a-w3-89a22a-serving-cert" not found
Nov 02 19:57:13.424400 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:57:13.424379    4463 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/787ed8bc-d29c-4e3b-b661-87d0712a506f-maas-api-tls podName:787ed8bc-d29c-4e3b-b661-87d0712a506f nodeName:}" failed. No retries permitted until 2026-11-02 19:57:13.924336869 +0000 UTC m=+1740.531370632 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/787ed8bc-d29c-4e3b-b661-87d0712a506f-maas-api-tls") pod "maas-api-e2e-shared-a-w3-89a22a-7587d7b466-6c9dq" (UID: "787ed8bc-d29c-4e3b-b661-87d0712a506f") : secret "maas-api-e2e-shared-a-w3-89a22a-serving-cert" not found
Nov 02 19:57:13.425277 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:13.425255    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/787ed8bc-d29c-4e3b-b661-87d0712a506f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-89a22a-7587d7b466-6c9dq\" (UID: \"787ed8bc-d29c-4e3b-b661-87d0712a506f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-89a22a-7587d7b466-6c9dq"
Nov 02 19:57:13.467696 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:13.467619    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9ksk\" (UniqueName: \"kubernetes.io/projected/787ed8bc-d29c-4e3b-b661-87d0712a506f-kube-api-access-j9ksk\") pod \"maas-api-e2e-shared-a-w3-89a22a-7587d7b466-6c9dq\" (UID: \"787ed8bc-d29c-4e3b-b661-87d0712a506f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-89a22a-7587d7b466-6c9dq"
Nov 02 19:57:13.931462 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:13.931430    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/787ed8bc-d29c-4e3b-b661-87d0712a506f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-89a22a-7587d7b466-6c9dq\" (UID: \"787ed8bc-d29c-4e3b-b661-87d0712a506f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-89a22a-7587d7b466-6c9dq"
Nov 02 19:57:13.933857 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:13.933837    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/787ed8bc-d29c-4e3b-b661-87d0712a506f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-89a22a-7587d7b466-6c9dq\" (UID: \"787ed8bc-d29c-4e3b-b661-87d0712a506f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-89a22a-7587d7b466-6c9dq"
Nov 02 19:57:14.099006 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:14.098983    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-89a22a-7587d7b466-6c9dq"
Nov 02 19:57:14.470987 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:14.470956    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-89a22a-7587d7b466-6c9dq"]
Nov 02 19:57:14.471431 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:57:14.471409    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod787ed8bc_d29c_4e3b_b661_87d0712a506f.slice/crio-e2d03f5046f85e975105f71e3cac6a14df3dfe50a5945f95000aaa941d969eef WatchSource:0}: Error finding container e2d03f5046f85e975105f71e3cac6a14df3dfe50a5945f95000aaa941d969eef: Status 404 returned error can't find the container with id e2d03f5046f85e975105f71e3cac6a14df3dfe50a5945f95000aaa941d969eef
Nov 02 19:57:15.329265 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:15.329235    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-89a22a-7587d7b466-6c9dq" event={"ID":"787ed8bc-d29c-4e3b-b661-87d0712a506f","Type":"ContainerStarted","Data":"bde8f6b1107382fef39131b4e08043be4dba831b15aac8c46f2e5d1142922261"}
Nov 02 19:57:15.329483 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:15.329462    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-89a22a-7587d7b466-6c9dq"
Nov 02 19:57:15.329545 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:15.329507    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-89a22a-7587d7b466-6c9dq" event={"ID":"787ed8bc-d29c-4e3b-b661-87d0712a506f","Type":"ContainerStarted","Data":"e2d03f5046f85e975105f71e3cac6a14df3dfe50a5945f95000aaa941d969eef"}
Nov 02 19:57:15.470065 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:15.470000    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-89a22a-7587d7b466-6c9dq" podStartSLOduration=1.6894907030000001 podStartE2EDuration="2.469987206s" totalImagesPullingTime="780.496503ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:57:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:57:15.467216611 +0000 UTC m=+1742.074250395" watchObservedRunningTime="2026-11-02 19:57:15.469987206 +0000 UTC m=+1742.077020973"
Nov 02 19:57:21.341853 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:21.341822    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-89a22a-7587d7b466-6c9dq"
Nov 02 19:57:22.221251 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:22.221214    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-qdr9m"]
Nov 02 19:57:22.221433 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:22.221401    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-qdr9m" podUID="115b980d-6906-4134-8657-9e02a2e6a478" containerName="istio-proxy" containerID="cri-o://5788700e73dcedd7189aa5f250e4ae86c920062691f401a9c44d7cdd40a672aa" gracePeriod=30
Nov 02 19:57:27.414560 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:27.414524    4463 generic.go:356] "Generic (PLEG): container finished" podID="115b980d-6906-4134-8657-9e02a2e6a478" containerID="5788700e73dcedd7189aa5f250e4ae86c920062691f401a9c44d7cdd40a672aa" exitCode=0
Nov 02 19:57:27.415557 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:27.414642    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-qdr9m" event={"ID":"115b980d-6906-4134-8657-9e02a2e6a478","Type":"ContainerDied","Data":"5788700e73dcedd7189aa5f250e4ae86c920062691f401a9c44d7cdd40a672aa"}
Nov 02 19:57:27.492589 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:27.492531    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-qdr9m"
Nov 02 19:57:27.513449 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:27.513417    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/115b980d-6906-4134-8657-9e02a2e6a478-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/115b980d-6906-4134-8657-9e02a2e6a478-istio-envoy\") pod \"115b980d-6906-4134-8657-9e02a2e6a478\" (UID: \"115b980d-6906-4134-8657-9e02a2e6a478\") "
Nov 02 19:57:27.513595 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:27.513532    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/115b980d-6906-4134-8657-9e02a2e6a478-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/115b980d-6906-4134-8657-9e02a2e6a478-istio-podinfo\") pod \"115b980d-6906-4134-8657-9e02a2e6a478\" (UID: \"115b980d-6906-4134-8657-9e02a2e6a478\") "
Nov 02 19:57:27.513663 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:27.513606    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/115b980d-6906-4134-8657-9e02a2e6a478-istio-token\" (UniqueName: \"kubernetes.io/projected/115b980d-6906-4134-8657-9e02a2e6a478-istio-token\") pod \"115b980d-6906-4134-8657-9e02a2e6a478\" (UID: \"115b980d-6906-4134-8657-9e02a2e6a478\") "
Nov 02 19:57:27.513663 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:27.513658    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/115b980d-6906-4134-8657-9e02a2e6a478-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/115b980d-6906-4134-8657-9e02a2e6a478-istiod-ca-cert\") pod \"115b980d-6906-4134-8657-9e02a2e6a478\" (UID: \"115b980d-6906-4134-8657-9e02a2e6a478\") "
Nov 02 19:57:27.513766 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:27.513707    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/115b980d-6906-4134-8657-9e02a2e6a478-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/115b980d-6906-4134-8657-9e02a2e6a478-credential-socket\") pod \"115b980d-6906-4134-8657-9e02a2e6a478\" (UID: \"115b980d-6906-4134-8657-9e02a2e6a478\") "
Nov 02 19:57:27.513766 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:27.513756    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/115b980d-6906-4134-8657-9e02a2e6a478-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/115b980d-6906-4134-8657-9e02a2e6a478-workload-socket\") pod \"115b980d-6906-4134-8657-9e02a2e6a478\" (UID: \"115b980d-6906-4134-8657-9e02a2e6a478\") "
Nov 02 19:57:27.513871 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:27.513803    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/115b980d-6906-4134-8657-9e02a2e6a478-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/115b980d-6906-4134-8657-9e02a2e6a478-workload-certs\") pod \"115b980d-6906-4134-8657-9e02a2e6a478\" (UID: \"115b980d-6906-4134-8657-9e02a2e6a478\") "
Nov 02 19:57:27.513871 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:27.513851    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/115b980d-6906-4134-8657-9e02a2e6a478-kube-api-access-h2n82\" (UniqueName: \"kubernetes.io/projected/115b980d-6906-4134-8657-9e02a2e6a478-kube-api-access-h2n82\") pod \"115b980d-6906-4134-8657-9e02a2e6a478\" (UID: \"115b980d-6906-4134-8657-9e02a2e6a478\") "
Nov 02 19:57:27.513970 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:27.513925    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/115b980d-6906-4134-8657-9e02a2e6a478-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/115b980d-6906-4134-8657-9e02a2e6a478-istio-data\") pod \"115b980d-6906-4134-8657-9e02a2e6a478\" (UID: \"115b980d-6906-4134-8657-9e02a2e6a478\") "
Nov 02 19:57:27.514162 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:27.514123    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/115b980d-6906-4134-8657-9e02a2e6a478-istiod-ca-cert" pod "115b980d-6906-4134-8657-9e02a2e6a478" (UID: "115b980d-6906-4134-8657-9e02a2e6a478"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 19:57:27.514718 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:27.514160    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/115b980d-6906-4134-8657-9e02a2e6a478-workload-certs" pod "115b980d-6906-4134-8657-9e02a2e6a478" (UID: "115b980d-6906-4134-8657-9e02a2e6a478"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:57:27.514718 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:27.514475    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/115b980d-6906-4134-8657-9e02a2e6a478-credential-socket" pod "115b980d-6906-4134-8657-9e02a2e6a478" (UID: "115b980d-6906-4134-8657-9e02a2e6a478"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:57:27.514906 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:27.514524    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/115b980d-6906-4134-8657-9e02a2e6a478-workload-socket" pod "115b980d-6906-4134-8657-9e02a2e6a478" (UID: "115b980d-6906-4134-8657-9e02a2e6a478"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:57:27.514906 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:27.514817    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/115b980d-6906-4134-8657-9e02a2e6a478-istio-data" pod "115b980d-6906-4134-8657-9e02a2e6a478" (UID: "115b980d-6906-4134-8657-9e02a2e6a478"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:57:27.515919 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:27.515900    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/115b980d-6906-4134-8657-9e02a2e6a478-istio-token" pod "115b980d-6906-4134-8657-9e02a2e6a478" (UID: "115b980d-6906-4134-8657-9e02a2e6a478"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:57:27.516053 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:27.516033    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/115b980d-6906-4134-8657-9e02a2e6a478-istio-envoy" pod "115b980d-6906-4134-8657-9e02a2e6a478" (UID: "115b980d-6906-4134-8657-9e02a2e6a478"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:57:27.516131 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:27.516096    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/115b980d-6906-4134-8657-9e02a2e6a478-istio-podinfo" pod "115b980d-6906-4134-8657-9e02a2e6a478" (UID: "115b980d-6906-4134-8657-9e02a2e6a478"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 19:57:27.516235 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:27.516221    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/115b980d-6906-4134-8657-9e02a2e6a478-kube-api-access-h2n82" pod "115b980d-6906-4134-8657-9e02a2e6a478" (UID: "115b980d-6906-4134-8657-9e02a2e6a478"). InnerVolumeSpecName "kube-api-access-h2n82". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:57:27.615849 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:27.615821    4463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/115b980d-6906-4134-8657-9e02a2e6a478-istiod-ca-cert\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:57:27.615849 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:27.615847    4463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/115b980d-6906-4134-8657-9e02a2e6a478-credential-socket\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:57:27.616035 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:27.615856    4463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/115b980d-6906-4134-8657-9e02a2e6a478-workload-socket\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:57:27.616035 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:27.615864    4463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/115b980d-6906-4134-8657-9e02a2e6a478-workload-certs\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:57:27.616035 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:27.615883    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2n82\" (UniqueName: \"kubernetes.io/projected/115b980d-6906-4134-8657-9e02a2e6a478-kube-api-access-h2n82\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:57:27.616035 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:27.615892    4463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/115b980d-6906-4134-8657-9e02a2e6a478-istio-data\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:57:27.616035 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:27.615900    4463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/115b980d-6906-4134-8657-9e02a2e6a478-istio-envoy\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:57:27.616035 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:27.615908    4463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/115b980d-6906-4134-8657-9e02a2e6a478-istio-podinfo\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:57:27.616035 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:27.615915    4463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/115b980d-6906-4134-8657-9e02a2e6a478-istio-token\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:57:28.423191 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:28.423164    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-qdr9m"
Nov 02 19:57:28.423191 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:28.423181    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-qdr9m" event={"ID":"115b980d-6906-4134-8657-9e02a2e6a478","Type":"ContainerDied","Data":"22cc4bfb2622b5d517a0c66f7e9553e58f45929c441787fd5b37aa9eae8da54f"}
Nov 02 19:57:28.423676 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:28.423238    4463 scope.go:122] "RemoveContainer" containerID="5788700e73dcedd7189aa5f250e4ae86c920062691f401a9c44d7cdd40a672aa"
Nov 02 19:57:28.628666 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:28.628637    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-qdr9m"]
Nov 02 19:57:28.675056 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:28.674953    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-qdr9m"]
Nov 02 19:57:29.585406 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:29.585374    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="115b980d-6906-4134-8657-9e02a2e6a478" path="/var/lib/kubelet/pods/115b980d-6906-4134-8657-9e02a2e6a478/volumes"
Nov 02 19:57:30.905848 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:30.905818    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b0cc24-75c69845d6-zhjrj"]
Nov 02 19:57:30.906513 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:30.906498    4463 cpu_manager.go:414] "containerMap: removing container" podUID="115b980d-6906-4134-8657-9e02a2e6a478" containerName="istio-proxy"
Nov 02 19:57:30.906513 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:30.906514    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="115b980d-6906-4134-8657-9e02a2e6a478" containerName="istio-proxy"
Nov 02 19:57:30.906612 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:30.906603    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="115b980d-6906-4134-8657-9e02a2e6a478" containerName="istio-proxy"
Nov 02 19:57:30.910567 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:30.910551    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b0cc24-75c69845d6-zhjrj"
Nov 02 19:57:30.958531 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:30.958508    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef952ef1-649a-4551-b974-d0b00d733f7e-metrics-certs\") pod \"maas-api-e2e-inf-a-b0cc24-75c69845d6-zhjrj\" (UID: \"ef952ef1-649a-4551-b974-d0b00d733f7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b0cc24-75c69845d6-zhjrj"
Nov 02 19:57:30.958648 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:30.958618    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6rbq\" (UniqueName: \"kubernetes.io/projected/ef952ef1-649a-4551-b974-d0b00d733f7e-kube-api-access-j6rbq\") pod \"maas-api-e2e-inf-a-b0cc24-75c69845d6-zhjrj\" (UID: \"ef952ef1-649a-4551-b974-d0b00d733f7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b0cc24-75c69845d6-zhjrj"
Nov 02 19:57:30.958788 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:30.958765    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef952ef1-649a-4551-b974-d0b00d733f7e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b0cc24-75c69845d6-zhjrj\" (UID: \"ef952ef1-649a-4551-b974-d0b00d733f7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b0cc24-75c69845d6-zhjrj"
Nov 02 19:57:30.958829 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:30.958816    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef952ef1-649a-4551-b974-d0b00d733f7e-maas-api-tls\") pod \"maas-api-e2e-inf-a-b0cc24-75c69845d6-zhjrj\" (UID: \"ef952ef1-649a-4551-b974-d0b00d733f7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b0cc24-75c69845d6-zhjrj"
Nov 02 19:57:31.006537 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:31.006505    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-b0cc24-serving-cert\""
Nov 02 19:57:31.012425 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:31.012398    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b0cc24-75c69845d6-zhjrj"]
Nov 02 19:57:31.060688 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:31.060659    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef952ef1-649a-4551-b974-d0b00d733f7e-metrics-certs\") pod \"maas-api-e2e-inf-a-b0cc24-75c69845d6-zhjrj\" (UID: \"ef952ef1-649a-4551-b974-d0b00d733f7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b0cc24-75c69845d6-zhjrj"
Nov 02 19:57:31.060838 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:31.060759    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6rbq\" (UniqueName: \"kubernetes.io/projected/ef952ef1-649a-4551-b974-d0b00d733f7e-kube-api-access-j6rbq\") pod \"maas-api-e2e-inf-a-b0cc24-75c69845d6-zhjrj\" (UID: \"ef952ef1-649a-4551-b974-d0b00d733f7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b0cc24-75c69845d6-zhjrj"
Nov 02 19:57:31.060908 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:31.060879    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef952ef1-649a-4551-b974-d0b00d733f7e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b0cc24-75c69845d6-zhjrj\" (UID: \"ef952ef1-649a-4551-b974-d0b00d733f7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b0cc24-75c69845d6-zhjrj"
Nov 02 19:57:31.060968 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:31.060921    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef952ef1-649a-4551-b974-d0b00d733f7e-maas-api-tls\") pod \"maas-api-e2e-inf-a-b0cc24-75c69845d6-zhjrj\" (UID: \"ef952ef1-649a-4551-b974-d0b00d733f7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b0cc24-75c69845d6-zhjrj"
Nov 02 19:57:31.061097 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:57:31.061079    4463 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-b0cc24-serving-cert: secret "maas-api-e2e-inf-a-b0cc24-serving-cert" not found
Nov 02 19:57:31.061170 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:57:31.061145    4463 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef952ef1-649a-4551-b974-d0b00d733f7e-maas-api-tls podName:ef952ef1-649a-4551-b974-d0b00d733f7e nodeName:}" failed. No retries permitted until 2026-11-02 19:57:31.561125946 +0000 UTC m=+1758.168159698 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ef952ef1-649a-4551-b974-d0b00d733f7e-maas-api-tls") pod "maas-api-e2e-inf-a-b0cc24-75c69845d6-zhjrj" (UID: "ef952ef1-649a-4551-b974-d0b00d733f7e") : secret "maas-api-e2e-inf-a-b0cc24-serving-cert" not found
Nov 02 19:57:31.061686 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:31.061659    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef952ef1-649a-4551-b974-d0b00d733f7e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b0cc24-75c69845d6-zhjrj\" (UID: \"ef952ef1-649a-4551-b974-d0b00d733f7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b0cc24-75c69845d6-zhjrj"
Nov 02 19:57:31.063222 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:31.063200    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef952ef1-649a-4551-b974-d0b00d733f7e-metrics-certs\") pod \"maas-api-e2e-inf-a-b0cc24-75c69845d6-zhjrj\" (UID: \"ef952ef1-649a-4551-b974-d0b00d733f7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b0cc24-75c69845d6-zhjrj"
Nov 02 19:57:31.116207 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:31.116182    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6rbq\" (UniqueName: \"kubernetes.io/projected/ef952ef1-649a-4551-b974-d0b00d733f7e-kube-api-access-j6rbq\") pod \"maas-api-e2e-inf-a-b0cc24-75c69845d6-zhjrj\" (UID: \"ef952ef1-649a-4551-b974-d0b00d733f7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b0cc24-75c69845d6-zhjrj"
Nov 02 19:57:31.567623 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:31.567591    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef952ef1-649a-4551-b974-d0b00d733f7e-maas-api-tls\") pod \"maas-api-e2e-inf-a-b0cc24-75c69845d6-zhjrj\" (UID: \"ef952ef1-649a-4551-b974-d0b00d733f7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b0cc24-75c69845d6-zhjrj"
Nov 02 19:57:31.569916 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:31.569892    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef952ef1-649a-4551-b974-d0b00d733f7e-maas-api-tls\") pod \"maas-api-e2e-inf-a-b0cc24-75c69845d6-zhjrj\" (UID: \"ef952ef1-649a-4551-b974-d0b00d733f7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b0cc24-75c69845d6-zhjrj"
Nov 02 19:57:31.823964 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:31.823882    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b0cc24-75c69845d6-zhjrj"
Nov 02 19:57:32.193587 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:32.193542    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b0cc24-75c69845d6-zhjrj"]
Nov 02 19:57:32.194388 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:57:32.194356    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef952ef1_649a_4551_b974_d0b00d733f7e.slice/crio-39ad1c16cc195b816f08159ea39596a79a7b618d6bdb342132f54044134404ef WatchSource:0}: Error finding container 39ad1c16cc195b816f08159ea39596a79a7b618d6bdb342132f54044134404ef: Status 404 returned error can't find the container with id 39ad1c16cc195b816f08159ea39596a79a7b618d6bdb342132f54044134404ef
Nov 02 19:57:32.303493 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:32.303465    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-66lh4"]
Nov 02 19:57:32.303658 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:32.303635    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-66lh4" podUID="307e0c79-e6b1-4751-b226-966f9b8f9283" containerName="discovery" containerID="cri-o://03510b69d335002bd8d59e325ccf69282f579d444c52e19dba82d6cb7f666884" gracePeriod=30
Nov 02 19:57:32.458873 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:32.458778    4463 generic.go:356] "Generic (PLEG): container finished" podID="307e0c79-e6b1-4751-b226-966f9b8f9283" containerID="03510b69d335002bd8d59e325ccf69282f579d444c52e19dba82d6cb7f666884" exitCode=0
Nov 02 19:57:32.459068 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:32.458856    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-66lh4" event={"ID":"307e0c79-e6b1-4751-b226-966f9b8f9283","Type":"ContainerDied","Data":"03510b69d335002bd8d59e325ccf69282f579d444c52e19dba82d6cb7f666884"}
Nov 02 19:57:32.460393 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:32.460347    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b0cc24-75c69845d6-zhjrj" event={"ID":"ef952ef1-649a-4551-b974-d0b00d733f7e","Type":"ContainerStarted","Data":"39ad1c16cc195b816f08159ea39596a79a7b618d6bdb342132f54044134404ef"}
Nov 02 19:57:32.604166 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:32.604141    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-66lh4"
Nov 02 19:57:32.685997 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:32.685967    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/307e0c79-e6b1-4751-b226-966f9b8f9283-kube-api-access-rmvvk\" (UniqueName: \"kubernetes.io/projected/307e0c79-e6b1-4751-b226-966f9b8f9283-kube-api-access-rmvvk\") pod \"307e0c79-e6b1-4751-b226-966f9b8f9283\" (UID: \"307e0c79-e6b1-4751-b226-966f9b8f9283\") "
Nov 02 19:57:32.686187 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:32.686070    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/307e0c79-e6b1-4751-b226-966f9b8f9283-istio-token\" (UniqueName: \"kubernetes.io/projected/307e0c79-e6b1-4751-b226-966f9b8f9283-istio-token\") pod \"307e0c79-e6b1-4751-b226-966f9b8f9283\" (UID: \"307e0c79-e6b1-4751-b226-966f9b8f9283\") "
Nov 02 19:57:32.686257 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:32.686202    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/307e0c79-e6b1-4751-b226-966f9b8f9283-cacerts\" (UniqueName: \"kubernetes.io/secret/307e0c79-e6b1-4751-b226-966f9b8f9283-cacerts\") pod \"307e0c79-e6b1-4751-b226-966f9b8f9283\" (UID: \"307e0c79-e6b1-4751-b226-966f9b8f9283\") "
Nov 02 19:57:32.686315 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:32.686264    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/307e0c79-e6b1-4751-b226-966f9b8f9283-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/307e0c79-e6b1-4751-b226-966f9b8f9283-local-certs\") pod \"307e0c79-e6b1-4751-b226-966f9b8f9283\" (UID: \"307e0c79-e6b1-4751-b226-966f9b8f9283\") "
Nov 02 19:57:32.686315 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:32.686303    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/307e0c79-e6b1-4751-b226-966f9b8f9283-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/307e0c79-e6b1-4751-b226-966f9b8f9283-istio-csr-dns-cert\") pod \"307e0c79-e6b1-4751-b226-966f9b8f9283\" (UID: \"307e0c79-e6b1-4751-b226-966f9b8f9283\") "
Nov 02 19:57:32.686429 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:32.686357    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/307e0c79-e6b1-4751-b226-966f9b8f9283-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/307e0c79-e6b1-4751-b226-966f9b8f9283-istio-csr-ca-configmap\") pod \"307e0c79-e6b1-4751-b226-966f9b8f9283\" (UID: \"307e0c79-e6b1-4751-b226-966f9b8f9283\") "
Nov 02 19:57:32.686480 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:32.686428    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/307e0c79-e6b1-4751-b226-966f9b8f9283-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/307e0c79-e6b1-4751-b226-966f9b8f9283-istio-kubeconfig\") pod \"307e0c79-e6b1-4751-b226-966f9b8f9283\" (UID: \"307e0c79-e6b1-4751-b226-966f9b8f9283\") "
Nov 02 19:57:32.687253 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:32.686848    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/307e0c79-e6b1-4751-b226-966f9b8f9283-istio-csr-ca-configmap" pod "307e0c79-e6b1-4751-b226-966f9b8f9283" (UID: "307e0c79-e6b1-4751-b226-966f9b8f9283"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 19:57:32.687253 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:32.686991    4463 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/307e0c79-e6b1-4751-b226-966f9b8f9283-istio-csr-ca-configmap\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:57:32.688784 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:32.688753    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/307e0c79-e6b1-4751-b226-966f9b8f9283-cacerts" pod "307e0c79-e6b1-4751-b226-966f9b8f9283" (UID: "307e0c79-e6b1-4751-b226-966f9b8f9283"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 19:57:32.688929 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:32.688816    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/307e0c79-e6b1-4751-b226-966f9b8f9283-local-certs" pod "307e0c79-e6b1-4751-b226-966f9b8f9283" (UID: "307e0c79-e6b1-4751-b226-966f9b8f9283"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:57:32.688929 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:32.688864    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/307e0c79-e6b1-4751-b226-966f9b8f9283-kube-api-access-rmvvk" pod "307e0c79-e6b1-4751-b226-966f9b8f9283" (UID: "307e0c79-e6b1-4751-b226-966f9b8f9283"). InnerVolumeSpecName "kube-api-access-rmvvk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:57:32.688929 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:32.688865    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/307e0c79-e6b1-4751-b226-966f9b8f9283-istio-csr-dns-cert" pod "307e0c79-e6b1-4751-b226-966f9b8f9283" (UID: "307e0c79-e6b1-4751-b226-966f9b8f9283"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 19:57:32.689248 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:32.689221    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/307e0c79-e6b1-4751-b226-966f9b8f9283-istio-kubeconfig" pod "307e0c79-e6b1-4751-b226-966f9b8f9283" (UID: "307e0c79-e6b1-4751-b226-966f9b8f9283"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 19:57:32.689248 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:32.689238    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/307e0c79-e6b1-4751-b226-966f9b8f9283-istio-token" pod "307e0c79-e6b1-4751-b226-966f9b8f9283" (UID: "307e0c79-e6b1-4751-b226-966f9b8f9283"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:57:32.788566 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:32.788490    4463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/307e0c79-e6b1-4751-b226-966f9b8f9283-istio-token\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:57:32.788566 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:32.788516    4463 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/307e0c79-e6b1-4751-b226-966f9b8f9283-cacerts\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:57:32.788566 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:32.788526    4463 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/307e0c79-e6b1-4751-b226-966f9b8f9283-local-certs\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:57:32.788566 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:32.788534    4463 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/307e0c79-e6b1-4751-b226-966f9b8f9283-istio-csr-dns-cert\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:57:32.788566 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:32.788543    4463 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/307e0c79-e6b1-4751-b226-966f9b8f9283-istio-kubeconfig\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:57:32.788566 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:32.788552    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmvvk\" (UniqueName: \"kubernetes.io/projected/307e0c79-e6b1-4751-b226-966f9b8f9283-kube-api-access-rmvvk\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:57:33.468325 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:33.468290    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-66lh4"
Nov 02 19:57:33.468325 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:33.468299    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-66lh4" event={"ID":"307e0c79-e6b1-4751-b226-966f9b8f9283","Type":"ContainerDied","Data":"5abc6126f1f9ae6666b81fd46a990d994d32b293996a0683600f0d80dd909bec"}
Nov 02 19:57:33.468887 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:33.468381    4463 scope.go:122] "RemoveContainer" containerID="03510b69d335002bd8d59e325ccf69282f579d444c52e19dba82d6cb7f666884"
Nov 02 19:57:33.470139 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:33.470119    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b0cc24-75c69845d6-zhjrj" event={"ID":"ef952ef1-649a-4551-b974-d0b00d733f7e","Type":"ContainerStarted","Data":"bb685e19cfa709de8c031dc6f63169978cb76015291cf349d0ca4117bef01039"}
Nov 02 19:57:33.470294 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:33.470279    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b0cc24-75c69845d6-zhjrj"
Nov 02 19:57:33.623515 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:33.623484    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-66lh4"]
Nov 02 19:57:33.693920 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:33.692240    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-66lh4"]
Nov 02 19:57:33.730380 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:33.730267    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b0cc24-75c69845d6-zhjrj" podStartSLOduration=2.910183979 podStartE2EDuration="3.730250098s" totalImagesPullingTime="820.066119ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:57:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:57:33.730213068 +0000 UTC m=+1760.337246847" watchObservedRunningTime="2026-11-02 19:57:33.730250098 +0000 UTC m=+1760.337283868"
Nov 02 19:57:34.294763 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:57:34.294733    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 19:57:34.841577 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:34.841526    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-kfx7k"]
Nov 02 19:57:34.842003 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:34.841733    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-kfx7k" podUID="0714f5c6-adfb-434e-8dd9-9bada5fbed29" containerName="istio-proxy" containerID="cri-o://802ced2434f4911faa946da31c34e5d042f675a5e745cbd07869139eab1971a0" gracePeriod=30
Nov 02 19:57:35.586075 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:35.586040    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="307e0c79-e6b1-4751-b226-966f9b8f9283" path="/var/lib/kubelet/pods/307e0c79-e6b1-4751-b226-966f9b8f9283/volumes"
Nov 02 19:57:37.188621 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:37.188589    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-bbt8n"]
Nov 02 19:57:37.189042 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:37.188771    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-bbt8n" podUID="0acfeac9-a741-45ec-afab-8fb4ecf5975f" containerName="istio-proxy" containerID="cri-o://1a3769a6f7367c62e1966d701f12fbf651e3b9dd1909b6e0b9f53ee37e507052" gracePeriod=30
Nov 02 19:57:37.287029 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:37.286985    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-2zrqj"]
Nov 02 19:57:37.287274 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:37.287245    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-2zrqj" podUID="a8216408-95d0-413b-a609-2ca8cf3299b6" containerName="istio-proxy" containerID="cri-o://f56ee5a76c4bf7aeafdf7fb898d4cbccecfe3d39d9505d83fa7eea05d94a0c75" gracePeriod=30
Nov 02 19:57:37.330157 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:37.330126    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fjnvm"]
Nov 02 19:57:37.330387 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:37.330363    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fjnvm" podUID="48eb73cb-231a-4462-9236-da01f4734826" containerName="istio-proxy" containerID="cri-o://c4b358dd25656c7e93235a7be81307b24c00d8675f0add06277a95d120bfbfc0" gracePeriod=30
Nov 02 19:57:39.549379 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:39.549351    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b0cc24-75c69845d6-zhjrj"
Nov 02 19:57:40.106393 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:40.106338    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-kfx7k"
Nov 02 19:57:40.190470 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:40.190443    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0714f5c6-adfb-434e-8dd9-9bada5fbed29-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0714f5c6-adfb-434e-8dd9-9bada5fbed29-istiod-ca-cert\") pod \"0714f5c6-adfb-434e-8dd9-9bada5fbed29\" (UID: \"0714f5c6-adfb-434e-8dd9-9bada5fbed29\") "
Nov 02 19:57:40.190601 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:40.190588    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0714f5c6-adfb-434e-8dd9-9bada5fbed29-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0714f5c6-adfb-434e-8dd9-9bada5fbed29-workload-certs\") pod \"0714f5c6-adfb-434e-8dd9-9bada5fbed29\" (UID: \"0714f5c6-adfb-434e-8dd9-9bada5fbed29\") "
Nov 02 19:57:40.190644 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:40.190635    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0714f5c6-adfb-434e-8dd9-9bada5fbed29-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0714f5c6-adfb-434e-8dd9-9bada5fbed29-workload-socket\") pod \"0714f5c6-adfb-434e-8dd9-9bada5fbed29\" (UID: \"0714f5c6-adfb-434e-8dd9-9bada5fbed29\") "
Nov 02 19:57:40.190715 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:40.190700    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0714f5c6-adfb-434e-8dd9-9bada5fbed29-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0714f5c6-adfb-434e-8dd9-9bada5fbed29-credential-socket\") pod \"0714f5c6-adfb-434e-8dd9-9bada5fbed29\" (UID: \"0714f5c6-adfb-434e-8dd9-9bada5fbed29\") "
Nov 02 19:57:40.190748 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:40.190732    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0714f5c6-adfb-434e-8dd9-9bada5fbed29-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0714f5c6-adfb-434e-8dd9-9bada5fbed29-istio-podinfo\") pod \"0714f5c6-adfb-434e-8dd9-9bada5fbed29\" (UID: \"0714f5c6-adfb-434e-8dd9-9bada5fbed29\") "
Nov 02 19:57:40.190787 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:40.190772    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0714f5c6-adfb-434e-8dd9-9bada5fbed29-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0714f5c6-adfb-434e-8dd9-9bada5fbed29-istio-data\") pod \"0714f5c6-adfb-434e-8dd9-9bada5fbed29\" (UID: \"0714f5c6-adfb-434e-8dd9-9bada5fbed29\") "
Nov 02 19:57:40.190831 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:40.190791    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0714f5c6-adfb-434e-8dd9-9bada5fbed29-istiod-ca-cert" pod "0714f5c6-adfb-434e-8dd9-9bada5fbed29" (UID: "0714f5c6-adfb-434e-8dd9-9bada5fbed29"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 19:57:40.190914 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:40.190893    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0714f5c6-adfb-434e-8dd9-9bada5fbed29-workload-certs" pod "0714f5c6-adfb-434e-8dd9-9bada5fbed29" (UID: "0714f5c6-adfb-434e-8dd9-9bada5fbed29"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:57:40.190980 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:40.190962    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0714f5c6-adfb-434e-8dd9-9bada5fbed29-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0714f5c6-adfb-434e-8dd9-9bada5fbed29-istio-envoy\") pod \"0714f5c6-adfb-434e-8dd9-9bada5fbed29\" (UID: \"0714f5c6-adfb-434e-8dd9-9bada5fbed29\") "
Nov 02 19:57:40.191063 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:40.190996    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0714f5c6-adfb-434e-8dd9-9bada5fbed29-workload-socket" pod "0714f5c6-adfb-434e-8dd9-9bada5fbed29" (UID: "0714f5c6-adfb-434e-8dd9-9bada5fbed29"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:57:40.191063 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:40.190983    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0714f5c6-adfb-434e-8dd9-9bada5fbed29-credential-socket" pod "0714f5c6-adfb-434e-8dd9-9bada5fbed29" (UID: "0714f5c6-adfb-434e-8dd9-9bada5fbed29"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:57:40.191063 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:40.190998    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0714f5c6-adfb-434e-8dd9-9bada5fbed29-istio-token\" (UniqueName: \"kubernetes.io/projected/0714f5c6-adfb-434e-8dd9-9bada5fbed29-istio-token\") pod \"0714f5c6-adfb-434e-8dd9-9bada5fbed29\" (UID: \"0714f5c6-adfb-434e-8dd9-9bada5fbed29\") "
Nov 02 19:57:40.191063 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:40.191049    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0714f5c6-adfb-434e-8dd9-9bada5fbed29-istio-data" pod "0714f5c6-adfb-434e-8dd9-9bada5fbed29" (UID: "0714f5c6-adfb-434e-8dd9-9bada5fbed29"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:57:40.191258 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:40.191091    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0714f5c6-adfb-434e-8dd9-9bada5fbed29-kube-api-access-4kjvg\" (UniqueName: \"kubernetes.io/projected/0714f5c6-adfb-434e-8dd9-9bada5fbed29-kube-api-access-4kjvg\") pod \"0714f5c6-adfb-434e-8dd9-9bada5fbed29\" (UID: \"0714f5c6-adfb-434e-8dd9-9bada5fbed29\") "
Nov 02 19:57:40.191827 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:40.191803    4463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0714f5c6-adfb-434e-8dd9-9bada5fbed29-workload-certs\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:57:40.191924 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:40.191831    4463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0714f5c6-adfb-434e-8dd9-9bada5fbed29-workload-socket\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:57:40.191924 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:40.191847    4463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0714f5c6-adfb-434e-8dd9-9bada5fbed29-credential-socket\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:57:40.191924 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:40.191863    4463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0714f5c6-adfb-434e-8dd9-9bada5fbed29-istio-data\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:57:40.191924 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:40.191877    4463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0714f5c6-adfb-434e-8dd9-9bada5fbed29-istiod-ca-cert\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:57:40.193101 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:40.193071    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0714f5c6-adfb-434e-8dd9-9bada5fbed29-istio-podinfo" pod "0714f5c6-adfb-434e-8dd9-9bada5fbed29" (UID: "0714f5c6-adfb-434e-8dd9-9bada5fbed29"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 19:57:40.193101 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:40.193080    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0714f5c6-adfb-434e-8dd9-9bada5fbed29-istio-token" pod "0714f5c6-adfb-434e-8dd9-9bada5fbed29" (UID: "0714f5c6-adfb-434e-8dd9-9bada5fbed29"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:57:40.193234 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:40.193135    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0714f5c6-adfb-434e-8dd9-9bada5fbed29-istio-envoy" pod "0714f5c6-adfb-434e-8dd9-9bada5fbed29" (UID: "0714f5c6-adfb-434e-8dd9-9bada5fbed29"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:57:40.193234 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:40.193181    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0714f5c6-adfb-434e-8dd9-9bada5fbed29-kube-api-access-4kjvg" pod "0714f5c6-adfb-434e-8dd9-9bada5fbed29" (UID: "0714f5c6-adfb-434e-8dd9-9bada5fbed29"). InnerVolumeSpecName "kube-api-access-4kjvg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:57:40.293078 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:40.293047    4463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0714f5c6-adfb-434e-8dd9-9bada5fbed29-istio-podinfo\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:57:40.293078 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:40.293072    4463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0714f5c6-adfb-434e-8dd9-9bada5fbed29-istio-envoy\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:57:40.293078 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:40.293081    4463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0714f5c6-adfb-434e-8dd9-9bada5fbed29-istio-token\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:57:40.293278 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:40.293090    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kjvg\" (UniqueName: \"kubernetes.io/projected/0714f5c6-adfb-434e-8dd9-9bada5fbed29-kube-api-access-4kjvg\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:57:40.586315 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:40.586280    4463 generic.go:356] "Generic (PLEG): container finished" podID="0714f5c6-adfb-434e-8dd9-9bada5fbed29" containerID="802ced2434f4911faa946da31c34e5d042f675a5e745cbd07869139eab1971a0" exitCode=0
Nov 02 19:57:40.586686 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:40.586355    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-kfx7k"
Nov 02 19:57:40.586686 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:40.586360    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-kfx7k" event={"ID":"0714f5c6-adfb-434e-8dd9-9bada5fbed29","Type":"ContainerDied","Data":"802ced2434f4911faa946da31c34e5d042f675a5e745cbd07869139eab1971a0"}
Nov 02 19:57:40.586686 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:40.586466    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-kfx7k" event={"ID":"0714f5c6-adfb-434e-8dd9-9bada5fbed29","Type":"ContainerDied","Data":"7fc900106940f9b3102f9499150872572164b1bf175f852d73a86c92e12dcd62"}
Nov 02 19:57:40.586686 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:40.586490    4463 scope.go:122] "RemoveContainer" containerID="802ced2434f4911faa946da31c34e5d042f675a5e745cbd07869139eab1971a0"
Nov 02 19:57:40.598855 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:40.598827    4463 scope.go:122] "RemoveContainer" containerID="802ced2434f4911faa946da31c34e5d042f675a5e745cbd07869139eab1971a0"
Nov 02 19:57:40.599106 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:57:40.599086    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"802ced2434f4911faa946da31c34e5d042f675a5e745cbd07869139eab1971a0\": container with ID starting with 802ced2434f4911faa946da31c34e5d042f675a5e745cbd07869139eab1971a0 not found: ID does not exist" containerID="802ced2434f4911faa946da31c34e5d042f675a5e745cbd07869139eab1971a0"
Nov 02 19:57:40.599178 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:40.599112    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"802ced2434f4911faa946da31c34e5d042f675a5e745cbd07869139eab1971a0"} err="failed to get container status \"802ced2434f4911faa946da31c34e5d042f675a5e745cbd07869139eab1971a0\": rpc error: code = NotFound desc = could not find container \"802ced2434f4911faa946da31c34e5d042f675a5e745cbd07869139eab1971a0\": container with ID starting with 802ced2434f4911faa946da31c34e5d042f675a5e745cbd07869139eab1971a0 not found: ID does not exist"
Nov 02 19:57:40.742755 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:40.742726    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-kfx7k"]
Nov 02 19:57:40.786792 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:40.786763    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-kfx7k"]
Nov 02 19:57:41.586006 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:41.585975    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0714f5c6-adfb-434e-8dd9-9bada5fbed29" path="/var/lib/kubelet/pods/0714f5c6-adfb-434e-8dd9-9bada5fbed29/volumes"
Nov 02 19:57:42.481033 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:42.480185    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-bbt8n"
Nov 02 19:57:42.605487 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:42.605460    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fjnvm"
Nov 02 19:57:42.606118 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:42.606090    4463 generic.go:356] "Generic (PLEG): container finished" podID="48eb73cb-231a-4462-9236-da01f4734826" containerID="c4b358dd25656c7e93235a7be81307b24c00d8675f0add06277a95d120bfbfc0" exitCode=0
Nov 02 19:57:42.606241 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:42.606124    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fjnvm" event={"ID":"48eb73cb-231a-4462-9236-da01f4734826","Type":"ContainerDied","Data":"c4b358dd25656c7e93235a7be81307b24c00d8675f0add06277a95d120bfbfc0"}
Nov 02 19:57:42.606241 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:42.606191    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fjnvm" event={"ID":"48eb73cb-231a-4462-9236-da01f4734826","Type":"ContainerDied","Data":"8175216fcbb7e57cbe0715d86f490807c1dc7e9cf01d2c2e1144bd1d260e83c4"}
Nov 02 19:57:42.606241 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:42.606216    4463 scope.go:122] "RemoveContainer" containerID="c4b358dd25656c7e93235a7be81307b24c00d8675f0add06277a95d120bfbfc0"
Nov 02 19:57:42.609291 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:42.609266    4463 generic.go:356] "Generic (PLEG): container finished" podID="a8216408-95d0-413b-a609-2ca8cf3299b6" containerID="f56ee5a76c4bf7aeafdf7fb898d4cbccecfe3d39d9505d83fa7eea05d94a0c75" exitCode=0
Nov 02 19:57:42.609471 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:42.609450    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-2zrqj" event={"ID":"a8216408-95d0-413b-a609-2ca8cf3299b6","Type":"ContainerDied","Data":"f56ee5a76c4bf7aeafdf7fb898d4cbccecfe3d39d9505d83fa7eea05d94a0c75"}
Nov 02 19:57:42.613182 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:42.613158    4463 generic.go:356] "Generic (PLEG): container finished" podID="0acfeac9-a741-45ec-afab-8fb4ecf5975f" containerID="1a3769a6f7367c62e1966d701f12fbf651e3b9dd1909b6e0b9f53ee37e507052" exitCode=0
Nov 02 19:57:42.613344 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:42.613265    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-bbt8n" event={"ID":"0acfeac9-a741-45ec-afab-8fb4ecf5975f","Type":"ContainerDied","Data":"1a3769a6f7367c62e1966d701f12fbf651e3b9dd1909b6e0b9f53ee37e507052"}
Nov 02 19:57:42.613344 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:42.613305    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-bbt8n" event={"ID":"0acfeac9-a741-45ec-afab-8fb4ecf5975f","Type":"ContainerDied","Data":"f86bde7b887e0d7ee739ddd45c6bdff4b405dfbb6ffca9e18aed1f7f8b993bec"}
Nov 02 19:57:42.613524 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:42.613371    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-bbt8n"
Nov 02 19:57:42.623242 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:42.623223    4463 scope.go:122] "RemoveContainer" containerID="c4b358dd25656c7e93235a7be81307b24c00d8675f0add06277a95d120bfbfc0"
Nov 02 19:57:42.623510 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:57:42.623491    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4b358dd25656c7e93235a7be81307b24c00d8675f0add06277a95d120bfbfc0\": container with ID starting with c4b358dd25656c7e93235a7be81307b24c00d8675f0add06277a95d120bfbfc0 not found: ID does not exist" containerID="c4b358dd25656c7e93235a7be81307b24c00d8675f0add06277a95d120bfbfc0"
Nov 02 19:57:42.623574 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:42.623516    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4b358dd25656c7e93235a7be81307b24c00d8675f0add06277a95d120bfbfc0"} err="failed to get container status \"c4b358dd25656c7e93235a7be81307b24c00d8675f0add06277a95d120bfbfc0\": rpc error: code = NotFound desc = could not find container \"c4b358dd25656c7e93235a7be81307b24c00d8675f0add06277a95d120bfbfc0\": container with ID starting with c4b358dd25656c7e93235a7be81307b24c00d8675f0add06277a95d120bfbfc0 not found: ID does not exist"
Nov 02 19:57:42.623574 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:42.623531    4463 scope.go:122] "RemoveContainer" containerID="22cf3df17c532e6521ea22300285e0798bc8f9e96c1c973b1d851db1f9dd597c"
Nov 02 19:57:42.625912 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:42.625891    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0acfeac9-a741-45ec-afab-8fb4ecf5975f-kube-api-access-vkqst\" (UniqueName: \"kubernetes.io/projected/0acfeac9-a741-45ec-afab-8fb4ecf5975f-kube-api-access-vkqst\") pod \"0acfeac9-a741-45ec-afab-8fb4ecf5975f\" (UID: \"0acfeac9-a741-45ec-afab-8fb4ecf5975f\") "
Nov 02 19:57:42.626039 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:42.625930    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0acfeac9-a741-45ec-afab-8fb4ecf5975f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0acfeac9-a741-45ec-afab-8fb4ecf5975f-istio-data\") pod \"0acfeac9-a741-45ec-afab-8fb4ecf5975f\" (UID: \"0acfeac9-a741-45ec-afab-8fb4ecf5975f\") "
Nov 02 19:57:42.626039 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:42.625963    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0acfeac9-a741-45ec-afab-8fb4ecf5975f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0acfeac9-a741-45ec-afab-8fb4ecf5975f-istiod-ca-cert\") pod \"0acfeac9-a741-45ec-afab-8fb4ecf5975f\" (UID: \"0acfeac9-a741-45ec-afab-8fb4ecf5975f\") "
Nov 02 19:57:42.626146 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:42.626082    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0acfeac9-a741-45ec-afab-8fb4ecf5975f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0acfeac9-a741-45ec-afab-8fb4ecf5975f-istio-podinfo\") pod \"0acfeac9-a741-45ec-afab-8fb4ecf5975f\" (UID: \"0acfeac9-a741-45ec-afab-8fb4ecf5975f\") "
Nov 02 19:57:42.626146 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:42.626137    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0acfeac9-a741-45ec-afab-8fb4ecf5975f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0acfeac9-a741-45ec-afab-8fb4ecf5975f-istio-envoy\") pod \"0acfeac9-a741-45ec-afab-8fb4ecf5975f\" (UID: \"0acfeac9-a741-45ec-afab-8fb4ecf5975f\") "
Nov 02 19:57:42.626256 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:42.626197    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0acfeac9-a741-45ec-afab-8fb4ecf5975f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0acfeac9-a741-45ec-afab-8fb4ecf5975f-credential-socket\") pod \"0acfeac9-a741-45ec-afab-8fb4ecf5975f\" (UID: \"0acfeac9-a741-45ec-afab-8fb4ecf5975f\") "
Nov 02 19:57:42.626312 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:42.626292    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0acfeac9-a741-45ec-afab-8fb4ecf5975f-istio-token\" (UniqueName: \"kubernetes.io/projected/0acfeac9-a741-45ec-afab-8fb4ecf5975f-istio-token\") pod \"0acfeac9-a741-45ec-afab-8fb4ecf5975f\" (UID: \"0acfeac9-a741-45ec-afab-8fb4ecf5975f\") "
Nov 02 19:57:42.626366 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:42.626316    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0acfeac9-a741-45ec-afab-8fb4ecf5975f-istio-data" pod "0acfeac9-a741-45ec-afab-8fb4ecf5975f" (UID: "0acfeac9-a741-45ec-afab-8fb4ecf5975f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:57:42.626419 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:42.626392    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0acfeac9-a741-45ec-afab-8fb4ecf5975f-istiod-ca-cert" pod "0acfeac9-a741-45ec-afab-8fb4ecf5975f" (UID: "0acfeac9-a741-45ec-afab-8fb4ecf5975f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 19:57:42.626480 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:42.626412    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0acfeac9-a741-45ec-afab-8fb4ecf5975f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0acfeac9-a741-45ec-afab-8fb4ecf5975f-workload-socket\") pod \"0acfeac9-a741-45ec-afab-8fb4ecf5975f\" (UID: \"0acfeac9-a741-45ec-afab-8fb4ecf5975f\") "
Nov 02 19:57:42.626480 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:42.626463    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0acfeac9-a741-45ec-afab-8fb4ecf5975f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0acfeac9-a741-45ec-afab-8fb4ecf5975f-workload-certs\") pod \"0acfeac9-a741-45ec-afab-8fb4ecf5975f\" (UID: \"0acfeac9-a741-45ec-afab-8fb4ecf5975f\") "
Nov 02 19:57:42.626926 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:42.626672    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0acfeac9-a741-45ec-afab-8fb4ecf5975f-credential-socket" pod "0acfeac9-a741-45ec-afab-8fb4ecf5975f" (UID: "0acfeac9-a741-45ec-afab-8fb4ecf5975f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:57:42.626926 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:42.626893    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0acfeac9-a741-45ec-afab-8fb4ecf5975f-workload-socket" pod "0acfeac9-a741-45ec-afab-8fb4ecf5975f" (UID: "0acfeac9-a741-45ec-afab-8fb4ecf5975f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:57:42.627152 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:42.627075    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0acfeac9-a741-45ec-afab-8fb4ecf5975f-workload-certs" pod "0acfeac9-a741-45ec-afab-8fb4ecf5975f" (UID: "0acfeac9-a741-45ec-afab-8fb4ecf5975f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:57:42.628081 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:42.628056    4463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0acfeac9-a741-45ec-afab-8fb4ecf5975f-credential-socket\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:57:42.628081 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:42.628080    4463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0acfeac9-a741-45ec-afab-8fb4ecf5975f-workload-socket\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:57:42.628218 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:42.628092    4463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0acfeac9-a741-45ec-afab-8fb4ecf5975f-workload-certs\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:57:42.628218 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:42.628100    4463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0acfeac9-a741-45ec-afab-8fb4ecf5975f-istio-data\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:57:42.628218 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:42.628114    4463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0acfeac9-a741-45ec-afab-8fb4ecf5975f-istiod-ca-cert\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:57:42.628401 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:42.628351    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0acfeac9-a741-45ec-afab-8fb4ecf5975f-istio-podinfo" pod "0acfeac9-a741-45ec-afab-8fb4ecf5975f" (UID: "0acfeac9-a741-45ec-afab-8fb4ecf5975f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 19:57:42.628519 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:42.628498    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0acfeac9-a741-45ec-afab-8fb4ecf5975f-kube-api-access-vkqst" pod "0acfeac9-a741-45ec-afab-8fb4ecf5975f" (UID: "0acfeac9-a741-45ec-afab-8fb4ecf5975f"). InnerVolumeSpecName "kube-api-access-vkqst". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:57:42.628899 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:42.628872    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0acfeac9-a741-45ec-afab-8fb4ecf5975f-istio-token" pod "0acfeac9-a741-45ec-afab-8fb4ecf5975f" (UID: "0acfeac9-a741-45ec-afab-8fb4ecf5975f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:57:42.629160 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:42.629138    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0acfeac9-a741-45ec-afab-8fb4ecf5975f-istio-envoy" pod "0acfeac9-a741-45ec-afab-8fb4ecf5975f" (UID: "0acfeac9-a741-45ec-afab-8fb4ecf5975f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:57:42.641307 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:42.641234    4463 scope.go:122] "RemoveContainer" containerID="1a3769a6f7367c62e1966d701f12fbf651e3b9dd1909b6e0b9f53ee37e507052"
Nov 02 19:57:42.647008 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:42.646988    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-2zrqj"
Nov 02 19:57:42.651411 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:42.651394    4463 scope.go:122] "RemoveContainer" containerID="1a3769a6f7367c62e1966d701f12fbf651e3b9dd1909b6e0b9f53ee37e507052"
Nov 02 19:57:42.651772 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:57:42.651752    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a3769a6f7367c62e1966d701f12fbf651e3b9dd1909b6e0b9f53ee37e507052\": container with ID starting with 1a3769a6f7367c62e1966d701f12fbf651e3b9dd1909b6e0b9f53ee37e507052 not found: ID does not exist" containerID="1a3769a6f7367c62e1966d701f12fbf651e3b9dd1909b6e0b9f53ee37e507052"
Nov 02 19:57:42.651849 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:42.651782    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a3769a6f7367c62e1966d701f12fbf651e3b9dd1909b6e0b9f53ee37e507052"} err="failed to get container status \"1a3769a6f7367c62e1966d701f12fbf651e3b9dd1909b6e0b9f53ee37e507052\": rpc error: code = NotFound desc = could not find container \"1a3769a6f7367c62e1966d701f12fbf651e3b9dd1909b6e0b9f53ee37e507052\": container with ID starting with 1a3769a6f7367c62e1966d701f12fbf651e3b9dd1909b6e0b9f53ee37e507052 not found: ID does not exist"
Nov 02 19:57:42.729254 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:42.729184    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/48eb73cb-231a-4462-9236-da01f4734826-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48eb73cb-231a-4462-9236-da01f4734826-istio-podinfo\") pod \"48eb73cb-231a-4462-9236-da01f4734826\" (UID: \"48eb73cb-231a-4462-9236-da01f4734826\") "
Nov 02 19:57:42.729254 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:42.729227    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48eb73cb-231a-4462-9236-da01f4734826-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48eb73cb-231a-4462-9236-da01f4734826-workload-socket\") pod \"48eb73cb-231a-4462-9236-da01f4734826\" (UID: \"48eb73cb-231a-4462-9236-da01f4734826\") "
Nov 02 19:57:42.729432 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:42.729272    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48eb73cb-231a-4462-9236-da01f4734826-kube-api-access-l2p4c\" (UniqueName: \"kubernetes.io/projected/48eb73cb-231a-4462-9236-da01f4734826-kube-api-access-l2p4c\") pod \"48eb73cb-231a-4462-9236-da01f4734826\" (UID: \"48eb73cb-231a-4462-9236-da01f4734826\") "
Nov 02 19:57:42.729432 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:42.729424    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48eb73cb-231a-4462-9236-da01f4734826-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48eb73cb-231a-4462-9236-da01f4734826-istiod-ca-cert\") pod \"48eb73cb-231a-4462-9236-da01f4734826\" (UID: \"48eb73cb-231a-4462-9236-da01f4734826\") "
Nov 02 19:57:42.729522 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:42.729486    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48eb73cb-231a-4462-9236-da01f4734826-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48eb73cb-231a-4462-9236-da01f4734826-credential-socket\") pod \"48eb73cb-231a-4462-9236-da01f4734826\" (UID: \"48eb73cb-231a-4462-9236-da01f4734826\") "
Nov 02 19:57:42.729593 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:42.729531    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48eb73cb-231a-4462-9236-da01f4734826-workload-socket" pod "48eb73cb-231a-4462-9236-da01f4734826" (UID: "48eb73cb-231a-4462-9236-da01f4734826"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:57:42.729593 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:42.729561    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48eb73cb-231a-4462-9236-da01f4734826-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48eb73cb-231a-4462-9236-da01f4734826-istio-data\") pod \"48eb73cb-231a-4462-9236-da01f4734826\" (UID: \"48eb73cb-231a-4462-9236-da01f4734826\") "
Nov 02 19:57:42.729750 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:42.729640    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48eb73cb-231a-4462-9236-da01f4734826-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48eb73cb-231a-4462-9236-da01f4734826-workload-certs\") pod \"48eb73cb-231a-4462-9236-da01f4734826\" (UID: \"48eb73cb-231a-4462-9236-da01f4734826\") "
Nov 02 19:57:42.729750 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:42.729673    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48eb73cb-231a-4462-9236-da01f4734826-istio-token\" (UniqueName: \"kubernetes.io/projected/48eb73cb-231a-4462-9236-da01f4734826-istio-token\") pod \"48eb73cb-231a-4462-9236-da01f4734826\" (UID: \"48eb73cb-231a-4462-9236-da01f4734826\") "
Nov 02 19:57:42.729750 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:42.729738    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48eb73cb-231a-4462-9236-da01f4734826-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48eb73cb-231a-4462-9236-da01f4734826-istio-envoy\") pod \"48eb73cb-231a-4462-9236-da01f4734826\" (UID: \"48eb73cb-231a-4462-9236-da01f4734826\") "
Nov 02 19:57:42.729888 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:42.729829    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48eb73cb-231a-4462-9236-da01f4734826-istiod-ca-cert" pod "48eb73cb-231a-4462-9236-da01f4734826" (UID: "48eb73cb-231a-4462-9236-da01f4734826"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 19:57:42.729888 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:42.729860    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48eb73cb-231a-4462-9236-da01f4734826-istio-data" pod "48eb73cb-231a-4462-9236-da01f4734826" (UID: "48eb73cb-231a-4462-9236-da01f4734826"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:57:42.730253 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:42.730077    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48eb73cb-231a-4462-9236-da01f4734826-credential-socket" pod "48eb73cb-231a-4462-9236-da01f4734826" (UID: "48eb73cb-231a-4462-9236-da01f4734826"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:57:42.730253 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:42.730209    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48eb73cb-231a-4462-9236-da01f4734826-workload-certs" pod "48eb73cb-231a-4462-9236-da01f4734826" (UID: "48eb73cb-231a-4462-9236-da01f4734826"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:57:42.730678 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:42.730653    4463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48eb73cb-231a-4462-9236-da01f4734826-istiod-ca-cert\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:57:42.730801 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:42.730681    4463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48eb73cb-231a-4462-9236-da01f4734826-credential-socket\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:57:42.730801 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:42.730691    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkqst\" (UniqueName: \"kubernetes.io/projected/0acfeac9-a741-45ec-afab-8fb4ecf5975f-kube-api-access-vkqst\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:57:42.730801 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:42.730703    4463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48eb73cb-231a-4462-9236-da01f4734826-istio-data\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:57:42.730801 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:42.730725    4463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48eb73cb-231a-4462-9236-da01f4734826-workload-certs\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:57:42.730801 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:42.730739    4463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0acfeac9-a741-45ec-afab-8fb4ecf5975f-istio-podinfo\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:57:42.730801 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:42.730749    4463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0acfeac9-a741-45ec-afab-8fb4ecf5975f-istio-envoy\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:57:42.730801 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:42.730756    4463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0acfeac9-a741-45ec-afab-8fb4ecf5975f-istio-token\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:57:42.730801 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:42.730764    4463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48eb73cb-231a-4462-9236-da01f4734826-workload-socket\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:57:42.731745 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:42.731722    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48eb73cb-231a-4462-9236-da01f4734826-kube-api-access-l2p4c" pod "48eb73cb-231a-4462-9236-da01f4734826" (UID: "48eb73cb-231a-4462-9236-da01f4734826"). InnerVolumeSpecName "kube-api-access-l2p4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:57:42.731907 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:42.731891    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/48eb73cb-231a-4462-9236-da01f4734826-istio-podinfo" pod "48eb73cb-231a-4462-9236-da01f4734826" (UID: "48eb73cb-231a-4462-9236-da01f4734826"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 19:57:42.732196 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:42.732172    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48eb73cb-231a-4462-9236-da01f4734826-istio-token" pod "48eb73cb-231a-4462-9236-da01f4734826" (UID: "48eb73cb-231a-4462-9236-da01f4734826"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:57:42.732254 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:42.732243    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48eb73cb-231a-4462-9236-da01f4734826-istio-envoy" pod "48eb73cb-231a-4462-9236-da01f4734826" (UID: "48eb73cb-231a-4462-9236-da01f4734826"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:57:42.831299 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:42.831266    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8216408-95d0-413b-a609-2ca8cf3299b6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8216408-95d0-413b-a609-2ca8cf3299b6-istio-data\") pod \"a8216408-95d0-413b-a609-2ca8cf3299b6\" (UID: \"a8216408-95d0-413b-a609-2ca8cf3299b6\") "
Nov 02 19:57:42.831474 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:42.831331    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8216408-95d0-413b-a609-2ca8cf3299b6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8216408-95d0-413b-a609-2ca8cf3299b6-istiod-ca-cert\") pod \"a8216408-95d0-413b-a609-2ca8cf3299b6\" (UID: \"a8216408-95d0-413b-a609-2ca8cf3299b6\") "
Nov 02 19:57:42.831474 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:42.831392    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8216408-95d0-413b-a609-2ca8cf3299b6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8216408-95d0-413b-a609-2ca8cf3299b6-workload-certs\") pod \"a8216408-95d0-413b-a609-2ca8cf3299b6\" (UID: \"a8216408-95d0-413b-a609-2ca8cf3299b6\") "
Nov 02 19:57:42.831474 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:42.831461    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8216408-95d0-413b-a609-2ca8cf3299b6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8216408-95d0-413b-a609-2ca8cf3299b6-workload-socket\") pod \"a8216408-95d0-413b-a609-2ca8cf3299b6\" (UID: \"a8216408-95d0-413b-a609-2ca8cf3299b6\") "
Nov 02 19:57:42.831623 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:42.831586    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a8216408-95d0-413b-a609-2ca8cf3299b6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8216408-95d0-413b-a609-2ca8cf3299b6-istio-podinfo\") pod \"a8216408-95d0-413b-a609-2ca8cf3299b6\" (UID: \"a8216408-95d0-413b-a609-2ca8cf3299b6\") "
Nov 02 19:57:42.831623 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:42.831593    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8216408-95d0-413b-a609-2ca8cf3299b6-istio-data" pod "a8216408-95d0-413b-a609-2ca8cf3299b6" (UID: "a8216408-95d0-413b-a609-2ca8cf3299b6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:57:42.831730 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:42.831632    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8216408-95d0-413b-a609-2ca8cf3299b6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8216408-95d0-413b-a609-2ca8cf3299b6-istio-envoy\") pod \"a8216408-95d0-413b-a609-2ca8cf3299b6\" (UID: \"a8216408-95d0-413b-a609-2ca8cf3299b6\") "
Nov 02 19:57:42.831730 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:42.831659    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8216408-95d0-413b-a609-2ca8cf3299b6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8216408-95d0-413b-a609-2ca8cf3299b6-credential-socket\") pod \"a8216408-95d0-413b-a609-2ca8cf3299b6\" (UID: \"a8216408-95d0-413b-a609-2ca8cf3299b6\") "
Nov 02 19:57:42.831835 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:42.831767    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8216408-95d0-413b-a609-2ca8cf3299b6-istio-token\" (UniqueName: \"kubernetes.io/projected/a8216408-95d0-413b-a609-2ca8cf3299b6-istio-token\") pod \"a8216408-95d0-413b-a609-2ca8cf3299b6\" (UID: \"a8216408-95d0-413b-a609-2ca8cf3299b6\") "
Nov 02 19:57:42.831884 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:42.831856    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8216408-95d0-413b-a609-2ca8cf3299b6-workload-certs" pod "a8216408-95d0-413b-a609-2ca8cf3299b6" (UID: "a8216408-95d0-413b-a609-2ca8cf3299b6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:57:42.831884 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:42.831870    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8216408-95d0-413b-a609-2ca8cf3299b6-kube-api-access-qlf5m\" (UniqueName: \"kubernetes.io/projected/a8216408-95d0-413b-a609-2ca8cf3299b6-kube-api-access-qlf5m\") pod \"a8216408-95d0-413b-a609-2ca8cf3299b6\" (UID: \"a8216408-95d0-413b-a609-2ca8cf3299b6\") "
Nov 02 19:57:42.831990 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:42.831878    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8216408-95d0-413b-a609-2ca8cf3299b6-workload-socket" pod "a8216408-95d0-413b-a609-2ca8cf3299b6" (UID: "a8216408-95d0-413b-a609-2ca8cf3299b6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:57:42.831990 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:42.831875    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8216408-95d0-413b-a609-2ca8cf3299b6-istiod-ca-cert" pod "a8216408-95d0-413b-a609-2ca8cf3299b6" (UID: "a8216408-95d0-413b-a609-2ca8cf3299b6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 19:57:42.832845 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:42.832227    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8216408-95d0-413b-a609-2ca8cf3299b6-credential-socket" pod "a8216408-95d0-413b-a609-2ca8cf3299b6" (UID: "a8216408-95d0-413b-a609-2ca8cf3299b6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:57:42.832845 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:42.832689    4463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48eb73cb-231a-4462-9236-da01f4734826-istio-token\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:57:42.832845 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:42.832709    4463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48eb73cb-231a-4462-9236-da01f4734826-istio-envoy\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:57:42.832845 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:42.832723    4463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8216408-95d0-413b-a609-2ca8cf3299b6-istio-data\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:57:42.832845 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:42.832737    4463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48eb73cb-231a-4462-9236-da01f4734826-istio-podinfo\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:57:42.832845 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:42.832750    4463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8216408-95d0-413b-a609-2ca8cf3299b6-istiod-ca-cert\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:57:42.832845 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:42.832765    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2p4c\" (UniqueName: \"kubernetes.io/projected/48eb73cb-231a-4462-9236-da01f4734826-kube-api-access-l2p4c\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:57:42.832845 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:42.832786    4463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8216408-95d0-413b-a609-2ca8cf3299b6-workload-certs\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:57:42.832845 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:42.832799    4463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8216408-95d0-413b-a609-2ca8cf3299b6-workload-socket\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:57:42.832845 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:42.832813    4463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8216408-95d0-413b-a609-2ca8cf3299b6-credential-socket\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:57:42.834440 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:42.834395    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a8216408-95d0-413b-a609-2ca8cf3299b6-istio-podinfo" pod "a8216408-95d0-413b-a609-2ca8cf3299b6" (UID: "a8216408-95d0-413b-a609-2ca8cf3299b6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 19:57:42.834678 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:42.834657    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8216408-95d0-413b-a609-2ca8cf3299b6-istio-token" pod "a8216408-95d0-413b-a609-2ca8cf3299b6" (UID: "a8216408-95d0-413b-a609-2ca8cf3299b6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:57:42.834758 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:42.834695    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8216408-95d0-413b-a609-2ca8cf3299b6-kube-api-access-qlf5m" pod "a8216408-95d0-413b-a609-2ca8cf3299b6" (UID: "a8216408-95d0-413b-a609-2ca8cf3299b6"). InnerVolumeSpecName "kube-api-access-qlf5m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:57:42.834758 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:42.834705    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8216408-95d0-413b-a609-2ca8cf3299b6-istio-envoy" pod "a8216408-95d0-413b-a609-2ca8cf3299b6" (UID: "a8216408-95d0-413b-a609-2ca8cf3299b6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:57:42.934430 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:42.934393    4463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8216408-95d0-413b-a609-2ca8cf3299b6-istio-podinfo\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:57:42.934430 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:42.934428    4463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8216408-95d0-413b-a609-2ca8cf3299b6-istio-envoy\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:57:42.934635 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:42.934444    4463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8216408-95d0-413b-a609-2ca8cf3299b6-istio-token\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:57:42.934635 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:42.934457    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlf5m\" (UniqueName: \"kubernetes.io/projected/a8216408-95d0-413b-a609-2ca8cf3299b6-kube-api-access-qlf5m\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:57:43.042733 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:43.036966    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-bbt8n"]
Nov 02 19:57:43.043306 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:43.043276    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-bbt8n"]
Nov 02 19:57:43.586929 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:43.586899    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0acfeac9-a741-45ec-afab-8fb4ecf5975f" path="/var/lib/kubelet/pods/0acfeac9-a741-45ec-afab-8fb4ecf5975f/volumes"
Nov 02 19:57:43.619975 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:43.619952    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fjnvm"
Nov 02 19:57:43.621514 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:43.621491    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-2zrqj"
Nov 02 19:57:43.621642 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:43.621507    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-2zrqj" event={"ID":"a8216408-95d0-413b-a609-2ca8cf3299b6","Type":"ContainerDied","Data":"b486ffccbc9e9095dffa4ae2b260b109089598ac6276ef7362508dbc5c095ae2"}
Nov 02 19:57:43.621642 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:43.621560    4463 scope.go:122] "RemoveContainer" containerID="f56ee5a76c4bf7aeafdf7fb898d4cbccecfe3d39d9505d83fa7eea05d94a0c75"
Nov 02 19:57:43.773968 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:43.773938    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-2zrqj"]
Nov 02 19:57:43.819794 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:43.819769    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-2zrqj"]
Nov 02 19:57:43.879791 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:43.879721    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fjnvm"]
Nov 02 19:57:43.884488 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:43.884461    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fjnvm"]
Nov 02 19:57:45.589400 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:45.589365    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48eb73cb-231a-4462-9236-da01f4734826" path="/var/lib/kubelet/pods/48eb73cb-231a-4462-9236-da01f4734826/volumes"
Nov 02 19:57:45.590231 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:45.590210    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8216408-95d0-413b-a609-2ca8cf3299b6" path="/var/lib/kubelet/pods/a8216408-95d0-413b-a609-2ca8cf3299b6/volumes"
Nov 02 19:57:48.293418 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:48.293383    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-8479ff-openshift-default-6bb884bd67-wnm6b"]
Nov 02 19:57:48.293982 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:48.293968    4463 cpu_manager.go:414] "containerMap: removing container" podUID="48eb73cb-231a-4462-9236-da01f4734826" containerName="istio-proxy"
Nov 02 19:57:48.294034 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:48.293983    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48eb73cb-231a-4462-9236-da01f4734826" containerName="istio-proxy"
Nov 02 19:57:48.294034 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:48.294005    4463 cpu_manager.go:414] "containerMap: removing container" podUID="a8216408-95d0-413b-a609-2ca8cf3299b6" containerName="istio-proxy"
Nov 02 19:57:48.294034 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:48.294024    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8216408-95d0-413b-a609-2ca8cf3299b6" containerName="istio-proxy"
Nov 02 19:57:48.294149 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:48.294063    4463 cpu_manager.go:414] "containerMap: removing container" podUID="307e0c79-e6b1-4751-b226-966f9b8f9283" containerName="discovery"
Nov 02 19:57:48.294149 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:48.294071    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="307e0c79-e6b1-4751-b226-966f9b8f9283" containerName="discovery"
Nov 02 19:57:48.294149 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:48.294084    4463 cpu_manager.go:414] "containerMap: removing container" podUID="0acfeac9-a741-45ec-afab-8fb4ecf5975f" containerName="istio-proxy"
Nov 02 19:57:48.294149 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:48.294091    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0acfeac9-a741-45ec-afab-8fb4ecf5975f" containerName="istio-proxy"
Nov 02 19:57:48.294149 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:48.294105    4463 cpu_manager.go:414] "containerMap: removing container" podUID="0714f5c6-adfb-434e-8dd9-9bada5fbed29" containerName="istio-proxy"
Nov 02 19:57:48.294149 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:48.294111    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0714f5c6-adfb-434e-8dd9-9bada5fbed29" containerName="istio-proxy"
Nov 02 19:57:48.294310 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:48.294251    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8216408-95d0-413b-a609-2ca8cf3299b6" containerName="istio-proxy"
Nov 02 19:57:48.294310 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:48.294268    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="307e0c79-e6b1-4751-b226-966f9b8f9283" containerName="discovery"
Nov 02 19:57:48.294310 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:48.294279    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="0acfeac9-a741-45ec-afab-8fb4ecf5975f" containerName="istio-proxy"
Nov 02 19:57:48.294310 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:48.294289    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="0714f5c6-adfb-434e-8dd9-9bada5fbed29" containerName="istio-proxy"
Nov 02 19:57:48.294310 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:48.294303    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="48eb73cb-231a-4462-9236-da01f4734826" containerName="istio-proxy"
Nov 02 19:57:48.299563 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:48.299542    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8479ff-openshift-default-6bb884bd67-wnm6b"
Nov 02 19:57:48.348336 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:48.348296    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-8479ff-openshift-default-dockercfg-l77lq\""
Nov 02 19:57:48.400084 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:48.399989    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-8479ff-openshift-default-6bb884bd67-wnm6b"]
Nov 02 19:57:48.407479 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:48.407450    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnvg8\" (UniqueName: \"kubernetes.io/projected/2f18cb37-937a-4cc9-9a20-f44f2f693d52-kube-api-access-qnvg8\") pod \"e2e-inf-a-8479ff-openshift-default-6bb884bd67-wnm6b\" (UID: \"2f18cb37-937a-4cc9-9a20-f44f2f693d52\") " pod="openshift-ingress/e2e-inf-a-8479ff-openshift-default-6bb884bd67-wnm6b"
Nov 02 19:57:48.407571 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:48.407560    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f18cb37-937a-4cc9-9a20-f44f2f693d52-istio-envoy\") pod \"e2e-inf-a-8479ff-openshift-default-6bb884bd67-wnm6b\" (UID: \"2f18cb37-937a-4cc9-9a20-f44f2f693d52\") " pod="openshift-ingress/e2e-inf-a-8479ff-openshift-default-6bb884bd67-wnm6b"
Nov 02 19:57:48.407653 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:48.407638    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f18cb37-937a-4cc9-9a20-f44f2f693d52-workload-socket\") pod \"e2e-inf-a-8479ff-openshift-default-6bb884bd67-wnm6b\" (UID: \"2f18cb37-937a-4cc9-9a20-f44f2f693d52\") " pod="openshift-ingress/e2e-inf-a-8479ff-openshift-default-6bb884bd67-wnm6b"
Nov 02 19:57:48.407697 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:48.407668    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f18cb37-937a-4cc9-9a20-f44f2f693d52-credential-socket\") pod \"e2e-inf-a-8479ff-openshift-default-6bb884bd67-wnm6b\" (UID: \"2f18cb37-937a-4cc9-9a20-f44f2f693d52\") " pod="openshift-ingress/e2e-inf-a-8479ff-openshift-default-6bb884bd67-wnm6b"
Nov 02 19:57:48.407753 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:48.407739    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f18cb37-937a-4cc9-9a20-f44f2f693d52-istio-data\") pod \"e2e-inf-a-8479ff-openshift-default-6bb884bd67-wnm6b\" (UID: \"2f18cb37-937a-4cc9-9a20-f44f2f693d52\") " pod="openshift-ingress/e2e-inf-a-8479ff-openshift-default-6bb884bd67-wnm6b"
Nov 02 19:57:48.407827 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:48.407813    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f18cb37-937a-4cc9-9a20-f44f2f693d52-workload-certs\") pod \"e2e-inf-a-8479ff-openshift-default-6bb884bd67-wnm6b\" (UID: \"2f18cb37-937a-4cc9-9a20-f44f2f693d52\") " pod="openshift-ingress/e2e-inf-a-8479ff-openshift-default-6bb884bd67-wnm6b"
Nov 02 19:57:48.407863 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:48.407838    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f18cb37-937a-4cc9-9a20-f44f2f693d52-istio-token\") pod \"e2e-inf-a-8479ff-openshift-default-6bb884bd67-wnm6b\" (UID: \"2f18cb37-937a-4cc9-9a20-f44f2f693d52\") " pod="openshift-ingress/e2e-inf-a-8479ff-openshift-default-6bb884bd67-wnm6b"
Nov 02 19:57:48.407905 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:48.407884    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f18cb37-937a-4cc9-9a20-f44f2f693d52-istio-podinfo\") pod \"e2e-inf-a-8479ff-openshift-default-6bb884bd67-wnm6b\" (UID: \"2f18cb37-937a-4cc9-9a20-f44f2f693d52\") " pod="openshift-ingress/e2e-inf-a-8479ff-openshift-default-6bb884bd67-wnm6b"
Nov 02 19:57:48.407937 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:48.407908    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f18cb37-937a-4cc9-9a20-f44f2f693d52-istiod-ca-cert\") pod \"e2e-inf-a-8479ff-openshift-default-6bb884bd67-wnm6b\" (UID: \"2f18cb37-937a-4cc9-9a20-f44f2f693d52\") " pod="openshift-ingress/e2e-inf-a-8479ff-openshift-default-6bb884bd67-wnm6b"
Nov 02 19:57:48.509541 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:48.509503    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f18cb37-937a-4cc9-9a20-f44f2f693d52-istio-token\") pod \"e2e-inf-a-8479ff-openshift-default-6bb884bd67-wnm6b\" (UID: \"2f18cb37-937a-4cc9-9a20-f44f2f693d52\") " pod="openshift-ingress/e2e-inf-a-8479ff-openshift-default-6bb884bd67-wnm6b"
Nov 02 19:57:48.509734 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:48.509583    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f18cb37-937a-4cc9-9a20-f44f2f693d52-istio-podinfo\") pod \"e2e-inf-a-8479ff-openshift-default-6bb884bd67-wnm6b\" (UID: \"2f18cb37-937a-4cc9-9a20-f44f2f693d52\") " pod="openshift-ingress/e2e-inf-a-8479ff-openshift-default-6bb884bd67-wnm6b"
Nov 02 19:57:48.509734 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:48.509642    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f18cb37-937a-4cc9-9a20-f44f2f693d52-istiod-ca-cert\") pod \"e2e-inf-a-8479ff-openshift-default-6bb884bd67-wnm6b\" (UID: \"2f18cb37-937a-4cc9-9a20-f44f2f693d52\") " pod="openshift-ingress/e2e-inf-a-8479ff-openshift-default-6bb884bd67-wnm6b"
Nov 02 19:57:48.509875 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:48.509752    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnvg8\" (UniqueName: \"kubernetes.io/projected/2f18cb37-937a-4cc9-9a20-f44f2f693d52-kube-api-access-qnvg8\") pod \"e2e-inf-a-8479ff-openshift-default-6bb884bd67-wnm6b\" (UID: \"2f18cb37-937a-4cc9-9a20-f44f2f693d52\") " pod="openshift-ingress/e2e-inf-a-8479ff-openshift-default-6bb884bd67-wnm6b"
Nov 02 19:57:48.509875 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:48.509847    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f18cb37-937a-4cc9-9a20-f44f2f693d52-istio-envoy\") pod \"e2e-inf-a-8479ff-openshift-default-6bb884bd67-wnm6b\" (UID: \"2f18cb37-937a-4cc9-9a20-f44f2f693d52\") " pod="openshift-ingress/e2e-inf-a-8479ff-openshift-default-6bb884bd67-wnm6b"
Nov 02 19:57:48.509988 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:48.509956    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f18cb37-937a-4cc9-9a20-f44f2f693d52-workload-socket\") pod \"e2e-inf-a-8479ff-openshift-default-6bb884bd67-wnm6b\" (UID: \"2f18cb37-937a-4cc9-9a20-f44f2f693d52\") " pod="openshift-ingress/e2e-inf-a-8479ff-openshift-default-6bb884bd67-wnm6b"
Nov 02 19:57:48.509988 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:48.509983    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f18cb37-937a-4cc9-9a20-f44f2f693d52-credential-socket\") pod \"e2e-inf-a-8479ff-openshift-default-6bb884bd67-wnm6b\" (UID: \"2f18cb37-937a-4cc9-9a20-f44f2f693d52\") " pod="openshift-ingress/e2e-inf-a-8479ff-openshift-default-6bb884bd67-wnm6b"
Nov 02 19:57:48.510262 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:48.510079    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f18cb37-937a-4cc9-9a20-f44f2f693d52-istio-data\") pod \"e2e-inf-a-8479ff-openshift-default-6bb884bd67-wnm6b\" (UID: \"2f18cb37-937a-4cc9-9a20-f44f2f693d52\") " pod="openshift-ingress/e2e-inf-a-8479ff-openshift-default-6bb884bd67-wnm6b"
Nov 02 19:57:48.510262 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:48.510155    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f18cb37-937a-4cc9-9a20-f44f2f693d52-workload-certs\") pod \"e2e-inf-a-8479ff-openshift-default-6bb884bd67-wnm6b\" (UID: \"2f18cb37-937a-4cc9-9a20-f44f2f693d52\") " pod="openshift-ingress/e2e-inf-a-8479ff-openshift-default-6bb884bd67-wnm6b"
Nov 02 19:57:48.510484 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:48.510459    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f18cb37-937a-4cc9-9a20-f44f2f693d52-credential-socket\") pod \"e2e-inf-a-8479ff-openshift-default-6bb884bd67-wnm6b\" (UID: \"2f18cb37-937a-4cc9-9a20-f44f2f693d52\") " pod="openshift-ingress/e2e-inf-a-8479ff-openshift-default-6bb884bd67-wnm6b"
Nov 02 19:57:48.510667 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:48.510620    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f18cb37-937a-4cc9-9a20-f44f2f693d52-istiod-ca-cert\") pod \"e2e-inf-a-8479ff-openshift-default-6bb884bd67-wnm6b\" (UID: \"2f18cb37-937a-4cc9-9a20-f44f2f693d52\") " pod="openshift-ingress/e2e-inf-a-8479ff-openshift-default-6bb884bd67-wnm6b"
Nov 02 19:57:48.510784 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:48.510721    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f18cb37-937a-4cc9-9a20-f44f2f693d52-istio-data\") pod \"e2e-inf-a-8479ff-openshift-default-6bb884bd67-wnm6b\" (UID: \"2f18cb37-937a-4cc9-9a20-f44f2f693d52\") " pod="openshift-ingress/e2e-inf-a-8479ff-openshift-default-6bb884bd67-wnm6b"
Nov 02 19:57:48.510853 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:48.510787    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f18cb37-937a-4cc9-9a20-f44f2f693d52-workload-certs\") pod \"e2e-inf-a-8479ff-openshift-default-6bb884bd67-wnm6b\" (UID: \"2f18cb37-937a-4cc9-9a20-f44f2f693d52\") " pod="openshift-ingress/e2e-inf-a-8479ff-openshift-default-6bb884bd67-wnm6b"
Nov 02 19:57:48.510909 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:48.510861    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f18cb37-937a-4cc9-9a20-f44f2f693d52-workload-socket\") pod \"e2e-inf-a-8479ff-openshift-default-6bb884bd67-wnm6b\" (UID: \"2f18cb37-937a-4cc9-9a20-f44f2f693d52\") " pod="openshift-ingress/e2e-inf-a-8479ff-openshift-default-6bb884bd67-wnm6b"
Nov 02 19:57:48.512645 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:48.512625    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f18cb37-937a-4cc9-9a20-f44f2f693d52-istio-podinfo\") pod \"e2e-inf-a-8479ff-openshift-default-6bb884bd67-wnm6b\" (UID: \"2f18cb37-937a-4cc9-9a20-f44f2f693d52\") " pod="openshift-ingress/e2e-inf-a-8479ff-openshift-default-6bb884bd67-wnm6b"
Nov 02 19:57:48.513109 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:48.513079    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f18cb37-937a-4cc9-9a20-f44f2f693d52-istio-envoy\") pod \"e2e-inf-a-8479ff-openshift-default-6bb884bd67-wnm6b\" (UID: \"2f18cb37-937a-4cc9-9a20-f44f2f693d52\") " pod="openshift-ingress/e2e-inf-a-8479ff-openshift-default-6bb884bd67-wnm6b"
Nov 02 19:57:48.563698 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:48.563633    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f18cb37-937a-4cc9-9a20-f44f2f693d52-istio-token\") pod \"e2e-inf-a-8479ff-openshift-default-6bb884bd67-wnm6b\" (UID: \"2f18cb37-937a-4cc9-9a20-f44f2f693d52\") " pod="openshift-ingress/e2e-inf-a-8479ff-openshift-default-6bb884bd67-wnm6b"
Nov 02 19:57:48.563698 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:48.563648    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnvg8\" (UniqueName: \"kubernetes.io/projected/2f18cb37-937a-4cc9-9a20-f44f2f693d52-kube-api-access-qnvg8\") pod \"e2e-inf-a-8479ff-openshift-default-6bb884bd67-wnm6b\" (UID: \"2f18cb37-937a-4cc9-9a20-f44f2f693d52\") " pod="openshift-ingress/e2e-inf-a-8479ff-openshift-default-6bb884bd67-wnm6b"
Nov 02 19:57:48.612837 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:48.612806    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8479ff-openshift-default-6bb884bd67-wnm6b"
Nov 02 19:57:49.045509 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:57:49.045474    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f18cb37_937a_4cc9_9a20_f44f2f693d52.slice/crio-4c287599baffdd2f0a706c1a91006fadec80acb1568f93a0ae53965c9e0e221a WatchSource:0}: Error finding container 4c287599baffdd2f0a706c1a91006fadec80acb1568f93a0ae53965c9e0e221a: Status 404 returned error can't find the container with id 4c287599baffdd2f0a706c1a91006fadec80acb1568f93a0ae53965c9e0e221a
Nov 02 19:57:49.046420 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:49.046392    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-8479ff-openshift-default-6bb884bd67-wnm6b"]
Nov 02 19:57:49.048256 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:49.048226    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:57:49.048336 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:49.048288    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:57:49.048336 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:49.048317    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:57:49.669205 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:49.669171    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8479ff-openshift-default-6bb884bd67-wnm6b" event={"ID":"2f18cb37-937a-4cc9-9a20-f44f2f693d52","Type":"ContainerStarted","Data":"d43aa87648a2daa10355da0054488ebb075beb0699084bcd11eb0888142a0db5"}
Nov 02 19:57:49.669615 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:49.669226    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8479ff-openshift-default-6bb884bd67-wnm6b" event={"ID":"2f18cb37-937a-4cc9-9a20-f44f2f693d52","Type":"ContainerStarted","Data":"4c287599baffdd2f0a706c1a91006fadec80acb1568f93a0ae53965c9e0e221a"}
Nov 02 19:57:49.779942 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:49.779888    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-8479ff-openshift-default-6bb884bd67-wnm6b" podStartSLOduration=1.779876139 podStartE2EDuration="1.779876139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:57:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:57:49.776261118 +0000 UTC m=+1776.383294905" watchObservedRunningTime="2026-11-02 19:57:49.779876139 +0000 UTC m=+1776.386909965"
Nov 02 19:57:50.613359 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:50.613329    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-8479ff-openshift-default-6bb884bd67-wnm6b"
Nov 02 19:57:50.619677 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:50.619652    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-8479ff-openshift-default-6bb884bd67-wnm6b"
Nov 02 19:57:50.684920 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:50.684886    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-8479ff-openshift-default-6bb884bd67-wnm6b"
Nov 02 19:57:50.685814 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:50.685794    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-8479ff-openshift-default-6bb884bd67-wnm6b"
Nov 02 19:57:56.188738 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:56.188705    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8479ff-7fd9b68fc5-plq28"]
Nov 02 19:57:56.189515 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:56.189494    4463 cpu_manager.go:414] "containerMap: removing container" podUID="a8216408-95d0-413b-a609-2ca8cf3299b6" containerName="istio-proxy"
Nov 02 19:57:56.189619 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:56.189516    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8216408-95d0-413b-a609-2ca8cf3299b6" containerName="istio-proxy"
Nov 02 19:57:56.189732 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:56.189718    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8216408-95d0-413b-a609-2ca8cf3299b6" containerName="istio-proxy"
Nov 02 19:57:56.193470 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:56.193448    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8479ff-7fd9b68fc5-plq28"
Nov 02 19:57:56.246742 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:56.246711    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8479ff-7fd9b68fc5-plq28"]
Nov 02 19:57:56.288306 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:56.288277    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-8479ff\""
Nov 02 19:57:56.288478 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:56.288400    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-8479ff-dockercfg-fhdd2\""
Nov 02 19:57:56.316034 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:56.309242    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a211d74d-a663-4c8e-a451-f3253236d6f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-8479ff-7fd9b68fc5-plq28\" (UID: \"a211d74d-a663-4c8e-a451-f3253236d6f6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8479ff-7fd9b68fc5-plq28"
Nov 02 19:57:56.316034 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:56.309453    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wnds\" (UniqueName: \"kubernetes.io/projected/a211d74d-a663-4c8e-a451-f3253236d6f6-kube-api-access-9wnds\") pod \"payload-pre-processing-e2e-inf-a-8479ff-7fd9b68fc5-plq28\" (UID: \"a211d74d-a663-4c8e-a451-f3253236d6f6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8479ff-7fd9b68fc5-plq28"
Nov 02 19:57:56.356608 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:56.350339    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8479ff-d99f8dbc7-9bvj7"]
Nov 02 19:57:56.356608 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:56.355899    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-8479ff-d99f8dbc7-9bvj7"
Nov 02 19:57:56.405270 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:56.405237    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8479ff-d99f8dbc7-9bvj7"]
Nov 02 19:57:56.411052 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:56.410992    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9045c38-104c-4065-b369-6f3803fae81e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-8479ff-d99f8dbc7-9bvj7\" (UID: \"f9045c38-104c-4065-b369-6f3803fae81e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8479ff-d99f8dbc7-9bvj7"
Nov 02 19:57:56.411209 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:56.411106    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wnds\" (UniqueName: \"kubernetes.io/projected/a211d74d-a663-4c8e-a451-f3253236d6f6-kube-api-access-9wnds\") pod \"payload-pre-processing-e2e-inf-a-8479ff-7fd9b68fc5-plq28\" (UID: \"a211d74d-a663-4c8e-a451-f3253236d6f6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8479ff-7fd9b68fc5-plq28"
Nov 02 19:57:56.411267 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:56.411248    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a211d74d-a663-4c8e-a451-f3253236d6f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-8479ff-7fd9b68fc5-plq28\" (UID: \"a211d74d-a663-4c8e-a451-f3253236d6f6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8479ff-7fd9b68fc5-plq28"
Nov 02 19:57:56.411348 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:56.411331    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xd2c\" (UniqueName: \"kubernetes.io/projected/f9045c38-104c-4065-b369-6f3803fae81e-kube-api-access-8xd2c\") pod \"payload-processing-e2e-inf-a-8479ff-d99f8dbc7-9bvj7\" (UID: \"f9045c38-104c-4065-b369-6f3803fae81e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8479ff-d99f8dbc7-9bvj7"
Nov 02 19:57:56.412387 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:56.412342    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a211d74d-a663-4c8e-a451-f3253236d6f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-8479ff-7fd9b68fc5-plq28\" (UID: \"a211d74d-a663-4c8e-a451-f3253236d6f6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8479ff-7fd9b68fc5-plq28"
Nov 02 19:57:56.465820 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:56.465746    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wnds\" (UniqueName: \"kubernetes.io/projected/a211d74d-a663-4c8e-a451-f3253236d6f6-kube-api-access-9wnds\") pod \"payload-pre-processing-e2e-inf-a-8479ff-7fd9b68fc5-plq28\" (UID: \"a211d74d-a663-4c8e-a451-f3253236d6f6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8479ff-7fd9b68fc5-plq28"
Nov 02 19:57:56.508412 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:56.508379    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8479ff-7fd9b68fc5-plq28"
Nov 02 19:57:56.512345 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:56.512312    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xd2c\" (UniqueName: \"kubernetes.io/projected/f9045c38-104c-4065-b369-6f3803fae81e-kube-api-access-8xd2c\") pod \"payload-processing-e2e-inf-a-8479ff-d99f8dbc7-9bvj7\" (UID: \"f9045c38-104c-4065-b369-6f3803fae81e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8479ff-d99f8dbc7-9bvj7"
Nov 02 19:57:56.512465 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:56.512454    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9045c38-104c-4065-b369-6f3803fae81e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-8479ff-d99f8dbc7-9bvj7\" (UID: \"f9045c38-104c-4065-b369-6f3803fae81e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8479ff-d99f8dbc7-9bvj7"
Nov 02 19:57:56.513227 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:56.513182    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9045c38-104c-4065-b369-6f3803fae81e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-8479ff-d99f8dbc7-9bvj7\" (UID: \"f9045c38-104c-4065-b369-6f3803fae81e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8479ff-d99f8dbc7-9bvj7"
Nov 02 19:57:56.567945 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:56.567919    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xd2c\" (UniqueName: \"kubernetes.io/projected/f9045c38-104c-4065-b369-6f3803fae81e-kube-api-access-8xd2c\") pod \"payload-processing-e2e-inf-a-8479ff-d99f8dbc7-9bvj7\" (UID: \"f9045c38-104c-4065-b369-6f3803fae81e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8479ff-d99f8dbc7-9bvj7"
Nov 02 19:57:56.675657 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:56.675627    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-8479ff-d99f8dbc7-9bvj7"
Nov 02 19:57:56.879939 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:57:56.879906    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda211d74d_a663_4c8e_a451_f3253236d6f6.slice/crio-aebb8cd9f3961ec66b11f0867d3ac75bbe1ec741fe32582b91b67580ed9323c5 WatchSource:0}: Error finding container aebb8cd9f3961ec66b11f0867d3ac75bbe1ec741fe32582b91b67580ed9323c5: Status 404 returned error can't find the container with id aebb8cd9f3961ec66b11f0867d3ac75bbe1ec741fe32582b91b67580ed9323c5
Nov 02 19:57:56.880524 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:56.880463    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8479ff-7fd9b68fc5-plq28"]
Nov 02 19:57:57.044669 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:57.044642    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8479ff-d99f8dbc7-9bvj7"]
Nov 02 19:57:57.045449 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:57:57.045424    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9045c38_104c_4065_b369_6f3803fae81e.slice/crio-421c5c73036908c3df98b6d89a1b0efc56c6063bced163a773253c999d034649 WatchSource:0}: Error finding container 421c5c73036908c3df98b6d89a1b0efc56c6063bced163a773253c999d034649: Status 404 returned error can't find the container with id 421c5c73036908c3df98b6d89a1b0efc56c6063bced163a773253c999d034649
Nov 02 19:57:57.729595 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:57.729541    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8479ff-7fd9b68fc5-plq28" event={"ID":"a211d74d-a663-4c8e-a451-f3253236d6f6","Type":"ContainerStarted","Data":"d9664da1d681a6b868086ab2121ce732a967f09c2dc9a57792d7912da37a4c14"}
Nov 02 19:57:57.730095 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:57.729616    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8479ff-7fd9b68fc5-plq28" event={"ID":"a211d74d-a663-4c8e-a451-f3253236d6f6","Type":"ContainerStarted","Data":"aebb8cd9f3961ec66b11f0867d3ac75bbe1ec741fe32582b91b67580ed9323c5"}
Nov 02 19:57:57.730095 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:57.729643    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8479ff-7fd9b68fc5-plq28"
Nov 02 19:57:57.731455 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:57.731427    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-8479ff-d99f8dbc7-9bvj7" event={"ID":"f9045c38-104c-4065-b369-6f3803fae81e","Type":"ContainerStarted","Data":"0e49b64e6970d2fc3d80e8bcd74f61a64fe6ad70654643f5c418c4344b186865"}
Nov 02 19:57:57.731578 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:57.731472    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-8479ff-d99f8dbc7-9bvj7" event={"ID":"f9045c38-104c-4065-b369-6f3803fae81e","Type":"ContainerStarted","Data":"421c5c73036908c3df98b6d89a1b0efc56c6063bced163a773253c999d034649"}
Nov 02 19:57:57.731578 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:57.731493    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-8479ff-d99f8dbc7-9bvj7"
Nov 02 19:57:57.836856 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:57.836810    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8479ff-7fd9b68fc5-plq28" podStartSLOduration=1.836797197 podStartE2EDuration="1.836797197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:57:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:57:57.83431487 +0000 UTC m=+1784.441348656" watchObservedRunningTime="2026-11-02 19:57:57.836797197 +0000 UTC m=+1784.443830965"
Nov 02 19:57:57.942600 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:57:57.942551    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-8479ff-d99f8dbc7-9bvj7" podStartSLOduration=1.9425394040000001 podStartE2EDuration="1.942539404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:57:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:57:57.938736617 +0000 UTC m=+1784.545770406" watchObservedRunningTime="2026-11-02 19:57:57.942539404 +0000 UTC m=+1784.549573172"
Nov 02 19:58:03.018069 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:03.018031    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-24ef8b-7fff7cb988-vllfk"]
Nov 02 19:58:03.022632 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:03.022609    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-24ef8b-7fff7cb988-vllfk"
Nov 02 19:58:03.071698 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:03.071670    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w6-24ef8b-serving-cert\""
Nov 02 19:58:03.121409 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:03.121372    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-24ef8b-7fff7cb988-vllfk"]
Nov 02 19:58:03.198387 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:03.198355    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2ktt\" (UniqueName: \"kubernetes.io/projected/a053056d-0fa4-4ea9-8091-85979931330f-kube-api-access-s2ktt\") pod \"maas-api-e2e-shared-b-w6-24ef8b-7fff7cb988-vllfk\" (UID: \"a053056d-0fa4-4ea9-8091-85979931330f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-24ef8b-7fff7cb988-vllfk"
Nov 02 19:58:03.198554 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:03.198398    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a053056d-0fa4-4ea9-8091-85979931330f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-24ef8b-7fff7cb988-vllfk\" (UID: \"a053056d-0fa4-4ea9-8091-85979931330f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-24ef8b-7fff7cb988-vllfk"
Nov 02 19:58:03.198554 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:03.198488    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a053056d-0fa4-4ea9-8091-85979931330f-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-24ef8b-7fff7cb988-vllfk\" (UID: \"a053056d-0fa4-4ea9-8091-85979931330f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-24ef8b-7fff7cb988-vllfk"
Nov 02 19:58:03.198657 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:03.198644    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a053056d-0fa4-4ea9-8091-85979931330f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-24ef8b-7fff7cb988-vllfk\" (UID: \"a053056d-0fa4-4ea9-8091-85979931330f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-24ef8b-7fff7cb988-vllfk"
Nov 02 19:58:03.299738 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:03.299656    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2ktt\" (UniqueName: \"kubernetes.io/projected/a053056d-0fa4-4ea9-8091-85979931330f-kube-api-access-s2ktt\") pod \"maas-api-e2e-shared-b-w6-24ef8b-7fff7cb988-vllfk\" (UID: \"a053056d-0fa4-4ea9-8091-85979931330f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-24ef8b-7fff7cb988-vllfk"
Nov 02 19:58:03.299738 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:03.299702    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a053056d-0fa4-4ea9-8091-85979931330f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-24ef8b-7fff7cb988-vllfk\" (UID: \"a053056d-0fa4-4ea9-8091-85979931330f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-24ef8b-7fff7cb988-vllfk"
Nov 02 19:58:03.299955 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:03.299755    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a053056d-0fa4-4ea9-8091-85979931330f-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-24ef8b-7fff7cb988-vllfk\" (UID: \"a053056d-0fa4-4ea9-8091-85979931330f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-24ef8b-7fff7cb988-vllfk"
Nov 02 19:58:03.299955 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:03.299908    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a053056d-0fa4-4ea9-8091-85979931330f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-24ef8b-7fff7cb988-vllfk\" (UID: \"a053056d-0fa4-4ea9-8091-85979931330f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-24ef8b-7fff7cb988-vllfk"
Nov 02 19:58:03.300541 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:03.300516    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a053056d-0fa4-4ea9-8091-85979931330f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-24ef8b-7fff7cb988-vllfk\" (UID: \"a053056d-0fa4-4ea9-8091-85979931330f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-24ef8b-7fff7cb988-vllfk"
Nov 02 19:58:03.302238 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:03.302215    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a053056d-0fa4-4ea9-8091-85979931330f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-24ef8b-7fff7cb988-vllfk\" (UID: \"a053056d-0fa4-4ea9-8091-85979931330f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-24ef8b-7fff7cb988-vllfk"
Nov 02 19:58:03.302330 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:03.302256    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a053056d-0fa4-4ea9-8091-85979931330f-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-24ef8b-7fff7cb988-vllfk\" (UID: \"a053056d-0fa4-4ea9-8091-85979931330f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-24ef8b-7fff7cb988-vllfk"
Nov 02 19:58:03.352964 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:03.352934    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2ktt\" (UniqueName: \"kubernetes.io/projected/a053056d-0fa4-4ea9-8091-85979931330f-kube-api-access-s2ktt\") pod \"maas-api-e2e-shared-b-w6-24ef8b-7fff7cb988-vllfk\" (UID: \"a053056d-0fa4-4ea9-8091-85979931330f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-24ef8b-7fff7cb988-vllfk"
Nov 02 19:58:03.636749 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:03.636670    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-24ef8b-7fff7cb988-vllfk"
Nov 02 19:58:04.004328 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:04.004293    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-24ef8b-7fff7cb988-vllfk"]
Nov 02 19:58:04.005573 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:58:04.005543    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda053056d_0fa4_4ea9_8091_85979931330f.slice/crio-630a4f2e0ca646400c14f58a9f98e12976d2caaf2ee69dcec693248417a9a11e WatchSource:0}: Error finding container 630a4f2e0ca646400c14f58a9f98e12976d2caaf2ee69dcec693248417a9a11e: Status 404 returned error can't find the container with id 630a4f2e0ca646400c14f58a9f98e12976d2caaf2ee69dcec693248417a9a11e
Nov 02 19:58:04.006874 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:04.006853    4463 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 19:58:04.294565 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:58:04.294485    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 19:58:04.778849 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:04.778814    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-24ef8b-7fff7cb988-vllfk" event={"ID":"a053056d-0fa4-4ea9-8091-85979931330f","Type":"ContainerStarted","Data":"630a4f2e0ca646400c14f58a9f98e12976d2caaf2ee69dcec693248417a9a11e"}
Nov 02 19:58:05.787065 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:05.787032    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-24ef8b-7fff7cb988-vllfk" event={"ID":"a053056d-0fa4-4ea9-8091-85979931330f","Type":"ContainerStarted","Data":"42ca1f33848771f5acc256713cae4d411f5302377cb12b633e8ab34ba79e94f5"}
Nov 02 19:58:05.787464 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:05.787165    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-24ef8b-7fff7cb988-vllfk"
Nov 02 19:58:05.899122 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:05.899052    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-24ef8b-7fff7cb988-vllfk" podStartSLOduration=3.059916753 podStartE2EDuration="3.899031951s" totalImagesPullingTime="839.115198ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:58:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:58:05.893765434 +0000 UTC m=+1792.500799203" watchObservedRunningTime="2026-11-02 19:58:05.899031951 +0000 UTC m=+1792.506065719"
Nov 02 19:58:08.740486 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:08.740455    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8479ff-7fd9b68fc5-plq28"
Nov 02 19:58:08.740886 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:08.740530    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-8479ff-d99f8dbc7-9bvj7"
Nov 02 19:58:11.802442 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:11.802405    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-24ef8b-7fff7cb988-vllfk"
Nov 02 19:58:12.104546 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:12.104466    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 19:58:12.439290 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:12.439261    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-23a9f5-openshift-default-6d49545b45-h5trw"]
Nov 02 19:58:12.443990 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:12.443964    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-23a9f5-openshift-default-6d49545b45-h5trw"
Nov 02 19:58:12.535256 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:12.528520    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-23a9f5-openshift-default-dockercfg-psbtx\""
Nov 02 19:58:12.535256 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:12.529530    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84a329f5-3f72-46b0-a7f6-eba84a47db5a-istio-envoy\") pod \"e2e-api-a-23a9f5-openshift-default-6d49545b45-h5trw\" (UID: \"84a329f5-3f72-46b0-a7f6-eba84a47db5a\") " pod="openshift-ingress/e2e-api-a-23a9f5-openshift-default-6d49545b45-h5trw"
Nov 02 19:58:12.535256 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:12.529598    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84a329f5-3f72-46b0-a7f6-eba84a47db5a-workload-socket\") pod \"e2e-api-a-23a9f5-openshift-default-6d49545b45-h5trw\" (UID: \"84a329f5-3f72-46b0-a7f6-eba84a47db5a\") " pod="openshift-ingress/e2e-api-a-23a9f5-openshift-default-6d49545b45-h5trw"
Nov 02 19:58:12.535256 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:12.529660    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84a329f5-3f72-46b0-a7f6-eba84a47db5a-istiod-ca-cert\") pod \"e2e-api-a-23a9f5-openshift-default-6d49545b45-h5trw\" (UID: \"84a329f5-3f72-46b0-a7f6-eba84a47db5a\") " pod="openshift-ingress/e2e-api-a-23a9f5-openshift-default-6d49545b45-h5trw"
Nov 02 19:58:12.535256 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:12.529787    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqw5x\" (UniqueName: \"kubernetes.io/projected/84a329f5-3f72-46b0-a7f6-eba84a47db5a-kube-api-access-wqw5x\") pod \"e2e-api-a-23a9f5-openshift-default-6d49545b45-h5trw\" (UID: \"84a329f5-3f72-46b0-a7f6-eba84a47db5a\") " pod="openshift-ingress/e2e-api-a-23a9f5-openshift-default-6d49545b45-h5trw"
Nov 02 19:58:12.535256 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:12.529834    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84a329f5-3f72-46b0-a7f6-eba84a47db5a-istio-token\") pod \"e2e-api-a-23a9f5-openshift-default-6d49545b45-h5trw\" (UID: \"84a329f5-3f72-46b0-a7f6-eba84a47db5a\") " pod="openshift-ingress/e2e-api-a-23a9f5-openshift-default-6d49545b45-h5trw"
Nov 02 19:58:12.535256 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:12.529885    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84a329f5-3f72-46b0-a7f6-eba84a47db5a-workload-certs\") pod \"e2e-api-a-23a9f5-openshift-default-6d49545b45-h5trw\" (UID: \"84a329f5-3f72-46b0-a7f6-eba84a47db5a\") " pod="openshift-ingress/e2e-api-a-23a9f5-openshift-default-6d49545b45-h5trw"
Nov 02 19:58:12.535256 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:12.529908    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84a329f5-3f72-46b0-a7f6-eba84a47db5a-istio-podinfo\") pod \"e2e-api-a-23a9f5-openshift-default-6d49545b45-h5trw\" (UID: \"84a329f5-3f72-46b0-a7f6-eba84a47db5a\") " pod="openshift-ingress/e2e-api-a-23a9f5-openshift-default-6d49545b45-h5trw"
Nov 02 19:58:12.535256 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:12.529938    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84a329f5-3f72-46b0-a7f6-eba84a47db5a-credential-socket\") pod \"e2e-api-a-23a9f5-openshift-default-6d49545b45-h5trw\" (UID: \"84a329f5-3f72-46b0-a7f6-eba84a47db5a\") " pod="openshift-ingress/e2e-api-a-23a9f5-openshift-default-6d49545b45-h5trw"
Nov 02 19:58:12.535256 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:12.530067    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84a329f5-3f72-46b0-a7f6-eba84a47db5a-istio-data\") pod \"e2e-api-a-23a9f5-openshift-default-6d49545b45-h5trw\" (UID: \"84a329f5-3f72-46b0-a7f6-eba84a47db5a\") " pod="openshift-ingress/e2e-api-a-23a9f5-openshift-default-6d49545b45-h5trw"
Nov 02 19:58:12.595036 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:12.594581    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-23a9f5-openshift-default-6d49545b45-h5trw"]
Nov 02 19:58:12.634039 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:12.632835    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84a329f5-3f72-46b0-a7f6-eba84a47db5a-istio-data\") pod \"e2e-api-a-23a9f5-openshift-default-6d49545b45-h5trw\" (UID: \"84a329f5-3f72-46b0-a7f6-eba84a47db5a\") " pod="openshift-ingress/e2e-api-a-23a9f5-openshift-default-6d49545b45-h5trw"
Nov 02 19:58:12.634039 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:12.632887    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84a329f5-3f72-46b0-a7f6-eba84a47db5a-istio-envoy\") pod \"e2e-api-a-23a9f5-openshift-default-6d49545b45-h5trw\" (UID: \"84a329f5-3f72-46b0-a7f6-eba84a47db5a\") " pod="openshift-ingress/e2e-api-a-23a9f5-openshift-default-6d49545b45-h5trw"
Nov 02 19:58:12.634039 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:12.632938    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84a329f5-3f72-46b0-a7f6-eba84a47db5a-workload-socket\") pod \"e2e-api-a-23a9f5-openshift-default-6d49545b45-h5trw\" (UID: \"84a329f5-3f72-46b0-a7f6-eba84a47db5a\") " pod="openshift-ingress/e2e-api-a-23a9f5-openshift-default-6d49545b45-h5trw"
Nov 02 19:58:12.634039 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:12.632987    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84a329f5-3f72-46b0-a7f6-eba84a47db5a-istiod-ca-cert\") pod \"e2e-api-a-23a9f5-openshift-default-6d49545b45-h5trw\" (UID: \"84a329f5-3f72-46b0-a7f6-eba84a47db5a\") " pod="openshift-ingress/e2e-api-a-23a9f5-openshift-default-6d49545b45-h5trw"
Nov 02 19:58:12.634039 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:12.633144    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqw5x\" (UniqueName: \"kubernetes.io/projected/84a329f5-3f72-46b0-a7f6-eba84a47db5a-kube-api-access-wqw5x\") pod \"e2e-api-a-23a9f5-openshift-default-6d49545b45-h5trw\" (UID: \"84a329f5-3f72-46b0-a7f6-eba84a47db5a\") " pod="openshift-ingress/e2e-api-a-23a9f5-openshift-default-6d49545b45-h5trw"
Nov 02 19:58:12.634039 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:12.633224    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84a329f5-3f72-46b0-a7f6-eba84a47db5a-istio-token\") pod \"e2e-api-a-23a9f5-openshift-default-6d49545b45-h5trw\" (UID: \"84a329f5-3f72-46b0-a7f6-eba84a47db5a\") " pod="openshift-ingress/e2e-api-a-23a9f5-openshift-default-6d49545b45-h5trw"
Nov 02 19:58:12.634039 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:12.633282    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84a329f5-3f72-46b0-a7f6-eba84a47db5a-workload-certs\") pod \"e2e-api-a-23a9f5-openshift-default-6d49545b45-h5trw\" (UID: \"84a329f5-3f72-46b0-a7f6-eba84a47db5a\") " pod="openshift-ingress/e2e-api-a-23a9f5-openshift-default-6d49545b45-h5trw"
Nov 02 19:58:12.634039 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:12.633309    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84a329f5-3f72-46b0-a7f6-eba84a47db5a-istio-podinfo\") pod \"e2e-api-a-23a9f5-openshift-default-6d49545b45-h5trw\" (UID: \"84a329f5-3f72-46b0-a7f6-eba84a47db5a\") " pod="openshift-ingress/e2e-api-a-23a9f5-openshift-default-6d49545b45-h5trw"
Nov 02 19:58:12.634039 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:12.633343    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84a329f5-3f72-46b0-a7f6-eba84a47db5a-credential-socket\") pod \"e2e-api-a-23a9f5-openshift-default-6d49545b45-h5trw\" (UID: \"84a329f5-3f72-46b0-a7f6-eba84a47db5a\") " pod="openshift-ingress/e2e-api-a-23a9f5-openshift-default-6d49545b45-h5trw"
Nov 02 19:58:12.634039 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:12.633786    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84a329f5-3f72-46b0-a7f6-eba84a47db5a-credential-socket\") pod \"e2e-api-a-23a9f5-openshift-default-6d49545b45-h5trw\" (UID: \"84a329f5-3f72-46b0-a7f6-eba84a47db5a\") " pod="openshift-ingress/e2e-api-a-23a9f5-openshift-default-6d49545b45-h5trw"
Nov 02 19:58:12.634681 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:12.634500    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84a329f5-3f72-46b0-a7f6-eba84a47db5a-istio-data\") pod \"e2e-api-a-23a9f5-openshift-default-6d49545b45-h5trw\" (UID: \"84a329f5-3f72-46b0-a7f6-eba84a47db5a\") " pod="openshift-ingress/e2e-api-a-23a9f5-openshift-default-6d49545b45-h5trw"
Nov 02 19:58:12.639434 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:12.635240    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84a329f5-3f72-46b0-a7f6-eba84a47db5a-workload-socket\") pod \"e2e-api-a-23a9f5-openshift-default-6d49545b45-h5trw\" (UID: \"84a329f5-3f72-46b0-a7f6-eba84a47db5a\") " pod="openshift-ingress/e2e-api-a-23a9f5-openshift-default-6d49545b45-h5trw"
Nov 02 19:58:12.639434 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:12.635792    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84a329f5-3f72-46b0-a7f6-eba84a47db5a-istiod-ca-cert\") pod \"e2e-api-a-23a9f5-openshift-default-6d49545b45-h5trw\" (UID: \"84a329f5-3f72-46b0-a7f6-eba84a47db5a\") " pod="openshift-ingress/e2e-api-a-23a9f5-openshift-default-6d49545b45-h5trw"
Nov 02 19:58:12.639434 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:12.636645    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84a329f5-3f72-46b0-a7f6-eba84a47db5a-workload-certs\") pod \"e2e-api-a-23a9f5-openshift-default-6d49545b45-h5trw\" (UID: \"84a329f5-3f72-46b0-a7f6-eba84a47db5a\") " pod="openshift-ingress/e2e-api-a-23a9f5-openshift-default-6d49545b45-h5trw"
Nov 02 19:58:12.639434 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:12.637589    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84a329f5-3f72-46b0-a7f6-eba84a47db5a-istio-envoy\") pod \"e2e-api-a-23a9f5-openshift-default-6d49545b45-h5trw\" (UID: \"84a329f5-3f72-46b0-a7f6-eba84a47db5a\") " pod="openshift-ingress/e2e-api-a-23a9f5-openshift-default-6d49545b45-h5trw"
Nov 02 19:58:12.668956 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:12.647933    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84a329f5-3f72-46b0-a7f6-eba84a47db5a-istio-podinfo\") pod \"e2e-api-a-23a9f5-openshift-default-6d49545b45-h5trw\" (UID: \"84a329f5-3f72-46b0-a7f6-eba84a47db5a\") " pod="openshift-ingress/e2e-api-a-23a9f5-openshift-default-6d49545b45-h5trw"
Nov 02 19:58:12.689407 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:12.689326    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84a329f5-3f72-46b0-a7f6-eba84a47db5a-istio-token\") pod \"e2e-api-a-23a9f5-openshift-default-6d49545b45-h5trw\" (UID: \"84a329f5-3f72-46b0-a7f6-eba84a47db5a\") " pod="openshift-ingress/e2e-api-a-23a9f5-openshift-default-6d49545b45-h5trw"
Nov 02 19:58:12.689595 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:12.689574    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqw5x\" (UniqueName: \"kubernetes.io/projected/84a329f5-3f72-46b0-a7f6-eba84a47db5a-kube-api-access-wqw5x\") pod \"e2e-api-a-23a9f5-openshift-default-6d49545b45-h5trw\" (UID: \"84a329f5-3f72-46b0-a7f6-eba84a47db5a\") " pod="openshift-ingress/e2e-api-a-23a9f5-openshift-default-6d49545b45-h5trw"
Nov 02 19:58:12.758204 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:12.758160    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-23a9f5-openshift-default-6d49545b45-h5trw"
Nov 02 19:58:13.207389 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:13.207356    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-23a9f5-openshift-default-6d49545b45-h5trw"]
Nov 02 19:58:13.244199 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:58:13.244173    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84a329f5_3f72_46b0_a7f6_eba84a47db5a.slice/crio-e419e5b0c0c0a2cce9aaf1fd92cae60d9cc5bdff1d5fd0334fcfdd4667e23ca9 WatchSource:0}: Error finding container e419e5b0c0c0a2cce9aaf1fd92cae60d9cc5bdff1d5fd0334fcfdd4667e23ca9: Status 404 returned error can't find the container with id e419e5b0c0c0a2cce9aaf1fd92cae60d9cc5bdff1d5fd0334fcfdd4667e23ca9
Nov 02 19:58:13.246798 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:13.246769    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:58:13.246878 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:13.246837    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:58:13.246878 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:13.246866    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:58:13.844946 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:13.844905    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-23a9f5-openshift-default-6d49545b45-h5trw" event={"ID":"84a329f5-3f72-46b0-a7f6-eba84a47db5a","Type":"ContainerStarted","Data":"e228b4c4a6073ae7a665418ee7b5a8ea91597b65f6ae33f2c65128768a2f0972"}
Nov 02 19:58:13.845130 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:13.844989    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-23a9f5-openshift-default-6d49545b45-h5trw" event={"ID":"84a329f5-3f72-46b0-a7f6-eba84a47db5a","Type":"ContainerStarted","Data":"e419e5b0c0c0a2cce9aaf1fd92cae60d9cc5bdff1d5fd0334fcfdd4667e23ca9"}
Nov 02 19:58:13.957330 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:13.957283    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-23a9f5-openshift-default-6d49545b45-h5trw" podStartSLOduration=1.957266382 podStartE2EDuration="1.957266382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:58:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:58:13.953057072 +0000 UTC m=+1800.560090840" watchObservedRunningTime="2026-11-02 19:58:13.957266382 +0000 UTC m=+1800.564300153"
Nov 02 19:58:14.759257 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:14.759216    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-23a9f5-openshift-default-6d49545b45-h5trw"
Nov 02 19:58:14.765147 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:14.765119    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-23a9f5-openshift-default-6d49545b45-h5trw"
Nov 02 19:58:14.851782 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:14.851762    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-23a9f5-openshift-default-6d49545b45-h5trw"
Nov 02 19:58:14.852488 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:14.852469    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-23a9f5-openshift-default-6d49545b45-h5trw"
Nov 02 19:58:20.515209 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:20.515171    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-23a9f5-85b7bb67f-8f9nj"]
Nov 02 19:58:20.523434 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:20.523406    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-23a9f5-85b7bb67f-8f9nj"
Nov 02 19:58:20.573929 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:20.573893    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-23a9f5-85b7bb67f-8f9nj"]
Nov 02 19:58:20.612950 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:20.612885    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-23a9f5\""
Nov 02 19:58:20.613405 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:20.613380    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-23a9f5-dockercfg-n2nml\""
Nov 02 19:58:20.645155 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:20.645115    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98mvh\" (UniqueName: \"kubernetes.io/projected/a2862a4d-a528-4766-ac1e-958b2de9ca1d-kube-api-access-98mvh\") pod \"payload-processing-e2e-api-a-23a9f5-85b7bb67f-8f9nj\" (UID: \"a2862a4d-a528-4766-ac1e-958b2de9ca1d\") " pod="openshift-ingress/payload-processing-e2e-api-a-23a9f5-85b7bb67f-8f9nj"
Nov 02 19:58:20.645600 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:20.645579    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2862a4d-a528-4766-ac1e-958b2de9ca1d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-23a9f5-85b7bb67f-8f9nj\" (UID: \"a2862a4d-a528-4766-ac1e-958b2de9ca1d\") " pod="openshift-ingress/payload-processing-e2e-api-a-23a9f5-85b7bb67f-8f9nj"
Nov 02 19:58:20.702706 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:20.702670    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-23a9f5-6f4f945579-gplz2"]
Nov 02 19:58:20.707098 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:20.707077    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-23a9f5-6f4f945579-gplz2"
Nov 02 19:58:20.746853 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:20.746817    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2862a4d-a528-4766-ac1e-958b2de9ca1d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-23a9f5-85b7bb67f-8f9nj\" (UID: \"a2862a4d-a528-4766-ac1e-958b2de9ca1d\") " pod="openshift-ingress/payload-processing-e2e-api-a-23a9f5-85b7bb67f-8f9nj"
Nov 02 19:58:20.746997 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:20.746900    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/045e2518-6641-4e5d-a9b7-10326c74b26b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-23a9f5-6f4f945579-gplz2\" (UID: \"045e2518-6641-4e5d-a9b7-10326c74b26b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-23a9f5-6f4f945579-gplz2"
Nov 02 19:58:20.747100 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:20.747077    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wd79v\" (UniqueName: \"kubernetes.io/projected/045e2518-6641-4e5d-a9b7-10326c74b26b-kube-api-access-wd79v\") pod \"payload-pre-processing-e2e-api-a-23a9f5-6f4f945579-gplz2\" (UID: \"045e2518-6641-4e5d-a9b7-10326c74b26b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-23a9f5-6f4f945579-gplz2"
Nov 02 19:58:20.747164 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:20.747135    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98mvh\" (UniqueName: \"kubernetes.io/projected/a2862a4d-a528-4766-ac1e-958b2de9ca1d-kube-api-access-98mvh\") pod \"payload-processing-e2e-api-a-23a9f5-85b7bb67f-8f9nj\" (UID: \"a2862a4d-a528-4766-ac1e-958b2de9ca1d\") " pod="openshift-ingress/payload-processing-e2e-api-a-23a9f5-85b7bb67f-8f9nj"
Nov 02 19:58:20.747597 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:20.747574    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2862a4d-a528-4766-ac1e-958b2de9ca1d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-23a9f5-85b7bb67f-8f9nj\" (UID: \"a2862a4d-a528-4766-ac1e-958b2de9ca1d\") " pod="openshift-ingress/payload-processing-e2e-api-a-23a9f5-85b7bb67f-8f9nj"
Nov 02 19:58:20.759567 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:20.759540    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-23a9f5-6f4f945579-gplz2"]
Nov 02 19:58:20.800190 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:20.800124    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98mvh\" (UniqueName: \"kubernetes.io/projected/a2862a4d-a528-4766-ac1e-958b2de9ca1d-kube-api-access-98mvh\") pod \"payload-processing-e2e-api-a-23a9f5-85b7bb67f-8f9nj\" (UID: \"a2862a4d-a528-4766-ac1e-958b2de9ca1d\") " pod="openshift-ingress/payload-processing-e2e-api-a-23a9f5-85b7bb67f-8f9nj"
Nov 02 19:58:20.841999 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:20.841975    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-23a9f5-85b7bb67f-8f9nj"
Nov 02 19:58:20.856372 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:20.852531    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wd79v\" (UniqueName: \"kubernetes.io/projected/045e2518-6641-4e5d-a9b7-10326c74b26b-kube-api-access-wd79v\") pod \"payload-pre-processing-e2e-api-a-23a9f5-6f4f945579-gplz2\" (UID: \"045e2518-6641-4e5d-a9b7-10326c74b26b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-23a9f5-6f4f945579-gplz2"
Nov 02 19:58:20.856372 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:20.852832    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/045e2518-6641-4e5d-a9b7-10326c74b26b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-23a9f5-6f4f945579-gplz2\" (UID: \"045e2518-6641-4e5d-a9b7-10326c74b26b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-23a9f5-6f4f945579-gplz2"
Nov 02 19:58:20.856372 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:20.853803    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/045e2518-6641-4e5d-a9b7-10326c74b26b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-23a9f5-6f4f945579-gplz2\" (UID: \"045e2518-6641-4e5d-a9b7-10326c74b26b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-23a9f5-6f4f945579-gplz2"
Nov 02 19:58:20.907137 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:20.907109    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wd79v\" (UniqueName: \"kubernetes.io/projected/045e2518-6641-4e5d-a9b7-10326c74b26b-kube-api-access-wd79v\") pod \"payload-pre-processing-e2e-api-a-23a9f5-6f4f945579-gplz2\" (UID: \"045e2518-6641-4e5d-a9b7-10326c74b26b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-23a9f5-6f4f945579-gplz2"
Nov 02 19:58:21.024709 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:21.024673    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-23a9f5-6f4f945579-gplz2"
Nov 02 19:58:21.217836 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:21.217811    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-23a9f5-85b7bb67f-8f9nj"]
Nov 02 19:58:21.218760 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:58:21.218733    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2862a4d_a528_4766_ac1e_958b2de9ca1d.slice/crio-d94f5d57f8c3d2840bb38dba2d21551be1aaef066b20e106dab72b1fae2ace7d WatchSource:0}: Error finding container d94f5d57f8c3d2840bb38dba2d21551be1aaef066b20e106dab72b1fae2ace7d: Status 404 returned error can't find the container with id d94f5d57f8c3d2840bb38dba2d21551be1aaef066b20e106dab72b1fae2ace7d
Nov 02 19:58:21.415696 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:21.415667    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-23a9f5-6f4f945579-gplz2"]
Nov 02 19:58:21.416323 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:58:21.416295    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod045e2518_6641_4e5d_a9b7_10326c74b26b.slice/crio-35d3b31f4e0faf657765b4c6945ce7a71f7b5dd6738d2394b4a7996bac9e6b44 WatchSource:0}: Error finding container 35d3b31f4e0faf657765b4c6945ce7a71f7b5dd6738d2394b4a7996bac9e6b44: Status 404 returned error can't find the container with id 35d3b31f4e0faf657765b4c6945ce7a71f7b5dd6738d2394b4a7996bac9e6b44
Nov 02 19:58:21.904067 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:21.904027    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-23a9f5-6f4f945579-gplz2" event={"ID":"045e2518-6641-4e5d-a9b7-10326c74b26b","Type":"ContainerStarted","Data":"efed4e8269b553eae37ecf8879f8103c20b91f197f6b855251d5c6233fb78674"}
Nov 02 19:58:21.904519 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:21.904084    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-23a9f5-6f4f945579-gplz2" event={"ID":"045e2518-6641-4e5d-a9b7-10326c74b26b","Type":"ContainerStarted","Data":"35d3b31f4e0faf657765b4c6945ce7a71f7b5dd6738d2394b4a7996bac9e6b44"}
Nov 02 19:58:21.904519 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:21.904108    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-23a9f5-6f4f945579-gplz2"
Nov 02 19:58:21.905704 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:21.905676    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-23a9f5-85b7bb67f-8f9nj" event={"ID":"a2862a4d-a528-4766-ac1e-958b2de9ca1d","Type":"ContainerStarted","Data":"9a2b9d289d540b5db5c814f5c4b5f5187714aab1febde8fd536cf32217b3c234"}
Nov 02 19:58:21.905825 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:21.905719    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-23a9f5-85b7bb67f-8f9nj"
Nov 02 19:58:21.905825 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:21.905736    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-23a9f5-85b7bb67f-8f9nj" event={"ID":"a2862a4d-a528-4766-ac1e-958b2de9ca1d","Type":"ContainerStarted","Data":"d94f5d57f8c3d2840bb38dba2d21551be1aaef066b20e106dab72b1fae2ace7d"}
Nov 02 19:58:22.013410 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:22.013364    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-23a9f5-6f4f945579-gplz2" podStartSLOduration=2.013351461 podStartE2EDuration="2.013351461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:58:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:58:22.009757938 +0000 UTC m=+1808.616791714" watchObservedRunningTime="2026-11-02 19:58:22.013351461 +0000 UTC m=+1808.620385229"
Nov 02 19:58:22.117474 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:22.117426    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-23a9f5-85b7bb67f-8f9nj" podStartSLOduration=2.117414331 podStartE2EDuration="2.117414331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:58:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:58:22.114927384 +0000 UTC m=+1808.721961151" watchObservedRunningTime="2026-11-02 19:58:22.117414331 +0000 UTC m=+1808.724448125"
Nov 02 19:58:32.915443 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:32.915414    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-23a9f5-6f4f945579-gplz2"
Nov 02 19:58:32.915824 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:32.915478    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-23a9f5-85b7bb67f-8f9nj"
Nov 02 19:58:33.870195 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:33.870160    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-c60a0d-openshift-default-6464975799-g9tvr"]
Nov 02 19:58:33.876045 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:33.875993    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c60a0d-openshift-default-6464975799-g9tvr"
Nov 02 19:58:33.927652 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:33.927618    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-c60a0d-openshift-default-dockercfg-6bbqd\""
Nov 02 19:58:34.035824 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:34.035783    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/618bcdd7-16ca-40c3-9a41-feed4d5f7161-credential-socket\") pod \"e2e-api-b-c60a0d-openshift-default-6464975799-g9tvr\" (UID: \"618bcdd7-16ca-40c3-9a41-feed4d5f7161\") " pod="openshift-ingress/e2e-api-b-c60a0d-openshift-default-6464975799-g9tvr"
Nov 02 19:58:34.036145 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:34.036093    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/618bcdd7-16ca-40c3-9a41-feed4d5f7161-istiod-ca-cert\") pod \"e2e-api-b-c60a0d-openshift-default-6464975799-g9tvr\" (UID: \"618bcdd7-16ca-40c3-9a41-feed4d5f7161\") " pod="openshift-ingress/e2e-api-b-c60a0d-openshift-default-6464975799-g9tvr"
Nov 02 19:58:34.042533 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:34.042503    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/618bcdd7-16ca-40c3-9a41-feed4d5f7161-istio-token\") pod \"e2e-api-b-c60a0d-openshift-default-6464975799-g9tvr\" (UID: \"618bcdd7-16ca-40c3-9a41-feed4d5f7161\") " pod="openshift-ingress/e2e-api-b-c60a0d-openshift-default-6464975799-g9tvr"
Nov 02 19:58:34.043481 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:34.043458    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/618bcdd7-16ca-40c3-9a41-feed4d5f7161-istio-data\") pod \"e2e-api-b-c60a0d-openshift-default-6464975799-g9tvr\" (UID: \"618bcdd7-16ca-40c3-9a41-feed4d5f7161\") " pod="openshift-ingress/e2e-api-b-c60a0d-openshift-default-6464975799-g9tvr"
Nov 02 19:58:34.043672 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:34.043654    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/618bcdd7-16ca-40c3-9a41-feed4d5f7161-istio-envoy\") pod \"e2e-api-b-c60a0d-openshift-default-6464975799-g9tvr\" (UID: \"618bcdd7-16ca-40c3-9a41-feed4d5f7161\") " pod="openshift-ingress/e2e-api-b-c60a0d-openshift-default-6464975799-g9tvr"
Nov 02 19:58:34.043883 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:34.043865    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zj5k\" (UniqueName: \"kubernetes.io/projected/618bcdd7-16ca-40c3-9a41-feed4d5f7161-kube-api-access-8zj5k\") pod \"e2e-api-b-c60a0d-openshift-default-6464975799-g9tvr\" (UID: \"618bcdd7-16ca-40c3-9a41-feed4d5f7161\") " pod="openshift-ingress/e2e-api-b-c60a0d-openshift-default-6464975799-g9tvr"
Nov 02 19:58:34.048600 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:34.048575    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/618bcdd7-16ca-40c3-9a41-feed4d5f7161-istio-podinfo\") pod \"e2e-api-b-c60a0d-openshift-default-6464975799-g9tvr\" (UID: \"618bcdd7-16ca-40c3-9a41-feed4d5f7161\") " pod="openshift-ingress/e2e-api-b-c60a0d-openshift-default-6464975799-g9tvr"
Nov 02 19:58:34.053526 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:34.049755    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/618bcdd7-16ca-40c3-9a41-feed4d5f7161-workload-socket\") pod \"e2e-api-b-c60a0d-openshift-default-6464975799-g9tvr\" (UID: \"618bcdd7-16ca-40c3-9a41-feed4d5f7161\") " pod="openshift-ingress/e2e-api-b-c60a0d-openshift-default-6464975799-g9tvr"
Nov 02 19:58:34.053526 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:34.049835    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/618bcdd7-16ca-40c3-9a41-feed4d5f7161-workload-certs\") pod \"e2e-api-b-c60a0d-openshift-default-6464975799-g9tvr\" (UID: \"618bcdd7-16ca-40c3-9a41-feed4d5f7161\") " pod="openshift-ingress/e2e-api-b-c60a0d-openshift-default-6464975799-g9tvr"
Nov 02 19:58:34.053526 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:34.052119    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-c60a0d-openshift-default-6464975799-g9tvr"]
Nov 02 19:58:34.151289 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:34.151211    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zj5k\" (UniqueName: \"kubernetes.io/projected/618bcdd7-16ca-40c3-9a41-feed4d5f7161-kube-api-access-8zj5k\") pod \"e2e-api-b-c60a0d-openshift-default-6464975799-g9tvr\" (UID: \"618bcdd7-16ca-40c3-9a41-feed4d5f7161\") " pod="openshift-ingress/e2e-api-b-c60a0d-openshift-default-6464975799-g9tvr"
Nov 02 19:58:34.151289 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:34.151247    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/618bcdd7-16ca-40c3-9a41-feed4d5f7161-istio-podinfo\") pod \"e2e-api-b-c60a0d-openshift-default-6464975799-g9tvr\" (UID: \"618bcdd7-16ca-40c3-9a41-feed4d5f7161\") " pod="openshift-ingress/e2e-api-b-c60a0d-openshift-default-6464975799-g9tvr"
Nov 02 19:58:34.151563 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:34.151417    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/618bcdd7-16ca-40c3-9a41-feed4d5f7161-workload-socket\") pod \"e2e-api-b-c60a0d-openshift-default-6464975799-g9tvr\" (UID: \"618bcdd7-16ca-40c3-9a41-feed4d5f7161\") " pod="openshift-ingress/e2e-api-b-c60a0d-openshift-default-6464975799-g9tvr"
Nov 02 19:58:34.151563 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:34.151504    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/618bcdd7-16ca-40c3-9a41-feed4d5f7161-workload-certs\") pod \"e2e-api-b-c60a0d-openshift-default-6464975799-g9tvr\" (UID: \"618bcdd7-16ca-40c3-9a41-feed4d5f7161\") " pod="openshift-ingress/e2e-api-b-c60a0d-openshift-default-6464975799-g9tvr"
Nov 02 19:58:34.151676 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:34.151597    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/618bcdd7-16ca-40c3-9a41-feed4d5f7161-credential-socket\") pod \"e2e-api-b-c60a0d-openshift-default-6464975799-g9tvr\" (UID: \"618bcdd7-16ca-40c3-9a41-feed4d5f7161\") " pod="openshift-ingress/e2e-api-b-c60a0d-openshift-default-6464975799-g9tvr"
Nov 02 19:58:34.151676 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:34.151628    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/618bcdd7-16ca-40c3-9a41-feed4d5f7161-istiod-ca-cert\") pod \"e2e-api-b-c60a0d-openshift-default-6464975799-g9tvr\" (UID: \"618bcdd7-16ca-40c3-9a41-feed4d5f7161\") " pod="openshift-ingress/e2e-api-b-c60a0d-openshift-default-6464975799-g9tvr"
Nov 02 19:58:34.151676 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:34.151667    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/618bcdd7-16ca-40c3-9a41-feed4d5f7161-istio-token\") pod \"e2e-api-b-c60a0d-openshift-default-6464975799-g9tvr\" (UID: \"618bcdd7-16ca-40c3-9a41-feed4d5f7161\") " pod="openshift-ingress/e2e-api-b-c60a0d-openshift-default-6464975799-g9tvr"
Nov 02 19:58:34.151848 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:34.151736    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/618bcdd7-16ca-40c3-9a41-feed4d5f7161-istio-data\") pod \"e2e-api-b-c60a0d-openshift-default-6464975799-g9tvr\" (UID: \"618bcdd7-16ca-40c3-9a41-feed4d5f7161\") " pod="openshift-ingress/e2e-api-b-c60a0d-openshift-default-6464975799-g9tvr"
Nov 02 19:58:34.151848 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:34.151777    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/618bcdd7-16ca-40c3-9a41-feed4d5f7161-workload-socket\") pod \"e2e-api-b-c60a0d-openshift-default-6464975799-g9tvr\" (UID: \"618bcdd7-16ca-40c3-9a41-feed4d5f7161\") " pod="openshift-ingress/e2e-api-b-c60a0d-openshift-default-6464975799-g9tvr"
Nov 02 19:58:34.151945 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:34.151843    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/618bcdd7-16ca-40c3-9a41-feed4d5f7161-workload-certs\") pod \"e2e-api-b-c60a0d-openshift-default-6464975799-g9tvr\" (UID: \"618bcdd7-16ca-40c3-9a41-feed4d5f7161\") " pod="openshift-ingress/e2e-api-b-c60a0d-openshift-default-6464975799-g9tvr"
Nov 02 19:58:34.151945 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:34.151876    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/618bcdd7-16ca-40c3-9a41-feed4d5f7161-istio-envoy\") pod \"e2e-api-b-c60a0d-openshift-default-6464975799-g9tvr\" (UID: \"618bcdd7-16ca-40c3-9a41-feed4d5f7161\") " pod="openshift-ingress/e2e-api-b-c60a0d-openshift-default-6464975799-g9tvr"
Nov 02 19:58:34.152071 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:34.151969    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/618bcdd7-16ca-40c3-9a41-feed4d5f7161-credential-socket\") pod \"e2e-api-b-c60a0d-openshift-default-6464975799-g9tvr\" (UID: \"618bcdd7-16ca-40c3-9a41-feed4d5f7161\") " pod="openshift-ingress/e2e-api-b-c60a0d-openshift-default-6464975799-g9tvr"
Nov 02 19:58:34.152132 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:34.152074    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/618bcdd7-16ca-40c3-9a41-feed4d5f7161-istio-data\") pod \"e2e-api-b-c60a0d-openshift-default-6464975799-g9tvr\" (UID: \"618bcdd7-16ca-40c3-9a41-feed4d5f7161\") " pod="openshift-ingress/e2e-api-b-c60a0d-openshift-default-6464975799-g9tvr"
Nov 02 19:58:34.152434 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:34.152412    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/618bcdd7-16ca-40c3-9a41-feed4d5f7161-istiod-ca-cert\") pod \"e2e-api-b-c60a0d-openshift-default-6464975799-g9tvr\" (UID: \"618bcdd7-16ca-40c3-9a41-feed4d5f7161\") " pod="openshift-ingress/e2e-api-b-c60a0d-openshift-default-6464975799-g9tvr"
Nov 02 19:58:34.153897 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:34.153879    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/618bcdd7-16ca-40c3-9a41-feed4d5f7161-istio-podinfo\") pod \"e2e-api-b-c60a0d-openshift-default-6464975799-g9tvr\" (UID: \"618bcdd7-16ca-40c3-9a41-feed4d5f7161\") " pod="openshift-ingress/e2e-api-b-c60a0d-openshift-default-6464975799-g9tvr"
Nov 02 19:58:34.154066 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:34.154049    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/618bcdd7-16ca-40c3-9a41-feed4d5f7161-istio-envoy\") pod \"e2e-api-b-c60a0d-openshift-default-6464975799-g9tvr\" (UID: \"618bcdd7-16ca-40c3-9a41-feed4d5f7161\") " pod="openshift-ingress/e2e-api-b-c60a0d-openshift-default-6464975799-g9tvr"
Nov 02 19:58:34.204400 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:34.204348    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/618bcdd7-16ca-40c3-9a41-feed4d5f7161-istio-token\") pod \"e2e-api-b-c60a0d-openshift-default-6464975799-g9tvr\" (UID: \"618bcdd7-16ca-40c3-9a41-feed4d5f7161\") " pod="openshift-ingress/e2e-api-b-c60a0d-openshift-default-6464975799-g9tvr"
Nov 02 19:58:34.204547 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:34.204448    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zj5k\" (UniqueName: \"kubernetes.io/projected/618bcdd7-16ca-40c3-9a41-feed4d5f7161-kube-api-access-8zj5k\") pod \"e2e-api-b-c60a0d-openshift-default-6464975799-g9tvr\" (UID: \"618bcdd7-16ca-40c3-9a41-feed4d5f7161\") " pod="openshift-ingress/e2e-api-b-c60a0d-openshift-default-6464975799-g9tvr"
Nov 02 19:58:34.299122 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:58:34.299068    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 19:58:34.494363 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:34.494332    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c60a0d-openshift-default-6464975799-g9tvr"
Nov 02 19:58:34.926312 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:34.926283    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-c60a0d-openshift-default-6464975799-g9tvr"]
Nov 02 19:58:34.926428 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:58:34.926334    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod618bcdd7_16ca_40c3_9a41_feed4d5f7161.slice/crio-4708f7fbf0f67d0c675a8f43ada375205811bfd8c4096de071ec614d56908890 WatchSource:0}: Error finding container 4708f7fbf0f67d0c675a8f43ada375205811bfd8c4096de071ec614d56908890: Status 404 returned error can't find the container with id 4708f7fbf0f67d0c675a8f43ada375205811bfd8c4096de071ec614d56908890
Nov 02 19:58:34.929035 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:34.928988    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:58:34.929323 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:34.929068    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:58:34.929323 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:34.929099    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:58:35.090773 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:35.090728    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c60a0d-openshift-default-6464975799-g9tvr" event={"ID":"618bcdd7-16ca-40c3-9a41-feed4d5f7161","Type":"ContainerStarted","Data":"5479f80afbc79dbc92d7fdcbb032f1942e61a73148b8910c12f9330b98b69962"}
Nov 02 19:58:35.090952 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:35.090795    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c60a0d-openshift-default-6464975799-g9tvr" event={"ID":"618bcdd7-16ca-40c3-9a41-feed4d5f7161","Type":"ContainerStarted","Data":"4708f7fbf0f67d0c675a8f43ada375205811bfd8c4096de071ec614d56908890"}
Nov 02 19:58:35.204825 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:35.204734    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-c60a0d-openshift-default-6464975799-g9tvr" podStartSLOduration=2.20471986 podStartE2EDuration="2.20471986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:58:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:58:35.201043044 +0000 UTC m=+1821.808076813" watchObservedRunningTime="2026-11-02 19:58:35.20471986 +0000 UTC m=+1821.811753628"
Nov 02 19:58:35.495554 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:35.495470    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-c60a0d-openshift-default-6464975799-g9tvr"
Nov 02 19:58:35.501154 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:35.501130    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-c60a0d-openshift-default-6464975799-g9tvr"
Nov 02 19:58:36.098444 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:36.098415    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-c60a0d-openshift-default-6464975799-g9tvr"
Nov 02 19:58:36.099315 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:36.099292    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-c60a0d-openshift-default-6464975799-g9tvr"
Nov 02 19:58:39.752042 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:39.751107    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d869e/premium-w5-1d869e-kserve-7745799ff8-pv2th" podUID="1d784b10-99e4-4561-bfb7-6614877d2b6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:58:39.772035 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:39.755497    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d869e/distinct-2-w5-1d869e-kserve-574f55db46-4cv47" podUID="7e236108-328b-4170-be20-13f16b56c4fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:58:39.848195 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:39.842102    4463 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5c46af/distinct-w3-5c46af-kserve-68b6dff7f5-bp59f" podUID="8dd6362e-7dbd-489b-b6d8-0d9caab0fa69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:58:39.878034 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:39.877094    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-26cfbe/distinct-w1-26cfbe-kserve-685985bfb-tkgn2" podUID="7415aa62-467c-43af-94c8-d604f57ec1c3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:58:39.878034 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:39.877973    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-43ceaa/premium-w0-43ceaa-kserve-67b55659b7-rckfd" podUID="7be32df1-5049-4f79-be23-a827f3f0a85f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:58:39.879033 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:39.878338    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8795bf/distinct-w4-8795bf-kserve-64b4796df4-8qgqh" podUID="ee7d28f4-e9c7-43e2-b983-1427589f1785" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Nov 02 19:58:40.075519 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:40.075096    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-26cfbe/simulator-w1-26cfbe-kserve-74c5578fbd-t79hq" podUID="50dc76f4-aebd-41f4-9f23-799ddc8c70f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:58:40.137037 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:40.136312    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8795bf/distinct-2-w4-8795bf-kserve-69c8b6b885-2chhn" podUID="9f430131-4938-47ca-ba86-b6a236f8cc3b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:58:40.744036 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:40.450829    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ffdc4d-openshift-default-6995c74bfc-mt2f2"]
Nov 02 19:58:40.840037 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:40.718084    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5c46af/distinct-w3-5c46af-kserve-68b6dff7f5-bp59f" podUID="8dd6362e-7dbd-489b-b6d8-0d9caab0fa69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:58:41.018091 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:41.017519    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ffdc4d-openshift-default-6995c74bfc-mt2f2"
Nov 02 19:58:41.132068 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:41.126628    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ffdc4d-openshift-default-6995c74bfc-mt2f2"]
Nov 02 19:58:41.146030 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:41.142648    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d869e/premium-w5-1d869e-kserve-7745799ff8-pv2th" podUID="1d784b10-99e4-4561-bfb7-6614877d2b6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 02 19:58:41.185456 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:41.169425    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-ffdc4d-openshift-default-dockercfg-dvx98\""
Nov 02 19:58:41.185456 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:41.172288    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-1d869e/premium-w5-1d869e-kserve-7745799ff8-pv2th"
Nov 02 19:58:41.189398 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:41.188262    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-43ceaa/premium-w0-43ceaa-kserve-67b55659b7-rckfd" podUID="7be32df1-5049-4f79-be23-a827f3f0a85f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:58:41.189398 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:41.188934    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-43ceaa/premium-w0-43ceaa-kserve-67b55659b7-rckfd"
Nov 02 19:58:41.292090 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:41.291563    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5ffe43f-1a80-4126-9cf1-251fdde99de1-istio-envoy\") pod \"e2e-shared-b-w3-ffdc4d-openshift-default-6995c74bfc-mt2f2\" (UID: \"d5ffe43f-1a80-4126-9cf1-251fdde99de1\") " pod="openshift-ingress/e2e-shared-b-w3-ffdc4d-openshift-default-6995c74bfc-mt2f2"
Nov 02 19:58:41.292090 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:41.291619    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5ffe43f-1a80-4126-9cf1-251fdde99de1-istio-podinfo\") pod \"e2e-shared-b-w3-ffdc4d-openshift-default-6995c74bfc-mt2f2\" (UID: \"d5ffe43f-1a80-4126-9cf1-251fdde99de1\") " pod="openshift-ingress/e2e-shared-b-w3-ffdc4d-openshift-default-6995c74bfc-mt2f2"
Nov 02 19:58:41.292090 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:41.291680    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5ffe43f-1a80-4126-9cf1-251fdde99de1-credential-socket\") pod \"e2e-shared-b-w3-ffdc4d-openshift-default-6995c74bfc-mt2f2\" (UID: \"d5ffe43f-1a80-4126-9cf1-251fdde99de1\") " pod="openshift-ingress/e2e-shared-b-w3-ffdc4d-openshift-default-6995c74bfc-mt2f2"
Nov 02 19:58:41.292090 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:41.291722    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5ffe43f-1a80-4126-9cf1-251fdde99de1-istiod-ca-cert\") pod \"e2e-shared-b-w3-ffdc4d-openshift-default-6995c74bfc-mt2f2\" (UID: \"d5ffe43f-1a80-4126-9cf1-251fdde99de1\") " pod="openshift-ingress/e2e-shared-b-w3-ffdc4d-openshift-default-6995c74bfc-mt2f2"
Nov 02 19:58:41.292090 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:41.291792    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5ffe43f-1a80-4126-9cf1-251fdde99de1-workload-certs\") pod \"e2e-shared-b-w3-ffdc4d-openshift-default-6995c74bfc-mt2f2\" (UID: \"d5ffe43f-1a80-4126-9cf1-251fdde99de1\") " pod="openshift-ingress/e2e-shared-b-w3-ffdc4d-openshift-default-6995c74bfc-mt2f2"
Nov 02 19:58:41.292090 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:41.291839    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5ffe43f-1a80-4126-9cf1-251fdde99de1-workload-socket\") pod \"e2e-shared-b-w3-ffdc4d-openshift-default-6995c74bfc-mt2f2\" (UID: \"d5ffe43f-1a80-4126-9cf1-251fdde99de1\") " pod="openshift-ingress/e2e-shared-b-w3-ffdc4d-openshift-default-6995c74bfc-mt2f2"
Nov 02 19:58:41.292090 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:41.291903    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5ffe43f-1a80-4126-9cf1-251fdde99de1-istio-token\") pod \"e2e-shared-b-w3-ffdc4d-openshift-default-6995c74bfc-mt2f2\" (UID: \"d5ffe43f-1a80-4126-9cf1-251fdde99de1\") " pod="openshift-ingress/e2e-shared-b-w3-ffdc4d-openshift-default-6995c74bfc-mt2f2"
Nov 02 19:58:41.292090 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:41.291964    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5ffe43f-1a80-4126-9cf1-251fdde99de1-istio-data\") pod \"e2e-shared-b-w3-ffdc4d-openshift-default-6995c74bfc-mt2f2\" (UID: \"d5ffe43f-1a80-4126-9cf1-251fdde99de1\") " pod="openshift-ingress/e2e-shared-b-w3-ffdc4d-openshift-default-6995c74bfc-mt2f2"
Nov 02 19:58:41.292090 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:41.292065    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hj7kt\" (UniqueName: \"kubernetes.io/projected/d5ffe43f-1a80-4126-9cf1-251fdde99de1-kube-api-access-hj7kt\") pod \"e2e-shared-b-w3-ffdc4d-openshift-default-6995c74bfc-mt2f2\" (UID: \"d5ffe43f-1a80-4126-9cf1-251fdde99de1\") " pod="openshift-ingress/e2e-shared-b-w3-ffdc4d-openshift-default-6995c74bfc-mt2f2"
Nov 02 19:58:41.406766 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:41.400984    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5ffe43f-1a80-4126-9cf1-251fdde99de1-istio-envoy\") pod \"e2e-shared-b-w3-ffdc4d-openshift-default-6995c74bfc-mt2f2\" (UID: \"d5ffe43f-1a80-4126-9cf1-251fdde99de1\") " pod="openshift-ingress/e2e-shared-b-w3-ffdc4d-openshift-default-6995c74bfc-mt2f2"
Nov 02 19:58:41.406766 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:41.401059    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5ffe43f-1a80-4126-9cf1-251fdde99de1-istio-podinfo\") pod \"e2e-shared-b-w3-ffdc4d-openshift-default-6995c74bfc-mt2f2\" (UID: \"d5ffe43f-1a80-4126-9cf1-251fdde99de1\") " pod="openshift-ingress/e2e-shared-b-w3-ffdc4d-openshift-default-6995c74bfc-mt2f2"
Nov 02 19:58:41.406766 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:41.401168    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5ffe43f-1a80-4126-9cf1-251fdde99de1-credential-socket\") pod \"e2e-shared-b-w3-ffdc4d-openshift-default-6995c74bfc-mt2f2\" (UID: \"d5ffe43f-1a80-4126-9cf1-251fdde99de1\") " pod="openshift-ingress/e2e-shared-b-w3-ffdc4d-openshift-default-6995c74bfc-mt2f2"
Nov 02 19:58:41.406766 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:41.401228    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5ffe43f-1a80-4126-9cf1-251fdde99de1-istiod-ca-cert\") pod \"e2e-shared-b-w3-ffdc4d-openshift-default-6995c74bfc-mt2f2\" (UID: \"d5ffe43f-1a80-4126-9cf1-251fdde99de1\") " pod="openshift-ingress/e2e-shared-b-w3-ffdc4d-openshift-default-6995c74bfc-mt2f2"
Nov 02 19:58:41.406766 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:41.401318    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5ffe43f-1a80-4126-9cf1-251fdde99de1-workload-certs\") pod \"e2e-shared-b-w3-ffdc4d-openshift-default-6995c74bfc-mt2f2\" (UID: \"d5ffe43f-1a80-4126-9cf1-251fdde99de1\") " pod="openshift-ingress/e2e-shared-b-w3-ffdc4d-openshift-default-6995c74bfc-mt2f2"
Nov 02 19:58:41.406766 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:41.401354    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5ffe43f-1a80-4126-9cf1-251fdde99de1-workload-socket\") pod \"e2e-shared-b-w3-ffdc4d-openshift-default-6995c74bfc-mt2f2\" (UID: \"d5ffe43f-1a80-4126-9cf1-251fdde99de1\") " pod="openshift-ingress/e2e-shared-b-w3-ffdc4d-openshift-default-6995c74bfc-mt2f2"
Nov 02 19:58:41.406766 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:41.401441    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5ffe43f-1a80-4126-9cf1-251fdde99de1-istio-token\") pod \"e2e-shared-b-w3-ffdc4d-openshift-default-6995c74bfc-mt2f2\" (UID: \"d5ffe43f-1a80-4126-9cf1-251fdde99de1\") " pod="openshift-ingress/e2e-shared-b-w3-ffdc4d-openshift-default-6995c74bfc-mt2f2"
Nov 02 19:58:41.406766 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:41.401501    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5ffe43f-1a80-4126-9cf1-251fdde99de1-istio-data\") pod \"e2e-shared-b-w3-ffdc4d-openshift-default-6995c74bfc-mt2f2\" (UID: \"d5ffe43f-1a80-4126-9cf1-251fdde99de1\") " pod="openshift-ingress/e2e-shared-b-w3-ffdc4d-openshift-default-6995c74bfc-mt2f2"
Nov 02 19:58:41.406766 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:41.401618    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hj7kt\" (UniqueName: \"kubernetes.io/projected/d5ffe43f-1a80-4126-9cf1-251fdde99de1-kube-api-access-hj7kt\") pod \"e2e-shared-b-w3-ffdc4d-openshift-default-6995c74bfc-mt2f2\" (UID: \"d5ffe43f-1a80-4126-9cf1-251fdde99de1\") " pod="openshift-ingress/e2e-shared-b-w3-ffdc4d-openshift-default-6995c74bfc-mt2f2"
Nov 02 19:58:41.406766 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:41.406395    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5ffe43f-1a80-4126-9cf1-251fdde99de1-workload-certs\") pod \"e2e-shared-b-w3-ffdc4d-openshift-default-6995c74bfc-mt2f2\" (UID: \"d5ffe43f-1a80-4126-9cf1-251fdde99de1\") " pod="openshift-ingress/e2e-shared-b-w3-ffdc4d-openshift-default-6995c74bfc-mt2f2"
Nov 02 19:58:41.416380 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:41.409225    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5ffe43f-1a80-4126-9cf1-251fdde99de1-istio-data\") pod \"e2e-shared-b-w3-ffdc4d-openshift-default-6995c74bfc-mt2f2\" (UID: \"d5ffe43f-1a80-4126-9cf1-251fdde99de1\") " pod="openshift-ingress/e2e-shared-b-w3-ffdc4d-openshift-default-6995c74bfc-mt2f2"
Nov 02 19:58:41.416380 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:41.409472    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5ffe43f-1a80-4126-9cf1-251fdde99de1-workload-socket\") pod \"e2e-shared-b-w3-ffdc4d-openshift-default-6995c74bfc-mt2f2\" (UID: \"d5ffe43f-1a80-4126-9cf1-251fdde99de1\") " pod="openshift-ingress/e2e-shared-b-w3-ffdc4d-openshift-default-6995c74bfc-mt2f2"
Nov 02 19:58:41.416380 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:41.410092    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5ffe43f-1a80-4126-9cf1-251fdde99de1-credential-socket\") pod \"e2e-shared-b-w3-ffdc4d-openshift-default-6995c74bfc-mt2f2\" (UID: \"d5ffe43f-1a80-4126-9cf1-251fdde99de1\") " pod="openshift-ingress/e2e-shared-b-w3-ffdc4d-openshift-default-6995c74bfc-mt2f2"
Nov 02 19:58:41.416380 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:41.410513    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-1d869e/premium-w5-1d869e-kserve-7745799ff8-pv2th"
Nov 02 19:58:41.416380 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:41.410978    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5ffe43f-1a80-4126-9cf1-251fdde99de1-istiod-ca-cert\") pod \"e2e-shared-b-w3-ffdc4d-openshift-default-6995c74bfc-mt2f2\" (UID: \"d5ffe43f-1a80-4126-9cf1-251fdde99de1\") " pod="openshift-ingress/e2e-shared-b-w3-ffdc4d-openshift-default-6995c74bfc-mt2f2"
Nov 02 19:58:41.418594 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:41.418558    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5ffe43f-1a80-4126-9cf1-251fdde99de1-istio-envoy\") pod \"e2e-shared-b-w3-ffdc4d-openshift-default-6995c74bfc-mt2f2\" (UID: \"d5ffe43f-1a80-4126-9cf1-251fdde99de1\") " pod="openshift-ingress/e2e-shared-b-w3-ffdc4d-openshift-default-6995c74bfc-mt2f2"
Nov 02 19:58:41.442106 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:41.442081    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5ffe43f-1a80-4126-9cf1-251fdde99de1-istio-podinfo\") pod \"e2e-shared-b-w3-ffdc4d-openshift-default-6995c74bfc-mt2f2\" (UID: \"d5ffe43f-1a80-4126-9cf1-251fdde99de1\") " pod="openshift-ingress/e2e-shared-b-w3-ffdc4d-openshift-default-6995c74bfc-mt2f2"
Nov 02 19:58:41.446199 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:41.446178    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-43ceaa/premium-w0-43ceaa-kserve-67b55659b7-rckfd"
Nov 02 19:58:41.513196 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:41.511629    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5ffe43f-1a80-4126-9cf1-251fdde99de1-istio-token\") pod \"e2e-shared-b-w3-ffdc4d-openshift-default-6995c74bfc-mt2f2\" (UID: \"d5ffe43f-1a80-4126-9cf1-251fdde99de1\") " pod="openshift-ingress/e2e-shared-b-w3-ffdc4d-openshift-default-6995c74bfc-mt2f2"
Nov 02 19:58:41.516816 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:41.516706    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hj7kt\" (UniqueName: \"kubernetes.io/projected/d5ffe43f-1a80-4126-9cf1-251fdde99de1-kube-api-access-hj7kt\") pod \"e2e-shared-b-w3-ffdc4d-openshift-default-6995c74bfc-mt2f2\" (UID: \"d5ffe43f-1a80-4126-9cf1-251fdde99de1\") " pod="openshift-ingress/e2e-shared-b-w3-ffdc4d-openshift-default-6995c74bfc-mt2f2"
Nov 02 19:58:41.815413 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:41.815377    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ffdc4d-openshift-default-6995c74bfc-mt2f2"
Nov 02 19:58:42.277786 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:42.277717    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ffdc4d-openshift-default-6995c74bfc-mt2f2"]
Nov 02 19:58:42.278371 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:58:42.278343    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5ffe43f_1a80_4126_9cf1_251fdde99de1.slice/crio-2140b4dd69f01ef7bdc2dfdf6af35b9ef88a44c9f97bdacc41f146bf78f20d92 WatchSource:0}: Error finding container 2140b4dd69f01ef7bdc2dfdf6af35b9ef88a44c9f97bdacc41f146bf78f20d92: Status 404 returned error can't find the container with id 2140b4dd69f01ef7bdc2dfdf6af35b9ef88a44c9f97bdacc41f146bf78f20d92
Nov 02 19:58:42.281276 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:42.281240    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:58:42.281398 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:42.281317    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:58:42.281398 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:42.281362    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:58:42.332001 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:42.331974    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ffdc4d-openshift-default-6995c74bfc-mt2f2" event={"ID":"d5ffe43f-1a80-4126-9cf1-251fdde99de1","Type":"ContainerStarted","Data":"2140b4dd69f01ef7bdc2dfdf6af35b9ef88a44c9f97bdacc41f146bf78f20d92"}
Nov 02 19:58:43.340762 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:43.340721    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ffdc4d-openshift-default-6995c74bfc-mt2f2" event={"ID":"d5ffe43f-1a80-4126-9cf1-251fdde99de1","Type":"ContainerStarted","Data":"ef65e7eca4ab2cbb96b24402b81accd94d6be216c8f7eb7166b44e3d7db88333"}
Nov 02 19:58:43.497311 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:43.497254    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-ffdc4d-openshift-default-6995c74bfc-mt2f2" podStartSLOduration=3.497239812 podStartE2EDuration="3.497239812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:58:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:58:43.494356075 +0000 UTC m=+1830.101389863" watchObservedRunningTime="2026-11-02 19:58:43.497239812 +0000 UTC m=+1830.104273581"
Nov 02 19:58:43.815669 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:43.815635    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-ffdc4d-openshift-default-6995c74bfc-mt2f2"
Nov 02 19:58:43.821121 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:43.821096    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-ffdc4d-openshift-default-6995c74bfc-mt2f2"
Nov 02 19:58:44.347657 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:44.347621    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-ffdc4d-openshift-default-6995c74bfc-mt2f2"
Nov 02 19:58:44.348611 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:44.348586    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-ffdc4d-openshift-default-6995c74bfc-mt2f2"
Nov 02 19:58:47.078151 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:47.078108    4463 patch_prober.go:29] interesting pod/e2e-shared-b-w6-24ef8b-openshift-default-685d675f47-76xb9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 19:58:47.078545 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:47.078162    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w6-24ef8b-openshift-default-685d675f47-76xb9" podUID="b2c0f86f-3e9e-4996-8710-664c445e9113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:58:48.381689 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:48.381610    4463 generic.go:356] "Generic (PLEG): container finished" podID="b2c0f86f-3e9e-4996-8710-664c445e9113" containerID="36d39bfe6697d6d7172d4e0ead0d7f1b422687a8126f90f98969a12155c00d60" exitCode=0
Nov 02 19:58:48.381689 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:48.381655    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-24ef8b-openshift-default-685d675f47-76xb9" event={"ID":"b2c0f86f-3e9e-4996-8710-664c445e9113","Type":"ContainerDied","Data":"36d39bfe6697d6d7172d4e0ead0d7f1b422687a8126f90f98969a12155c00d60"}
Nov 02 19:58:48.382264 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:48.382142    4463 scope.go:122] "RemoveContainer" containerID="36d39bfe6697d6d7172d4e0ead0d7f1b422687a8126f90f98969a12155c00d60"
Nov 02 19:58:48.383678 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:48.383645    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:58:48.383784 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:48.383715    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:58:48.383784 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:48.383747    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:58:49.389985 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:49.389951    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-24ef8b-openshift-default-685d675f47-76xb9" event={"ID":"b2c0f86f-3e9e-4996-8710-664c445e9113","Type":"ContainerStarted","Data":"cf9f5f253bbb7e992b020a6d5848a9abe04be961c682b3b028fe46c434642e3c"}
Nov 02 19:58:49.601164 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:49.601132    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-24ef8b-openshift-default-685d675f47-76xb9"
Nov 02 19:58:49.601334 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:49.601272    4463 patch_prober.go:29] interesting pod/e2e-shared-b-w6-24ef8b-openshift-default-685d675f47-76xb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Nov 02 19:58:49.601334 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:49.601313    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-24ef8b-openshift-default-685d675f47-76xb9" podUID="b2c0f86f-3e9e-4996-8710-664c445e9113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Nov 02 19:58:50.600955 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:50.600921    4463 patch_prober.go:29] interesting pod/e2e-shared-b-w6-24ef8b-openshift-default-685d675f47-76xb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Nov 02 19:58:50.601546 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:50.601505    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-24ef8b-openshift-default-685d675f47-76xb9" podUID="b2c0f86f-3e9e-4996-8710-664c445e9113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Nov 02 19:58:51.601548 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:51.601518    4463 patch_prober.go:29] interesting pod/e2e-shared-b-w6-24ef8b-openshift-default-685d675f47-76xb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Nov 02 19:58:51.601942 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:51.601567    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-24ef8b-openshift-default-685d675f47-76xb9" podUID="b2c0f86f-3e9e-4996-8710-664c445e9113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Nov 02 19:58:52.601491 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:52.601457    4463 patch_prober.go:29] interesting pod/e2e-shared-b-w6-24ef8b-openshift-default-685d675f47-76xb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Nov 02 19:58:52.601742 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:52.601507    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-24ef8b-openshift-default-685d675f47-76xb9" podUID="b2c0f86f-3e9e-4996-8710-664c445e9113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Nov 02 19:58:53.601836 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:53.601802    4463 patch_prober.go:29] interesting pod/e2e-shared-b-w6-24ef8b-openshift-default-685d675f47-76xb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Nov 02 19:58:53.602240 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:53.601858    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-24ef8b-openshift-default-685d675f47-76xb9" podUID="b2c0f86f-3e9e-4996-8710-664c445e9113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Nov 02 19:58:54.624312 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:54.624279    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-24ef8b-openshift-default-685d675f47-76xb9"
Nov 02 19:58:54.626424 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:54.626401    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-24ef8b-openshift-default-685d675f47-76xb9"
Nov 02 19:58:54.641825 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:54.641113    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-24ef8b-openshift-default-685d675f47-76xb9"
Nov 02 19:58:59.706293 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:58:59.706105    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 19:59:01.691880 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:01.691846    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-4b0a12-openshift-default-8677ff4fb5-6css2"]
Nov 02 19:59:01.695874 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:01.695854    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4b0a12-openshift-default-8677ff4fb5-6css2"
Nov 02 19:59:01.707942 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:01.707918    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/277fb120-e642-4215-bc3d-1b9c08c57e81-istio-token\") pod \"e2e-inf-b-4b0a12-openshift-default-8677ff4fb5-6css2\" (UID: \"277fb120-e642-4215-bc3d-1b9c08c57e81\") " pod="openshift-ingress/e2e-inf-b-4b0a12-openshift-default-8677ff4fb5-6css2"
Nov 02 19:59:01.708092 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:01.707987    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/277fb120-e642-4215-bc3d-1b9c08c57e81-istio-envoy\") pod \"e2e-inf-b-4b0a12-openshift-default-8677ff4fb5-6css2\" (UID: \"277fb120-e642-4215-bc3d-1b9c08c57e81\") " pod="openshift-ingress/e2e-inf-b-4b0a12-openshift-default-8677ff4fb5-6css2"
Nov 02 19:59:01.708205 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:01.708183    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/277fb120-e642-4215-bc3d-1b9c08c57e81-credential-socket\") pod \"e2e-inf-b-4b0a12-openshift-default-8677ff4fb5-6css2\" (UID: \"277fb120-e642-4215-bc3d-1b9c08c57e81\") " pod="openshift-ingress/e2e-inf-b-4b0a12-openshift-default-8677ff4fb5-6css2"
Nov 02 19:59:01.708299 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:01.708285    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/277fb120-e642-4215-bc3d-1b9c08c57e81-workload-socket\") pod \"e2e-inf-b-4b0a12-openshift-default-8677ff4fb5-6css2\" (UID: \"277fb120-e642-4215-bc3d-1b9c08c57e81\") " pod="openshift-ingress/e2e-inf-b-4b0a12-openshift-default-8677ff4fb5-6css2"
Nov 02 19:59:01.708379 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:01.708315    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/277fb120-e642-4215-bc3d-1b9c08c57e81-istio-data\") pod \"e2e-inf-b-4b0a12-openshift-default-8677ff4fb5-6css2\" (UID: \"277fb120-e642-4215-bc3d-1b9c08c57e81\") " pod="openshift-ingress/e2e-inf-b-4b0a12-openshift-default-8677ff4fb5-6css2"
Nov 02 19:59:01.708379 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:01.708343    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/277fb120-e642-4215-bc3d-1b9c08c57e81-istiod-ca-cert\") pod \"e2e-inf-b-4b0a12-openshift-default-8677ff4fb5-6css2\" (UID: \"277fb120-e642-4215-bc3d-1b9c08c57e81\") " pod="openshift-ingress/e2e-inf-b-4b0a12-openshift-default-8677ff4fb5-6css2"
Nov 02 19:59:01.708497 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:01.708477    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/277fb120-e642-4215-bc3d-1b9c08c57e81-istio-podinfo\") pod \"e2e-inf-b-4b0a12-openshift-default-8677ff4fb5-6css2\" (UID: \"277fb120-e642-4215-bc3d-1b9c08c57e81\") " pod="openshift-ingress/e2e-inf-b-4b0a12-openshift-default-8677ff4fb5-6css2"
Nov 02 19:59:01.708554 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:01.708520    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/277fb120-e642-4215-bc3d-1b9c08c57e81-workload-certs\") pod \"e2e-inf-b-4b0a12-openshift-default-8677ff4fb5-6css2\" (UID: \"277fb120-e642-4215-bc3d-1b9c08c57e81\") " pod="openshift-ingress/e2e-inf-b-4b0a12-openshift-default-8677ff4fb5-6css2"
Nov 02 19:59:01.708663 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:01.708647    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqqkj\" (UniqueName: \"kubernetes.io/projected/277fb120-e642-4215-bc3d-1b9c08c57e81-kube-api-access-rqqkj\") pod \"e2e-inf-b-4b0a12-openshift-default-8677ff4fb5-6css2\" (UID: \"277fb120-e642-4215-bc3d-1b9c08c57e81\") " pod="openshift-ingress/e2e-inf-b-4b0a12-openshift-default-8677ff4fb5-6css2"
Nov 02 19:59:01.748044 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:01.746290    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-4b0a12-openshift-default-dockercfg-rcpk2\""
Nov 02 19:59:01.853223 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:01.847177    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/277fb120-e642-4215-bc3d-1b9c08c57e81-istio-podinfo\") pod \"e2e-inf-b-4b0a12-openshift-default-8677ff4fb5-6css2\" (UID: \"277fb120-e642-4215-bc3d-1b9c08c57e81\") " pod="openshift-ingress/e2e-inf-b-4b0a12-openshift-default-8677ff4fb5-6css2"
Nov 02 19:59:01.853223 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:01.847249    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/277fb120-e642-4215-bc3d-1b9c08c57e81-workload-certs\") pod \"e2e-inf-b-4b0a12-openshift-default-8677ff4fb5-6css2\" (UID: \"277fb120-e642-4215-bc3d-1b9c08c57e81\") " pod="openshift-ingress/e2e-inf-b-4b0a12-openshift-default-8677ff4fb5-6css2"
Nov 02 19:59:01.853223 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:01.847329    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqqkj\" (UniqueName: \"kubernetes.io/projected/277fb120-e642-4215-bc3d-1b9c08c57e81-kube-api-access-rqqkj\") pod \"e2e-inf-b-4b0a12-openshift-default-8677ff4fb5-6css2\" (UID: \"277fb120-e642-4215-bc3d-1b9c08c57e81\") " pod="openshift-ingress/e2e-inf-b-4b0a12-openshift-default-8677ff4fb5-6css2"
Nov 02 19:59:01.853223 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:01.847440    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/277fb120-e642-4215-bc3d-1b9c08c57e81-istio-token\") pod \"e2e-inf-b-4b0a12-openshift-default-8677ff4fb5-6css2\" (UID: \"277fb120-e642-4215-bc3d-1b9c08c57e81\") " pod="openshift-ingress/e2e-inf-b-4b0a12-openshift-default-8677ff4fb5-6css2"
Nov 02 19:59:01.853223 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:01.847515    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/277fb120-e642-4215-bc3d-1b9c08c57e81-istio-envoy\") pod \"e2e-inf-b-4b0a12-openshift-default-8677ff4fb5-6css2\" (UID: \"277fb120-e642-4215-bc3d-1b9c08c57e81\") " pod="openshift-ingress/e2e-inf-b-4b0a12-openshift-default-8677ff4fb5-6css2"
Nov 02 19:59:01.853223 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:01.847602    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/277fb120-e642-4215-bc3d-1b9c08c57e81-credential-socket\") pod \"e2e-inf-b-4b0a12-openshift-default-8677ff4fb5-6css2\" (UID: \"277fb120-e642-4215-bc3d-1b9c08c57e81\") " pod="openshift-ingress/e2e-inf-b-4b0a12-openshift-default-8677ff4fb5-6css2"
Nov 02 19:59:01.853223 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:01.847682    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/277fb120-e642-4215-bc3d-1b9c08c57e81-workload-socket\") pod \"e2e-inf-b-4b0a12-openshift-default-8677ff4fb5-6css2\" (UID: \"277fb120-e642-4215-bc3d-1b9c08c57e81\") " pod="openshift-ingress/e2e-inf-b-4b0a12-openshift-default-8677ff4fb5-6css2"
Nov 02 19:59:01.853223 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:01.847717    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/277fb120-e642-4215-bc3d-1b9c08c57e81-istio-data\") pod \"e2e-inf-b-4b0a12-openshift-default-8677ff4fb5-6css2\" (UID: \"277fb120-e642-4215-bc3d-1b9c08c57e81\") " pod="openshift-ingress/e2e-inf-b-4b0a12-openshift-default-8677ff4fb5-6css2"
Nov 02 19:59:01.853223 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:01.847749    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/277fb120-e642-4215-bc3d-1b9c08c57e81-istiod-ca-cert\") pod \"e2e-inf-b-4b0a12-openshift-default-8677ff4fb5-6css2\" (UID: \"277fb120-e642-4215-bc3d-1b9c08c57e81\") " pod="openshift-ingress/e2e-inf-b-4b0a12-openshift-default-8677ff4fb5-6css2"
Nov 02 19:59:01.853223 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:01.848702    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/277fb120-e642-4215-bc3d-1b9c08c57e81-istiod-ca-cert\") pod \"e2e-inf-b-4b0a12-openshift-default-8677ff4fb5-6css2\" (UID: \"277fb120-e642-4215-bc3d-1b9c08c57e81\") " pod="openshift-ingress/e2e-inf-b-4b0a12-openshift-default-8677ff4fb5-6css2"
Nov 02 19:59:01.878753 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:01.878392    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/277fb120-e642-4215-bc3d-1b9c08c57e81-istio-podinfo\") pod \"e2e-inf-b-4b0a12-openshift-default-8677ff4fb5-6css2\" (UID: \"277fb120-e642-4215-bc3d-1b9c08c57e81\") " pod="openshift-ingress/e2e-inf-b-4b0a12-openshift-default-8677ff4fb5-6css2"
Nov 02 19:59:01.879292 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:01.879270    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/277fb120-e642-4215-bc3d-1b9c08c57e81-workload-certs\") pod \"e2e-inf-b-4b0a12-openshift-default-8677ff4fb5-6css2\" (UID: \"277fb120-e642-4215-bc3d-1b9c08c57e81\") " pod="openshift-ingress/e2e-inf-b-4b0a12-openshift-default-8677ff4fb5-6css2"
Nov 02 19:59:01.932033 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:01.922921    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/277fb120-e642-4215-bc3d-1b9c08c57e81-workload-socket\") pod \"e2e-inf-b-4b0a12-openshift-default-8677ff4fb5-6css2\" (UID: \"277fb120-e642-4215-bc3d-1b9c08c57e81\") " pod="openshift-ingress/e2e-inf-b-4b0a12-openshift-default-8677ff4fb5-6css2"
Nov 02 19:59:01.932033 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:01.923899    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/277fb120-e642-4215-bc3d-1b9c08c57e81-credential-socket\") pod \"e2e-inf-b-4b0a12-openshift-default-8677ff4fb5-6css2\" (UID: \"277fb120-e642-4215-bc3d-1b9c08c57e81\") " pod="openshift-ingress/e2e-inf-b-4b0a12-openshift-default-8677ff4fb5-6css2"
Nov 02 19:59:01.932033 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:01.924128    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/277fb120-e642-4215-bc3d-1b9c08c57e81-istio-data\") pod \"e2e-inf-b-4b0a12-openshift-default-8677ff4fb5-6css2\" (UID: \"277fb120-e642-4215-bc3d-1b9c08c57e81\") " pod="openshift-ingress/e2e-inf-b-4b0a12-openshift-default-8677ff4fb5-6css2"
Nov 02 19:59:01.932033 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:01.929349    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/277fb120-e642-4215-bc3d-1b9c08c57e81-istio-envoy\") pod \"e2e-inf-b-4b0a12-openshift-default-8677ff4fb5-6css2\" (UID: \"277fb120-e642-4215-bc3d-1b9c08c57e81\") " pod="openshift-ingress/e2e-inf-b-4b0a12-openshift-default-8677ff4fb5-6css2"
Nov 02 19:59:01.935145 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:01.935050    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-4b0a12-openshift-default-8677ff4fb5-6css2"]
Nov 02 19:59:01.948092 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:01.946424    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/277fb120-e642-4215-bc3d-1b9c08c57e81-istio-token\") pod \"e2e-inf-b-4b0a12-openshift-default-8677ff4fb5-6css2\" (UID: \"277fb120-e642-4215-bc3d-1b9c08c57e81\") " pod="openshift-ingress/e2e-inf-b-4b0a12-openshift-default-8677ff4fb5-6css2"
Nov 02 19:59:01.948092 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:01.947678    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqqkj\" (UniqueName: \"kubernetes.io/projected/277fb120-e642-4215-bc3d-1b9c08c57e81-kube-api-access-rqqkj\") pod \"e2e-inf-b-4b0a12-openshift-default-8677ff4fb5-6css2\" (UID: \"277fb120-e642-4215-bc3d-1b9c08c57e81\") " pod="openshift-ingress/e2e-inf-b-4b0a12-openshift-default-8677ff4fb5-6css2"
Nov 02 19:59:02.008753 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:02.008711    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4b0a12-openshift-default-8677ff4fb5-6css2"
Nov 02 19:59:02.445861 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:02.445825    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-4b0a12-openshift-default-8677ff4fb5-6css2"]
Nov 02 19:59:02.446000 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:59:02.445885    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod277fb120_e642_4215_bc3d_1b9c08c57e81.slice/crio-5a3144b31cf58ab5df4104e6f83ea28974dd85ebef97eab70d2f77250079bfe4 WatchSource:0}: Error finding container 5a3144b31cf58ab5df4104e6f83ea28974dd85ebef97eab70d2f77250079bfe4: Status 404 returned error can't find the container with id 5a3144b31cf58ab5df4104e6f83ea28974dd85ebef97eab70d2f77250079bfe4
Nov 02 19:59:02.448845 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:02.448814    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:59:02.448939 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:02.448881    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:59:02.448939 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:02.448911    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:59:02.480626 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:02.480598    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4b0a12-openshift-default-8677ff4fb5-6css2" event={"ID":"277fb120-e642-4215-bc3d-1b9c08c57e81","Type":"ContainerStarted","Data":"5a3144b31cf58ab5df4104e6f83ea28974dd85ebef97eab70d2f77250079bfe4"}
Nov 02 19:59:03.495842 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:03.495804    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4b0a12-openshift-default-8677ff4fb5-6css2" event={"ID":"277fb120-e642-4215-bc3d-1b9c08c57e81","Type":"ContainerStarted","Data":"985dd2c51fbde5438cae0f74b3691c5bca1361a42cb033dc5bc62ed5c386ae25"}
Nov 02 19:59:03.651609 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:03.651562    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-4b0a12-openshift-default-8677ff4fb5-6css2" podStartSLOduration=2.651550068 podStartE2EDuration="2.651550068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:59:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:59:03.650097973 +0000 UTC m=+1850.257131783" watchObservedRunningTime="2026-11-02 19:59:03.651550068 +0000 UTC m=+1850.258583836"
Nov 02 19:59:04.009919 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:04.009885    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-4b0a12-openshift-default-8677ff4fb5-6css2"
Nov 02 19:59:04.015616 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:04.015591    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-4b0a12-openshift-default-8677ff4fb5-6css2"
Nov 02 19:59:04.295323 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:59:04.295244    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 19:59:04.503785 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:04.503556    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-4b0a12-openshift-default-8677ff4fb5-6css2"
Nov 02 19:59:04.504670 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:04.504643    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-4b0a12-openshift-default-8677ff4fb5-6css2"
Nov 02 19:59:34.295322 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:59:34.295284    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 19:59:41.975932 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:41.975895    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-23a9f5-798d467749-kxrt9"]
Nov 02 19:59:41.980561 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:41.980542    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-23a9f5-798d467749-kxrt9"
Nov 02 19:59:42.032457 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:42.032426    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-23a9f5-798d467749-kxrt9"]
Nov 02 19:59:42.074987 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:42.074956    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-23a9f5-serving-cert\""
Nov 02 19:59:42.106732 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:42.106697    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/254c92da-c945-42b8-931b-791c60405784-openshift-service-ca\") pod \"maas-api-e2e-api-a-23a9f5-798d467749-kxrt9\" (UID: \"254c92da-c945-42b8-931b-791c60405784\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-23a9f5-798d467749-kxrt9"
Nov 02 19:59:42.107295 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:42.107110    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/254c92da-c945-42b8-931b-791c60405784-maas-api-tls\") pod \"maas-api-e2e-api-a-23a9f5-798d467749-kxrt9\" (UID: \"254c92da-c945-42b8-931b-791c60405784\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-23a9f5-798d467749-kxrt9"
Nov 02 19:59:42.107441 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:42.107423    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knq2m\" (UniqueName: \"kubernetes.io/projected/254c92da-c945-42b8-931b-791c60405784-kube-api-access-knq2m\") pod \"maas-api-e2e-api-a-23a9f5-798d467749-kxrt9\" (UID: \"254c92da-c945-42b8-931b-791c60405784\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-23a9f5-798d467749-kxrt9"
Nov 02 19:59:42.107560 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:42.107545    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/254c92da-c945-42b8-931b-791c60405784-metrics-certs\") pod \"maas-api-e2e-api-a-23a9f5-798d467749-kxrt9\" (UID: \"254c92da-c945-42b8-931b-791c60405784\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-23a9f5-798d467749-kxrt9"
Nov 02 19:59:42.209843 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:42.209808    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/254c92da-c945-42b8-931b-791c60405784-maas-api-tls\") pod \"maas-api-e2e-api-a-23a9f5-798d467749-kxrt9\" (UID: \"254c92da-c945-42b8-931b-791c60405784\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-23a9f5-798d467749-kxrt9"
Nov 02 19:59:42.210067 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:42.209853    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knq2m\" (UniqueName: \"kubernetes.io/projected/254c92da-c945-42b8-931b-791c60405784-kube-api-access-knq2m\") pod \"maas-api-e2e-api-a-23a9f5-798d467749-kxrt9\" (UID: \"254c92da-c945-42b8-931b-791c60405784\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-23a9f5-798d467749-kxrt9"
Nov 02 19:59:42.210067 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:42.209880    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/254c92da-c945-42b8-931b-791c60405784-metrics-certs\") pod \"maas-api-e2e-api-a-23a9f5-798d467749-kxrt9\" (UID: \"254c92da-c945-42b8-931b-791c60405784\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-23a9f5-798d467749-kxrt9"
Nov 02 19:59:42.210194 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:59:42.210126    4463 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-23a9f5-serving-cert: secret "maas-api-e2e-api-a-23a9f5-serving-cert" not found
Nov 02 19:59:42.210245 ip-10-0-133-9 kubenswrapper[4463]: E1102 19:59:42.210195    4463 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/254c92da-c945-42b8-931b-791c60405784-maas-api-tls podName:254c92da-c945-42b8-931b-791c60405784 nodeName:}" failed. No retries permitted until 2026-11-02 19:59:42.710174003 +0000 UTC m=+1889.317207764 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/254c92da-c945-42b8-931b-791c60405784-maas-api-tls") pod "maas-api-e2e-api-a-23a9f5-798d467749-kxrt9" (UID: "254c92da-c945-42b8-931b-791c60405784") : secret "maas-api-e2e-api-a-23a9f5-serving-cert" not found
Nov 02 19:59:42.210684 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:42.210656    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/254c92da-c945-42b8-931b-791c60405784-openshift-service-ca\") pod \"maas-api-e2e-api-a-23a9f5-798d467749-kxrt9\" (UID: \"254c92da-c945-42b8-931b-791c60405784\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-23a9f5-798d467749-kxrt9"
Nov 02 19:59:42.211532 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:42.211506    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/254c92da-c945-42b8-931b-791c60405784-openshift-service-ca\") pod \"maas-api-e2e-api-a-23a9f5-798d467749-kxrt9\" (UID: \"254c92da-c945-42b8-931b-791c60405784\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-23a9f5-798d467749-kxrt9"
Nov 02 19:59:42.219662 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:42.219632    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/254c92da-c945-42b8-931b-791c60405784-metrics-certs\") pod \"maas-api-e2e-api-a-23a9f5-798d467749-kxrt9\" (UID: \"254c92da-c945-42b8-931b-791c60405784\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-23a9f5-798d467749-kxrt9"
Nov 02 19:59:42.273082 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:42.272426    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knq2m\" (UniqueName: \"kubernetes.io/projected/254c92da-c945-42b8-931b-791c60405784-kube-api-access-knq2m\") pod \"maas-api-e2e-api-a-23a9f5-798d467749-kxrt9\" (UID: \"254c92da-c945-42b8-931b-791c60405784\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-23a9f5-798d467749-kxrt9"
Nov 02 19:59:42.718986 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:42.718951    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/254c92da-c945-42b8-931b-791c60405784-maas-api-tls\") pod \"maas-api-e2e-api-a-23a9f5-798d467749-kxrt9\" (UID: \"254c92da-c945-42b8-931b-791c60405784\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-23a9f5-798d467749-kxrt9"
Nov 02 19:59:42.721466 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:42.721434    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/254c92da-c945-42b8-931b-791c60405784-maas-api-tls\") pod \"maas-api-e2e-api-a-23a9f5-798d467749-kxrt9\" (UID: \"254c92da-c945-42b8-931b-791c60405784\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-23a9f5-798d467749-kxrt9"
Nov 02 19:59:42.894439 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:42.894407    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-23a9f5-798d467749-kxrt9"
Nov 02 19:59:43.286598 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:43.286564    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-23a9f5-798d467749-kxrt9"]
Nov 02 19:59:43.287522 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:59:43.287494    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod254c92da_c945_42b8_931b_791c60405784.slice/crio-db8fcaf400e6cc245d65beaf5057ac98227a7482a8f2d60031867f9ad16c8886 WatchSource:0}: Error finding container db8fcaf400e6cc245d65beaf5057ac98227a7482a8f2d60031867f9ad16c8886: Status 404 returned error can't find the container with id db8fcaf400e6cc245d65beaf5057ac98227a7482a8f2d60031867f9ad16c8886
Nov 02 19:59:43.800392 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:43.800333    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-23a9f5-798d467749-kxrt9" event={"ID":"254c92da-c945-42b8-931b-791c60405784","Type":"ContainerStarted","Data":"db8fcaf400e6cc245d65beaf5057ac98227a7482a8f2d60031867f9ad16c8886"}
Nov 02 19:59:44.808745 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:44.808706    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-23a9f5-798d467749-kxrt9" event={"ID":"254c92da-c945-42b8-931b-791c60405784","Type":"ContainerStarted","Data":"ca1718832bf61951c9f35356a699b0b98184c520e414d2485f686408a576b53f"}
Nov 02 19:59:44.809258 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:44.808800    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-23a9f5-798d467749-kxrt9"
Nov 02 19:59:49.485084 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:49.485004    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-23a9f5-798d467749-kxrt9" podStartSLOduration=7.552821226 podStartE2EDuration="8.484985603s" totalImagesPullingTime="932.164377ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:59:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:59:44.935819311 +0000 UTC m=+1891.542853108" watchObservedRunningTime="2026-11-02 19:59:49.484985603 +0000 UTC m=+1896.092019372"
Nov 02 19:59:49.486227 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:49.486198    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-9f8f23-openshift-default-5dd8c74c6f-n75dv"]
Nov 02 19:59:49.493433 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:49.493409    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9f8f23-openshift-default-5dd8c74c6f-n75dv"
Nov 02 19:59:49.563719 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:49.563672    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-9f8f23-openshift-default-5dd8c74c6f-n75dv"]
Nov 02 19:59:49.565591 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:49.565564    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e054dacd-930b-430d-8719-3e22a12ce72f-istio-envoy\") pod \"e2e-api-a-9f8f23-openshift-default-5dd8c74c6f-n75dv\" (UID: \"e054dacd-930b-430d-8719-3e22a12ce72f\") " pod="openshift-ingress/e2e-api-a-9f8f23-openshift-default-5dd8c74c6f-n75dv"
Nov 02 19:59:49.565749 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:49.565612    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e054dacd-930b-430d-8719-3e22a12ce72f-istiod-ca-cert\") pod \"e2e-api-a-9f8f23-openshift-default-5dd8c74c6f-n75dv\" (UID: \"e054dacd-930b-430d-8719-3e22a12ce72f\") " pod="openshift-ingress/e2e-api-a-9f8f23-openshift-default-5dd8c74c6f-n75dv"
Nov 02 19:59:49.565749 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:49.565633    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e054dacd-930b-430d-8719-3e22a12ce72f-istio-data\") pod \"e2e-api-a-9f8f23-openshift-default-5dd8c74c6f-n75dv\" (UID: \"e054dacd-930b-430d-8719-3e22a12ce72f\") " pod="openshift-ingress/e2e-api-a-9f8f23-openshift-default-5dd8c74c6f-n75dv"
Nov 02 19:59:49.565749 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:49.565716    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e054dacd-930b-430d-8719-3e22a12ce72f-istio-token\") pod \"e2e-api-a-9f8f23-openshift-default-5dd8c74c6f-n75dv\" (UID: \"e054dacd-930b-430d-8719-3e22a12ce72f\") " pod="openshift-ingress/e2e-api-a-9f8f23-openshift-default-5dd8c74c6f-n75dv"
Nov 02 19:59:49.565901 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:49.565757    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e054dacd-930b-430d-8719-3e22a12ce72f-credential-socket\") pod \"e2e-api-a-9f8f23-openshift-default-5dd8c74c6f-n75dv\" (UID: \"e054dacd-930b-430d-8719-3e22a12ce72f\") " pod="openshift-ingress/e2e-api-a-9f8f23-openshift-default-5dd8c74c6f-n75dv"
Nov 02 19:59:49.565901 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:49.565818    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e054dacd-930b-430d-8719-3e22a12ce72f-workload-certs\") pod \"e2e-api-a-9f8f23-openshift-default-5dd8c74c6f-n75dv\" (UID: \"e054dacd-930b-430d-8719-3e22a12ce72f\") " pod="openshift-ingress/e2e-api-a-9f8f23-openshift-default-5dd8c74c6f-n75dv"
Nov 02 19:59:49.565901 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:49.565867    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e054dacd-930b-430d-8719-3e22a12ce72f-istio-podinfo\") pod \"e2e-api-a-9f8f23-openshift-default-5dd8c74c6f-n75dv\" (UID: \"e054dacd-930b-430d-8719-3e22a12ce72f\") " pod="openshift-ingress/e2e-api-a-9f8f23-openshift-default-5dd8c74c6f-n75dv"
Nov 02 19:59:49.565901 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:49.565892    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e054dacd-930b-430d-8719-3e22a12ce72f-workload-socket\") pod \"e2e-api-a-9f8f23-openshift-default-5dd8c74c6f-n75dv\" (UID: \"e054dacd-930b-430d-8719-3e22a12ce72f\") " pod="openshift-ingress/e2e-api-a-9f8f23-openshift-default-5dd8c74c6f-n75dv"
Nov 02 19:59:49.566136 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:49.565926    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gzvk\" (UniqueName: \"kubernetes.io/projected/e054dacd-930b-430d-8719-3e22a12ce72f-kube-api-access-2gzvk\") pod \"e2e-api-a-9f8f23-openshift-default-5dd8c74c6f-n75dv\" (UID: \"e054dacd-930b-430d-8719-3e22a12ce72f\") " pod="openshift-ingress/e2e-api-a-9f8f23-openshift-default-5dd8c74c6f-n75dv"
Nov 02 19:59:49.584418 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:49.584390    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-9f8f23-openshift-default-dockercfg-zk7xd\""
Nov 02 19:59:49.667228 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:49.667178    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e054dacd-930b-430d-8719-3e22a12ce72f-istio-token\") pod \"e2e-api-a-9f8f23-openshift-default-5dd8c74c6f-n75dv\" (UID: \"e054dacd-930b-430d-8719-3e22a12ce72f\") " pod="openshift-ingress/e2e-api-a-9f8f23-openshift-default-5dd8c74c6f-n75dv"
Nov 02 19:59:49.667425 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:49.667250    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e054dacd-930b-430d-8719-3e22a12ce72f-credential-socket\") pod \"e2e-api-a-9f8f23-openshift-default-5dd8c74c6f-n75dv\" (UID: \"e054dacd-930b-430d-8719-3e22a12ce72f\") " pod="openshift-ingress/e2e-api-a-9f8f23-openshift-default-5dd8c74c6f-n75dv"
Nov 02 19:59:49.667425 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:49.667373    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e054dacd-930b-430d-8719-3e22a12ce72f-workload-certs\") pod \"e2e-api-a-9f8f23-openshift-default-5dd8c74c6f-n75dv\" (UID: \"e054dacd-930b-430d-8719-3e22a12ce72f\") " pod="openshift-ingress/e2e-api-a-9f8f23-openshift-default-5dd8c74c6f-n75dv"
Nov 02 19:59:49.667545 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:49.667460    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e054dacd-930b-430d-8719-3e22a12ce72f-istio-podinfo\") pod \"e2e-api-a-9f8f23-openshift-default-5dd8c74c6f-n75dv\" (UID: \"e054dacd-930b-430d-8719-3e22a12ce72f\") " pod="openshift-ingress/e2e-api-a-9f8f23-openshift-default-5dd8c74c6f-n75dv"
Nov 02 19:59:49.667545 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:49.667488    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e054dacd-930b-430d-8719-3e22a12ce72f-workload-socket\") pod \"e2e-api-a-9f8f23-openshift-default-5dd8c74c6f-n75dv\" (UID: \"e054dacd-930b-430d-8719-3e22a12ce72f\") " pod="openshift-ingress/e2e-api-a-9f8f23-openshift-default-5dd8c74c6f-n75dv"
Nov 02 19:59:49.667656 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:49.667608    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gzvk\" (UniqueName: \"kubernetes.io/projected/e054dacd-930b-430d-8719-3e22a12ce72f-kube-api-access-2gzvk\") pod \"e2e-api-a-9f8f23-openshift-default-5dd8c74c6f-n75dv\" (UID: \"e054dacd-930b-430d-8719-3e22a12ce72f\") " pod="openshift-ingress/e2e-api-a-9f8f23-openshift-default-5dd8c74c6f-n75dv"
Nov 02 19:59:49.667713 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:49.667699    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e054dacd-930b-430d-8719-3e22a12ce72f-istio-envoy\") pod \"e2e-api-a-9f8f23-openshift-default-5dd8c74c6f-n75dv\" (UID: \"e054dacd-930b-430d-8719-3e22a12ce72f\") " pod="openshift-ingress/e2e-api-a-9f8f23-openshift-default-5dd8c74c6f-n75dv"
Nov 02 19:59:49.667767 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:49.667747    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e054dacd-930b-430d-8719-3e22a12ce72f-istiod-ca-cert\") pod \"e2e-api-a-9f8f23-openshift-default-5dd8c74c6f-n75dv\" (UID: \"e054dacd-930b-430d-8719-3e22a12ce72f\") " pod="openshift-ingress/e2e-api-a-9f8f23-openshift-default-5dd8c74c6f-n75dv"
Nov 02 19:59:49.667821 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:49.667776    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e054dacd-930b-430d-8719-3e22a12ce72f-istio-data\") pod \"e2e-api-a-9f8f23-openshift-default-5dd8c74c6f-n75dv\" (UID: \"e054dacd-930b-430d-8719-3e22a12ce72f\") " pod="openshift-ingress/e2e-api-a-9f8f23-openshift-default-5dd8c74c6f-n75dv"
Nov 02 19:59:49.667821 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:49.667783    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e054dacd-930b-430d-8719-3e22a12ce72f-workload-socket\") pod \"e2e-api-a-9f8f23-openshift-default-5dd8c74c6f-n75dv\" (UID: \"e054dacd-930b-430d-8719-3e22a12ce72f\") " pod="openshift-ingress/e2e-api-a-9f8f23-openshift-default-5dd8c74c6f-n75dv"
Nov 02 19:59:49.668806 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:49.668778    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e054dacd-930b-430d-8719-3e22a12ce72f-credential-socket\") pod \"e2e-api-a-9f8f23-openshift-default-5dd8c74c6f-n75dv\" (UID: \"e054dacd-930b-430d-8719-3e22a12ce72f\") " pod="openshift-ingress/e2e-api-a-9f8f23-openshift-default-5dd8c74c6f-n75dv"
Nov 02 19:59:49.668967 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:49.668827    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e054dacd-930b-430d-8719-3e22a12ce72f-workload-certs\") pod \"e2e-api-a-9f8f23-openshift-default-5dd8c74c6f-n75dv\" (UID: \"e054dacd-930b-430d-8719-3e22a12ce72f\") " pod="openshift-ingress/e2e-api-a-9f8f23-openshift-default-5dd8c74c6f-n75dv"
Nov 02 19:59:49.669078 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:49.669061    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e054dacd-930b-430d-8719-3e22a12ce72f-istio-data\") pod \"e2e-api-a-9f8f23-openshift-default-5dd8c74c6f-n75dv\" (UID: \"e054dacd-930b-430d-8719-3e22a12ce72f\") " pod="openshift-ingress/e2e-api-a-9f8f23-openshift-default-5dd8c74c6f-n75dv"
Nov 02 19:59:49.671085 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:49.669223    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e054dacd-930b-430d-8719-3e22a12ce72f-istiod-ca-cert\") pod \"e2e-api-a-9f8f23-openshift-default-5dd8c74c6f-n75dv\" (UID: \"e054dacd-930b-430d-8719-3e22a12ce72f\") " pod="openshift-ingress/e2e-api-a-9f8f23-openshift-default-5dd8c74c6f-n75dv"
Nov 02 19:59:49.672617 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:49.672595    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e054dacd-930b-430d-8719-3e22a12ce72f-istio-envoy\") pod \"e2e-api-a-9f8f23-openshift-default-5dd8c74c6f-n75dv\" (UID: \"e054dacd-930b-430d-8719-3e22a12ce72f\") " pod="openshift-ingress/e2e-api-a-9f8f23-openshift-default-5dd8c74c6f-n75dv"
Nov 02 19:59:49.685813 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:49.685780    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e054dacd-930b-430d-8719-3e22a12ce72f-istio-podinfo\") pod \"e2e-api-a-9f8f23-openshift-default-5dd8c74c6f-n75dv\" (UID: \"e054dacd-930b-430d-8719-3e22a12ce72f\") " pod="openshift-ingress/e2e-api-a-9f8f23-openshift-default-5dd8c74c6f-n75dv"
Nov 02 19:59:49.721745 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:49.721713    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e054dacd-930b-430d-8719-3e22a12ce72f-istio-token\") pod \"e2e-api-a-9f8f23-openshift-default-5dd8c74c6f-n75dv\" (UID: \"e054dacd-930b-430d-8719-3e22a12ce72f\") " pod="openshift-ingress/e2e-api-a-9f8f23-openshift-default-5dd8c74c6f-n75dv"
Nov 02 19:59:49.722082 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:49.722046    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gzvk\" (UniqueName: \"kubernetes.io/projected/e054dacd-930b-430d-8719-3e22a12ce72f-kube-api-access-2gzvk\") pod \"e2e-api-a-9f8f23-openshift-default-5dd8c74c6f-n75dv\" (UID: \"e054dacd-930b-430d-8719-3e22a12ce72f\") " pod="openshift-ingress/e2e-api-a-9f8f23-openshift-default-5dd8c74c6f-n75dv"
Nov 02 19:59:49.808548 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:49.808450    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9f8f23-openshift-default-5dd8c74c6f-n75dv"
Nov 02 19:59:49.846128 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:49.846093    4463 generic.go:356] "Generic (PLEG): container finished" podID="2f18cb37-937a-4cc9-9a20-f44f2f693d52" containerID="d43aa87648a2daa10355da0054488ebb075beb0699084bcd11eb0888142a0db5" exitCode=0
Nov 02 19:59:49.846303 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:49.846173    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8479ff-openshift-default-6bb884bd67-wnm6b" event={"ID":"2f18cb37-937a-4cc9-9a20-f44f2f693d52","Type":"ContainerDied","Data":"d43aa87648a2daa10355da0054488ebb075beb0699084bcd11eb0888142a0db5"}
Nov 02 19:59:49.846708 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:49.846683    4463 scope.go:122] "RemoveContainer" containerID="d43aa87648a2daa10355da0054488ebb075beb0699084bcd11eb0888142a0db5"
Nov 02 19:59:49.849294 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:49.849255    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:59:49.849402 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:49.849361    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:59:49.849448 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:49.849418    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:59:50.241356 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:50.241322    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-9f8f23-openshift-default-5dd8c74c6f-n75dv"]
Nov 02 19:59:50.241506 ip-10-0-133-9 kubenswrapper[4463]: W1102 19:59:50.241479    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode054dacd_930b_430d_8719_3e22a12ce72f.slice/crio-40e0bcda198591038254056420805466c3e479d08246882cb7bd8537bc189c5f WatchSource:0}: Error finding container 40e0bcda198591038254056420805466c3e479d08246882cb7bd8537bc189c5f: Status 404 returned error can't find the container with id 40e0bcda198591038254056420805466c3e479d08246882cb7bd8537bc189c5f
Nov 02 19:59:50.244766 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:50.244732    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:59:50.244885 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:50.244791    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:59:50.244885 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:50.244818    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:59:50.677246 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:50.677210    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-8479ff-openshift-default-6bb884bd67-wnm6b"
Nov 02 19:59:50.820831 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:50.820804    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-23a9f5-798d467749-kxrt9"
Nov 02 19:59:50.877422 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:50.877379    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9f8f23-openshift-default-5dd8c74c6f-n75dv" event={"ID":"e054dacd-930b-430d-8719-3e22a12ce72f","Type":"ContainerStarted","Data":"6a0ec9645fe5619f3279385f6b436203289097e22f54c65f82cefec790674ba9"}
Nov 02 19:59:50.877648 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:50.877447    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9f8f23-openshift-default-5dd8c74c6f-n75dv" event={"ID":"e054dacd-930b-430d-8719-3e22a12ce72f","Type":"ContainerStarted","Data":"40e0bcda198591038254056420805466c3e479d08246882cb7bd8537bc189c5f"}
Nov 02 19:59:50.879483 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:50.879448    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8479ff-openshift-default-6bb884bd67-wnm6b" event={"ID":"2f18cb37-937a-4cc9-9a20-f44f2f693d52","Type":"ContainerStarted","Data":"85b8be1ea4a46ccdb495ac05bb1eeb498f9506f7b9c3241f4085a200ed733f5e"}
Nov 02 19:59:51.045787 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:51.045647    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-9f8f23-openshift-default-5dd8c74c6f-n75dv" podStartSLOduration=2.045630992 podStartE2EDuration="2.045630992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:59:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:59:51.041036541 +0000 UTC m=+1897.648070311" watchObservedRunningTime="2026-11-02 19:59:51.045630992 +0000 UTC m=+1897.652664760"
Nov 02 19:59:51.613467 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:51.613433    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-8479ff-openshift-default-6bb884bd67-wnm6b"
Nov 02 19:59:51.614743 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:51.614718    4463 patch_prober.go:29] interesting pod/e2e-inf-a-8479ff-openshift-default-6bb884bd67-wnm6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Nov 02 19:59:51.614831 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:51.614782    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8479ff-openshift-default-6bb884bd67-wnm6b" podUID="2f18cb37-937a-4cc9-9a20-f44f2f693d52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Nov 02 19:59:51.808772 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:51.808734    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-9f8f23-openshift-default-5dd8c74c6f-n75dv"
Nov 02 19:59:51.814891 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:51.814867    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-9f8f23-openshift-default-5dd8c74c6f-n75dv"
Nov 02 19:59:51.887212 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:51.887140    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-9f8f23-openshift-default-5dd8c74c6f-n75dv"
Nov 02 19:59:51.887844 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:51.887825    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-9f8f23-openshift-default-5dd8c74c6f-n75dv"
Nov 02 19:59:52.613213 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:52.613180    4463 patch_prober.go:29] interesting pod/e2e-inf-a-8479ff-openshift-default-6bb884bd67-wnm6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Nov 02 19:59:52.613424 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:52.613232    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8479ff-openshift-default-6bb884bd67-wnm6b" podUID="2f18cb37-937a-4cc9-9a20-f44f2f693d52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Nov 02 19:59:53.613787 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:53.613756    4463 patch_prober.go:29] interesting pod/e2e-inf-a-8479ff-openshift-default-6bb884bd67-wnm6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Nov 02 19:59:53.614174 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:53.613806    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8479ff-openshift-default-6bb884bd67-wnm6b" podUID="2f18cb37-937a-4cc9-9a20-f44f2f693d52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Nov 02 19:59:54.613178 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:54.613147    4463 patch_prober.go:29] interesting pod/e2e-inf-a-8479ff-openshift-default-6bb884bd67-wnm6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Nov 02 19:59:54.613379 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:54.613198    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8479ff-openshift-default-6bb884bd67-wnm6b" podUID="2f18cb37-937a-4cc9-9a20-f44f2f693d52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Nov 02 19:59:55.613280 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:55.613248    4463 patch_prober.go:29] interesting pod/e2e-inf-a-8479ff-openshift-default-6bb884bd67-wnm6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Nov 02 19:59:55.613712 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:55.613295    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8479ff-openshift-default-6bb884bd67-wnm6b" podUID="2f18cb37-937a-4cc9-9a20-f44f2f693d52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Nov 02 19:59:56.661515 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:56.661485    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-8479ff-openshift-default-6bb884bd67-wnm6b"
Nov 02 19:59:56.663131 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:56.663103    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-8479ff-openshift-default-6bb884bd67-wnm6b"
Nov 02 19:59:56.664789 ip-10-0-133-9 kubenswrapper[4463]: I1102 19:59:56.664739    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-8479ff-openshift-default-6bb884bd67-wnm6b"
Nov 02 20:00:00.210746 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:00.210711    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-28623b-29894160-cspqn"]
Nov 02 20:00:00.215429 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:00.215412    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-28623b-29894160-cspqn"
Nov 02 20:00:00.226472 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:00.226447    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5c46af-29894160-cw6qx"]
Nov 02 20:00:00.230748 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:00.230728    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5c46af-29894160-cw6qx"
Nov 02 20:00:00.232821 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:00.232799    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894160-k9hjx"]
Nov 02 20:00:00.237799 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:00.237781    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894160-k9hjx"
Nov 02 20:00:00.316805 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:00.316768    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlmb2\" (UniqueName: \"kubernetes.io/projected/5a3f250a-3664-4b0b-b07e-0690f1ad231c-kube-api-access-tlmb2\") pod \"maas-api-key-cleanup-e2e-worker-w3-5c46af-29894160-cw6qx\" (UID: \"5a3f250a-3664-4b0b-b07e-0690f1ad231c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5c46af-29894160-cw6qx"
Nov 02 20:00:00.317002 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:00.316825    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a3f250a-3664-4b0b-b07e-0690f1ad231c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-5c46af-29894160-cw6qx\" (UID: \"5a3f250a-3664-4b0b-b07e-0690f1ad231c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5c46af-29894160-cw6qx"
Nov 02 20:00:00.317002 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:00.316856    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/027cb607-a331-4bc2-86b7-39c85b92860a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-28623b-29894160-cspqn\" (UID: \"027cb607-a331-4bc2-86b7-39c85b92860a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-28623b-29894160-cspqn"
Nov 02 20:00:00.317188 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:00.317003    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6wsw\" (UniqueName: \"kubernetes.io/projected/027cb607-a331-4bc2-86b7-39c85b92860a-kube-api-access-v6wsw\") pod \"maas-api-key-cleanup-e2e-worker-w2-28623b-29894160-cspqn\" (UID: \"027cb607-a331-4bc2-86b7-39c85b92860a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-28623b-29894160-cspqn"
Nov 02 20:00:00.317250 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:00.317224    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzr62\" (UniqueName: \"kubernetes.io/projected/5cddfa53-4cf6-49fd-b883-c50e95ff138a-kube-api-access-nzr62\") pod \"maas-api-key-cleanup-29894160-k9hjx\" (UID: \"5cddfa53-4cf6-49fd-b883-c50e95ff138a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894160-k9hjx"
Nov 02 20:00:00.317410 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:00.317392    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cddfa53-4cf6-49fd-b883-c50e95ff138a-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894160-k9hjx\" (UID: \"5cddfa53-4cf6-49fd-b883-c50e95ff138a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894160-k9hjx"
Nov 02 20:00:00.323095 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:00.323068    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\""
Nov 02 20:00:00.349710 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:00.349679    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894160-k9hjx"]
Nov 02 20:00:00.418714 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:00.418675    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cddfa53-4cf6-49fd-b883-c50e95ff138a-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894160-k9hjx\" (UID: \"5cddfa53-4cf6-49fd-b883-c50e95ff138a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894160-k9hjx"
Nov 02 20:00:00.418891 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:00.418776    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlmb2\" (UniqueName: \"kubernetes.io/projected/5a3f250a-3664-4b0b-b07e-0690f1ad231c-kube-api-access-tlmb2\") pod \"maas-api-key-cleanup-e2e-worker-w3-5c46af-29894160-cw6qx\" (UID: \"5a3f250a-3664-4b0b-b07e-0690f1ad231c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5c46af-29894160-cw6qx"
Nov 02 20:00:00.418891 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:00.418830    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a3f250a-3664-4b0b-b07e-0690f1ad231c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-5c46af-29894160-cw6qx\" (UID: \"5a3f250a-3664-4b0b-b07e-0690f1ad231c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5c46af-29894160-cw6qx"
Nov 02 20:00:00.419242 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:00.419176    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/027cb607-a331-4bc2-86b7-39c85b92860a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-28623b-29894160-cspqn\" (UID: \"027cb607-a331-4bc2-86b7-39c85b92860a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-28623b-29894160-cspqn"
Nov 02 20:00:00.419242 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:00.419237    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6wsw\" (UniqueName: \"kubernetes.io/projected/027cb607-a331-4bc2-86b7-39c85b92860a-kube-api-access-v6wsw\") pod \"maas-api-key-cleanup-e2e-worker-w2-28623b-29894160-cspqn\" (UID: \"027cb607-a331-4bc2-86b7-39c85b92860a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-28623b-29894160-cspqn"
Nov 02 20:00:00.419427 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:00.419380    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzr62\" (UniqueName: \"kubernetes.io/projected/5cddfa53-4cf6-49fd-b883-c50e95ff138a-kube-api-access-nzr62\") pod \"maas-api-key-cleanup-29894160-k9hjx\" (UID: \"5cddfa53-4cf6-49fd-b883-c50e95ff138a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894160-k9hjx"
Nov 02 20:00:00.419512 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:00.419491    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cddfa53-4cf6-49fd-b883-c50e95ff138a-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894160-k9hjx\" (UID: \"5cddfa53-4cf6-49fd-b883-c50e95ff138a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894160-k9hjx"
Nov 02 20:00:00.419629 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:00.419606    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a3f250a-3664-4b0b-b07e-0690f1ad231c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-5c46af-29894160-cw6qx\" (UID: \"5a3f250a-3664-4b0b-b07e-0690f1ad231c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5c46af-29894160-cw6qx"
Nov 02 20:00:00.420127 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:00.420106    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/027cb607-a331-4bc2-86b7-39c85b92860a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-28623b-29894160-cspqn\" (UID: \"027cb607-a331-4bc2-86b7-39c85b92860a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-28623b-29894160-cspqn"
Nov 02 20:00:00.433075 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:00.433050    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-28623b-29894160-cspqn"]
Nov 02 20:00:00.474434 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:00.474364    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlmb2\" (UniqueName: \"kubernetes.io/projected/5a3f250a-3664-4b0b-b07e-0690f1ad231c-kube-api-access-tlmb2\") pod \"maas-api-key-cleanup-e2e-worker-w3-5c46af-29894160-cw6qx\" (UID: \"5a3f250a-3664-4b0b-b07e-0690f1ad231c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5c46af-29894160-cw6qx"
Nov 02 20:00:00.476337 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:00.476306    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzr62\" (UniqueName: \"kubernetes.io/projected/5cddfa53-4cf6-49fd-b883-c50e95ff138a-kube-api-access-nzr62\") pod \"maas-api-key-cleanup-29894160-k9hjx\" (UID: \"5cddfa53-4cf6-49fd-b883-c50e95ff138a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894160-k9hjx"
Nov 02 20:00:00.480007 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:00.479977    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6wsw\" (UniqueName: \"kubernetes.io/projected/027cb607-a331-4bc2-86b7-39c85b92860a-kube-api-access-v6wsw\") pod \"maas-api-key-cleanup-e2e-worker-w2-28623b-29894160-cspqn\" (UID: \"027cb607-a331-4bc2-86b7-39c85b92860a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-28623b-29894160-cspqn"
Nov 02 20:00:00.529861 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:00.529838    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-28623b-29894160-cspqn"
Nov 02 20:00:00.546573 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:00.546549    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5c46af-29894160-cw6qx"
Nov 02 20:00:00.554825 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:00.554801    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5c46af-29894160-cw6qx"]
Nov 02 20:00:00.554950 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:00.554928    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894160-k9hjx"
Nov 02 20:00:00.941896 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:00.941860    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-28623b-29894160-cspqn"]
Nov 02 20:00:00.948915 ip-10-0-133-9 kubenswrapper[4463]: W1102 20:00:00.948883    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod027cb607_a331_4bc2_86b7_39c85b92860a.slice/crio-59bf53a08f424306ec42b563a344c92d48ac9db9cd675c96566c6686091880da WatchSource:0}: Error finding container 59bf53a08f424306ec42b563a344c92d48ac9db9cd675c96566c6686091880da: Status 404 returned error can't find the container with id 59bf53a08f424306ec42b563a344c92d48ac9db9cd675c96566c6686091880da
Nov 02 20:00:00.990826 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:00.990795    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5c46af-29894160-cw6qx"]
Nov 02 20:00:00.991598 ip-10-0-133-9 kubenswrapper[4463]: W1102 20:00:00.991567    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cddfa53_4cf6_49fd_b883_c50e95ff138a.slice/crio-1fe2d83d7ab4af01f9cdaafc0b5608a5b57f1496747b9921900b7bf41b59450d WatchSource:0}: Error finding container 1fe2d83d7ab4af01f9cdaafc0b5608a5b57f1496747b9921900b7bf41b59450d: Status 404 returned error can't find the container with id 1fe2d83d7ab4af01f9cdaafc0b5608a5b57f1496747b9921900b7bf41b59450d
Nov 02 20:00:00.992616 ip-10-0-133-9 kubenswrapper[4463]: W1102 20:00:00.992583    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a3f250a_3664_4b0b_b07e_0690f1ad231c.slice/crio-73d1e0db61d3e61ca65a2191693720fabc11f5c60a58e9110d7f1cf9a803f4ef WatchSource:0}: Error finding container 73d1e0db61d3e61ca65a2191693720fabc11f5c60a58e9110d7f1cf9a803f4ef: Status 404 returned error can't find the container with id 73d1e0db61d3e61ca65a2191693720fabc11f5c60a58e9110d7f1cf9a803f4ef
Nov 02 20:00:00.995843 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:00.995817    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894160-k9hjx"]
Nov 02 20:00:01.970138 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:01.970109    4463 generic.go:356] "Generic (PLEG): container finished" podID="5a3f250a-3664-4b0b-b07e-0690f1ad231c" containerID="10ee939ed96d7b3e3a7d83b83b181e441dc8bb99d82d4522e340b9aa0a69412b" exitCode=60
Nov 02 20:00:01.970529 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:01.970213    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5c46af-29894160-cw6qx" event={"ID":"5a3f250a-3664-4b0b-b07e-0690f1ad231c","Type":"ContainerDied","Data":"10ee939ed96d7b3e3a7d83b83b181e441dc8bb99d82d4522e340b9aa0a69412b"}
Nov 02 20:00:01.970529 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:01.970276    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5c46af-29894160-cw6qx" event={"ID":"5a3f250a-3664-4b0b-b07e-0690f1ad231c","Type":"ContainerStarted","Data":"73d1e0db61d3e61ca65a2191693720fabc11f5c60a58e9110d7f1cf9a803f4ef"}
Nov 02 20:00:01.970529 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:01.970494    4463 scope.go:122] "RemoveContainer" containerID="10ee939ed96d7b3e3a7d83b83b181e441dc8bb99d82d4522e340b9aa0a69412b"
Nov 02 20:00:01.972163 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:01.972131    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894160-k9hjx" event={"ID":"5cddfa53-4cf6-49fd-b883-c50e95ff138a","Type":"ContainerStarted","Data":"589a60b7454dabbd74f387b1d8c93dd9a8754e5594dc01a3b88af96a64f34c62"}
Nov 02 20:00:01.972283 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:01.972175    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894160-k9hjx" event={"ID":"5cddfa53-4cf6-49fd-b883-c50e95ff138a","Type":"ContainerStarted","Data":"1fe2d83d7ab4af01f9cdaafc0b5608a5b57f1496747b9921900b7bf41b59450d"}
Nov 02 20:00:01.972429 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:01.972405    4463 scope.go:122] "RemoveContainer" containerID="589a60b7454dabbd74f387b1d8c93dd9a8754e5594dc01a3b88af96a64f34c62"
Nov 02 20:00:01.973860 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:01.973840    4463 generic.go:356] "Generic (PLEG): container finished" podID="027cb607-a331-4bc2-86b7-39c85b92860a" containerID="519654da8cc8c5c44c250393531d69692884a460053533a7df32b2172f29af4c" exitCode=60
Nov 02 20:00:01.973916 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:01.973909    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-28623b-29894160-cspqn" event={"ID":"027cb607-a331-4bc2-86b7-39c85b92860a","Type":"ContainerDied","Data":"519654da8cc8c5c44c250393531d69692884a460053533a7df32b2172f29af4c"}
Nov 02 20:00:01.973952 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:01.973936    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-28623b-29894160-cspqn" event={"ID":"027cb607-a331-4bc2-86b7-39c85b92860a","Type":"ContainerStarted","Data":"59bf53a08f424306ec42b563a344c92d48ac9db9cd675c96566c6686091880da"}
Nov 02 20:00:01.974200 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:01.974187    4463 scope.go:122] "RemoveContainer" containerID="519654da8cc8c5c44c250393531d69692884a460053533a7df32b2172f29af4c"
Nov 02 20:00:02.982286 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:02.982250    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5c46af-29894160-cw6qx" event={"ID":"5a3f250a-3664-4b0b-b07e-0690f1ad231c","Type":"ContainerStarted","Data":"33ce8ddcdeb348f3ad5effe0b2c3f3a5cd4934b0c4a5b7f2f72b60c69dcabbdc"}
Nov 02 20:00:02.984343 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:02.984313    4463 generic.go:356] "Generic (PLEG): container finished" podID="5cddfa53-4cf6-49fd-b883-c50e95ff138a" containerID="589a60b7454dabbd74f387b1d8c93dd9a8754e5594dc01a3b88af96a64f34c62" exitCode=60
Nov 02 20:00:02.984515 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:02.984421    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894160-k9hjx" event={"ID":"5cddfa53-4cf6-49fd-b883-c50e95ff138a","Type":"ContainerDied","Data":"589a60b7454dabbd74f387b1d8c93dd9a8754e5594dc01a3b88af96a64f34c62"}
Nov 02 20:00:02.984515 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:02.984472    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894160-k9hjx" event={"ID":"5cddfa53-4cf6-49fd-b883-c50e95ff138a","Type":"ContainerStarted","Data":"ed2c2b819855c965fd031d84772ea90171fbab67a996dadc474f8084c2ab2eb1"}
Nov 02 20:00:02.986700 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:02.986678    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-28623b-29894160-cspqn" event={"ID":"027cb607-a331-4bc2-86b7-39c85b92860a","Type":"ContainerStarted","Data":"dab4725ccd7ed36e93d8564b972dbae4a1f3ba8687b5e37c95a08098c774c8b6"}
Nov 02 20:00:03.093461 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:03.093399    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5c46af-29894160-cw6qx" podStartSLOduration=3.093380599 podStartE2EDuration="3.093380599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:00:03.091715972 +0000 UTC m=+1909.698749742" watchObservedRunningTime="2026-11-02 20:00:03.093380599 +0000 UTC m=+1909.700414369"
Nov 02 20:00:03.128727 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:00:03.128698    4463 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cddfa53_4cf6_49fd_b883_c50e95ff138a.slice/crio-ed2c2b819855c965fd031d84772ea90171fbab67a996dadc474f8084c2ab2eb1.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 20:00:03.201365 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:03.201314    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-28623b-29894160-cspqn" podStartSLOduration=3.201302188 podStartE2EDuration="3.201302188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:00:03.197691318 +0000 UTC m=+1909.804725105" watchObservedRunningTime="2026-11-02 20:00:03.201302188 +0000 UTC m=+1909.808335981"
Nov 02 20:00:03.302788 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:03.302741    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894160-k9hjx" podStartSLOduration=3.302726252 podStartE2EDuration="3.302726252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:00:03.301653051 +0000 UTC m=+1909.908686819" watchObservedRunningTime="2026-11-02 20:00:03.302726252 +0000 UTC m=+1909.909760020"
Nov 02 20:00:03.995679 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:03.995646    4463 generic.go:356] "Generic (PLEG): container finished" podID="5a3f250a-3664-4b0b-b07e-0690f1ad231c" containerID="33ce8ddcdeb348f3ad5effe0b2c3f3a5cd4934b0c4a5b7f2f72b60c69dcabbdc" exitCode=60
Nov 02 20:00:03.996130 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:03.995730    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5c46af-29894160-cw6qx" event={"ID":"5a3f250a-3664-4b0b-b07e-0690f1ad231c","Type":"ContainerDied","Data":"33ce8ddcdeb348f3ad5effe0b2c3f3a5cd4934b0c4a5b7f2f72b60c69dcabbdc"}
Nov 02 20:00:03.996130 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:03.995796    4463 scope.go:122] "RemoveContainer" containerID="10ee939ed96d7b3e3a7d83b83b181e441dc8bb99d82d4522e340b9aa0a69412b"
Nov 02 20:00:03.996130 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:03.996085    4463 scope.go:122] "RemoveContainer" containerID="33ce8ddcdeb348f3ad5effe0b2c3f3a5cd4934b0c4a5b7f2f72b60c69dcabbdc"
Nov 02 20:00:03.996480 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:00:03.996455    4463 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-5c46af-29894160-cw6qx_odh-ai-gateway-infra(5a3f250a-3664-4b0b-b07e-0690f1ad231c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5c46af-29894160-cw6qx" podUID="5a3f250a-3664-4b0b-b07e-0690f1ad231c"
Nov 02 20:00:03.997970 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:03.997947    4463 generic.go:356] "Generic (PLEG): container finished" podID="5cddfa53-4cf6-49fd-b883-c50e95ff138a" containerID="ed2c2b819855c965fd031d84772ea90171fbab67a996dadc474f8084c2ab2eb1" exitCode=60
Nov 02 20:00:03.998078 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:03.998054    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894160-k9hjx" event={"ID":"5cddfa53-4cf6-49fd-b883-c50e95ff138a","Type":"ContainerDied","Data":"ed2c2b819855c965fd031d84772ea90171fbab67a996dadc474f8084c2ab2eb1"}
Nov 02 20:00:03.998331 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:03.998315    4463 scope.go:122] "RemoveContainer" containerID="ed2c2b819855c965fd031d84772ea90171fbab67a996dadc474f8084c2ab2eb1"
Nov 02 20:00:03.998719 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:00:03.998670    4463 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894160-k9hjx_odh-ai-gateway-infra(5cddfa53-4cf6-49fd-b883-c50e95ff138a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894160-k9hjx" podUID="5cddfa53-4cf6-49fd-b883-c50e95ff138a"
Nov 02 20:00:03.999886 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:03.999865    4463 generic.go:356] "Generic (PLEG): container finished" podID="027cb607-a331-4bc2-86b7-39c85b92860a" containerID="dab4725ccd7ed36e93d8564b972dbae4a1f3ba8687b5e37c95a08098c774c8b6" exitCode=60
Nov 02 20:00:03.999970 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:03.999949    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-28623b-29894160-cspqn" event={"ID":"027cb607-a331-4bc2-86b7-39c85b92860a","Type":"ContainerDied","Data":"dab4725ccd7ed36e93d8564b972dbae4a1f3ba8687b5e37c95a08098c774c8b6"}
Nov 02 20:00:04.000179 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:04.000166    4463 scope.go:122] "RemoveContainer" containerID="dab4725ccd7ed36e93d8564b972dbae4a1f3ba8687b5e37c95a08098c774c8b6"
Nov 02 20:00:04.000492 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:00:04.000474    4463 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-28623b-29894160-cspqn_odh-ai-gateway-infra(027cb607-a331-4bc2-86b7-39c85b92860a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-28623b-29894160-cspqn" podUID="027cb607-a331-4bc2-86b7-39c85b92860a"
Nov 02 20:00:04.009374 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:04.009350    4463 scope.go:122] "RemoveContainer" containerID="589a60b7454dabbd74f387b1d8c93dd9a8754e5594dc01a3b88af96a64f34c62"
Nov 02 20:00:04.022236 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:04.022216    4463 scope.go:122] "RemoveContainer" containerID="519654da8cc8c5c44c250393531d69692884a460053533a7df32b2172f29af4c"
Nov 02 20:00:04.296217 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:00:04.296136    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 20:00:05.008632 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:05.008608    4463 scope.go:122] "RemoveContainer" containerID="33ce8ddcdeb348f3ad5effe0b2c3f3a5cd4934b0c4a5b7f2f72b60c69dcabbdc"
Nov 02 20:00:05.009032 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:00:05.008976    4463 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-5c46af-29894160-cw6qx_odh-ai-gateway-infra(5a3f250a-3664-4b0b-b07e-0690f1ad231c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5c46af-29894160-cw6qx" podUID="5a3f250a-3664-4b0b-b07e-0690f1ad231c"
Nov 02 20:00:05.010299 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:05.010277    4463 scope.go:122] "RemoveContainer" containerID="ed2c2b819855c965fd031d84772ea90171fbab67a996dadc474f8084c2ab2eb1"
Nov 02 20:00:05.010623 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:00:05.010603    4463 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894160-k9hjx_odh-ai-gateway-infra(5cddfa53-4cf6-49fd-b883-c50e95ff138a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894160-k9hjx" podUID="5cddfa53-4cf6-49fd-b883-c50e95ff138a"
Nov 02 20:00:05.011729 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:05.011710    4463 scope.go:122] "RemoveContainer" containerID="dab4725ccd7ed36e93d8564b972dbae4a1f3ba8687b5e37c95a08098c774c8b6"
Nov 02 20:00:05.011943 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:00:05.011926    4463 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-28623b-29894160-cspqn_odh-ai-gateway-infra(027cb607-a331-4bc2-86b7-39c85b92860a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-28623b-29894160-cspqn" podUID="027cb607-a331-4bc2-86b7-39c85b92860a"
Nov 02 20:00:07.085277 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:07.085242    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 20:00:15.893137 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:15.893102    4463 patch_prober.go:29] interesting pod/e2e-api-a-23a9f5-openshift-default-6d49545b45-h5trw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 20:00:15.893644 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:15.893158    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-23a9f5-openshift-default-6d49545b45-h5trw" podUID="84a329f5-3f72-46b0-a7f6-eba84a47db5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:00:16.577859 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:16.577830    4463 scope.go:122] "RemoveContainer" containerID="33ce8ddcdeb348f3ad5effe0b2c3f3a5cd4934b0c4a5b7f2f72b60c69dcabbdc"
Nov 02 20:00:17.131721 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:17.131687    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5c46af-29894160-cw6qx" event={"ID":"5a3f250a-3664-4b0b-b07e-0690f1ad231c","Type":"ContainerStarted","Data":"281c7c139f7445169a85b8adecae479ae147e653bfe12dcf1fe2f20f6e6b2cc2"}
Nov 02 20:00:17.577093 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:17.577063    4463 scope.go:122] "RemoveContainer" containerID="ed2c2b819855c965fd031d84772ea90171fbab67a996dadc474f8084c2ab2eb1"
Nov 02 20:00:17.577282 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:17.577226    4463 scope.go:122] "RemoveContainer" containerID="dab4725ccd7ed36e93d8564b972dbae4a1f3ba8687b5e37c95a08098c774c8b6"
Nov 02 20:00:18.140738 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:18.140663    4463 generic.go:356] "Generic (PLEG): container finished" podID="84a329f5-3f72-46b0-a7f6-eba84a47db5a" containerID="e228b4c4a6073ae7a665418ee7b5a8ea91597b65f6ae33f2c65128768a2f0972" exitCode=0
Nov 02 20:00:18.141198 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:18.140736    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-23a9f5-openshift-default-6d49545b45-h5trw" event={"ID":"84a329f5-3f72-46b0-a7f6-eba84a47db5a","Type":"ContainerDied","Data":"e228b4c4a6073ae7a665418ee7b5a8ea91597b65f6ae33f2c65128768a2f0972"}
Nov 02 20:00:18.141198 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:18.141172    4463 scope.go:122] "RemoveContainer" containerID="e228b4c4a6073ae7a665418ee7b5a8ea91597b65f6ae33f2c65128768a2f0972"
Nov 02 20:00:18.142893 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:18.142872    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894160-k9hjx" event={"ID":"5cddfa53-4cf6-49fd-b883-c50e95ff138a","Type":"ContainerStarted","Data":"d09dcc08bcf72030ac9d008d962ebe81a43feb4cbebc8ec96fee5e3a071dd58d"}
Nov 02 20:00:18.143007 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:18.142922    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:00:18.143113 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:18.143008    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:00:18.143113 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:18.143091    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:00:18.145633 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:18.145608    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-28623b-29894160-cspqn" event={"ID":"027cb607-a331-4bc2-86b7-39c85b92860a","Type":"ContainerStarted","Data":"5b456e6e7131c945fb238ef1cad5111a8afd00fe55a0a6a7737c53aa734c95bc"}
Nov 02 20:00:18.147515 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:18.147491    4463 generic.go:356] "Generic (PLEG): container finished" podID="5a3f250a-3664-4b0b-b07e-0690f1ad231c" containerID="281c7c139f7445169a85b8adecae479ae147e653bfe12dcf1fe2f20f6e6b2cc2" exitCode=60
Nov 02 20:00:18.147616 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:18.147541    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5c46af-29894160-cw6qx" event={"ID":"5a3f250a-3664-4b0b-b07e-0690f1ad231c","Type":"ContainerDied","Data":"281c7c139f7445169a85b8adecae479ae147e653bfe12dcf1fe2f20f6e6b2cc2"}
Nov 02 20:00:18.147616 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:18.147582    4463 scope.go:122] "RemoveContainer" containerID="33ce8ddcdeb348f3ad5effe0b2c3f3a5cd4934b0c4a5b7f2f72b60c69dcabbdc"
Nov 02 20:00:18.147885 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:18.147867    4463 scope.go:122] "RemoveContainer" containerID="281c7c139f7445169a85b8adecae479ae147e653bfe12dcf1fe2f20f6e6b2cc2"
Nov 02 20:00:18.148175 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:00:18.148153    4463 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-5c46af-29894160-cw6qx_odh-ai-gateway-infra(5a3f250a-3664-4b0b-b07e-0690f1ad231c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5c46af-29894160-cw6qx" podUID="5a3f250a-3664-4b0b-b07e-0690f1ad231c"
Nov 02 20:00:18.249359 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:18.249331    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5c46af-29894160-cw6qx"]
Nov 02 20:00:19.187881 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:19.187838    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-23a9f5-openshift-default-6d49545b45-h5trw" event={"ID":"84a329f5-3f72-46b0-a7f6-eba84a47db5a","Type":"ContainerStarted","Data":"c15762394abfc9e1844aa9814ca9284c852d0d4e7ed1cda6c682519795e1e1e7"}
Nov 02 20:00:19.190521 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:19.190493    4463 generic.go:356] "Generic (PLEG): container finished" podID="5cddfa53-4cf6-49fd-b883-c50e95ff138a" containerID="d09dcc08bcf72030ac9d008d962ebe81a43feb4cbebc8ec96fee5e3a071dd58d" exitCode=60
Nov 02 20:00:19.190645 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:19.190582    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894160-k9hjx" event={"ID":"5cddfa53-4cf6-49fd-b883-c50e95ff138a","Type":"ContainerDied","Data":"d09dcc08bcf72030ac9d008d962ebe81a43feb4cbebc8ec96fee5e3a071dd58d"}
Nov 02 20:00:19.190713 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:19.190666    4463 scope.go:122] "RemoveContainer" containerID="ed2c2b819855c965fd031d84772ea90171fbab67a996dadc474f8084c2ab2eb1"
Nov 02 20:00:19.191177 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:19.191102    4463 scope.go:122] "RemoveContainer" containerID="d09dcc08bcf72030ac9d008d962ebe81a43feb4cbebc8ec96fee5e3a071dd58d"
Nov 02 20:00:19.191517 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:00:19.191490    4463 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894160-k9hjx_odh-ai-gateway-infra(5cddfa53-4cf6-49fd-b883-c50e95ff138a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894160-k9hjx" podUID="5cddfa53-4cf6-49fd-b883-c50e95ff138a"
Nov 02 20:00:19.196186 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:19.196160    4463 generic.go:356] "Generic (PLEG): container finished" podID="027cb607-a331-4bc2-86b7-39c85b92860a" containerID="5b456e6e7131c945fb238ef1cad5111a8afd00fe55a0a6a7737c53aa734c95bc" exitCode=60
Nov 02 20:00:19.196542 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:19.196280    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-28623b-29894160-cspqn" event={"ID":"027cb607-a331-4bc2-86b7-39c85b92860a","Type":"ContainerDied","Data":"5b456e6e7131c945fb238ef1cad5111a8afd00fe55a0a6a7737c53aa734c95bc"}
Nov 02 20:00:19.196762 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:19.196686    4463 scope.go:122] "RemoveContainer" containerID="5b456e6e7131c945fb238ef1cad5111a8afd00fe55a0a6a7737c53aa734c95bc"
Nov 02 20:00:19.197072 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:00:19.197046    4463 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-28623b-29894160-cspqn_odh-ai-gateway-infra(027cb607-a331-4bc2-86b7-39c85b92860a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-28623b-29894160-cspqn" podUID="027cb607-a331-4bc2-86b7-39c85b92860a"
Nov 02 20:00:19.210191 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:19.210167    4463 scope.go:122] "RemoveContainer" containerID="dab4725ccd7ed36e93d8564b972dbae4a1f3ba8687b5e37c95a08098c774c8b6"
Nov 02 20:00:19.403400 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:19.403373    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5c46af-29894160-cw6qx"
Nov 02 20:00:19.416175 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:19.416150    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894160-k9hjx"]
Nov 02 20:00:19.518364 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:19.518286    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-28623b-29894160-cspqn"]
Nov 02 20:00:19.586166 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:19.586036    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a3f250a-3664-4b0b-b07e-0690f1ad231c-kube-api-access-tlmb2\" (UniqueName: \"kubernetes.io/projected/5a3f250a-3664-4b0b-b07e-0690f1ad231c-kube-api-access-tlmb2\") pod \"5a3f250a-3664-4b0b-b07e-0690f1ad231c\" (UID: \"5a3f250a-3664-4b0b-b07e-0690f1ad231c\") "
Nov 02 20:00:19.588024 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:19.586378    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a3f250a-3664-4b0b-b07e-0690f1ad231c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a3f250a-3664-4b0b-b07e-0690f1ad231c-maas-api-service-ca\") pod \"5a3f250a-3664-4b0b-b07e-0690f1ad231c\" (UID: \"5a3f250a-3664-4b0b-b07e-0690f1ad231c\") "
Nov 02 20:00:19.592402 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:19.592347    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a3f250a-3664-4b0b-b07e-0690f1ad231c-maas-api-service-ca" pod "5a3f250a-3664-4b0b-b07e-0690f1ad231c" (UID: "5a3f250a-3664-4b0b-b07e-0690f1ad231c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:00:19.606332 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:19.605230    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a3f250a-3664-4b0b-b07e-0690f1ad231c-kube-api-access-tlmb2" pod "5a3f250a-3664-4b0b-b07e-0690f1ad231c" (UID: "5a3f250a-3664-4b0b-b07e-0690f1ad231c"). InnerVolumeSpecName "kube-api-access-tlmb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:00:19.688659 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:19.688620    4463 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a3f250a-3664-4b0b-b07e-0690f1ad231c-maas-api-service-ca\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:00:19.688659 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:19.688648    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlmb2\" (UniqueName: \"kubernetes.io/projected/5a3f250a-3664-4b0b-b07e-0690f1ad231c-kube-api-access-tlmb2\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:00:19.790185 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:19.783538    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-23a9f5-openshift-default-6d49545b45-h5trw"
Nov 02 20:00:19.790185 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:19.783659    4463 patch_prober.go:29] interesting pod/e2e-api-a-23a9f5-openshift-default-6d49545b45-h5trw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Nov 02 20:00:19.790185 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:19.783716    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-23a9f5-openshift-default-6d49545b45-h5trw" podUID="84a329f5-3f72-46b0-a7f6-eba84a47db5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Nov 02 20:00:20.208596 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:20.208567    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5c46af-29894160-cw6qx"
Nov 02 20:00:20.209005 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:20.208609    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5c46af-29894160-cw6qx" event={"ID":"5a3f250a-3664-4b0b-b07e-0690f1ad231c","Type":"ContainerDied","Data":"73d1e0db61d3e61ca65a2191693720fabc11f5c60a58e9110d7f1cf9a803f4ef"}
Nov 02 20:00:20.209005 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:20.208680    4463 scope.go:122] "RemoveContainer" containerID="281c7c139f7445169a85b8adecae479ae147e653bfe12dcf1fe2f20f6e6b2cc2"
Nov 02 20:00:20.430221 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:20.430194    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894160-k9hjx"
Nov 02 20:00:20.496477 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:20.496451    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-28623b-29894160-cspqn"
Nov 02 20:00:20.515644 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:20.515608    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cddfa53-4cf6-49fd-b883-c50e95ff138a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cddfa53-4cf6-49fd-b883-c50e95ff138a-maas-api-service-ca\") pod \"5cddfa53-4cf6-49fd-b883-c50e95ff138a\" (UID: \"5cddfa53-4cf6-49fd-b883-c50e95ff138a\") "
Nov 02 20:00:20.515783 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:20.515653    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cddfa53-4cf6-49fd-b883-c50e95ff138a-kube-api-access-nzr62\" (UniqueName: \"kubernetes.io/projected/5cddfa53-4cf6-49fd-b883-c50e95ff138a-kube-api-access-nzr62\") pod \"5cddfa53-4cf6-49fd-b883-c50e95ff138a\" (UID: \"5cddfa53-4cf6-49fd-b883-c50e95ff138a\") "
Nov 02 20:00:20.515955 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:20.515930    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/027cb607-a331-4bc2-86b7-39c85b92860a-kube-api-access-v6wsw\" (UniqueName: \"kubernetes.io/projected/027cb607-a331-4bc2-86b7-39c85b92860a-kube-api-access-v6wsw\") pod \"027cb607-a331-4bc2-86b7-39c85b92860a\" (UID: \"027cb607-a331-4bc2-86b7-39c85b92860a\") "
Nov 02 20:00:20.516092 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:20.515974    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/027cb607-a331-4bc2-86b7-39c85b92860a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/027cb607-a331-4bc2-86b7-39c85b92860a-maas-api-service-ca\") pod \"027cb607-a331-4bc2-86b7-39c85b92860a\" (UID: \"027cb607-a331-4bc2-86b7-39c85b92860a\") "
Nov 02 20:00:20.516201 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:20.516175    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cddfa53-4cf6-49fd-b883-c50e95ff138a-maas-api-service-ca" pod "5cddfa53-4cf6-49fd-b883-c50e95ff138a" (UID: "5cddfa53-4cf6-49fd-b883-c50e95ff138a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:00:20.516842 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:20.516817    4463 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cddfa53-4cf6-49fd-b883-c50e95ff138a-maas-api-service-ca\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:00:20.517161 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:20.517135    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/027cb607-a331-4bc2-86b7-39c85b92860a-maas-api-service-ca" pod "027cb607-a331-4bc2-86b7-39c85b92860a" (UID: "027cb607-a331-4bc2-86b7-39c85b92860a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:00:20.520775 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:20.518581    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cddfa53-4cf6-49fd-b883-c50e95ff138a-kube-api-access-nzr62" pod "5cddfa53-4cf6-49fd-b883-c50e95ff138a" (UID: "5cddfa53-4cf6-49fd-b883-c50e95ff138a"). InnerVolumeSpecName "kube-api-access-nzr62". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:00:20.520775 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:20.519039    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/027cb607-a331-4bc2-86b7-39c85b92860a-kube-api-access-v6wsw" pod "027cb607-a331-4bc2-86b7-39c85b92860a" (UID: "027cb607-a331-4bc2-86b7-39c85b92860a"). InnerVolumeSpecName "kube-api-access-v6wsw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:00:20.544923 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:20.544891    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5c46af-29894160-cw6qx"]
Nov 02 20:00:20.576651 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:20.576619    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5c46af-29894160-cw6qx"]
Nov 02 20:00:20.619183 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:20.619153    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6wsw\" (UniqueName: \"kubernetes.io/projected/027cb607-a331-4bc2-86b7-39c85b92860a-kube-api-access-v6wsw\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:00:20.619183 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:20.619184    4463 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/027cb607-a331-4bc2-86b7-39c85b92860a-maas-api-service-ca\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:00:20.619429 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:20.619199    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzr62\" (UniqueName: \"kubernetes.io/projected/5cddfa53-4cf6-49fd-b883-c50e95ff138a-kube-api-access-nzr62\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:00:20.779460 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:20.779369    4463 patch_prober.go:29] interesting pod/e2e-api-a-23a9f5-openshift-default-6d49545b45-h5trw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Nov 02 20:00:20.779460 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:20.779428    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-23a9f5-openshift-default-6d49545b45-h5trw" podUID="84a329f5-3f72-46b0-a7f6-eba84a47db5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Nov 02 20:00:21.216432 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:21.216399    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894160-k9hjx"
Nov 02 20:00:21.216900 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:21.216398    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894160-k9hjx" event={"ID":"5cddfa53-4cf6-49fd-b883-c50e95ff138a","Type":"ContainerDied","Data":"1fe2d83d7ab4af01f9cdaafc0b5608a5b57f1496747b9921900b7bf41b59450d"}
Nov 02 20:00:21.216900 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:21.216544    4463 scope.go:122] "RemoveContainer" containerID="d09dcc08bcf72030ac9d008d962ebe81a43feb4cbebc8ec96fee5e3a071dd58d"
Nov 02 20:00:21.217881 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:21.217838    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-28623b-29894160-cspqn"
Nov 02 20:00:21.217881 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:21.217848    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-28623b-29894160-cspqn" event={"ID":"027cb607-a331-4bc2-86b7-39c85b92860a","Type":"ContainerDied","Data":"59bf53a08f424306ec42b563a344c92d48ac9db9cd675c96566c6686091880da"}
Nov 02 20:00:21.227176 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:21.227156    4463 scope.go:122] "RemoveContainer" containerID="5b456e6e7131c945fb238ef1cad5111a8afd00fe55a0a6a7737c53aa734c95bc"
Nov 02 20:00:21.396144 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:21.396095    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894160-k9hjx"]
Nov 02 20:00:21.402966 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:21.402602    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894160-k9hjx"]
Nov 02 20:00:21.495095 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:21.494979    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-28623b-29894160-cspqn"]
Nov 02 20:00:21.499494 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:21.499471    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-28623b-29894160-cspqn"]
Nov 02 20:00:21.587726 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:21.587692    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="027cb607-a331-4bc2-86b7-39c85b92860a" path="/var/lib/kubelet/pods/027cb607-a331-4bc2-86b7-39c85b92860a/volumes"
Nov 02 20:00:21.588208 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:21.588190    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a3f250a-3664-4b0b-b07e-0690f1ad231c" path="/var/lib/kubelet/pods/5a3f250a-3664-4b0b-b07e-0690f1ad231c/volumes"
Nov 02 20:00:21.588614 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:21.588600    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cddfa53-4cf6-49fd-b883-c50e95ff138a" path="/var/lib/kubelet/pods/5cddfa53-4cf6-49fd-b883-c50e95ff138a/volumes"
Nov 02 20:00:21.774413 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:21.774330    4463 patch_prober.go:29] interesting pod/e2e-api-a-23a9f5-openshift-default-6d49545b45-h5trw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Nov 02 20:00:21.774571 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:21.774407    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-23a9f5-openshift-default-6d49545b45-h5trw" podUID="84a329f5-3f72-46b0-a7f6-eba84a47db5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Nov 02 20:00:22.774038 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:22.773752    4463 patch_prober.go:29] interesting pod/e2e-api-a-23a9f5-openshift-default-6d49545b45-h5trw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Nov 02 20:00:22.774038 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:22.773844    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-23a9f5-openshift-default-6d49545b45-h5trw" podUID="84a329f5-3f72-46b0-a7f6-eba84a47db5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Nov 02 20:00:23.774410 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:23.774371    4463 patch_prober.go:29] interesting pod/e2e-api-a-23a9f5-openshift-default-6d49545b45-h5trw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Nov 02 20:00:23.774851 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:23.774439    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-23a9f5-openshift-default-6d49545b45-h5trw" podUID="84a329f5-3f72-46b0-a7f6-eba84a47db5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Nov 02 20:00:24.763426 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:24.763390    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-23a9f5-openshift-default-6d49545b45-h5trw"
Nov 02 20:00:24.763779 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:24.763754    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-23a9f5-openshift-default-6d49545b45-h5trw"
Nov 02 20:00:24.764542 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:24.764521    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-23a9f5-openshift-default-6d49545b45-h5trw"
Nov 02 20:00:34.296851 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:00:34.296770    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 20:00:51.344781 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:00:51.344747    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 20:01:00.389249 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:00.389208    4463 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ffdc4d-openshift-default-6995c74bfc-mt2f2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 20:01:00.389859 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:00.389822    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-ffdc4d-openshift-default-6995c74bfc-mt2f2" podUID="d5ffe43f-1a80-4126-9cf1-251fdde99de1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:01:02.579420 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:02.579344    4463 generic.go:356] "Generic (PLEG): container finished" podID="d5ffe43f-1a80-4126-9cf1-251fdde99de1" containerID="ef65e7eca4ab2cbb96b24402b81accd94d6be216c8f7eb7166b44e3d7db88333" exitCode=0
Nov 02 20:01:02.579828 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:02.579417    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ffdc4d-openshift-default-6995c74bfc-mt2f2" event={"ID":"d5ffe43f-1a80-4126-9cf1-251fdde99de1","Type":"ContainerDied","Data":"ef65e7eca4ab2cbb96b24402b81accd94d6be216c8f7eb7166b44e3d7db88333"}
Nov 02 20:01:02.579828 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:02.579822    4463 scope.go:122] "RemoveContainer" containerID="ef65e7eca4ab2cbb96b24402b81accd94d6be216c8f7eb7166b44e3d7db88333"
Nov 02 20:01:02.581701 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:02.581666    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:01:02.581838 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:02.581750    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:01:02.581838 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:02.581799    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:01:03.603357 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:03.603321    4463 generic.go:356] "Generic (PLEG): container finished" podID="618bcdd7-16ca-40c3-9a41-feed4d5f7161" containerID="5479f80afbc79dbc92d7fdcbb032f1942e61a73148b8910c12f9330b98b69962" exitCode=0
Nov 02 20:01:03.604484 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:03.604452    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c60a0d-openshift-default-6464975799-g9tvr" event={"ID":"618bcdd7-16ca-40c3-9a41-feed4d5f7161","Type":"ContainerDied","Data":"5479f80afbc79dbc92d7fdcbb032f1942e61a73148b8910c12f9330b98b69962"}
Nov 02 20:01:03.604874 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:03.604854    4463 scope.go:122] "RemoveContainer" containerID="5479f80afbc79dbc92d7fdcbb032f1942e61a73148b8910c12f9330b98b69962"
Nov 02 20:01:03.605800 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:03.605577    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ffdc4d-openshift-default-6995c74bfc-mt2f2" event={"ID":"d5ffe43f-1a80-4126-9cf1-251fdde99de1","Type":"ContainerStarted","Data":"fe477fa7a8d2626d231a2ee729ee2204560465ab75763a9dba0ec97543d12d40"}
Nov 02 20:01:03.606572 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:03.606541    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:01:03.606667 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:03.606625    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:01:03.606731 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:03.606670    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:01:03.817089 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:03.816173    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-ffdc4d-openshift-default-6995c74bfc-mt2f2"
Nov 02 20:01:03.817089 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:03.816263    4463 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ffdc4d-openshift-default-6995c74bfc-mt2f2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Nov 02 20:01:03.817089 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:03.816294    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ffdc4d-openshift-default-6995c74bfc-mt2f2" podUID="d5ffe43f-1a80-4126-9cf1-251fdde99de1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Nov 02 20:01:04.305433 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:01:04.305107    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 20:01:04.618603 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:04.618303    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c60a0d-openshift-default-6464975799-g9tvr" event={"ID":"618bcdd7-16ca-40c3-9a41-feed4d5f7161","Type":"ContainerStarted","Data":"8d1069229b6457b0b97e70fd618dd7fcfca991676d6fe15413782732e24db6d9"}
Nov 02 20:01:04.816707 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:04.816670    4463 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ffdc4d-openshift-default-6995c74bfc-mt2f2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Nov 02 20:01:04.816913 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:04.816731    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ffdc4d-openshift-default-6995c74bfc-mt2f2" podUID="d5ffe43f-1a80-4126-9cf1-251fdde99de1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Nov 02 20:01:05.495113 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:05.495078    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-c60a0d-openshift-default-6464975799-g9tvr"
Nov 02 20:01:05.498524 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:05.498049    4463 patch_prober.go:29] interesting pod/e2e-api-b-c60a0d-openshift-default-6464975799-g9tvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Nov 02 20:01:05.498524 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:05.498134    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c60a0d-openshift-default-6464975799-g9tvr" podUID="618bcdd7-16ca-40c3-9a41-feed4d5f7161" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Nov 02 20:01:05.815973 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:05.815888    4463 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ffdc4d-openshift-default-6995c74bfc-mt2f2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Nov 02 20:01:05.815973 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:05.815948    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ffdc4d-openshift-default-6995c74bfc-mt2f2" podUID="d5ffe43f-1a80-4126-9cf1-251fdde99de1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Nov 02 20:01:06.101093 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:06.100170    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-c60a0d-openshift-default-6464975799-g9tvr"
Nov 02 20:01:06.497396 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:06.497358    4463 patch_prober.go:29] interesting pod/e2e-api-b-c60a0d-openshift-default-6464975799-g9tvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Nov 02 20:01:06.497662 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:06.497634    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c60a0d-openshift-default-6464975799-g9tvr" podUID="618bcdd7-16ca-40c3-9a41-feed4d5f7161" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Nov 02 20:01:06.816402 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:06.816323    4463 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ffdc4d-openshift-default-6995c74bfc-mt2f2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Nov 02 20:01:06.816402 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:06.816378    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ffdc4d-openshift-default-6995c74bfc-mt2f2" podUID="d5ffe43f-1a80-4126-9cf1-251fdde99de1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Nov 02 20:01:07.497044 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:07.496406    4463 patch_prober.go:29] interesting pod/e2e-api-b-c60a0d-openshift-default-6464975799-g9tvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Nov 02 20:01:07.497044 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:07.496468    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c60a0d-openshift-default-6464975799-g9tvr" podUID="618bcdd7-16ca-40c3-9a41-feed4d5f7161" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Nov 02 20:01:07.816857 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:07.816770    4463 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ffdc4d-openshift-default-6995c74bfc-mt2f2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Nov 02 20:01:07.816857 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:07.816834    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ffdc4d-openshift-default-6995c74bfc-mt2f2" podUID="d5ffe43f-1a80-4126-9cf1-251fdde99de1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Nov 02 20:01:08.501035 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:08.497373    4463 patch_prober.go:29] interesting pod/e2e-api-b-c60a0d-openshift-default-6464975799-g9tvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Nov 02 20:01:08.501035 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:08.497436    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c60a0d-openshift-default-6464975799-g9tvr" podUID="618bcdd7-16ca-40c3-9a41-feed4d5f7161" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Nov 02 20:01:08.827902 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:08.827822    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-ffdc4d-openshift-default-6995c74bfc-mt2f2"
Nov 02 20:01:08.828668 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:08.828645    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-ffdc4d-openshift-default-6995c74bfc-mt2f2"
Nov 02 20:01:08.829351 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:08.829329    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-ffdc4d-openshift-default-6995c74bfc-mt2f2"
Nov 02 20:01:09.494879 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:09.494844    4463 patch_prober.go:29] interesting pod/e2e-api-b-c60a0d-openshift-default-6464975799-g9tvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Nov 02 20:01:09.495087 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:09.494906    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c60a0d-openshift-default-6464975799-g9tvr" podUID="618bcdd7-16ca-40c3-9a41-feed4d5f7161" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Nov 02 20:01:10.501476 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:10.501439    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-c60a0d-openshift-default-6464975799-g9tvr"
Nov 02 20:01:10.502519 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:10.502496    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-c60a0d-openshift-default-6464975799-g9tvr"
Nov 02 20:01:10.658530 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:10.658499    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8479ff-d99f8dbc7-9bvj7"]
Nov 02 20:01:10.770991 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:10.770907    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8479ff-7fd9b68fc5-plq28"]
Nov 02 20:01:15.659440 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:15.659379    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-8479ff-d99f8dbc7-9bvj7" podUID="f9045c38-104c-4065-b369-6f3803fae81e" containerName="payload-processing" containerID="cri-o://0e49b64e6970d2fc3d80e8bcd74f61a64fe6ad70654643f5c418c4344b186865" gracePeriod=25
Nov 02 20:01:15.771565 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:15.771530    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8479ff-7fd9b68fc5-plq28" podUID="a211d74d-a663-4c8e-a451-f3253236d6f6" containerName="payload-pre-processing" containerID="cri-o://d9664da1d681a6b868086ab2121ce732a967f09c2dc9a57792d7912da37a4c14" gracePeriod=25
Nov 02 20:01:15.965292 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:15.964336    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-8479ff-d99f8dbc7-9bvj7"
Nov 02 20:01:16.051381 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:16.051357    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8479ff-7fd9b68fc5-plq28"
Nov 02 20:01:16.136969 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:16.136941    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a211d74d-a663-4c8e-a451-f3253236d6f6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a211d74d-a663-4c8e-a451-f3253236d6f6-plugins-config-volume\") pod \"a211d74d-a663-4c8e-a451-f3253236d6f6\" (UID: \"a211d74d-a663-4c8e-a451-f3253236d6f6\") "
Nov 02 20:01:16.137135 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:16.137069    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9045c38-104c-4065-b369-6f3803fae81e-kube-api-access-8xd2c\" (UniqueName: \"kubernetes.io/projected/f9045c38-104c-4065-b369-6f3803fae81e-kube-api-access-8xd2c\") pod \"f9045c38-104c-4065-b369-6f3803fae81e\" (UID: \"f9045c38-104c-4065-b369-6f3803fae81e\") "
Nov 02 20:01:16.137279 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:16.137261    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a211d74d-a663-4c8e-a451-f3253236d6f6-kube-api-access-9wnds\" (UniqueName: \"kubernetes.io/projected/a211d74d-a663-4c8e-a451-f3253236d6f6-kube-api-access-9wnds\") pod \"a211d74d-a663-4c8e-a451-f3253236d6f6\" (UID: \"a211d74d-a663-4c8e-a451-f3253236d6f6\") "
Nov 02 20:01:16.137421 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:16.137403    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9045c38-104c-4065-b369-6f3803fae81e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9045c38-104c-4065-b369-6f3803fae81e-plugins-config-volume\") pod \"f9045c38-104c-4065-b369-6f3803fae81e\" (UID: \"f9045c38-104c-4065-b369-6f3803fae81e\") "
Nov 02 20:01:16.137605 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:16.137404    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a211d74d-a663-4c8e-a451-f3253236d6f6-plugins-config-volume" pod "a211d74d-a663-4c8e-a451-f3253236d6f6" (UID: "a211d74d-a663-4c8e-a451-f3253236d6f6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:01:16.137802 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:16.137782    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9045c38-104c-4065-b369-6f3803fae81e-plugins-config-volume" pod "f9045c38-104c-4065-b369-6f3803fae81e" (UID: "f9045c38-104c-4065-b369-6f3803fae81e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:01:16.138055 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:16.138037    4463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a211d74d-a663-4c8e-a451-f3253236d6f6-plugins-config-volume\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:01:16.138122 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:16.138060    4463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9045c38-104c-4065-b369-6f3803fae81e-plugins-config-volume\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:01:16.139421 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:16.139400    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a211d74d-a663-4c8e-a451-f3253236d6f6-kube-api-access-9wnds" pod "a211d74d-a663-4c8e-a451-f3253236d6f6" (UID: "a211d74d-a663-4c8e-a451-f3253236d6f6"). InnerVolumeSpecName "kube-api-access-9wnds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:01:16.139565 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:16.139546    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9045c38-104c-4065-b369-6f3803fae81e-kube-api-access-8xd2c" pod "f9045c38-104c-4065-b369-6f3803fae81e" (UID: "f9045c38-104c-4065-b369-6f3803fae81e"). InnerVolumeSpecName "kube-api-access-8xd2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:01:16.239225 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:16.239195    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xd2c\" (UniqueName: \"kubernetes.io/projected/f9045c38-104c-4065-b369-6f3803fae81e-kube-api-access-8xd2c\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:01:16.239225 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:16.239222    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wnds\" (UniqueName: \"kubernetes.io/projected/a211d74d-a663-4c8e-a451-f3253236d6f6-kube-api-access-9wnds\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:01:16.727890 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:16.727857    4463 generic.go:356] "Generic (PLEG): container finished" podID="a211d74d-a663-4c8e-a451-f3253236d6f6" containerID="d9664da1d681a6b868086ab2121ce732a967f09c2dc9a57792d7912da37a4c14" exitCode=0
Nov 02 20:01:16.728404 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:16.727902    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8479ff-7fd9b68fc5-plq28"
Nov 02 20:01:16.728404 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:16.727938    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8479ff-7fd9b68fc5-plq28" event={"ID":"a211d74d-a663-4c8e-a451-f3253236d6f6","Type":"ContainerDied","Data":"d9664da1d681a6b868086ab2121ce732a967f09c2dc9a57792d7912da37a4c14"}
Nov 02 20:01:16.728404 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:16.727991    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8479ff-7fd9b68fc5-plq28" event={"ID":"a211d74d-a663-4c8e-a451-f3253236d6f6","Type":"ContainerDied","Data":"aebb8cd9f3961ec66b11f0867d3ac75bbe1ec741fe32582b91b67580ed9323c5"}
Nov 02 20:01:16.728404 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:16.728039    4463 scope.go:122] "RemoveContainer" containerID="d9664da1d681a6b868086ab2121ce732a967f09c2dc9a57792d7912da37a4c14"
Nov 02 20:01:16.729758 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:16.729618    4463 generic.go:356] "Generic (PLEG): container finished" podID="f9045c38-104c-4065-b369-6f3803fae81e" containerID="0e49b64e6970d2fc3d80e8bcd74f61a64fe6ad70654643f5c418c4344b186865" exitCode=0
Nov 02 20:01:16.729758 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:16.729716    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-8479ff-d99f8dbc7-9bvj7" event={"ID":"f9045c38-104c-4065-b369-6f3803fae81e","Type":"ContainerDied","Data":"0e49b64e6970d2fc3d80e8bcd74f61a64fe6ad70654643f5c418c4344b186865"}
Nov 02 20:01:16.729758 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:16.729734    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-8479ff-d99f8dbc7-9bvj7"
Nov 02 20:01:16.729758 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:16.729752    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-8479ff-d99f8dbc7-9bvj7" event={"ID":"f9045c38-104c-4065-b369-6f3803fae81e","Type":"ContainerDied","Data":"421c5c73036908c3df98b6d89a1b0efc56c6063bced163a773253c999d034649"}
Nov 02 20:01:16.738674 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:16.738656    4463 scope.go:122] "RemoveContainer" containerID="d9664da1d681a6b868086ab2121ce732a967f09c2dc9a57792d7912da37a4c14"
Nov 02 20:01:16.738977 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:01:16.738962    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9664da1d681a6b868086ab2121ce732a967f09c2dc9a57792d7912da37a4c14\": container with ID starting with d9664da1d681a6b868086ab2121ce732a967f09c2dc9a57792d7912da37a4c14 not found: ID does not exist" containerID="d9664da1d681a6b868086ab2121ce732a967f09c2dc9a57792d7912da37a4c14"
Nov 02 20:01:16.739083 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:16.738987    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9664da1d681a6b868086ab2121ce732a967f09c2dc9a57792d7912da37a4c14"} err="failed to get container status \"d9664da1d681a6b868086ab2121ce732a967f09c2dc9a57792d7912da37a4c14\": rpc error: code = NotFound desc = could not find container \"d9664da1d681a6b868086ab2121ce732a967f09c2dc9a57792d7912da37a4c14\": container with ID starting with d9664da1d681a6b868086ab2121ce732a967f09c2dc9a57792d7912da37a4c14 not found: ID does not exist"
Nov 02 20:01:16.739083 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:16.739037    4463 scope.go:122] "RemoveContainer" containerID="0e49b64e6970d2fc3d80e8bcd74f61a64fe6ad70654643f5c418c4344b186865"
Nov 02 20:01:16.748684 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:16.748666    4463 scope.go:122] "RemoveContainer" containerID="0e49b64e6970d2fc3d80e8bcd74f61a64fe6ad70654643f5c418c4344b186865"
Nov 02 20:01:16.748944 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:01:16.748918    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e49b64e6970d2fc3d80e8bcd74f61a64fe6ad70654643f5c418c4344b186865\": container with ID starting with 0e49b64e6970d2fc3d80e8bcd74f61a64fe6ad70654643f5c418c4344b186865 not found: ID does not exist" containerID="0e49b64e6970d2fc3d80e8bcd74f61a64fe6ad70654643f5c418c4344b186865"
Nov 02 20:01:16.748989 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:16.748956    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e49b64e6970d2fc3d80e8bcd74f61a64fe6ad70654643f5c418c4344b186865"} err="failed to get container status \"0e49b64e6970d2fc3d80e8bcd74f61a64fe6ad70654643f5c418c4344b186865\": rpc error: code = NotFound desc = could not find container \"0e49b64e6970d2fc3d80e8bcd74f61a64fe6ad70654643f5c418c4344b186865\": container with ID starting with 0e49b64e6970d2fc3d80e8bcd74f61a64fe6ad70654643f5c418c4344b186865 not found: ID does not exist"
Nov 02 20:01:16.882024 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:16.881977    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8479ff-7fd9b68fc5-plq28"]
Nov 02 20:01:16.927842 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:16.927754    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8479ff-7fd9b68fc5-plq28"]
Nov 02 20:01:16.988671 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:16.988635    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8479ff-d99f8dbc7-9bvj7"]
Nov 02 20:01:16.992047 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:16.992022    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8479ff-d99f8dbc7-9bvj7"]
Nov 02 20:01:17.587205 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:17.587164    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a211d74d-a663-4c8e-a451-f3253236d6f6" path="/var/lib/kubelet/pods/a211d74d-a663-4c8e-a451-f3253236d6f6/volumes"
Nov 02 20:01:17.587611 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:17.587596    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9045c38-104c-4065-b369-6f3803fae81e" path="/var/lib/kubelet/pods/f9045c38-104c-4065-b369-6f3803fae81e/volumes"
Nov 02 20:01:28.094928 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:28.094887    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-23a9f5-85b7bb67f-8f9nj"]
Nov 02 20:01:28.207379 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:28.204939    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-23a9f5-6f4f945579-gplz2"]
Nov 02 20:01:29.848751 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:29.848717    4463 generic.go:356] "Generic (PLEG): container finished" podID="277fb120-e642-4215-bc3d-1b9c08c57e81" containerID="985dd2c51fbde5438cae0f74b3691c5bca1361a42cb033dc5bc62ed5c386ae25" exitCode=0
Nov 02 20:01:29.849239 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:29.848773    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4b0a12-openshift-default-8677ff4fb5-6css2" event={"ID":"277fb120-e642-4215-bc3d-1b9c08c57e81","Type":"ContainerDied","Data":"985dd2c51fbde5438cae0f74b3691c5bca1361a42cb033dc5bc62ed5c386ae25"}
Nov 02 20:01:29.849239 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:29.849227    4463 scope.go:122] "RemoveContainer" containerID="985dd2c51fbde5438cae0f74b3691c5bca1361a42cb033dc5bc62ed5c386ae25"
Nov 02 20:01:29.851107 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:29.851064    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:01:29.851225 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:29.851164    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:01:29.851225 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:29.851219    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:01:30.883917 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:30.883883    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4b0a12-openshift-default-8677ff4fb5-6css2" event={"ID":"277fb120-e642-4215-bc3d-1b9c08c57e81","Type":"ContainerStarted","Data":"0ef0bbf558b30756c144f89346b9e29b5c6e55ffaf486fbc717d4fdd6d245ed0"}
Nov 02 20:01:31.009369 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:31.009335    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-4b0a12-openshift-default-8677ff4fb5-6css2"
Nov 02 20:01:31.010860 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:31.010835    4463 patch_prober.go:29] interesting pod/e2e-inf-b-4b0a12-openshift-default-8677ff4fb5-6css2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Nov 02 20:01:31.010970 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:31.010894    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4b0a12-openshift-default-8677ff4fb5-6css2" podUID="277fb120-e642-4215-bc3d-1b9c08c57e81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Nov 02 20:01:32.010057 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:32.010004    4463 patch_prober.go:29] interesting pod/e2e-inf-b-4b0a12-openshift-default-8677ff4fb5-6css2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Nov 02 20:01:32.010489 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:32.010094    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4b0a12-openshift-default-8677ff4fb5-6css2" podUID="277fb120-e642-4215-bc3d-1b9c08c57e81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Nov 02 20:01:33.009045 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:33.008978    4463 patch_prober.go:29] interesting pod/e2e-inf-b-4b0a12-openshift-default-8677ff4fb5-6css2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Nov 02 20:01:33.009216 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:33.009056    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4b0a12-openshift-default-8677ff4fb5-6css2" podUID="277fb120-e642-4215-bc3d-1b9c08c57e81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Nov 02 20:01:33.095450 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:33.095407    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-23a9f5-85b7bb67f-8f9nj" podUID="a2862a4d-a528-4766-ac1e-958b2de9ca1d" containerName="payload-processing" containerID="cri-o://9a2b9d289d540b5db5c814f5c4b5f5187714aab1febde8fd536cf32217b3c234" gracePeriod=25
Nov 02 20:01:33.208222 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:33.208184    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-23a9f5-6f4f945579-gplz2" podUID="045e2518-6641-4e5d-a9b7-10326c74b26b" containerName="payload-pre-processing" containerID="cri-o://efed4e8269b553eae37ecf8879f8103c20b91f197f6b855251d5c6233fb78674" gracePeriod=25
Nov 02 20:01:33.423056 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:33.422752    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-23a9f5-85b7bb67f-8f9nj"
Nov 02 20:01:33.493877 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:33.493840    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2862a4d-a528-4766-ac1e-958b2de9ca1d-kube-api-access-98mvh\" (UniqueName: \"kubernetes.io/projected/a2862a4d-a528-4766-ac1e-958b2de9ca1d-kube-api-access-98mvh\") pod \"a2862a4d-a528-4766-ac1e-958b2de9ca1d\" (UID: \"a2862a4d-a528-4766-ac1e-958b2de9ca1d\") "
Nov 02 20:01:33.494528 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:33.494498    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2862a4d-a528-4766-ac1e-958b2de9ca1d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2862a4d-a528-4766-ac1e-958b2de9ca1d-plugins-config-volume\") pod \"a2862a4d-a528-4766-ac1e-958b2de9ca1d\" (UID: \"a2862a4d-a528-4766-ac1e-958b2de9ca1d\") "
Nov 02 20:01:33.494895 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:33.494872    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2862a4d-a528-4766-ac1e-958b2de9ca1d-plugins-config-volume" pod "a2862a4d-a528-4766-ac1e-958b2de9ca1d" (UID: "a2862a4d-a528-4766-ac1e-958b2de9ca1d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:01:33.495256 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:33.495238    4463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2862a4d-a528-4766-ac1e-958b2de9ca1d-plugins-config-volume\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:01:33.502904 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:33.502883    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2862a4d-a528-4766-ac1e-958b2de9ca1d-kube-api-access-98mvh" pod "a2862a4d-a528-4766-ac1e-958b2de9ca1d" (UID: "a2862a4d-a528-4766-ac1e-958b2de9ca1d"). InnerVolumeSpecName "kube-api-access-98mvh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:01:33.541003 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:33.540146    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-23a9f5-6f4f945579-gplz2"
Nov 02 20:01:33.595946 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:33.595914    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/045e2518-6641-4e5d-a9b7-10326c74b26b-kube-api-access-wd79v\" (UniqueName: \"kubernetes.io/projected/045e2518-6641-4e5d-a9b7-10326c74b26b-kube-api-access-wd79v\") pod \"045e2518-6641-4e5d-a9b7-10326c74b26b\" (UID: \"045e2518-6641-4e5d-a9b7-10326c74b26b\") "
Nov 02 20:01:33.596143 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:33.596080    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/045e2518-6641-4e5d-a9b7-10326c74b26b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/045e2518-6641-4e5d-a9b7-10326c74b26b-plugins-config-volume\") pod \"045e2518-6641-4e5d-a9b7-10326c74b26b\" (UID: \"045e2518-6641-4e5d-a9b7-10326c74b26b\") "
Nov 02 20:01:33.596545 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:33.596517    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/045e2518-6641-4e5d-a9b7-10326c74b26b-plugins-config-volume" pod "045e2518-6641-4e5d-a9b7-10326c74b26b" (UID: "045e2518-6641-4e5d-a9b7-10326c74b26b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:01:33.596798 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:33.596780    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98mvh\" (UniqueName: \"kubernetes.io/projected/a2862a4d-a528-4766-ac1e-958b2de9ca1d-kube-api-access-98mvh\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:01:33.598549 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:33.598525    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/045e2518-6641-4e5d-a9b7-10326c74b26b-kube-api-access-wd79v" pod "045e2518-6641-4e5d-a9b7-10326c74b26b" (UID: "045e2518-6641-4e5d-a9b7-10326c74b26b"). InnerVolumeSpecName "kube-api-access-wd79v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:01:33.698568 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:33.698532    4463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/045e2518-6641-4e5d-a9b7-10326c74b26b-plugins-config-volume\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:01:33.698954 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:33.698741    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wd79v\" (UniqueName: \"kubernetes.io/projected/045e2518-6641-4e5d-a9b7-10326c74b26b-kube-api-access-wd79v\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:01:33.906376 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:33.906289    4463 generic.go:356] "Generic (PLEG): container finished" podID="045e2518-6641-4e5d-a9b7-10326c74b26b" containerID="efed4e8269b553eae37ecf8879f8103c20b91f197f6b855251d5c6233fb78674" exitCode=0
Nov 02 20:01:33.906376 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:33.906345    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-23a9f5-6f4f945579-gplz2"
Nov 02 20:01:33.906376 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:33.906367    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-23a9f5-6f4f945579-gplz2" event={"ID":"045e2518-6641-4e5d-a9b7-10326c74b26b","Type":"ContainerDied","Data":"efed4e8269b553eae37ecf8879f8103c20b91f197f6b855251d5c6233fb78674"}
Nov 02 20:01:33.906623 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:33.906447    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-23a9f5-6f4f945579-gplz2" event={"ID":"045e2518-6641-4e5d-a9b7-10326c74b26b","Type":"ContainerDied","Data":"35d3b31f4e0faf657765b4c6945ce7a71f7b5dd6738d2394b4a7996bac9e6b44"}
Nov 02 20:01:33.906623 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:33.906477    4463 scope.go:122] "RemoveContainer" containerID="efed4e8269b553eae37ecf8879f8103c20b91f197f6b855251d5c6233fb78674"
Nov 02 20:01:33.907904 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:33.907881    4463 generic.go:356] "Generic (PLEG): container finished" podID="a2862a4d-a528-4766-ac1e-958b2de9ca1d" containerID="9a2b9d289d540b5db5c814f5c4b5f5187714aab1febde8fd536cf32217b3c234" exitCode=0
Nov 02 20:01:33.908070 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:33.907907    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-23a9f5-85b7bb67f-8f9nj" event={"ID":"a2862a4d-a528-4766-ac1e-958b2de9ca1d","Type":"ContainerDied","Data":"9a2b9d289d540b5db5c814f5c4b5f5187714aab1febde8fd536cf32217b3c234"}
Nov 02 20:01:33.908070 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:33.907939    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-23a9f5-85b7bb67f-8f9nj"
Nov 02 20:01:33.908070 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:33.907966    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-23a9f5-85b7bb67f-8f9nj" event={"ID":"a2862a4d-a528-4766-ac1e-958b2de9ca1d","Type":"ContainerDied","Data":"d94f5d57f8c3d2840bb38dba2d21551be1aaef066b20e106dab72b1fae2ace7d"}
Nov 02 20:01:33.918240 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:33.918221    4463 scope.go:122] "RemoveContainer" containerID="efed4e8269b553eae37ecf8879f8103c20b91f197f6b855251d5c6233fb78674"
Nov 02 20:01:33.918503 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:01:33.918484    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efed4e8269b553eae37ecf8879f8103c20b91f197f6b855251d5c6233fb78674\": container with ID starting with efed4e8269b553eae37ecf8879f8103c20b91f197f6b855251d5c6233fb78674 not found: ID does not exist" containerID="efed4e8269b553eae37ecf8879f8103c20b91f197f6b855251d5c6233fb78674"
Nov 02 20:01:33.918585 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:33.918513    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efed4e8269b553eae37ecf8879f8103c20b91f197f6b855251d5c6233fb78674"} err="failed to get container status \"efed4e8269b553eae37ecf8879f8103c20b91f197f6b855251d5c6233fb78674\": rpc error: code = NotFound desc = could not find container \"efed4e8269b553eae37ecf8879f8103c20b91f197f6b855251d5c6233fb78674\": container with ID starting with efed4e8269b553eae37ecf8879f8103c20b91f197f6b855251d5c6233fb78674 not found: ID does not exist"
Nov 02 20:01:33.918585 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:33.918538    4463 scope.go:122] "RemoveContainer" containerID="9a2b9d289d540b5db5c814f5c4b5f5187714aab1febde8fd536cf32217b3c234"
Nov 02 20:01:33.927385 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:33.927370    4463 scope.go:122] "RemoveContainer" containerID="9a2b9d289d540b5db5c814f5c4b5f5187714aab1febde8fd536cf32217b3c234"
Nov 02 20:01:33.927618 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:01:33.927598    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a2b9d289d540b5db5c814f5c4b5f5187714aab1febde8fd536cf32217b3c234\": container with ID starting with 9a2b9d289d540b5db5c814f5c4b5f5187714aab1febde8fd536cf32217b3c234 not found: ID does not exist" containerID="9a2b9d289d540b5db5c814f5c4b5f5187714aab1febde8fd536cf32217b3c234"
Nov 02 20:01:33.927662 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:33.927625    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a2b9d289d540b5db5c814f5c4b5f5187714aab1febde8fd536cf32217b3c234"} err="failed to get container status \"9a2b9d289d540b5db5c814f5c4b5f5187714aab1febde8fd536cf32217b3c234\": rpc error: code = NotFound desc = could not find container \"9a2b9d289d540b5db5c814f5c4b5f5187714aab1febde8fd536cf32217b3c234\": container with ID starting with 9a2b9d289d540b5db5c814f5c4b5f5187714aab1febde8fd536cf32217b3c234 not found: ID does not exist"
Nov 02 20:01:34.009364 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:34.009329    4463 patch_prober.go:29] interesting pod/e2e-inf-b-4b0a12-openshift-default-8677ff4fb5-6css2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Nov 02 20:01:34.009539 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:34.009379    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4b0a12-openshift-default-8677ff4fb5-6css2" podUID="277fb120-e642-4215-bc3d-1b9c08c57e81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Nov 02 20:01:34.077430 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:34.077400    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-23a9f5-85b7bb67f-8f9nj"]
Nov 02 20:01:34.082224 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:34.082195    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-23a9f5-85b7bb67f-8f9nj"]
Nov 02 20:01:34.180565 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:34.180533    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-23a9f5-6f4f945579-gplz2"]
Nov 02 20:01:34.184618 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:34.184591    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-23a9f5-6f4f945579-gplz2"]
Nov 02 20:01:34.302189 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:01:34.302149    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 20:01:34.503892 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:34.503791    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-4b0a12-openshift-default-8677ff4fb5-6css2"
Nov 02 20:01:34.841305 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:34.841216    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-csdm8"]
Nov 02 20:01:34.841484 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:34.841446    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-csdm8" podUID="6b51fb09-651f-4dc4-afbd-8734da30febe" containerName="istio-proxy" containerID="cri-o://1a90b50b9718d62aeebd3d406eff50c97330d794e7bf083bf26e8f3b1bf4962a" gracePeriod=30
Nov 02 20:01:35.009271 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:35.009237    4463 patch_prober.go:29] interesting pod/e2e-inf-b-4b0a12-openshift-default-8677ff4fb5-6css2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Nov 02 20:01:35.009458 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:35.009296    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4b0a12-openshift-default-8677ff4fb5-6css2" podUID="277fb120-e642-4215-bc3d-1b9c08c57e81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Nov 02 20:01:35.594916 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:35.594878    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="045e2518-6641-4e5d-a9b7-10326c74b26b" path="/var/lib/kubelet/pods/045e2518-6641-4e5d-a9b7-10326c74b26b/volumes"
Nov 02 20:01:35.595429 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:35.595413    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2862a4d-a528-4766-ac1e-958b2de9ca1d" path="/var/lib/kubelet/pods/a2862a4d-a528-4766-ac1e-958b2de9ca1d/volumes"
Nov 02 20:01:36.009227 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:36.009192    4463 patch_prober.go:29] interesting pod/e2e-inf-b-4b0a12-openshift-default-8677ff4fb5-6css2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Nov 02 20:01:36.009407 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:36.009245    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4b0a12-openshift-default-8677ff4fb5-6css2" podUID="277fb120-e642-4215-bc3d-1b9c08c57e81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Nov 02 20:01:37.014100 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:37.014067    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-4b0a12-openshift-default-8677ff4fb5-6css2"
Nov 02 20:01:37.015223 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:37.015204    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-4b0a12-openshift-default-8677ff4fb5-6css2"
Nov 02 20:01:40.111224 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:40.111159    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-csdm8"
Nov 02 20:01:40.193069 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:40.193002    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6b51fb09-651f-4dc4-afbd-8734da30febe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b51fb09-651f-4dc4-afbd-8734da30febe-istio-podinfo\") pod \"6b51fb09-651f-4dc4-afbd-8734da30febe\" (UID: \"6b51fb09-651f-4dc4-afbd-8734da30febe\") "
Nov 02 20:01:40.193244 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:40.193148    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b51fb09-651f-4dc4-afbd-8734da30febe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b51fb09-651f-4dc4-afbd-8734da30febe-credential-socket\") pod \"6b51fb09-651f-4dc4-afbd-8734da30febe\" (UID: \"6b51fb09-651f-4dc4-afbd-8734da30febe\") "
Nov 02 20:01:40.193357 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:40.193249    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b51fb09-651f-4dc4-afbd-8734da30febe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b51fb09-651f-4dc4-afbd-8734da30febe-workload-socket\") pod \"6b51fb09-651f-4dc4-afbd-8734da30febe\" (UID: \"6b51fb09-651f-4dc4-afbd-8734da30febe\") "
Nov 02 20:01:40.193412 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:40.193370    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b51fb09-651f-4dc4-afbd-8734da30febe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b51fb09-651f-4dc4-afbd-8734da30febe-istio-data\") pod \"6b51fb09-651f-4dc4-afbd-8734da30febe\" (UID: \"6b51fb09-651f-4dc4-afbd-8734da30febe\") "
Nov 02 20:01:40.193467 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:40.193428    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b51fb09-651f-4dc4-afbd-8734da30febe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b51fb09-651f-4dc4-afbd-8734da30febe-workload-certs\") pod \"6b51fb09-651f-4dc4-afbd-8734da30febe\" (UID: \"6b51fb09-651f-4dc4-afbd-8734da30febe\") "
Nov 02 20:01:40.193467 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:40.193458    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b51fb09-651f-4dc4-afbd-8734da30febe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b51fb09-651f-4dc4-afbd-8734da30febe-istio-envoy\") pod \"6b51fb09-651f-4dc4-afbd-8734da30febe\" (UID: \"6b51fb09-651f-4dc4-afbd-8734da30febe\") "
Nov 02 20:01:40.193557 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:40.193499    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b51fb09-651f-4dc4-afbd-8734da30febe-kube-api-access-jggsm\" (UniqueName: \"kubernetes.io/projected/6b51fb09-651f-4dc4-afbd-8734da30febe-kube-api-access-jggsm\") pod \"6b51fb09-651f-4dc4-afbd-8734da30febe\" (UID: \"6b51fb09-651f-4dc4-afbd-8734da30febe\") "
Nov 02 20:01:40.193615 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:40.193587    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b51fb09-651f-4dc4-afbd-8734da30febe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b51fb09-651f-4dc4-afbd-8734da30febe-istiod-ca-cert\") pod \"6b51fb09-651f-4dc4-afbd-8734da30febe\" (UID: \"6b51fb09-651f-4dc4-afbd-8734da30febe\") "
Nov 02 20:01:40.193673 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:40.193457    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b51fb09-651f-4dc4-afbd-8734da30febe-credential-socket" pod "6b51fb09-651f-4dc4-afbd-8734da30febe" (UID: "6b51fb09-651f-4dc4-afbd-8734da30febe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:01:40.193673 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:40.193619    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b51fb09-651f-4dc4-afbd-8734da30febe-istio-token\" (UniqueName: \"kubernetes.io/projected/6b51fb09-651f-4dc4-afbd-8734da30febe-istio-token\") pod \"6b51fb09-651f-4dc4-afbd-8734da30febe\" (UID: \"6b51fb09-651f-4dc4-afbd-8734da30febe\") "
Nov 02 20:01:40.193673 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:40.193524    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b51fb09-651f-4dc4-afbd-8734da30febe-workload-socket" pod "6b51fb09-651f-4dc4-afbd-8734da30febe" (UID: "6b51fb09-651f-4dc4-afbd-8734da30febe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:01:40.193673 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:40.193656    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b51fb09-651f-4dc4-afbd-8734da30febe-workload-certs" pod "6b51fb09-651f-4dc4-afbd-8734da30febe" (UID: "6b51fb09-651f-4dc4-afbd-8734da30febe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:01:40.193855 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:40.193679    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b51fb09-651f-4dc4-afbd-8734da30febe-istio-data" pod "6b51fb09-651f-4dc4-afbd-8734da30febe" (UID: "6b51fb09-651f-4dc4-afbd-8734da30febe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:01:40.193929 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:40.193906    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b51fb09-651f-4dc4-afbd-8734da30febe-istiod-ca-cert" pod "6b51fb09-651f-4dc4-afbd-8734da30febe" (UID: "6b51fb09-651f-4dc4-afbd-8734da30febe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:01:40.194308 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:40.194285    4463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b51fb09-651f-4dc4-afbd-8734da30febe-credential-socket\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:01:40.194409 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:40.194313    4463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b51fb09-651f-4dc4-afbd-8734da30febe-workload-socket\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:01:40.194409 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:40.194342    4463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b51fb09-651f-4dc4-afbd-8734da30febe-istio-data\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:01:40.194409 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:40.194357    4463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b51fb09-651f-4dc4-afbd-8734da30febe-workload-certs\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:01:40.194409 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:40.194371    4463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b51fb09-651f-4dc4-afbd-8734da30febe-istiod-ca-cert\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:01:40.195493 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:40.195469    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6b51fb09-651f-4dc4-afbd-8734da30febe-istio-podinfo" pod "6b51fb09-651f-4dc4-afbd-8734da30febe" (UID: "6b51fb09-651f-4dc4-afbd-8734da30febe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:01:40.195617 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:40.195584    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b51fb09-651f-4dc4-afbd-8734da30febe-istio-envoy" pod "6b51fb09-651f-4dc4-afbd-8734da30febe" (UID: "6b51fb09-651f-4dc4-afbd-8734da30febe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:01:40.195780 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:40.195766    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b51fb09-651f-4dc4-afbd-8734da30febe-istio-token" pod "6b51fb09-651f-4dc4-afbd-8734da30febe" (UID: "6b51fb09-651f-4dc4-afbd-8734da30febe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:01:40.195960 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:40.195941    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b51fb09-651f-4dc4-afbd-8734da30febe-kube-api-access-jggsm" pod "6b51fb09-651f-4dc4-afbd-8734da30febe" (UID: "6b51fb09-651f-4dc4-afbd-8734da30febe"). InnerVolumeSpecName "kube-api-access-jggsm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:01:40.295479 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:40.295436    4463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b51fb09-651f-4dc4-afbd-8734da30febe-istio-envoy\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:01:40.295479 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:40.295474    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jggsm\" (UniqueName: \"kubernetes.io/projected/6b51fb09-651f-4dc4-afbd-8734da30febe-kube-api-access-jggsm\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:01:40.295676 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:40.295488    4463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b51fb09-651f-4dc4-afbd-8734da30febe-istio-token\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:01:40.295676 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:40.295501    4463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b51fb09-651f-4dc4-afbd-8734da30febe-istio-podinfo\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:01:40.960247 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:40.960201    4463 generic.go:356] "Generic (PLEG): container finished" podID="6b51fb09-651f-4dc4-afbd-8734da30febe" containerID="1a90b50b9718d62aeebd3d406eff50c97330d794e7bf083bf26e8f3b1bf4962a" exitCode=0
Nov 02 20:01:40.960453 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:40.960271    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-csdm8"
Nov 02 20:01:40.960453 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:40.960267    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-csdm8" event={"ID":"6b51fb09-651f-4dc4-afbd-8734da30febe","Type":"ContainerDied","Data":"1a90b50b9718d62aeebd3d406eff50c97330d794e7bf083bf26e8f3b1bf4962a"}
Nov 02 20:01:40.960453 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:40.960400    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-csdm8" event={"ID":"6b51fb09-651f-4dc4-afbd-8734da30febe","Type":"ContainerDied","Data":"e05444dc75bda49409548c3c40ae9f5ec3006fdd182fc21c8cebf8bdc36dc4f5"}
Nov 02 20:01:40.960453 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:40.960426    4463 scope.go:122] "RemoveContainer" containerID="1a90b50b9718d62aeebd3d406eff50c97330d794e7bf083bf26e8f3b1bf4962a"
Nov 02 20:01:40.971748 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:40.971728    4463 scope.go:122] "RemoveContainer" containerID="1a90b50b9718d62aeebd3d406eff50c97330d794e7bf083bf26e8f3b1bf4962a"
Nov 02 20:01:40.972020 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:01:40.971984    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a90b50b9718d62aeebd3d406eff50c97330d794e7bf083bf26e8f3b1bf4962a\": container with ID starting with 1a90b50b9718d62aeebd3d406eff50c97330d794e7bf083bf26e8f3b1bf4962a not found: ID does not exist" containerID="1a90b50b9718d62aeebd3d406eff50c97330d794e7bf083bf26e8f3b1bf4962a"
Nov 02 20:01:40.972099 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:40.972043    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a90b50b9718d62aeebd3d406eff50c97330d794e7bf083bf26e8f3b1bf4962a"} err="failed to get container status \"1a90b50b9718d62aeebd3d406eff50c97330d794e7bf083bf26e8f3b1bf4962a\": rpc error: code = NotFound desc = could not find container \"1a90b50b9718d62aeebd3d406eff50c97330d794e7bf083bf26e8f3b1bf4962a\": container with ID starting with 1a90b50b9718d62aeebd3d406eff50c97330d794e7bf083bf26e8f3b1bf4962a not found: ID does not exist"
Nov 02 20:01:41.176447 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:41.176417    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-csdm8"]
Nov 02 20:01:41.221839 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:41.221768    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-csdm8"]
Nov 02 20:01:41.589149 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:41.589071    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b51fb09-651f-4dc4-afbd-8734da30febe" path="/var/lib/kubelet/pods/6b51fb09-651f-4dc4-afbd-8734da30febe/volumes"
Nov 02 20:01:42.471385 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:42.471351    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 20:01:45.425068 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:45.425031    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9f8f23-bbc4d9487-gl2f2"]
Nov 02 20:01:45.425673 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:45.425659    4463 cpu_manager.go:414] "containerMap: removing container" podUID="027cb607-a331-4bc2-86b7-39c85b92860a" containerName="cleanup"
Nov 02 20:01:45.425728 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:45.425674    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="027cb607-a331-4bc2-86b7-39c85b92860a" containerName="cleanup"
Nov 02 20:01:45.425728 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:45.425686    4463 cpu_manager.go:414] "containerMap: removing container" podUID="027cb607-a331-4bc2-86b7-39c85b92860a" containerName="cleanup"
Nov 02 20:01:45.425728 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:45.425692    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="027cb607-a331-4bc2-86b7-39c85b92860a" containerName="cleanup"
Nov 02 20:01:45.425728 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:45.425701    4463 cpu_manager.go:414] "containerMap: removing container" podUID="a211d74d-a663-4c8e-a451-f3253236d6f6" containerName="payload-pre-processing"
Nov 02 20:01:45.425728 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:45.425706    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a211d74d-a663-4c8e-a451-f3253236d6f6" containerName="payload-pre-processing"
Nov 02 20:01:45.425728 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:45.425721    4463 cpu_manager.go:414] "containerMap: removing container" podUID="f9045c38-104c-4065-b369-6f3803fae81e" containerName="payload-processing"
Nov 02 20:01:45.425728 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:45.425726    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9045c38-104c-4065-b369-6f3803fae81e" containerName="payload-processing"
Nov 02 20:01:45.425929 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:45.425735    4463 cpu_manager.go:414] "containerMap: removing container" podUID="5a3f250a-3664-4b0b-b07e-0690f1ad231c" containerName="cleanup"
Nov 02 20:01:45.425929 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:45.425739    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a3f250a-3664-4b0b-b07e-0690f1ad231c" containerName="cleanup"
Nov 02 20:01:45.425929 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:45.425747    4463 cpu_manager.go:414] "containerMap: removing container" podUID="045e2518-6641-4e5d-a9b7-10326c74b26b" containerName="payload-pre-processing"
Nov 02 20:01:45.425929 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:45.425752    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="045e2518-6641-4e5d-a9b7-10326c74b26b" containerName="payload-pre-processing"
Nov 02 20:01:45.425929 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:45.425761    4463 cpu_manager.go:414] "containerMap: removing container" podUID="027cb607-a331-4bc2-86b7-39c85b92860a" containerName="cleanup"
Nov 02 20:01:45.425929 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:45.425765    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="027cb607-a331-4bc2-86b7-39c85b92860a" containerName="cleanup"
Nov 02 20:01:45.425929 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:45.425774    4463 cpu_manager.go:414] "containerMap: removing container" podUID="6b51fb09-651f-4dc4-afbd-8734da30febe" containerName="istio-proxy"
Nov 02 20:01:45.425929 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:45.425778    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b51fb09-651f-4dc4-afbd-8734da30febe" containerName="istio-proxy"
Nov 02 20:01:45.425929 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:45.425786    4463 cpu_manager.go:414] "containerMap: removing container" podUID="5cddfa53-4cf6-49fd-b883-c50e95ff138a" containerName="cleanup"
Nov 02 20:01:45.425929 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:45.425791    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cddfa53-4cf6-49fd-b883-c50e95ff138a" containerName="cleanup"
Nov 02 20:01:45.425929 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:45.425805    4463 cpu_manager.go:414] "containerMap: removing container" podUID="5cddfa53-4cf6-49fd-b883-c50e95ff138a" containerName="cleanup"
Nov 02 20:01:45.425929 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:45.425810    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cddfa53-4cf6-49fd-b883-c50e95ff138a" containerName="cleanup"
Nov 02 20:01:45.425929 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:45.425817    4463 cpu_manager.go:414] "containerMap: removing container" podUID="5a3f250a-3664-4b0b-b07e-0690f1ad231c" containerName="cleanup"
Nov 02 20:01:45.425929 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:45.425821    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a3f250a-3664-4b0b-b07e-0690f1ad231c" containerName="cleanup"
Nov 02 20:01:45.425929 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:45.425827    4463 cpu_manager.go:414] "containerMap: removing container" podUID="5cddfa53-4cf6-49fd-b883-c50e95ff138a" containerName="cleanup"
Nov 02 20:01:45.425929 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:45.425831    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cddfa53-4cf6-49fd-b883-c50e95ff138a" containerName="cleanup"
Nov 02 20:01:45.425929 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:45.425843    4463 cpu_manager.go:414] "containerMap: removing container" podUID="a2862a4d-a528-4766-ac1e-958b2de9ca1d" containerName="payload-processing"
Nov 02 20:01:45.425929 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:45.425847    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2862a4d-a528-4766-ac1e-958b2de9ca1d" containerName="payload-processing"
Nov 02 20:01:45.425929 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:45.425862    4463 cpu_manager.go:414] "containerMap: removing container" podUID="5a3f250a-3664-4b0b-b07e-0690f1ad231c" containerName="cleanup"
Nov 02 20:01:45.425929 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:45.425866    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a3f250a-3664-4b0b-b07e-0690f1ad231c" containerName="cleanup"
Nov 02 20:01:45.426541 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:45.425957    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="027cb607-a331-4bc2-86b7-39c85b92860a" containerName="cleanup"
Nov 02 20:01:45.426541 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:45.425965    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="027cb607-a331-4bc2-86b7-39c85b92860a" containerName="cleanup"
Nov 02 20:01:45.426541 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:45.425972    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="a211d74d-a663-4c8e-a451-f3253236d6f6" containerName="payload-pre-processing"
Nov 02 20:01:45.426541 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:45.425980    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9045c38-104c-4065-b369-6f3803fae81e" containerName="payload-processing"
Nov 02 20:01:45.426541 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:45.425987    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a3f250a-3664-4b0b-b07e-0690f1ad231c" containerName="cleanup"
Nov 02 20:01:45.426541 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:45.425992    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="045e2518-6641-4e5d-a9b7-10326c74b26b" containerName="payload-pre-processing"
Nov 02 20:01:45.426541 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:45.425999    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="027cb607-a331-4bc2-86b7-39c85b92860a" containerName="cleanup"
Nov 02 20:01:45.426541 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:45.426006    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b51fb09-651f-4dc4-afbd-8734da30febe" containerName="istio-proxy"
Nov 02 20:01:45.426541 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:45.426029    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cddfa53-4cf6-49fd-b883-c50e95ff138a" containerName="cleanup"
Nov 02 20:01:45.426541 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:45.426040    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cddfa53-4cf6-49fd-b883-c50e95ff138a" containerName="cleanup"
Nov 02 20:01:45.426541 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:45.426048    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cddfa53-4cf6-49fd-b883-c50e95ff138a" containerName="cleanup"
Nov 02 20:01:45.426541 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:45.426054    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2862a4d-a528-4766-ac1e-958b2de9ca1d" containerName="payload-processing"
Nov 02 20:01:45.426541 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:45.426062    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a3f250a-3664-4b0b-b07e-0690f1ad231c" containerName="cleanup"
Nov 02 20:01:45.429423 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:45.429400    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9f8f23-bbc4d9487-gl2f2"
Nov 02 20:01:45.524143 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:45.524108    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-9f8f23-serving-cert\""
Nov 02 20:01:45.528190 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:45.528162    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9f8f23-bbc4d9487-gl2f2"]
Nov 02 20:01:45.573971 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:45.573935    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6f2209a-00fa-4e46-a0be-85ed51d89369-openshift-service-ca\") pod \"maas-api-e2e-api-a-9f8f23-bbc4d9487-gl2f2\" (UID: \"b6f2209a-00fa-4e46-a0be-85ed51d89369\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9f8f23-bbc4d9487-gl2f2"
Nov 02 20:01:45.574167 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:45.574098    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bl985\" (UniqueName: \"kubernetes.io/projected/b6f2209a-00fa-4e46-a0be-85ed51d89369-kube-api-access-bl985\") pod \"maas-api-e2e-api-a-9f8f23-bbc4d9487-gl2f2\" (UID: \"b6f2209a-00fa-4e46-a0be-85ed51d89369\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9f8f23-bbc4d9487-gl2f2"
Nov 02 20:01:45.574337 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:45.574294    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6f2209a-00fa-4e46-a0be-85ed51d89369-maas-api-tls\") pod \"maas-api-e2e-api-a-9f8f23-bbc4d9487-gl2f2\" (UID: \"b6f2209a-00fa-4e46-a0be-85ed51d89369\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9f8f23-bbc4d9487-gl2f2"
Nov 02 20:01:45.574448 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:45.574432    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6f2209a-00fa-4e46-a0be-85ed51d89369-metrics-certs\") pod \"maas-api-e2e-api-a-9f8f23-bbc4d9487-gl2f2\" (UID: \"b6f2209a-00fa-4e46-a0be-85ed51d89369\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9f8f23-bbc4d9487-gl2f2"
Nov 02 20:01:45.676269 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:45.676172    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6f2209a-00fa-4e46-a0be-85ed51d89369-metrics-certs\") pod \"maas-api-e2e-api-a-9f8f23-bbc4d9487-gl2f2\" (UID: \"b6f2209a-00fa-4e46-a0be-85ed51d89369\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9f8f23-bbc4d9487-gl2f2"
Nov 02 20:01:45.676457 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:45.676398    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6f2209a-00fa-4e46-a0be-85ed51d89369-openshift-service-ca\") pod \"maas-api-e2e-api-a-9f8f23-bbc4d9487-gl2f2\" (UID: \"b6f2209a-00fa-4e46-a0be-85ed51d89369\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9f8f23-bbc4d9487-gl2f2"
Nov 02 20:01:45.676546 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:45.676523    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bl985\" (UniqueName: \"kubernetes.io/projected/b6f2209a-00fa-4e46-a0be-85ed51d89369-kube-api-access-bl985\") pod \"maas-api-e2e-api-a-9f8f23-bbc4d9487-gl2f2\" (UID: \"b6f2209a-00fa-4e46-a0be-85ed51d89369\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9f8f23-bbc4d9487-gl2f2"
Nov 02 20:01:45.676773 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:45.676751    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6f2209a-00fa-4e46-a0be-85ed51d89369-maas-api-tls\") pod \"maas-api-e2e-api-a-9f8f23-bbc4d9487-gl2f2\" (UID: \"b6f2209a-00fa-4e46-a0be-85ed51d89369\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9f8f23-bbc4d9487-gl2f2"
Nov 02 20:01:45.677118 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:45.677100    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6f2209a-00fa-4e46-a0be-85ed51d89369-openshift-service-ca\") pod \"maas-api-e2e-api-a-9f8f23-bbc4d9487-gl2f2\" (UID: \"b6f2209a-00fa-4e46-a0be-85ed51d89369\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9f8f23-bbc4d9487-gl2f2"
Nov 02 20:01:45.678852 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:45.678831    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6f2209a-00fa-4e46-a0be-85ed51d89369-metrics-certs\") pod \"maas-api-e2e-api-a-9f8f23-bbc4d9487-gl2f2\" (UID: \"b6f2209a-00fa-4e46-a0be-85ed51d89369\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9f8f23-bbc4d9487-gl2f2"
Nov 02 20:01:45.678933 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:45.678914    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6f2209a-00fa-4e46-a0be-85ed51d89369-maas-api-tls\") pod \"maas-api-e2e-api-a-9f8f23-bbc4d9487-gl2f2\" (UID: \"b6f2209a-00fa-4e46-a0be-85ed51d89369\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9f8f23-bbc4d9487-gl2f2"
Nov 02 20:01:45.734836 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:45.734799    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bl985\" (UniqueName: \"kubernetes.io/projected/b6f2209a-00fa-4e46-a0be-85ed51d89369-kube-api-access-bl985\") pod \"maas-api-e2e-api-a-9f8f23-bbc4d9487-gl2f2\" (UID: \"b6f2209a-00fa-4e46-a0be-85ed51d89369\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9f8f23-bbc4d9487-gl2f2"
Nov 02 20:01:45.755993 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:45.755962    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9f8f23-bbc4d9487-gl2f2"
Nov 02 20:01:46.132688 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:46.132658    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9f8f23-bbc4d9487-gl2f2"]
Nov 02 20:01:46.133677 ip-10-0-133-9 kubenswrapper[4463]: W1102 20:01:46.133645    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6f2209a_00fa_4e46_a0be_85ed51d89369.slice/crio-f4cafbbf0b11a679d2225021989b4dcb65772ee95a01bd7b5a77858107aa7fa7 WatchSource:0}: Error finding container f4cafbbf0b11a679d2225021989b4dcb65772ee95a01bd7b5a77858107aa7fa7: Status 404 returned error can't find the container with id f4cafbbf0b11a679d2225021989b4dcb65772ee95a01bd7b5a77858107aa7fa7
Nov 02 20:01:47.005497 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:47.005462    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9f8f23-bbc4d9487-gl2f2" event={"ID":"b6f2209a-00fa-4e46-a0be-85ed51d89369","Type":"ContainerStarted","Data":"933cf812d1fe173a7237b4755651b3f9ae1ccbf353c67f48f05b53968c445038"}
Nov 02 20:01:47.005865 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:47.005531    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9f8f23-bbc4d9487-gl2f2" event={"ID":"b6f2209a-00fa-4e46-a0be-85ed51d89369","Type":"ContainerStarted","Data":"f4cafbbf0b11a679d2225021989b4dcb65772ee95a01bd7b5a77858107aa7fa7"}
Nov 02 20:01:47.005865 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:47.005563    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9f8f23-bbc4d9487-gl2f2"
Nov 02 20:01:47.134652 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:47.134599    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9f8f23-bbc4d9487-gl2f2" podStartSLOduration=1.355048607 podStartE2EDuration="2.134586497s" totalImagesPullingTime="779.53789ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:01:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:01:47.133732617 +0000 UTC m=+2013.740766417" watchObservedRunningTime="2026-11-02 20:01:47.134586497 +0000 UTC m=+2013.741620303"
Nov 02 20:01:53.018816 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:53.018781    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9f8f23-bbc4d9487-gl2f2"
Nov 02 20:01:54.154442 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:54.154399    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5hkwc"]
Nov 02 20:01:54.154834 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:54.154701    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5hkwc" podUID="d288950b-84fb-4cef-bb78-0a713bcd14c0" containerName="istio-proxy" containerID="cri-o://9b44e60b60bba2de29171381c72094b7a1d37cc7a93f575d15756f7c4b1f6ad1" gracePeriod=30
Nov 02 20:01:59.445570 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:59.445546    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5hkwc"
Nov 02 20:01:59.609116 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:59.609084    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d288950b-84fb-4cef-bb78-0a713bcd14c0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d288950b-84fb-4cef-bb78-0a713bcd14c0-istiod-ca-cert\") pod \"d288950b-84fb-4cef-bb78-0a713bcd14c0\" (UID: \"d288950b-84fb-4cef-bb78-0a713bcd14c0\") "
Nov 02 20:01:59.609269 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:59.609124    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d288950b-84fb-4cef-bb78-0a713bcd14c0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d288950b-84fb-4cef-bb78-0a713bcd14c0-istio-envoy\") pod \"d288950b-84fb-4cef-bb78-0a713bcd14c0\" (UID: \"d288950b-84fb-4cef-bb78-0a713bcd14c0\") "
Nov 02 20:01:59.609269 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:59.609203    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d288950b-84fb-4cef-bb78-0a713bcd14c0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d288950b-84fb-4cef-bb78-0a713bcd14c0-workload-socket\") pod \"d288950b-84fb-4cef-bb78-0a713bcd14c0\" (UID: \"d288950b-84fb-4cef-bb78-0a713bcd14c0\") "
Nov 02 20:01:59.609269 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:59.609234    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d288950b-84fb-4cef-bb78-0a713bcd14c0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d288950b-84fb-4cef-bb78-0a713bcd14c0-istio-podinfo\") pod \"d288950b-84fb-4cef-bb78-0a713bcd14c0\" (UID: \"d288950b-84fb-4cef-bb78-0a713bcd14c0\") "
Nov 02 20:01:59.609421 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:59.609266    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d288950b-84fb-4cef-bb78-0a713bcd14c0-kube-api-access-zjzb8\" (UniqueName: \"kubernetes.io/projected/d288950b-84fb-4cef-bb78-0a713bcd14c0-kube-api-access-zjzb8\") pod \"d288950b-84fb-4cef-bb78-0a713bcd14c0\" (UID: \"d288950b-84fb-4cef-bb78-0a713bcd14c0\") "
Nov 02 20:01:59.609460 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:59.609421    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d288950b-84fb-4cef-bb78-0a713bcd14c0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d288950b-84fb-4cef-bb78-0a713bcd14c0-credential-socket\") pod \"d288950b-84fb-4cef-bb78-0a713bcd14c0\" (UID: \"d288950b-84fb-4cef-bb78-0a713bcd14c0\") "
Nov 02 20:01:59.609512 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:59.609460    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d288950b-84fb-4cef-bb78-0a713bcd14c0-istiod-ca-cert" pod "d288950b-84fb-4cef-bb78-0a713bcd14c0" (UID: "d288950b-84fb-4cef-bb78-0a713bcd14c0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:01:59.609512 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:59.609467    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d288950b-84fb-4cef-bb78-0a713bcd14c0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d288950b-84fb-4cef-bb78-0a713bcd14c0-istio-data\") pod \"d288950b-84fb-4cef-bb78-0a713bcd14c0\" (UID: \"d288950b-84fb-4cef-bb78-0a713bcd14c0\") "
Nov 02 20:01:59.609622 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:59.609562    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d288950b-84fb-4cef-bb78-0a713bcd14c0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d288950b-84fb-4cef-bb78-0a713bcd14c0-workload-certs\") pod \"d288950b-84fb-4cef-bb78-0a713bcd14c0\" (UID: \"d288950b-84fb-4cef-bb78-0a713bcd14c0\") "
Nov 02 20:01:59.610715 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:59.609706    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d288950b-84fb-4cef-bb78-0a713bcd14c0-credential-socket" pod "d288950b-84fb-4cef-bb78-0a713bcd14c0" (UID: "d288950b-84fb-4cef-bb78-0a713bcd14c0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:01:59.610715 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:59.609795    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d288950b-84fb-4cef-bb78-0a713bcd14c0-istio-token\" (UniqueName: \"kubernetes.io/projected/d288950b-84fb-4cef-bb78-0a713bcd14c0-istio-token\") pod \"d288950b-84fb-4cef-bb78-0a713bcd14c0\" (UID: \"d288950b-84fb-4cef-bb78-0a713bcd14c0\") "
Nov 02 20:01:59.610715 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:59.609795    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d288950b-84fb-4cef-bb78-0a713bcd14c0-workload-socket" pod "d288950b-84fb-4cef-bb78-0a713bcd14c0" (UID: "d288950b-84fb-4cef-bb78-0a713bcd14c0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:01:59.610715 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:59.610086    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d288950b-84fb-4cef-bb78-0a713bcd14c0-workload-certs" pod "d288950b-84fb-4cef-bb78-0a713bcd14c0" (UID: "d288950b-84fb-4cef-bb78-0a713bcd14c0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:01:59.610715 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:59.610157    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d288950b-84fb-4cef-bb78-0a713bcd14c0-istio-data" pod "d288950b-84fb-4cef-bb78-0a713bcd14c0" (UID: "d288950b-84fb-4cef-bb78-0a713bcd14c0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:01:59.610715 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:59.610613    4463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d288950b-84fb-4cef-bb78-0a713bcd14c0-istiod-ca-cert\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:01:59.610715 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:59.610637    4463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d288950b-84fb-4cef-bb78-0a713bcd14c0-workload-socket\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:01:59.610715 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:59.610653    4463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d288950b-84fb-4cef-bb78-0a713bcd14c0-credential-socket\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:01:59.610715 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:59.610668    4463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d288950b-84fb-4cef-bb78-0a713bcd14c0-istio-data\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:01:59.610715 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:59.610683    4463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d288950b-84fb-4cef-bb78-0a713bcd14c0-workload-certs\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:01:59.611721 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:59.611697    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d288950b-84fb-4cef-bb78-0a713bcd14c0-istio-envoy" pod "d288950b-84fb-4cef-bb78-0a713bcd14c0" (UID: "d288950b-84fb-4cef-bb78-0a713bcd14c0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:01:59.612148 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:59.612126    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d288950b-84fb-4cef-bb78-0a713bcd14c0-istio-podinfo" pod "d288950b-84fb-4cef-bb78-0a713bcd14c0" (UID: "d288950b-84fb-4cef-bb78-0a713bcd14c0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:01:59.612237 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:59.612155    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d288950b-84fb-4cef-bb78-0a713bcd14c0-kube-api-access-zjzb8" pod "d288950b-84fb-4cef-bb78-0a713bcd14c0" (UID: "d288950b-84fb-4cef-bb78-0a713bcd14c0"). InnerVolumeSpecName "kube-api-access-zjzb8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:01:59.612283 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:59.612260    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d288950b-84fb-4cef-bb78-0a713bcd14c0-istio-token" pod "d288950b-84fb-4cef-bb78-0a713bcd14c0" (UID: "d288950b-84fb-4cef-bb78-0a713bcd14c0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:01:59.712108 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:59.712022    4463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d288950b-84fb-4cef-bb78-0a713bcd14c0-istio-podinfo\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:01:59.712108 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:59.712058    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjzb8\" (UniqueName: \"kubernetes.io/projected/d288950b-84fb-4cef-bb78-0a713bcd14c0-kube-api-access-zjzb8\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:01:59.712108 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:59.712072    4463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d288950b-84fb-4cef-bb78-0a713bcd14c0-istio-token\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:01:59.712108 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:01:59.712084    4463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d288950b-84fb-4cef-bb78-0a713bcd14c0-istio-envoy\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:02:00.100532 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:02:00.100434    4463 generic.go:356] "Generic (PLEG): container finished" podID="d288950b-84fb-4cef-bb78-0a713bcd14c0" containerID="9b44e60b60bba2de29171381c72094b7a1d37cc7a93f575d15756f7c4b1f6ad1" exitCode=0
Nov 02 20:02:00.100532 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:02:00.100484    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5hkwc"
Nov 02 20:02:00.100532 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:02:00.100504    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5hkwc" event={"ID":"d288950b-84fb-4cef-bb78-0a713bcd14c0","Type":"ContainerDied","Data":"9b44e60b60bba2de29171381c72094b7a1d37cc7a93f575d15756f7c4b1f6ad1"}
Nov 02 20:02:00.100813 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:02:00.100546    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5hkwc" event={"ID":"d288950b-84fb-4cef-bb78-0a713bcd14c0","Type":"ContainerDied","Data":"3dfa420e211f7a5f23c4d4198766667fdc88c1a5875287fb2ffe1b9f2675d4a1"}
Nov 02 20:02:00.100813 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:02:00.100567    4463 scope.go:122] "RemoveContainer" containerID="9b44e60b60bba2de29171381c72094b7a1d37cc7a93f575d15756f7c4b1f6ad1"
Nov 02 20:02:00.113416 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:02:00.113384    4463 scope.go:122] "RemoveContainer" containerID="9b44e60b60bba2de29171381c72094b7a1d37cc7a93f575d15756f7c4b1f6ad1"
Nov 02 20:02:00.113736 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:02:00.113717    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b44e60b60bba2de29171381c72094b7a1d37cc7a93f575d15756f7c4b1f6ad1\": container with ID starting with 9b44e60b60bba2de29171381c72094b7a1d37cc7a93f575d15756f7c4b1f6ad1 not found: ID does not exist" containerID="9b44e60b60bba2de29171381c72094b7a1d37cc7a93f575d15756f7c4b1f6ad1"
Nov 02 20:02:00.113807 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:02:00.113744    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b44e60b60bba2de29171381c72094b7a1d37cc7a93f575d15756f7c4b1f6ad1"} err="failed to get container status \"9b44e60b60bba2de29171381c72094b7a1d37cc7a93f575d15756f7c4b1f6ad1\": rpc error: code = NotFound desc = could not find container \"9b44e60b60bba2de29171381c72094b7a1d37cc7a93f575d15756f7c4b1f6ad1\": container with ID starting with 9b44e60b60bba2de29171381c72094b7a1d37cc7a93f575d15756f7c4b1f6ad1 not found: ID does not exist"
Nov 02 20:02:00.255123 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:02:00.255090    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5hkwc"]
Nov 02 20:02:00.301785 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:02:00.301754    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5hkwc"]
Nov 02 20:02:01.591831 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:02:01.591797    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d288950b-84fb-4cef-bb78-0a713bcd14c0" path="/var/lib/kubelet/pods/d288950b-84fb-4cef-bb78-0a713bcd14c0/volumes"
Nov 02 20:02:04.296419 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:02:04.296383    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 20:02:20.264260 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:02:20.264227    4463 generic.go:356] "Generic (PLEG): container finished" podID="e054dacd-930b-430d-8719-3e22a12ce72f" containerID="6a0ec9645fe5619f3279385f6b436203289097e22f54c65f82cefec790674ba9" exitCode=0
Nov 02 20:02:20.264718 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:02:20.264297    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9f8f23-openshift-default-5dd8c74c6f-n75dv" event={"ID":"e054dacd-930b-430d-8719-3e22a12ce72f","Type":"ContainerDied","Data":"6a0ec9645fe5619f3279385f6b436203289097e22f54c65f82cefec790674ba9"}
Nov 02 20:02:20.264718 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:02:20.264664    4463 scope.go:122] "RemoveContainer" containerID="6a0ec9645fe5619f3279385f6b436203289097e22f54c65f82cefec790674ba9"
Nov 02 20:02:20.266486 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:02:20.266459    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:02:20.266618 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:02:20.266543    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:02:20.266618 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:02:20.266591    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:02:21.273447 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:02:21.273405    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9f8f23-openshift-default-5dd8c74c6f-n75dv" event={"ID":"e054dacd-930b-430d-8719-3e22a12ce72f","Type":"ContainerStarted","Data":"4c3ce16179c1418f5628c810b9784add7afa7f26f147540968e796ed8e35ee07"}
Nov 02 20:02:21.808737 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:02:21.808698    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-9f8f23-openshift-default-5dd8c74c6f-n75dv"
Nov 02 20:02:21.809863 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:02:21.809830    4463 patch_prober.go:29] interesting pod/e2e-api-a-9f8f23-openshift-default-5dd8c74c6f-n75dv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Nov 02 20:02:21.809990 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:02:21.809889    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9f8f23-openshift-default-5dd8c74c6f-n75dv" podUID="e054dacd-930b-430d-8719-3e22a12ce72f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Nov 02 20:02:21.887130 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:02:21.887095    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-9f8f23-openshift-default-5dd8c74c6f-n75dv"
Nov 02 20:02:22.809244 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:02:22.809206    4463 patch_prober.go:29] interesting pod/e2e-api-a-9f8f23-openshift-default-5dd8c74c6f-n75dv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Nov 02 20:02:22.809702 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:02:22.809269    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9f8f23-openshift-default-5dd8c74c6f-n75dv" podUID="e054dacd-930b-430d-8719-3e22a12ce72f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Nov 02 20:02:23.809181 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:02:23.809145    4463 patch_prober.go:29] interesting pod/e2e-api-a-9f8f23-openshift-default-5dd8c74c6f-n75dv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Nov 02 20:02:23.809382 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:02:23.809236    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9f8f23-openshift-default-5dd8c74c6f-n75dv" podUID="e054dacd-930b-430d-8719-3e22a12ce72f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Nov 02 20:02:24.809164 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:02:24.809125    4463 patch_prober.go:29] interesting pod/e2e-api-a-9f8f23-openshift-default-5dd8c74c6f-n75dv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Nov 02 20:02:24.809339 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:02:24.809177    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9f8f23-openshift-default-5dd8c74c6f-n75dv" podUID="e054dacd-930b-430d-8719-3e22a12ce72f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Nov 02 20:02:25.809423 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:02:25.809385    4463 patch_prober.go:29] interesting pod/e2e-api-a-9f8f23-openshift-default-5dd8c74c6f-n75dv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Nov 02 20:02:25.809809 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:02:25.809436    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9f8f23-openshift-default-5dd8c74c6f-n75dv" podUID="e054dacd-930b-430d-8719-3e22a12ce72f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Nov 02 20:02:26.809089 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:02:26.809047    4463 patch_prober.go:29] interesting pod/e2e-api-a-9f8f23-openshift-default-5dd8c74c6f-n75dv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Nov 02 20:02:26.809288 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:02:26.809108    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9f8f23-openshift-default-5dd8c74c6f-n75dv" podUID="e054dacd-930b-430d-8719-3e22a12ce72f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Nov 02 20:02:27.813513 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:02:27.813480    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-9f8f23-openshift-default-5dd8c74c6f-n75dv"
Nov 02 20:02:27.814634 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:02:27.814614    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-9f8f23-openshift-default-5dd8c74c6f-n75dv"
Nov 02 20:02:34.295495 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:02:34.295460    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 20:02:53.943328 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:02:53.943292    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-8479ff-openshift-default-6bb884bd67-wnm6b"]
Nov 02 20:02:53.943854 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:02:53.943506    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-8479ff-openshift-default-6bb884bd67-wnm6b" podUID="2f18cb37-937a-4cc9-9a20-f44f2f693d52" containerName="istio-proxy" containerID="cri-o://85b8be1ea4a46ccdb495ac05bb1eeb498f9506f7b9c3241f4085a200ed733f5e" gracePeriod=30
Nov 02 20:02:54.903204 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:02:54.903168    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 20:02:56.665967 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:02:56.665930    4463 patch_prober.go:29] interesting pod/e2e-inf-a-8479ff-openshift-default-6bb884bd67-wnm6b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 20:02:56.666450 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:02:56.666002    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-8479ff-openshift-default-6bb884bd67-wnm6b" podUID="2f18cb37-937a-4cc9-9a20-f44f2f693d52" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 20:02:59.226369 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:02:59.226344    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8479ff-openshift-default-6bb884bd67-wnm6b"
Nov 02 20:02:59.256376 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:02:59.256348    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f18cb37-937a-4cc9-9a20-f44f2f693d52-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f18cb37-937a-4cc9-9a20-f44f2f693d52-credential-socket\") pod \"2f18cb37-937a-4cc9-9a20-f44f2f693d52\" (UID: \"2f18cb37-937a-4cc9-9a20-f44f2f693d52\") "
Nov 02 20:02:59.256486 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:02:59.256391    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f18cb37-937a-4cc9-9a20-f44f2f693d52-istio-token\" (UniqueName: \"kubernetes.io/projected/2f18cb37-937a-4cc9-9a20-f44f2f693d52-istio-token\") pod \"2f18cb37-937a-4cc9-9a20-f44f2f693d52\" (UID: \"2f18cb37-937a-4cc9-9a20-f44f2f693d52\") "
Nov 02 20:02:59.256535 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:02:59.256518    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f18cb37-937a-4cc9-9a20-f44f2f693d52-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f18cb37-937a-4cc9-9a20-f44f2f693d52-workload-certs\") pod \"2f18cb37-937a-4cc9-9a20-f44f2f693d52\" (UID: \"2f18cb37-937a-4cc9-9a20-f44f2f693d52\") "
Nov 02 20:02:59.256571 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:02:59.256555    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f18cb37-937a-4cc9-9a20-f44f2f693d52-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f18cb37-937a-4cc9-9a20-f44f2f693d52-workload-socket\") pod \"2f18cb37-937a-4cc9-9a20-f44f2f693d52\" (UID: \"2f18cb37-937a-4cc9-9a20-f44f2f693d52\") "
Nov 02 20:02:59.256653 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:02:59.256623    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f18cb37-937a-4cc9-9a20-f44f2f693d52-credential-socket" pod "2f18cb37-937a-4cc9-9a20-f44f2f693d52" (UID: "2f18cb37-937a-4cc9-9a20-f44f2f693d52"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:02:59.256783 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:02:59.256624    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2f18cb37-937a-4cc9-9a20-f44f2f693d52-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f18cb37-937a-4cc9-9a20-f44f2f693d52-istio-podinfo\") pod \"2f18cb37-937a-4cc9-9a20-f44f2f693d52\" (UID: \"2f18cb37-937a-4cc9-9a20-f44f2f693d52\") "
Nov 02 20:02:59.256783 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:02:59.256750    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f18cb37-937a-4cc9-9a20-f44f2f693d52-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f18cb37-937a-4cc9-9a20-f44f2f693d52-istio-envoy\") pod \"2f18cb37-937a-4cc9-9a20-f44f2f693d52\" (UID: \"2f18cb37-937a-4cc9-9a20-f44f2f693d52\") "
Nov 02 20:02:59.256886 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:02:59.256807    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f18cb37-937a-4cc9-9a20-f44f2f693d52-kube-api-access-qnvg8\" (UniqueName: \"kubernetes.io/projected/2f18cb37-937a-4cc9-9a20-f44f2f693d52-kube-api-access-qnvg8\") pod \"2f18cb37-937a-4cc9-9a20-f44f2f693d52\" (UID: \"2f18cb37-937a-4cc9-9a20-f44f2f693d52\") "
Nov 02 20:02:59.256886 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:02:59.256818    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f18cb37-937a-4cc9-9a20-f44f2f693d52-workload-certs" pod "2f18cb37-937a-4cc9-9a20-f44f2f693d52" (UID: "2f18cb37-937a-4cc9-9a20-f44f2f693d52"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:02:59.256886 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:02:59.256843    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f18cb37-937a-4cc9-9a20-f44f2f693d52-workload-socket" pod "2f18cb37-937a-4cc9-9a20-f44f2f693d52" (UID: "2f18cb37-937a-4cc9-9a20-f44f2f693d52"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:02:59.257074 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:02:59.256904    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f18cb37-937a-4cc9-9a20-f44f2f693d52-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f18cb37-937a-4cc9-9a20-f44f2f693d52-istiod-ca-cert\") pod \"2f18cb37-937a-4cc9-9a20-f44f2f693d52\" (UID: \"2f18cb37-937a-4cc9-9a20-f44f2f693d52\") "
Nov 02 20:02:59.257074 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:02:59.256954    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f18cb37-937a-4cc9-9a20-f44f2f693d52-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f18cb37-937a-4cc9-9a20-f44f2f693d52-istio-data\") pod \"2f18cb37-937a-4cc9-9a20-f44f2f693d52\" (UID: \"2f18cb37-937a-4cc9-9a20-f44f2f693d52\") "
Nov 02 20:02:59.257261 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:02:59.257237    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f18cb37-937a-4cc9-9a20-f44f2f693d52-istio-data" pod "2f18cb37-937a-4cc9-9a20-f44f2f693d52" (UID: "2f18cb37-937a-4cc9-9a20-f44f2f693d52"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:02:59.257326 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:02:59.257262    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f18cb37-937a-4cc9-9a20-f44f2f693d52-istiod-ca-cert" pod "2f18cb37-937a-4cc9-9a20-f44f2f693d52" (UID: "2f18cb37-937a-4cc9-9a20-f44f2f693d52"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:02:59.257796 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:02:59.257772    4463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f18cb37-937a-4cc9-9a20-f44f2f693d52-workload-certs\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:02:59.257847 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:02:59.257804    4463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f18cb37-937a-4cc9-9a20-f44f2f693d52-workload-socket\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:02:59.257847 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:02:59.257821    4463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f18cb37-937a-4cc9-9a20-f44f2f693d52-istiod-ca-cert\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:02:59.257847 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:02:59.257835    4463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f18cb37-937a-4cc9-9a20-f44f2f693d52-istio-data\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:02:59.257952 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:02:59.257849    4463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f18cb37-937a-4cc9-9a20-f44f2f693d52-credential-socket\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:02:59.258779 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:02:59.258754    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f18cb37-937a-4cc9-9a20-f44f2f693d52-istio-envoy" pod "2f18cb37-937a-4cc9-9a20-f44f2f693d52" (UID: "2f18cb37-937a-4cc9-9a20-f44f2f693d52"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:02:59.258779 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:02:59.258769    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2f18cb37-937a-4cc9-9a20-f44f2f693d52-istio-podinfo" pod "2f18cb37-937a-4cc9-9a20-f44f2f693d52" (UID: "2f18cb37-937a-4cc9-9a20-f44f2f693d52"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:02:59.259099 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:02:59.259078    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f18cb37-937a-4cc9-9a20-f44f2f693d52-istio-token" pod "2f18cb37-937a-4cc9-9a20-f44f2f693d52" (UID: "2f18cb37-937a-4cc9-9a20-f44f2f693d52"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:02:59.259156 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:02:59.259108    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f18cb37-937a-4cc9-9a20-f44f2f693d52-kube-api-access-qnvg8" pod "2f18cb37-937a-4cc9-9a20-f44f2f693d52" (UID: "2f18cb37-937a-4cc9-9a20-f44f2f693d52"). InnerVolumeSpecName "kube-api-access-qnvg8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:02:59.358848 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:02:59.358817    4463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f18cb37-937a-4cc9-9a20-f44f2f693d52-istio-podinfo\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:02:59.358848 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:02:59.358841    4463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f18cb37-937a-4cc9-9a20-f44f2f693d52-istio-envoy\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:02:59.358848 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:02:59.358852    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnvg8\" (UniqueName: \"kubernetes.io/projected/2f18cb37-937a-4cc9-9a20-f44f2f693d52-kube-api-access-qnvg8\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:02:59.359118 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:02:59.358861    4463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f18cb37-937a-4cc9-9a20-f44f2f693d52-istio-token\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:02:59.597051 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:02:59.596992    4463 generic.go:356] "Generic (PLEG): container finished" podID="2f18cb37-937a-4cc9-9a20-f44f2f693d52" containerID="85b8be1ea4a46ccdb495ac05bb1eeb498f9506f7b9c3241f4085a200ed733f5e" exitCode=0
Nov 02 20:02:59.597202 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:02:59.597061    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8479ff-openshift-default-6bb884bd67-wnm6b" event={"ID":"2f18cb37-937a-4cc9-9a20-f44f2f693d52","Type":"ContainerDied","Data":"85b8be1ea4a46ccdb495ac05bb1eeb498f9506f7b9c3241f4085a200ed733f5e"}
Nov 02 20:02:59.597202 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:02:59.597116    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8479ff-openshift-default-6bb884bd67-wnm6b" event={"ID":"2f18cb37-937a-4cc9-9a20-f44f2f693d52","Type":"ContainerDied","Data":"4c287599baffdd2f0a706c1a91006fadec80acb1568f93a0ae53965c9e0e221a"}
Nov 02 20:02:59.597202 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:02:59.597138    4463 scope.go:122] "RemoveContainer" containerID="85b8be1ea4a46ccdb495ac05bb1eeb498f9506f7b9c3241f4085a200ed733f5e"
Nov 02 20:02:59.597202 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:02:59.597072    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8479ff-openshift-default-6bb884bd67-wnm6b"
Nov 02 20:02:59.607519 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:02:59.607504    4463 scope.go:122] "RemoveContainer" containerID="d43aa87648a2daa10355da0054488ebb075beb0699084bcd11eb0888142a0db5"
Nov 02 20:02:59.617092 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:02:59.617001    4463 scope.go:122] "RemoveContainer" containerID="85b8be1ea4a46ccdb495ac05bb1eeb498f9506f7b9c3241f4085a200ed733f5e"
Nov 02 20:02:59.617366 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:02:59.617345    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85b8be1ea4a46ccdb495ac05bb1eeb498f9506f7b9c3241f4085a200ed733f5e\": container with ID starting with 85b8be1ea4a46ccdb495ac05bb1eeb498f9506f7b9c3241f4085a200ed733f5e not found: ID does not exist" containerID="85b8be1ea4a46ccdb495ac05bb1eeb498f9506f7b9c3241f4085a200ed733f5e"
Nov 02 20:02:59.617414 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:02:59.617375    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85b8be1ea4a46ccdb495ac05bb1eeb498f9506f7b9c3241f4085a200ed733f5e"} err="failed to get container status \"85b8be1ea4a46ccdb495ac05bb1eeb498f9506f7b9c3241f4085a200ed733f5e\": rpc error: code = NotFound desc = could not find container \"85b8be1ea4a46ccdb495ac05bb1eeb498f9506f7b9c3241f4085a200ed733f5e\": container with ID starting with 85b8be1ea4a46ccdb495ac05bb1eeb498f9506f7b9c3241f4085a200ed733f5e not found: ID does not exist"
Nov 02 20:02:59.617414 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:02:59.617394    4463 scope.go:122] "RemoveContainer" containerID="d43aa87648a2daa10355da0054488ebb075beb0699084bcd11eb0888142a0db5"
Nov 02 20:02:59.617633 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:02:59.617615    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d43aa87648a2daa10355da0054488ebb075beb0699084bcd11eb0888142a0db5\": container with ID starting with d43aa87648a2daa10355da0054488ebb075beb0699084bcd11eb0888142a0db5 not found: ID does not exist" containerID="d43aa87648a2daa10355da0054488ebb075beb0699084bcd11eb0888142a0db5"
Nov 02 20:02:59.617678 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:02:59.617640    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d43aa87648a2daa10355da0054488ebb075beb0699084bcd11eb0888142a0db5"} err="failed to get container status \"d43aa87648a2daa10355da0054488ebb075beb0699084bcd11eb0888142a0db5\": rpc error: code = NotFound desc = could not find container \"d43aa87648a2daa10355da0054488ebb075beb0699084bcd11eb0888142a0db5\": container with ID starting with d43aa87648a2daa10355da0054488ebb075beb0699084bcd11eb0888142a0db5 not found: ID does not exist"
Nov 02 20:02:59.751284 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:02:59.751255    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-8479ff-openshift-default-6bb884bd67-wnm6b"]
Nov 02 20:02:59.796921 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:02:59.796893    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-8479ff-openshift-default-6bb884bd67-wnm6b"]
Nov 02 20:03:01.585670 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:01.585636    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f18cb37-937a-4cc9-9a20-f44f2f693d52" path="/var/lib/kubelet/pods/2f18cb37-937a-4cc9-9a20-f44f2f693d52/volumes"
Nov 02 20:03:04.297817 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:03:04.297780    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 20:03:09.162690 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:09.162610    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-23a9f5-798d467749-kxrt9"]
Nov 02 20:03:09.163145 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:09.162813    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-23a9f5-798d467749-kxrt9" podUID="254c92da-c945-42b8-931b-791c60405784" containerName="maas-api" containerID="cri-o://ca1718832bf61951c9f35356a699b0b98184c520e414d2485f686408a576b53f" gracePeriod=30
Nov 02 20:03:09.589355 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:09.589325    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-23a9f5-798d467749-kxrt9"
Nov 02 20:03:09.668116 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:09.668090    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-23a9f5-798d467749-kxrt9"
Nov 02 20:03:09.668728 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:09.668080    4463 generic.go:356] "Generic (PLEG): container finished" podID="254c92da-c945-42b8-931b-791c60405784" containerID="ca1718832bf61951c9f35356a699b0b98184c520e414d2485f686408a576b53f" exitCode=0
Nov 02 20:03:09.668728 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:09.668114    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-23a9f5-798d467749-kxrt9" event={"ID":"254c92da-c945-42b8-931b-791c60405784","Type":"ContainerDied","Data":"ca1718832bf61951c9f35356a699b0b98184c520e414d2485f686408a576b53f"}
Nov 02 20:03:09.668728 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:09.668620    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-23a9f5-798d467749-kxrt9" event={"ID":"254c92da-c945-42b8-931b-791c60405784","Type":"ContainerDied","Data":"db8fcaf400e6cc245d65beaf5057ac98227a7482a8f2d60031867f9ad16c8886"}
Nov 02 20:03:09.668728 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:09.668676    4463 scope.go:122] "RemoveContainer" containerID="ca1718832bf61951c9f35356a699b0b98184c520e414d2485f686408a576b53f"
Nov 02 20:03:09.679956 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:09.679935    4463 scope.go:122] "RemoveContainer" containerID="ca1718832bf61951c9f35356a699b0b98184c520e414d2485f686408a576b53f"
Nov 02 20:03:09.680352 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:03:09.680316    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca1718832bf61951c9f35356a699b0b98184c520e414d2485f686408a576b53f\": container with ID starting with ca1718832bf61951c9f35356a699b0b98184c520e414d2485f686408a576b53f not found: ID does not exist" containerID="ca1718832bf61951c9f35356a699b0b98184c520e414d2485f686408a576b53f"
Nov 02 20:03:09.680487 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:09.680360    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca1718832bf61951c9f35356a699b0b98184c520e414d2485f686408a576b53f"} err="failed to get container status \"ca1718832bf61951c9f35356a699b0b98184c520e414d2485f686408a576b53f\": rpc error: code = NotFound desc = could not find container \"ca1718832bf61951c9f35356a699b0b98184c520e414d2485f686408a576b53f\": container with ID starting with ca1718832bf61951c9f35356a699b0b98184c520e414d2485f686408a576b53f not found: ID does not exist"
Nov 02 20:03:09.721656 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:09.721619    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/254c92da-c945-42b8-931b-791c60405784-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/254c92da-c945-42b8-931b-791c60405784-openshift-service-ca\") pod \"254c92da-c945-42b8-931b-791c60405784\" (UID: \"254c92da-c945-42b8-931b-791c60405784\") "
Nov 02 20:03:09.721829 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:09.721778    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/254c92da-c945-42b8-931b-791c60405784-kube-api-access-knq2m\" (UniqueName: \"kubernetes.io/projected/254c92da-c945-42b8-931b-791c60405784-kube-api-access-knq2m\") pod \"254c92da-c945-42b8-931b-791c60405784\" (UID: \"254c92da-c945-42b8-931b-791c60405784\") "
Nov 02 20:03:09.721925 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:09.721908    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/254c92da-c945-42b8-931b-791c60405784-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/254c92da-c945-42b8-931b-791c60405784-maas-api-tls\") pod \"254c92da-c945-42b8-931b-791c60405784\" (UID: \"254c92da-c945-42b8-931b-791c60405784\") "
Nov 02 20:03:09.721989 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:09.721972    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/254c92da-c945-42b8-931b-791c60405784-metrics-certs\" (UniqueName: \"kubernetes.io/secret/254c92da-c945-42b8-931b-791c60405784-metrics-certs\") pod \"254c92da-c945-42b8-931b-791c60405784\" (UID: \"254c92da-c945-42b8-931b-791c60405784\") "
Nov 02 20:03:09.722113 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:09.722043    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/254c92da-c945-42b8-931b-791c60405784-openshift-service-ca" pod "254c92da-c945-42b8-931b-791c60405784" (UID: "254c92da-c945-42b8-931b-791c60405784"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:03:09.723130 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:09.723104    4463 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/254c92da-c945-42b8-931b-791c60405784-openshift-service-ca\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:03:09.724573 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:09.724552    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/254c92da-c945-42b8-931b-791c60405784-kube-api-access-knq2m" pod "254c92da-c945-42b8-931b-791c60405784" (UID: "254c92da-c945-42b8-931b-791c60405784"). InnerVolumeSpecName "kube-api-access-knq2m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:03:09.726136 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:09.726103    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/254c92da-c945-42b8-931b-791c60405784-metrics-certs" pod "254c92da-c945-42b8-931b-791c60405784" (UID: "254c92da-c945-42b8-931b-791c60405784"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:03:09.726220 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:09.726154    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/254c92da-c945-42b8-931b-791c60405784-maas-api-tls" pod "254c92da-c945-42b8-931b-791c60405784" (UID: "254c92da-c945-42b8-931b-791c60405784"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:03:09.825313 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:09.825235    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knq2m\" (UniqueName: \"kubernetes.io/projected/254c92da-c945-42b8-931b-791c60405784-kube-api-access-knq2m\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:03:09.825313 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:09.825263    4463 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/254c92da-c945-42b8-931b-791c60405784-maas-api-tls\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:03:09.825313 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:09.825275    4463 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/254c92da-c945-42b8-931b-791c60405784-metrics-certs\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:03:10.165904 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:10.165807    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-23a9f5-798d467749-kxrt9"]
Nov 02 20:03:10.214665 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:10.213530    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-23a9f5-798d467749-kxrt9"]
Nov 02 20:03:10.324116 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:10.324080    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-4fd0d5-5b9fb6dd49-cp9lc"]
Nov 02 20:03:11.586282 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:11.586248    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="254c92da-c945-42b8-931b-791c60405784" path="/var/lib/kubelet/pods/254c92da-c945-42b8-931b-791c60405784/volumes"
Nov 02 20:03:15.324729 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:15.324688    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-4fd0d5-5b9fb6dd49-cp9lc" podUID="fb9cc982-8fba-4a27-a0c4-71504d971705" containerName="payload-processing" containerID="cri-o://882a9118b1def71280d9eefac68a4ec97868197d521bed5497d193d296651d9a" gracePeriod=25
Nov 02 20:03:15.632491 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:15.632466    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-4fd0d5-5b9fb6dd49-cp9lc"
Nov 02 20:03:15.674604 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:15.674579    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb9cc982-8fba-4a27-a0c4-71504d971705-kube-api-access-sj47m\" (UniqueName: \"kubernetes.io/projected/fb9cc982-8fba-4a27-a0c4-71504d971705-kube-api-access-sj47m\") pod \"fb9cc982-8fba-4a27-a0c4-71504d971705\" (UID: \"fb9cc982-8fba-4a27-a0c4-71504d971705\") "
Nov 02 20:03:15.674739 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:15.674716    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb9cc982-8fba-4a27-a0c4-71504d971705-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb9cc982-8fba-4a27-a0c4-71504d971705-plugins-config-volume\") pod \"fb9cc982-8fba-4a27-a0c4-71504d971705\" (UID: \"fb9cc982-8fba-4a27-a0c4-71504d971705\") "
Nov 02 20:03:15.675112 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:15.675089    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb9cc982-8fba-4a27-a0c4-71504d971705-plugins-config-volume" pod "fb9cc982-8fba-4a27-a0c4-71504d971705" (UID: "fb9cc982-8fba-4a27-a0c4-71504d971705"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:03:15.675289 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:15.675271    4463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb9cc982-8fba-4a27-a0c4-71504d971705-plugins-config-volume\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:03:15.676698 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:15.676673    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb9cc982-8fba-4a27-a0c4-71504d971705-kube-api-access-sj47m" pod "fb9cc982-8fba-4a27-a0c4-71504d971705" (UID: "fb9cc982-8fba-4a27-a0c4-71504d971705"). InnerVolumeSpecName "kube-api-access-sj47m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:03:15.711333 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:15.711305    4463 generic.go:356] "Generic (PLEG): container finished" podID="fb9cc982-8fba-4a27-a0c4-71504d971705" containerID="882a9118b1def71280d9eefac68a4ec97868197d521bed5497d193d296651d9a" exitCode=0
Nov 02 20:03:15.711431 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:15.711352    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-4fd0d5-5b9fb6dd49-cp9lc"
Nov 02 20:03:15.711431 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:15.711393    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-4fd0d5-5b9fb6dd49-cp9lc" event={"ID":"fb9cc982-8fba-4a27-a0c4-71504d971705","Type":"ContainerDied","Data":"882a9118b1def71280d9eefac68a4ec97868197d521bed5497d193d296651d9a"}
Nov 02 20:03:15.711522 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:15.711447    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-4fd0d5-5b9fb6dd49-cp9lc" event={"ID":"fb9cc982-8fba-4a27-a0c4-71504d971705","Type":"ContainerDied","Data":"985f2f9e64be592c74f6e4e83990af302563cc2cd801e59986e6524d371f9862"}
Nov 02 20:03:15.711522 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:15.711472    4463 scope.go:122] "RemoveContainer" containerID="882a9118b1def71280d9eefac68a4ec97868197d521bed5497d193d296651d9a"
Nov 02 20:03:15.721678 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:15.721662    4463 scope.go:122] "RemoveContainer" containerID="882a9118b1def71280d9eefac68a4ec97868197d521bed5497d193d296651d9a"
Nov 02 20:03:15.721927 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:03:15.721911    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"882a9118b1def71280d9eefac68a4ec97868197d521bed5497d193d296651d9a\": container with ID starting with 882a9118b1def71280d9eefac68a4ec97868197d521bed5497d193d296651d9a not found: ID does not exist" containerID="882a9118b1def71280d9eefac68a4ec97868197d521bed5497d193d296651d9a"
Nov 02 20:03:15.721983 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:15.721932    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"882a9118b1def71280d9eefac68a4ec97868197d521bed5497d193d296651d9a"} err="failed to get container status \"882a9118b1def71280d9eefac68a4ec97868197d521bed5497d193d296651d9a\": rpc error: code = NotFound desc = could not find container \"882a9118b1def71280d9eefac68a4ec97868197d521bed5497d193d296651d9a\": container with ID starting with 882a9118b1def71280d9eefac68a4ec97868197d521bed5497d193d296651d9a not found: ID does not exist"
Nov 02 20:03:15.777359 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:15.777325    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sj47m\" (UniqueName: \"kubernetes.io/projected/fb9cc982-8fba-4a27-a0c4-71504d971705-kube-api-access-sj47m\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:03:15.844231 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:15.844199    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-4fd0d5-5b9fb6dd49-cp9lc"]
Nov 02 20:03:15.848153 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:15.848128    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-4fd0d5-5b9fb6dd49-cp9lc"]
Nov 02 20:03:17.588035 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:17.587985    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb9cc982-8fba-4a27-a0c4-71504d971705" path="/var/lib/kubelet/pods/fb9cc982-8fba-4a27-a0c4-71504d971705/volumes"
Nov 02 20:03:17.953065 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:17.953006    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-23a9f5-openshift-default-6d49545b45-h5trw"]
Nov 02 20:03:17.953308 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:17.953249    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-23a9f5-openshift-default-6d49545b45-h5trw" podUID="84a329f5-3f72-46b0-a7f6-eba84a47db5a" containerName="istio-proxy" containerID="cri-o://c15762394abfc9e1844aa9814ca9284c852d0d4e7ed1cda6c682519795e1e1e7" gracePeriod=30
Nov 02 20:03:20.888059 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:20.888008    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9f8f23-bbc4d9487-gl2f2"]
Nov 02 20:03:20.888576 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:20.888208    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9f8f23-bbc4d9487-gl2f2" podUID="b6f2209a-00fa-4e46-a0be-85ed51d89369" containerName="maas-api" containerID="cri-o://933cf812d1fe173a7237b4755651b3f9ae1ccbf353c67f48f05b53968c445038" gracePeriod=30
Nov 02 20:03:21.175323 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:21.175288    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9f8f23-bbc4d9487-gl2f2"
Nov 02 20:03:21.264155 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:21.264126    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6f2209a-00fa-4e46-a0be-85ed51d89369-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6f2209a-00fa-4e46-a0be-85ed51d89369-openshift-service-ca\") pod \"b6f2209a-00fa-4e46-a0be-85ed51d89369\" (UID: \"b6f2209a-00fa-4e46-a0be-85ed51d89369\") "
Nov 02 20:03:21.264344 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:21.264169    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6f2209a-00fa-4e46-a0be-85ed51d89369-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6f2209a-00fa-4e46-a0be-85ed51d89369-maas-api-tls\") pod \"b6f2209a-00fa-4e46-a0be-85ed51d89369\" (UID: \"b6f2209a-00fa-4e46-a0be-85ed51d89369\") "
Nov 02 20:03:21.264344 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:21.264204    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6f2209a-00fa-4e46-a0be-85ed51d89369-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6f2209a-00fa-4e46-a0be-85ed51d89369-metrics-certs\") pod \"b6f2209a-00fa-4e46-a0be-85ed51d89369\" (UID: \"b6f2209a-00fa-4e46-a0be-85ed51d89369\") "
Nov 02 20:03:21.264344 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:21.264250    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6f2209a-00fa-4e46-a0be-85ed51d89369-kube-api-access-bl985\" (UniqueName: \"kubernetes.io/projected/b6f2209a-00fa-4e46-a0be-85ed51d89369-kube-api-access-bl985\") pod \"b6f2209a-00fa-4e46-a0be-85ed51d89369\" (UID: \"b6f2209a-00fa-4e46-a0be-85ed51d89369\") "
Nov 02 20:03:21.264579 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:21.264545    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6f2209a-00fa-4e46-a0be-85ed51d89369-openshift-service-ca" pod "b6f2209a-00fa-4e46-a0be-85ed51d89369" (UID: "b6f2209a-00fa-4e46-a0be-85ed51d89369"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:03:21.265822 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:21.265201    4463 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6f2209a-00fa-4e46-a0be-85ed51d89369-openshift-service-ca\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:03:21.266498 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:21.266473    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6f2209a-00fa-4e46-a0be-85ed51d89369-metrics-certs" pod "b6f2209a-00fa-4e46-a0be-85ed51d89369" (UID: "b6f2209a-00fa-4e46-a0be-85ed51d89369"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:03:21.266827 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:21.266809    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6f2209a-00fa-4e46-a0be-85ed51d89369-kube-api-access-bl985" pod "b6f2209a-00fa-4e46-a0be-85ed51d89369" (UID: "b6f2209a-00fa-4e46-a0be-85ed51d89369"). InnerVolumeSpecName "kube-api-access-bl985". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:03:21.267488 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:21.267466    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6f2209a-00fa-4e46-a0be-85ed51d89369-maas-api-tls" pod "b6f2209a-00fa-4e46-a0be-85ed51d89369" (UID: "b6f2209a-00fa-4e46-a0be-85ed51d89369"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:03:21.367022 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:21.366990    4463 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6f2209a-00fa-4e46-a0be-85ed51d89369-maas-api-tls\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:03:21.367177 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:21.367050    4463 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6f2209a-00fa-4e46-a0be-85ed51d89369-metrics-certs\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:03:21.367177 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:21.367062    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bl985\" (UniqueName: \"kubernetes.io/projected/b6f2209a-00fa-4e46-a0be-85ed51d89369-kube-api-access-bl985\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:03:21.427955 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:21.427925    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-a5b64d-openshift-default-6f8c4599cf-vztxx"]
Nov 02 20:03:21.428609 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:21.428592    4463 cpu_manager.go:414] "containerMap: removing container" podUID="fb9cc982-8fba-4a27-a0c4-71504d971705" containerName="payload-processing"
Nov 02 20:03:21.428609 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:21.428609    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb9cc982-8fba-4a27-a0c4-71504d971705" containerName="payload-processing"
Nov 02 20:03:21.428708 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:21.428631    4463 cpu_manager.go:414] "containerMap: removing container" podUID="254c92da-c945-42b8-931b-791c60405784" containerName="maas-api"
Nov 02 20:03:21.428708 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:21.428636    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="254c92da-c945-42b8-931b-791c60405784" containerName="maas-api"
Nov 02 20:03:21.428708 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:21.428646    4463 cpu_manager.go:414] "containerMap: removing container" podUID="2f18cb37-937a-4cc9-9a20-f44f2f693d52" containerName="istio-proxy"
Nov 02 20:03:21.428708 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:21.428652    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f18cb37-937a-4cc9-9a20-f44f2f693d52" containerName="istio-proxy"
Nov 02 20:03:21.428708 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:21.428666    4463 cpu_manager.go:414] "containerMap: removing container" podUID="d288950b-84fb-4cef-bb78-0a713bcd14c0" containerName="istio-proxy"
Nov 02 20:03:21.428708 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:21.428671    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d288950b-84fb-4cef-bb78-0a713bcd14c0" containerName="istio-proxy"
Nov 02 20:03:21.428708 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:21.428687    4463 cpu_manager.go:414] "containerMap: removing container" podUID="b6f2209a-00fa-4e46-a0be-85ed51d89369" containerName="maas-api"
Nov 02 20:03:21.428708 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:21.428691    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6f2209a-00fa-4e46-a0be-85ed51d89369" containerName="maas-api"
Nov 02 20:03:21.428940 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:21.428791    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6f2209a-00fa-4e46-a0be-85ed51d89369" containerName="maas-api"
Nov 02 20:03:21.428940 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:21.428802    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f18cb37-937a-4cc9-9a20-f44f2f693d52" containerName="istio-proxy"
Nov 02 20:03:21.428940 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:21.428809    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb9cc982-8fba-4a27-a0c4-71504d971705" containerName="payload-processing"
Nov 02 20:03:21.428940 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:21.428819    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="254c92da-c945-42b8-931b-791c60405784" containerName="maas-api"
Nov 02 20:03:21.428940 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:21.428825    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f18cb37-937a-4cc9-9a20-f44f2f693d52" containerName="istio-proxy"
Nov 02 20:03:21.428940 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:21.428832    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="d288950b-84fb-4cef-bb78-0a713bcd14c0" containerName="istio-proxy"
Nov 02 20:03:21.428940 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:21.428838    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a3f250a-3664-4b0b-b07e-0690f1ad231c" containerName="cleanup"
Nov 02 20:03:21.431773 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:21.431756    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a5b64d-openshift-default-6f8c4599cf-vztxx"
Nov 02 20:03:21.546213 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:21.546178    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-a5b64d-openshift-default-dockercfg-7kt6s\""
Nov 02 20:03:21.578031 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:21.573720    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8709c9fb-f685-43fd-a3d5-9cb73cf8329d-istio-data\") pod \"e2e-inf-a-a5b64d-openshift-default-6f8c4599cf-vztxx\" (UID: \"8709c9fb-f685-43fd-a3d5-9cb73cf8329d\") " pod="openshift-ingress/e2e-inf-a-a5b64d-openshift-default-6f8c4599cf-vztxx"
Nov 02 20:03:21.578031 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:21.573793    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8709c9fb-f685-43fd-a3d5-9cb73cf8329d-workload-socket\") pod \"e2e-inf-a-a5b64d-openshift-default-6f8c4599cf-vztxx\" (UID: \"8709c9fb-f685-43fd-a3d5-9cb73cf8329d\") " pod="openshift-ingress/e2e-inf-a-a5b64d-openshift-default-6f8c4599cf-vztxx"
Nov 02 20:03:21.578031 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:21.573929    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8709c9fb-f685-43fd-a3d5-9cb73cf8329d-istio-podinfo\") pod \"e2e-inf-a-a5b64d-openshift-default-6f8c4599cf-vztxx\" (UID: \"8709c9fb-f685-43fd-a3d5-9cb73cf8329d\") " pod="openshift-ingress/e2e-inf-a-a5b64d-openshift-default-6f8c4599cf-vztxx"
Nov 02 20:03:21.578031 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:21.573996    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8709c9fb-f685-43fd-a3d5-9cb73cf8329d-istio-envoy\") pod \"e2e-inf-a-a5b64d-openshift-default-6f8c4599cf-vztxx\" (UID: \"8709c9fb-f685-43fd-a3d5-9cb73cf8329d\") " pod="openshift-ingress/e2e-inf-a-a5b64d-openshift-default-6f8c4599cf-vztxx"
Nov 02 20:03:21.578031 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:21.574064    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8709c9fb-f685-43fd-a3d5-9cb73cf8329d-credential-socket\") pod \"e2e-inf-a-a5b64d-openshift-default-6f8c4599cf-vztxx\" (UID: \"8709c9fb-f685-43fd-a3d5-9cb73cf8329d\") " pod="openshift-ingress/e2e-inf-a-a5b64d-openshift-default-6f8c4599cf-vztxx"
Nov 02 20:03:21.578031 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:21.574094    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8709c9fb-f685-43fd-a3d5-9cb73cf8329d-workload-certs\") pod \"e2e-inf-a-a5b64d-openshift-default-6f8c4599cf-vztxx\" (UID: \"8709c9fb-f685-43fd-a3d5-9cb73cf8329d\") " pod="openshift-ingress/e2e-inf-a-a5b64d-openshift-default-6f8c4599cf-vztxx"
Nov 02 20:03:21.578031 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:21.574164    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8709c9fb-f685-43fd-a3d5-9cb73cf8329d-istio-token\") pod \"e2e-inf-a-a5b64d-openshift-default-6f8c4599cf-vztxx\" (UID: \"8709c9fb-f685-43fd-a3d5-9cb73cf8329d\") " pod="openshift-ingress/e2e-inf-a-a5b64d-openshift-default-6f8c4599cf-vztxx"
Nov 02 20:03:21.578031 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:21.574189    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8709c9fb-f685-43fd-a3d5-9cb73cf8329d-istiod-ca-cert\") pod \"e2e-inf-a-a5b64d-openshift-default-6f8c4599cf-vztxx\" (UID: \"8709c9fb-f685-43fd-a3d5-9cb73cf8329d\") " pod="openshift-ingress/e2e-inf-a-a5b64d-openshift-default-6f8c4599cf-vztxx"
Nov 02 20:03:21.578031 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:21.574244    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tv4f7\" (UniqueName: \"kubernetes.io/projected/8709c9fb-f685-43fd-a3d5-9cb73cf8329d-kube-api-access-tv4f7\") pod \"e2e-inf-a-a5b64d-openshift-default-6f8c4599cf-vztxx\" (UID: \"8709c9fb-f685-43fd-a3d5-9cb73cf8329d\") " pod="openshift-ingress/e2e-inf-a-a5b64d-openshift-default-6f8c4599cf-vztxx"
Nov 02 20:03:21.578031 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:21.575740    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a5b64d-openshift-default-6f8c4599cf-vztxx"]
Nov 02 20:03:21.678113 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:21.677609    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8709c9fb-f685-43fd-a3d5-9cb73cf8329d-workload-socket\") pod \"e2e-inf-a-a5b64d-openshift-default-6f8c4599cf-vztxx\" (UID: \"8709c9fb-f685-43fd-a3d5-9cb73cf8329d\") " pod="openshift-ingress/e2e-inf-a-a5b64d-openshift-default-6f8c4599cf-vztxx"
Nov 02 20:03:21.678113 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:21.677781    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8709c9fb-f685-43fd-a3d5-9cb73cf8329d-istio-podinfo\") pod \"e2e-inf-a-a5b64d-openshift-default-6f8c4599cf-vztxx\" (UID: \"8709c9fb-f685-43fd-a3d5-9cb73cf8329d\") " pod="openshift-ingress/e2e-inf-a-a5b64d-openshift-default-6f8c4599cf-vztxx"
Nov 02 20:03:21.678113 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:21.677829    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8709c9fb-f685-43fd-a3d5-9cb73cf8329d-istio-envoy\") pod \"e2e-inf-a-a5b64d-openshift-default-6f8c4599cf-vztxx\" (UID: \"8709c9fb-f685-43fd-a3d5-9cb73cf8329d\") " pod="openshift-ingress/e2e-inf-a-a5b64d-openshift-default-6f8c4599cf-vztxx"
Nov 02 20:03:21.678113 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:21.677879    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8709c9fb-f685-43fd-a3d5-9cb73cf8329d-credential-socket\") pod \"e2e-inf-a-a5b64d-openshift-default-6f8c4599cf-vztxx\" (UID: \"8709c9fb-f685-43fd-a3d5-9cb73cf8329d\") " pod="openshift-ingress/e2e-inf-a-a5b64d-openshift-default-6f8c4599cf-vztxx"
Nov 02 20:03:21.678113 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:21.677909    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8709c9fb-f685-43fd-a3d5-9cb73cf8329d-workload-certs\") pod \"e2e-inf-a-a5b64d-openshift-default-6f8c4599cf-vztxx\" (UID: \"8709c9fb-f685-43fd-a3d5-9cb73cf8329d\") " pod="openshift-ingress/e2e-inf-a-a5b64d-openshift-default-6f8c4599cf-vztxx"
Nov 02 20:03:21.678113 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:21.678004    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8709c9fb-f685-43fd-a3d5-9cb73cf8329d-istio-token\") pod \"e2e-inf-a-a5b64d-openshift-default-6f8c4599cf-vztxx\" (UID: \"8709c9fb-f685-43fd-a3d5-9cb73cf8329d\") " pod="openshift-ingress/e2e-inf-a-a5b64d-openshift-default-6f8c4599cf-vztxx"
Nov 02 20:03:21.678113 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:21.678051    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8709c9fb-f685-43fd-a3d5-9cb73cf8329d-istiod-ca-cert\") pod \"e2e-inf-a-a5b64d-openshift-default-6f8c4599cf-vztxx\" (UID: \"8709c9fb-f685-43fd-a3d5-9cb73cf8329d\") " pod="openshift-ingress/e2e-inf-a-a5b64d-openshift-default-6f8c4599cf-vztxx"
Nov 02 20:03:21.678113 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:21.678094    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv4f7\" (UniqueName: \"kubernetes.io/projected/8709c9fb-f685-43fd-a3d5-9cb73cf8329d-kube-api-access-tv4f7\") pod \"e2e-inf-a-a5b64d-openshift-default-6f8c4599cf-vztxx\" (UID: \"8709c9fb-f685-43fd-a3d5-9cb73cf8329d\") " pod="openshift-ingress/e2e-inf-a-a5b64d-openshift-default-6f8c4599cf-vztxx"
Nov 02 20:03:21.678635 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:21.678216    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8709c9fb-f685-43fd-a3d5-9cb73cf8329d-workload-socket\") pod \"e2e-inf-a-a5b64d-openshift-default-6f8c4599cf-vztxx\" (UID: \"8709c9fb-f685-43fd-a3d5-9cb73cf8329d\") " pod="openshift-ingress/e2e-inf-a-a5b64d-openshift-default-6f8c4599cf-vztxx"
Nov 02 20:03:21.678635 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:21.678275    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8709c9fb-f685-43fd-a3d5-9cb73cf8329d-istio-data\") pod \"e2e-inf-a-a5b64d-openshift-default-6f8c4599cf-vztxx\" (UID: \"8709c9fb-f685-43fd-a3d5-9cb73cf8329d\") " pod="openshift-ingress/e2e-inf-a-a5b64d-openshift-default-6f8c4599cf-vztxx"
Nov 02 20:03:21.678635 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:21.678505    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8709c9fb-f685-43fd-a3d5-9cb73cf8329d-istio-data\") pod \"e2e-inf-a-a5b64d-openshift-default-6f8c4599cf-vztxx\" (UID: \"8709c9fb-f685-43fd-a3d5-9cb73cf8329d\") " pod="openshift-ingress/e2e-inf-a-a5b64d-openshift-default-6f8c4599cf-vztxx"
Nov 02 20:03:21.678797 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:21.678665    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8709c9fb-f685-43fd-a3d5-9cb73cf8329d-credential-socket\") pod \"e2e-inf-a-a5b64d-openshift-default-6f8c4599cf-vztxx\" (UID: \"8709c9fb-f685-43fd-a3d5-9cb73cf8329d\") " pod="openshift-ingress/e2e-inf-a-a5b64d-openshift-default-6f8c4599cf-vztxx"
Nov 02 20:03:21.678797 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:21.678741    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8709c9fb-f685-43fd-a3d5-9cb73cf8329d-workload-certs\") pod \"e2e-inf-a-a5b64d-openshift-default-6f8c4599cf-vztxx\" (UID: \"8709c9fb-f685-43fd-a3d5-9cb73cf8329d\") " pod="openshift-ingress/e2e-inf-a-a5b64d-openshift-default-6f8c4599cf-vztxx"
Nov 02 20:03:21.679994 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:21.679970    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8709c9fb-f685-43fd-a3d5-9cb73cf8329d-istiod-ca-cert\") pod \"e2e-inf-a-a5b64d-openshift-default-6f8c4599cf-vztxx\" (UID: \"8709c9fb-f685-43fd-a3d5-9cb73cf8329d\") " pod="openshift-ingress/e2e-inf-a-a5b64d-openshift-default-6f8c4599cf-vztxx"
Nov 02 20:03:21.681936 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:21.681912    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8709c9fb-f685-43fd-a3d5-9cb73cf8329d-istio-podinfo\") pod \"e2e-inf-a-a5b64d-openshift-default-6f8c4599cf-vztxx\" (UID: \"8709c9fb-f685-43fd-a3d5-9cb73cf8329d\") " pod="openshift-ingress/e2e-inf-a-a5b64d-openshift-default-6f8c4599cf-vztxx"
Nov 02 20:03:21.682781 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:21.682764    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8709c9fb-f685-43fd-a3d5-9cb73cf8329d-istio-envoy\") pod \"e2e-inf-a-a5b64d-openshift-default-6f8c4599cf-vztxx\" (UID: \"8709c9fb-f685-43fd-a3d5-9cb73cf8329d\") " pod="openshift-ingress/e2e-inf-a-a5b64d-openshift-default-6f8c4599cf-vztxx"
Nov 02 20:03:21.731716 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:21.731681    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8709c9fb-f685-43fd-a3d5-9cb73cf8329d-istio-token\") pod \"e2e-inf-a-a5b64d-openshift-default-6f8c4599cf-vztxx\" (UID: \"8709c9fb-f685-43fd-a3d5-9cb73cf8329d\") " pod="openshift-ingress/e2e-inf-a-a5b64d-openshift-default-6f8c4599cf-vztxx"
Nov 02 20:03:21.731865 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:21.731831    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tv4f7\" (UniqueName: \"kubernetes.io/projected/8709c9fb-f685-43fd-a3d5-9cb73cf8329d-kube-api-access-tv4f7\") pod \"e2e-inf-a-a5b64d-openshift-default-6f8c4599cf-vztxx\" (UID: \"8709c9fb-f685-43fd-a3d5-9cb73cf8329d\") " pod="openshift-ingress/e2e-inf-a-a5b64d-openshift-default-6f8c4599cf-vztxx"
Nov 02 20:03:21.745562 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:21.745538    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a5b64d-openshift-default-6f8c4599cf-vztxx"
Nov 02 20:03:21.754163 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:21.754134    4463 generic.go:356] "Generic (PLEG): container finished" podID="b6f2209a-00fa-4e46-a0be-85ed51d89369" containerID="933cf812d1fe173a7237b4755651b3f9ae1ccbf353c67f48f05b53968c445038" exitCode=0
Nov 02 20:03:21.754277 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:21.754173    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9f8f23-bbc4d9487-gl2f2"
Nov 02 20:03:21.754277 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:21.754208    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9f8f23-bbc4d9487-gl2f2" event={"ID":"b6f2209a-00fa-4e46-a0be-85ed51d89369","Type":"ContainerDied","Data":"933cf812d1fe173a7237b4755651b3f9ae1ccbf353c67f48f05b53968c445038"}
Nov 02 20:03:21.754277 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:21.754261    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9f8f23-bbc4d9487-gl2f2" event={"ID":"b6f2209a-00fa-4e46-a0be-85ed51d89369","Type":"ContainerDied","Data":"f4cafbbf0b11a679d2225021989b4dcb65772ee95a01bd7b5a77858107aa7fa7"}
Nov 02 20:03:21.754472 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:21.754283    4463 scope.go:122] "RemoveContainer" containerID="933cf812d1fe173a7237b4755651b3f9ae1ccbf353c67f48f05b53968c445038"
Nov 02 20:03:21.764576 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:21.764553    4463 scope.go:122] "RemoveContainer" containerID="933cf812d1fe173a7237b4755651b3f9ae1ccbf353c67f48f05b53968c445038"
Nov 02 20:03:21.764888 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:03:21.764866    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"933cf812d1fe173a7237b4755651b3f9ae1ccbf353c67f48f05b53968c445038\": container with ID starting with 933cf812d1fe173a7237b4755651b3f9ae1ccbf353c67f48f05b53968c445038 not found: ID does not exist" containerID="933cf812d1fe173a7237b4755651b3f9ae1ccbf353c67f48f05b53968c445038"
Nov 02 20:03:21.764970 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:21.764901    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"933cf812d1fe173a7237b4755651b3f9ae1ccbf353c67f48f05b53968c445038"} err="failed to get container status \"933cf812d1fe173a7237b4755651b3f9ae1ccbf353c67f48f05b53968c445038\": rpc error: code = NotFound desc = could not find container \"933cf812d1fe173a7237b4755651b3f9ae1ccbf353c67f48f05b53968c445038\": container with ID starting with 933cf812d1fe173a7237b4755651b3f9ae1ccbf353c67f48f05b53968c445038 not found: ID does not exist"
Nov 02 20:03:21.935804 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:21.935725    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9f8f23-bbc4d9487-gl2f2"]
Nov 02 20:03:21.938941 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:21.938917    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9f8f23-bbc4d9487-gl2f2"]
Nov 02 20:03:22.020155 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:22.020122    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-89a22a-7587d7b466-6c9dq"]
Nov 02 20:03:22.020423 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:22.020397    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-89a22a-7587d7b466-6c9dq" podUID="787ed8bc-d29c-4e3b-b661-87d0712a506f" containerName="maas-api" containerID="cri-o://bde8f6b1107382fef39131b4e08043be4dba831b15aac8c46f2e5d1142922261" gracePeriod=30
Nov 02 20:03:22.129854 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:22.129827    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a5b64d-openshift-default-6f8c4599cf-vztxx"]
Nov 02 20:03:22.131362 ip-10-0-133-9 kubenswrapper[4463]: W1102 20:03:22.131332    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8709c9fb_f685_43fd_a3d5_9cb73cf8329d.slice/crio-90e2c6101e009b31b950e182901d59a43a2acd9c64cf590f90ca42949d01c4b8 WatchSource:0}: Error finding container 90e2c6101e009b31b950e182901d59a43a2acd9c64cf590f90ca42949d01c4b8: Status 404 returned error can't find the container with id 90e2c6101e009b31b950e182901d59a43a2acd9c64cf590f90ca42949d01c4b8
Nov 02 20:03:22.134398 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:22.134362    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:03:22.134501 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:22.134430    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:03:22.134501 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:22.134466    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:03:22.317750 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:22.317724    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-89a22a-7587d7b466-6c9dq"
Nov 02 20:03:22.390384 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:22.390355    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/787ed8bc-d29c-4e3b-b661-87d0712a506f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/787ed8bc-d29c-4e3b-b661-87d0712a506f-openshift-service-ca\") pod \"787ed8bc-d29c-4e3b-b661-87d0712a506f\" (UID: \"787ed8bc-d29c-4e3b-b661-87d0712a506f\") "
Nov 02 20:03:22.390558 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:22.390543    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/787ed8bc-d29c-4e3b-b661-87d0712a506f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/787ed8bc-d29c-4e3b-b661-87d0712a506f-maas-api-tls\") pod \"787ed8bc-d29c-4e3b-b661-87d0712a506f\" (UID: \"787ed8bc-d29c-4e3b-b661-87d0712a506f\") "
Nov 02 20:03:22.390606 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:22.390572    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/787ed8bc-d29c-4e3b-b661-87d0712a506f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/787ed8bc-d29c-4e3b-b661-87d0712a506f-metrics-certs\") pod \"787ed8bc-d29c-4e3b-b661-87d0712a506f\" (UID: \"787ed8bc-d29c-4e3b-b661-87d0712a506f\") "
Nov 02 20:03:22.390606 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:22.390600    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/787ed8bc-d29c-4e3b-b661-87d0712a506f-kube-api-access-j9ksk\" (UniqueName: \"kubernetes.io/projected/787ed8bc-d29c-4e3b-b661-87d0712a506f-kube-api-access-j9ksk\") pod \"787ed8bc-d29c-4e3b-b661-87d0712a506f\" (UID: \"787ed8bc-d29c-4e3b-b661-87d0712a506f\") "
Nov 02 20:03:22.390709 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:22.390686    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/787ed8bc-d29c-4e3b-b661-87d0712a506f-openshift-service-ca" pod "787ed8bc-d29c-4e3b-b661-87d0712a506f" (UID: "787ed8bc-d29c-4e3b-b661-87d0712a506f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:03:22.391191 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:22.391168    4463 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/787ed8bc-d29c-4e3b-b661-87d0712a506f-openshift-service-ca\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:03:22.392721 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:22.392696    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/787ed8bc-d29c-4e3b-b661-87d0712a506f-kube-api-access-j9ksk" pod "787ed8bc-d29c-4e3b-b661-87d0712a506f" (UID: "787ed8bc-d29c-4e3b-b661-87d0712a506f"). InnerVolumeSpecName "kube-api-access-j9ksk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:03:22.393151 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:22.393133    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/787ed8bc-d29c-4e3b-b661-87d0712a506f-metrics-certs" pod "787ed8bc-d29c-4e3b-b661-87d0712a506f" (UID: "787ed8bc-d29c-4e3b-b661-87d0712a506f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:03:22.393205 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:22.393147    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/787ed8bc-d29c-4e3b-b661-87d0712a506f-maas-api-tls" pod "787ed8bc-d29c-4e3b-b661-87d0712a506f" (UID: "787ed8bc-d29c-4e3b-b661-87d0712a506f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:03:22.493367 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:22.493222    4463 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/787ed8bc-d29c-4e3b-b661-87d0712a506f-maas-api-tls\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:03:22.493367 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:22.493269    4463 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/787ed8bc-d29c-4e3b-b661-87d0712a506f-metrics-certs\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:03:22.493367 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:22.493285    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9ksk\" (UniqueName: \"kubernetes.io/projected/787ed8bc-d29c-4e3b-b661-87d0712a506f-kube-api-access-j9ksk\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:03:22.766430 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:22.766350    4463 generic.go:356] "Generic (PLEG): container finished" podID="787ed8bc-d29c-4e3b-b661-87d0712a506f" containerID="bde8f6b1107382fef39131b4e08043be4dba831b15aac8c46f2e5d1142922261" exitCode=0
Nov 02 20:03:22.766810 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:22.766656    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-89a22a-7587d7b466-6c9dq"
Nov 02 20:03:22.766960 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:22.766663    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-89a22a-7587d7b466-6c9dq" event={"ID":"787ed8bc-d29c-4e3b-b661-87d0712a506f","Type":"ContainerDied","Data":"bde8f6b1107382fef39131b4e08043be4dba831b15aac8c46f2e5d1142922261"}
Nov 02 20:03:22.767739 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:22.767616    4463 scope.go:122] "RemoveContainer" containerID="bde8f6b1107382fef39131b4e08043be4dba831b15aac8c46f2e5d1142922261"
Nov 02 20:03:22.767879 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:22.767584    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-89a22a-7587d7b466-6c9dq" event={"ID":"787ed8bc-d29c-4e3b-b661-87d0712a506f","Type":"ContainerDied","Data":"e2d03f5046f85e975105f71e3cac6a14df3dfe50a5945f95000aaa941d969eef"}
Nov 02 20:03:22.771280 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:22.770245    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a5b64d-openshift-default-6f8c4599cf-vztxx" event={"ID":"8709c9fb-f685-43fd-a3d5-9cb73cf8329d","Type":"ContainerStarted","Data":"0435c58e164777f9c6652c868d54e806e906fb0b1f35f6619af491e1c538d23b"}
Nov 02 20:03:22.771280 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:22.770313    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a5b64d-openshift-default-6f8c4599cf-vztxx" event={"ID":"8709c9fb-f685-43fd-a3d5-9cb73cf8329d","Type":"ContainerStarted","Data":"90e2c6101e009b31b950e182901d59a43a2acd9c64cf590f90ca42949d01c4b8"}
Nov 02 20:03:22.820186 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:22.819511    4463 scope.go:122] "RemoveContainer" containerID="bde8f6b1107382fef39131b4e08043be4dba831b15aac8c46f2e5d1142922261"
Nov 02 20:03:22.826415 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:03:22.826388    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bde8f6b1107382fef39131b4e08043be4dba831b15aac8c46f2e5d1142922261\": container with ID starting with bde8f6b1107382fef39131b4e08043be4dba831b15aac8c46f2e5d1142922261 not found: ID does not exist" containerID="bde8f6b1107382fef39131b4e08043be4dba831b15aac8c46f2e5d1142922261"
Nov 02 20:03:22.826545 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:22.826422    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bde8f6b1107382fef39131b4e08043be4dba831b15aac8c46f2e5d1142922261"} err="failed to get container status \"bde8f6b1107382fef39131b4e08043be4dba831b15aac8c46f2e5d1142922261\": rpc error: code = NotFound desc = could not find container \"bde8f6b1107382fef39131b4e08043be4dba831b15aac8c46f2e5d1142922261\": container with ID starting with bde8f6b1107382fef39131b4e08043be4dba831b15aac8c46f2e5d1142922261 not found: ID does not exist"
Nov 02 20:03:22.926001 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:22.925950    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-a5b64d-openshift-default-6f8c4599cf-vztxx" podStartSLOduration=1.925936501 podStartE2EDuration="1.925936501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:03:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:03:22.924561599 +0000 UTC m=+2109.531595392" watchObservedRunningTime="2026-11-02 20:03:22.925936501 +0000 UTC m=+2109.532970267"
Nov 02 20:03:23.029812 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:23.029740    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-89a22a-7587d7b466-6c9dq"]
Nov 02 20:03:23.033104 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:23.033079    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-89a22a-7587d7b466-6c9dq"]
Nov 02 20:03:23.333677 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:23.333654    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-23a9f5-openshift-default-6d49545b45-h5trw"
Nov 02 20:03:23.432632 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:23.432600    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84a329f5-3f72-46b0-a7f6-eba84a47db5a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84a329f5-3f72-46b0-a7f6-eba84a47db5a-istiod-ca-cert\") pod \"84a329f5-3f72-46b0-a7f6-eba84a47db5a\" (UID: \"84a329f5-3f72-46b0-a7f6-eba84a47db5a\") "
Nov 02 20:03:23.432807 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:23.432699    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84a329f5-3f72-46b0-a7f6-eba84a47db5a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84a329f5-3f72-46b0-a7f6-eba84a47db5a-workload-certs\") pod \"84a329f5-3f72-46b0-a7f6-eba84a47db5a\" (UID: \"84a329f5-3f72-46b0-a7f6-eba84a47db5a\") "
Nov 02 20:03:23.432807 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:23.432722    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84a329f5-3f72-46b0-a7f6-eba84a47db5a-kube-api-access-wqw5x\" (UniqueName: \"kubernetes.io/projected/84a329f5-3f72-46b0-a7f6-eba84a47db5a-kube-api-access-wqw5x\") pod \"84a329f5-3f72-46b0-a7f6-eba84a47db5a\" (UID: \"84a329f5-3f72-46b0-a7f6-eba84a47db5a\") "
Nov 02 20:03:23.432807 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:23.432760    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84a329f5-3f72-46b0-a7f6-eba84a47db5a-istio-token\" (UniqueName: \"kubernetes.io/projected/84a329f5-3f72-46b0-a7f6-eba84a47db5a-istio-token\") pod \"84a329f5-3f72-46b0-a7f6-eba84a47db5a\" (UID: \"84a329f5-3f72-46b0-a7f6-eba84a47db5a\") "
Nov 02 20:03:23.432939 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:23.432823    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84a329f5-3f72-46b0-a7f6-eba84a47db5a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84a329f5-3f72-46b0-a7f6-eba84a47db5a-istio-data\") pod \"84a329f5-3f72-46b0-a7f6-eba84a47db5a\" (UID: \"84a329f5-3f72-46b0-a7f6-eba84a47db5a\") "
Nov 02 20:03:23.432939 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:23.432846    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/84a329f5-3f72-46b0-a7f6-eba84a47db5a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84a329f5-3f72-46b0-a7f6-eba84a47db5a-istio-podinfo\") pod \"84a329f5-3f72-46b0-a7f6-eba84a47db5a\" (UID: \"84a329f5-3f72-46b0-a7f6-eba84a47db5a\") "
Nov 02 20:03:23.432939 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:23.432875    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84a329f5-3f72-46b0-a7f6-eba84a47db5a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84a329f5-3f72-46b0-a7f6-eba84a47db5a-credential-socket\") pod \"84a329f5-3f72-46b0-a7f6-eba84a47db5a\" (UID: \"84a329f5-3f72-46b0-a7f6-eba84a47db5a\") "
Nov 02 20:03:23.433143 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:23.432975    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84a329f5-3f72-46b0-a7f6-eba84a47db5a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84a329f5-3f72-46b0-a7f6-eba84a47db5a-istio-envoy\") pod \"84a329f5-3f72-46b0-a7f6-eba84a47db5a\" (UID: \"84a329f5-3f72-46b0-a7f6-eba84a47db5a\") "
Nov 02 20:03:23.433143 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:23.433037    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84a329f5-3f72-46b0-a7f6-eba84a47db5a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84a329f5-3f72-46b0-a7f6-eba84a47db5a-workload-socket\") pod \"84a329f5-3f72-46b0-a7f6-eba84a47db5a\" (UID: \"84a329f5-3f72-46b0-a7f6-eba84a47db5a\") "
Nov 02 20:03:23.433143 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:23.433058    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84a329f5-3f72-46b0-a7f6-eba84a47db5a-istiod-ca-cert" pod "84a329f5-3f72-46b0-a7f6-eba84a47db5a" (UID: "84a329f5-3f72-46b0-a7f6-eba84a47db5a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:03:23.433143 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:23.433057    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84a329f5-3f72-46b0-a7f6-eba84a47db5a-workload-certs" pod "84a329f5-3f72-46b0-a7f6-eba84a47db5a" (UID: "84a329f5-3f72-46b0-a7f6-eba84a47db5a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:03:23.433357 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:23.433183    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84a329f5-3f72-46b0-a7f6-eba84a47db5a-istio-data" pod "84a329f5-3f72-46b0-a7f6-eba84a47db5a" (UID: "84a329f5-3f72-46b0-a7f6-eba84a47db5a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:03:23.433467 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:23.433442    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84a329f5-3f72-46b0-a7f6-eba84a47db5a-workload-socket" pod "84a329f5-3f72-46b0-a7f6-eba84a47db5a" (UID: "84a329f5-3f72-46b0-a7f6-eba84a47db5a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:03:23.433578 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:23.433515    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84a329f5-3f72-46b0-a7f6-eba84a47db5a-credential-socket" pod "84a329f5-3f72-46b0-a7f6-eba84a47db5a" (UID: "84a329f5-3f72-46b0-a7f6-eba84a47db5a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:03:23.433869 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:23.433844    4463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84a329f5-3f72-46b0-a7f6-eba84a47db5a-istiod-ca-cert\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:03:23.433953 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:23.433875    4463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84a329f5-3f72-46b0-a7f6-eba84a47db5a-workload-certs\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:03:23.433953 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:23.433893    4463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84a329f5-3f72-46b0-a7f6-eba84a47db5a-istio-data\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:03:23.433953 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:23.433907    4463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84a329f5-3f72-46b0-a7f6-eba84a47db5a-credential-socket\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:03:23.433953 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:23.433923    4463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84a329f5-3f72-46b0-a7f6-eba84a47db5a-workload-socket\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:03:23.435155 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:23.435123    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84a329f5-3f72-46b0-a7f6-eba84a47db5a-istio-envoy" pod "84a329f5-3f72-46b0-a7f6-eba84a47db5a" (UID: "84a329f5-3f72-46b0-a7f6-eba84a47db5a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:03:23.435272 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:23.435210    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/84a329f5-3f72-46b0-a7f6-eba84a47db5a-istio-podinfo" pod "84a329f5-3f72-46b0-a7f6-eba84a47db5a" (UID: "84a329f5-3f72-46b0-a7f6-eba84a47db5a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:03:23.435272 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:23.435227    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84a329f5-3f72-46b0-a7f6-eba84a47db5a-kube-api-access-wqw5x" pod "84a329f5-3f72-46b0-a7f6-eba84a47db5a" (UID: "84a329f5-3f72-46b0-a7f6-eba84a47db5a"). InnerVolumeSpecName "kube-api-access-wqw5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:03:23.435272 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:23.435259    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84a329f5-3f72-46b0-a7f6-eba84a47db5a-istio-token" pod "84a329f5-3f72-46b0-a7f6-eba84a47db5a" (UID: "84a329f5-3f72-46b0-a7f6-eba84a47db5a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:03:23.535227 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:23.535196    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqw5x\" (UniqueName: \"kubernetes.io/projected/84a329f5-3f72-46b0-a7f6-eba84a47db5a-kube-api-access-wqw5x\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:03:23.535227 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:23.535223    4463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84a329f5-3f72-46b0-a7f6-eba84a47db5a-istio-token\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:03:23.535227 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:23.535232    4463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84a329f5-3f72-46b0-a7f6-eba84a47db5a-istio-podinfo\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:03:23.535454 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:23.535241    4463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84a329f5-3f72-46b0-a7f6-eba84a47db5a-istio-envoy\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:03:23.587221 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:23.587147    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="787ed8bc-d29c-4e3b-b661-87d0712a506f" path="/var/lib/kubelet/pods/787ed8bc-d29c-4e3b-b661-87d0712a506f/volumes"
Nov 02 20:03:23.587597 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:23.587583    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6f2209a-00fa-4e46-a0be-85ed51d89369" path="/var/lib/kubelet/pods/b6f2209a-00fa-4e46-a0be-85ed51d89369/volumes"
Nov 02 20:03:23.746539 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:23.746511    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a5b64d-openshift-default-6f8c4599cf-vztxx"
Nov 02 20:03:23.752798 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:23.752772    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a5b64d-openshift-default-6f8c4599cf-vztxx"
Nov 02 20:03:23.821034 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:23.820119    4463 generic.go:356] "Generic (PLEG): container finished" podID="84a329f5-3f72-46b0-a7f6-eba84a47db5a" containerID="c15762394abfc9e1844aa9814ca9284c852d0d4e7ed1cda6c682519795e1e1e7" exitCode=0
Nov 02 20:03:23.826899 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:23.821523    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-23a9f5-openshift-default-6d49545b45-h5trw" event={"ID":"84a329f5-3f72-46b0-a7f6-eba84a47db5a","Type":"ContainerDied","Data":"c15762394abfc9e1844aa9814ca9284c852d0d4e7ed1cda6c682519795e1e1e7"}
Nov 02 20:03:23.826899 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:23.821592    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-23a9f5-openshift-default-6d49545b45-h5trw" event={"ID":"84a329f5-3f72-46b0-a7f6-eba84a47db5a","Type":"ContainerDied","Data":"e419e5b0c0c0a2cce9aaf1fd92cae60d9cc5bdff1d5fd0334fcfdd4667e23ca9"}
Nov 02 20:03:23.826899 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:23.821626    4463 scope.go:122] "RemoveContainer" containerID="c15762394abfc9e1844aa9814ca9284c852d0d4e7ed1cda6c682519795e1e1e7"
Nov 02 20:03:23.826899 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:23.821739    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-23a9f5-openshift-default-6d49545b45-h5trw"
Nov 02 20:03:23.826899 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:23.826536    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a5b64d-openshift-default-6f8c4599cf-vztxx"
Nov 02 20:03:23.849081 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:23.844306    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a5b64d-openshift-default-6f8c4599cf-vztxx"
Nov 02 20:03:23.853741 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:23.853679    4463 scope.go:122] "RemoveContainer" containerID="e228b4c4a6073ae7a665418ee7b5a8ea91597b65f6ae33f2c65128768a2f0972"
Nov 02 20:03:23.866058 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:23.865998    4463 scope.go:122] "RemoveContainer" containerID="c15762394abfc9e1844aa9814ca9284c852d0d4e7ed1cda6c682519795e1e1e7"
Nov 02 20:03:23.866640 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:03:23.866454    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c15762394abfc9e1844aa9814ca9284c852d0d4e7ed1cda6c682519795e1e1e7\": container with ID starting with c15762394abfc9e1844aa9814ca9284c852d0d4e7ed1cda6c682519795e1e1e7 not found: ID does not exist" containerID="c15762394abfc9e1844aa9814ca9284c852d0d4e7ed1cda6c682519795e1e1e7"
Nov 02 20:03:23.866640 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:23.866516    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c15762394abfc9e1844aa9814ca9284c852d0d4e7ed1cda6c682519795e1e1e7"} err="failed to get container status \"c15762394abfc9e1844aa9814ca9284c852d0d4e7ed1cda6c682519795e1e1e7\": rpc error: code = NotFound desc = could not find container \"c15762394abfc9e1844aa9814ca9284c852d0d4e7ed1cda6c682519795e1e1e7\": container with ID starting with c15762394abfc9e1844aa9814ca9284c852d0d4e7ed1cda6c682519795e1e1e7 not found: ID does not exist"
Nov 02 20:03:23.866640 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:23.866555    4463 scope.go:122] "RemoveContainer" containerID="e228b4c4a6073ae7a665418ee7b5a8ea91597b65f6ae33f2c65128768a2f0972"
Nov 02 20:03:23.866930 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:03:23.866890    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e228b4c4a6073ae7a665418ee7b5a8ea91597b65f6ae33f2c65128768a2f0972\": container with ID starting with e228b4c4a6073ae7a665418ee7b5a8ea91597b65f6ae33f2c65128768a2f0972 not found: ID does not exist" containerID="e228b4c4a6073ae7a665418ee7b5a8ea91597b65f6ae33f2c65128768a2f0972"
Nov 02 20:03:23.867075 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:23.866922    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e228b4c4a6073ae7a665418ee7b5a8ea91597b65f6ae33f2c65128768a2f0972"} err="failed to get container status \"e228b4c4a6073ae7a665418ee7b5a8ea91597b65f6ae33f2c65128768a2f0972\": rpc error: code = NotFound desc = could not find container \"e228b4c4a6073ae7a665418ee7b5a8ea91597b65f6ae33f2c65128768a2f0972\": container with ID starting with e228b4c4a6073ae7a665418ee7b5a8ea91597b65f6ae33f2c65128768a2f0972 not found: ID does not exist"
Nov 02 20:03:24.093488 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:24.093454    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-23a9f5-openshift-default-6d49545b45-h5trw"]
Nov 02 20:03:24.101550 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:24.101486    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-23a9f5-openshift-default-6d49545b45-h5trw"]
Nov 02 20:03:25.587553 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:25.587521    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84a329f5-3f72-46b0-a7f6-eba84a47db5a" path="/var/lib/kubelet/pods/84a329f5-3f72-46b0-a7f6-eba84a47db5a/volumes"
Nov 02 20:03:29.501071 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:29.498050    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-9f8f23-openshift-default-5dd8c74c6f-n75dv"]
Nov 02 20:03:29.501071 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:29.498381    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-9f8f23-openshift-default-5dd8c74c6f-n75dv" podUID="e054dacd-930b-430d-8719-3e22a12ce72f" containerName="istio-proxy" containerID="cri-o://4c3ce16179c1418f5628c810b9784add7afa7f26f147540968e796ed8e35ee07" gracePeriod=30
Nov 02 20:03:29.858153 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:29.858078    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a5b64d-59fd995f49-4g9cm"]
Nov 02 20:03:29.858713 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:29.858697    4463 cpu_manager.go:414] "containerMap: removing container" podUID="84a329f5-3f72-46b0-a7f6-eba84a47db5a" containerName="istio-proxy"
Nov 02 20:03:29.858751 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:29.858716    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84a329f5-3f72-46b0-a7f6-eba84a47db5a" containerName="istio-proxy"
Nov 02 20:03:29.858751 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:29.858747    4463 cpu_manager.go:414] "containerMap: removing container" podUID="787ed8bc-d29c-4e3b-b661-87d0712a506f" containerName="maas-api"
Nov 02 20:03:29.858826 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:29.858753    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="787ed8bc-d29c-4e3b-b661-87d0712a506f" containerName="maas-api"
Nov 02 20:03:29.858826 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:29.858800    4463 cpu_manager.go:414] "containerMap: removing container" podUID="84a329f5-3f72-46b0-a7f6-eba84a47db5a" containerName="istio-proxy"
Nov 02 20:03:29.858826 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:29.858805    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84a329f5-3f72-46b0-a7f6-eba84a47db5a" containerName="istio-proxy"
Nov 02 20:03:29.858826 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:29.858818    4463 cpu_manager.go:414] "containerMap: removing container" podUID="2f18cb37-937a-4cc9-9a20-f44f2f693d52" containerName="istio-proxy"
Nov 02 20:03:29.858826 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:29.858822    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f18cb37-937a-4cc9-9a20-f44f2f693d52" containerName="istio-proxy"
Nov 02 20:03:29.858972 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:29.858912    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="787ed8bc-d29c-4e3b-b661-87d0712a506f" containerName="maas-api"
Nov 02 20:03:29.858972 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:29.858926    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="84a329f5-3f72-46b0-a7f6-eba84a47db5a" containerName="istio-proxy"
Nov 02 20:03:29.858972 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:29.858935    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="84a329f5-3f72-46b0-a7f6-eba84a47db5a" containerName="istio-proxy"
Nov 02 20:03:29.862591 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:29.862571    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a5b64d-59fd995f49-4g9cm"
Nov 02 20:03:29.912188 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:29.912164    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-a5b64d\""
Nov 02 20:03:29.915598 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:29.915573    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a5b64d-59fd995f49-4g9cm"]
Nov 02 20:03:29.955036 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:29.947282    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f250661-9b39-4d06-af9c-6f067b82f8ca-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a5b64d-59fd995f49-4g9cm\" (UID: \"7f250661-9b39-4d06-af9c-6f067b82f8ca\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a5b64d-59fd995f49-4g9cm"
Nov 02 20:03:29.955036 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:29.947343    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gml7s\" (UniqueName: \"kubernetes.io/projected/7f250661-9b39-4d06-af9c-6f067b82f8ca-kube-api-access-gml7s\") pod \"payload-processing-e2e-inf-a-a5b64d-59fd995f49-4g9cm\" (UID: \"7f250661-9b39-4d06-af9c-6f067b82f8ca\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a5b64d-59fd995f49-4g9cm"
Nov 02 20:03:30.021031 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:30.020400    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-a5b64d-dockercfg-jvk7c\""
Nov 02 20:03:30.054035 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:30.050258    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f250661-9b39-4d06-af9c-6f067b82f8ca-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a5b64d-59fd995f49-4g9cm\" (UID: \"7f250661-9b39-4d06-af9c-6f067b82f8ca\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a5b64d-59fd995f49-4g9cm"
Nov 02 20:03:30.054035 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:30.050315    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gml7s\" (UniqueName: \"kubernetes.io/projected/7f250661-9b39-4d06-af9c-6f067b82f8ca-kube-api-access-gml7s\") pod \"payload-processing-e2e-inf-a-a5b64d-59fd995f49-4g9cm\" (UID: \"7f250661-9b39-4d06-af9c-6f067b82f8ca\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a5b64d-59fd995f49-4g9cm"
Nov 02 20:03:30.054035 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:30.051609    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f250661-9b39-4d06-af9c-6f067b82f8ca-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a5b64d-59fd995f49-4g9cm\" (UID: \"7f250661-9b39-4d06-af9c-6f067b82f8ca\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a5b64d-59fd995f49-4g9cm"
Nov 02 20:03:30.104487 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:30.104439    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gml7s\" (UniqueName: \"kubernetes.io/projected/7f250661-9b39-4d06-af9c-6f067b82f8ca-kube-api-access-gml7s\") pod \"payload-processing-e2e-inf-a-a5b64d-59fd995f49-4g9cm\" (UID: \"7f250661-9b39-4d06-af9c-6f067b82f8ca\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a5b64d-59fd995f49-4g9cm"
Nov 02 20:03:30.177286 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:30.177264    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a5b64d-59fd995f49-4g9cm"
Nov 02 20:03:30.545520 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:30.545492    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a5b64d-59fd995f49-4g9cm"]
Nov 02 20:03:30.547663 ip-10-0-133-9 kubenswrapper[4463]: W1102 20:03:30.547631    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f250661_9b39_4d06_af9c_6f067b82f8ca.slice/crio-37d1ac2dc15c600f8a6a59d962017946a104cfd1dd7aa87d088c7fb5ebc350bf WatchSource:0}: Error finding container 37d1ac2dc15c600f8a6a59d962017946a104cfd1dd7aa87d088c7fb5ebc350bf: Status 404 returned error can't find the container with id 37d1ac2dc15c600f8a6a59d962017946a104cfd1dd7aa87d088c7fb5ebc350bf
Nov 02 20:03:30.905086 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:30.904297    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a5b64d-59fd995f49-4g9cm" event={"ID":"7f250661-9b39-4d06-af9c-6f067b82f8ca","Type":"ContainerStarted","Data":"a8df302061409f44d31d2fdfe07afae4293717c73ec6ea4b1168d83f289eefe4"}
Nov 02 20:03:30.905086 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:30.904362    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a5b64d-59fd995f49-4g9cm" event={"ID":"7f250661-9b39-4d06-af9c-6f067b82f8ca","Type":"ContainerStarted","Data":"37d1ac2dc15c600f8a6a59d962017946a104cfd1dd7aa87d088c7fb5ebc350bf"}
Nov 02 20:03:30.906075 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:30.905421    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-a5b64d-59fd995f49-4g9cm"
Nov 02 20:03:31.014916 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:31.014853    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-a5b64d-59fd995f49-4g9cm" podStartSLOduration=2.014835839 podStartE2EDuration="2.014835839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:03:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:03:31.010362599 +0000 UTC m=+2117.617396362" watchObservedRunningTime="2026-11-02 20:03:31.014835839 +0000 UTC m=+2117.621869603"
Nov 02 20:03:32.000260 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:32.000228    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-838bc1-openshift-default-5fbf6db645-sx9sd"]
Nov 02 20:03:32.003626 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:32.003607    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-838bc1-openshift-default-5fbf6db645-sx9sd"
Nov 02 20:03:32.114420 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:32.114384    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-838bc1-openshift-default-dockercfg-mxlwn\""
Nov 02 20:03:32.142453 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:32.142424    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-838bc1-openshift-default-5fbf6db645-sx9sd"]
Nov 02 20:03:32.217482 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:32.217450    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9zbq\" (UniqueName: \"kubernetes.io/projected/c4fc2355-c8a3-4c51-9302-f93ced5c343e-kube-api-access-d9zbq\") pod \"e2e-inf-a-838bc1-openshift-default-5fbf6db645-sx9sd\" (UID: \"c4fc2355-c8a3-4c51-9302-f93ced5c343e\") " pod="openshift-ingress/e2e-inf-a-838bc1-openshift-default-5fbf6db645-sx9sd"
Nov 02 20:03:32.217482 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:32.217490    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4fc2355-c8a3-4c51-9302-f93ced5c343e-credential-socket\") pod \"e2e-inf-a-838bc1-openshift-default-5fbf6db645-sx9sd\" (UID: \"c4fc2355-c8a3-4c51-9302-f93ced5c343e\") " pod="openshift-ingress/e2e-inf-a-838bc1-openshift-default-5fbf6db645-sx9sd"
Nov 02 20:03:32.217712 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:32.217523    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4fc2355-c8a3-4c51-9302-f93ced5c343e-istio-envoy\") pod \"e2e-inf-a-838bc1-openshift-default-5fbf6db645-sx9sd\" (UID: \"c4fc2355-c8a3-4c51-9302-f93ced5c343e\") " pod="openshift-ingress/e2e-inf-a-838bc1-openshift-default-5fbf6db645-sx9sd"
Nov 02 20:03:32.217712 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:32.217677    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4fc2355-c8a3-4c51-9302-f93ced5c343e-workload-socket\") pod \"e2e-inf-a-838bc1-openshift-default-5fbf6db645-sx9sd\" (UID: \"c4fc2355-c8a3-4c51-9302-f93ced5c343e\") " pod="openshift-ingress/e2e-inf-a-838bc1-openshift-default-5fbf6db645-sx9sd"
Nov 02 20:03:32.217820 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:32.217733    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4fc2355-c8a3-4c51-9302-f93ced5c343e-workload-certs\") pod \"e2e-inf-a-838bc1-openshift-default-5fbf6db645-sx9sd\" (UID: \"c4fc2355-c8a3-4c51-9302-f93ced5c343e\") " pod="openshift-ingress/e2e-inf-a-838bc1-openshift-default-5fbf6db645-sx9sd"
Nov 02 20:03:32.217897 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:32.217868    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4fc2355-c8a3-4c51-9302-f93ced5c343e-istio-token\") pod \"e2e-inf-a-838bc1-openshift-default-5fbf6db645-sx9sd\" (UID: \"c4fc2355-c8a3-4c51-9302-f93ced5c343e\") " pod="openshift-ingress/e2e-inf-a-838bc1-openshift-default-5fbf6db645-sx9sd"
Nov 02 20:03:32.218061 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:32.218004    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4fc2355-c8a3-4c51-9302-f93ced5c343e-istio-podinfo\") pod \"e2e-inf-a-838bc1-openshift-default-5fbf6db645-sx9sd\" (UID: \"c4fc2355-c8a3-4c51-9302-f93ced5c343e\") " pod="openshift-ingress/e2e-inf-a-838bc1-openshift-default-5fbf6db645-sx9sd"
Nov 02 20:03:32.218151 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:32.218117    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4fc2355-c8a3-4c51-9302-f93ced5c343e-istiod-ca-cert\") pod \"e2e-inf-a-838bc1-openshift-default-5fbf6db645-sx9sd\" (UID: \"c4fc2355-c8a3-4c51-9302-f93ced5c343e\") " pod="openshift-ingress/e2e-inf-a-838bc1-openshift-default-5fbf6db645-sx9sd"
Nov 02 20:03:32.218213 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:32.218198    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4fc2355-c8a3-4c51-9302-f93ced5c343e-istio-data\") pod \"e2e-inf-a-838bc1-openshift-default-5fbf6db645-sx9sd\" (UID: \"c4fc2355-c8a3-4c51-9302-f93ced5c343e\") " pod="openshift-ingress/e2e-inf-a-838bc1-openshift-default-5fbf6db645-sx9sd"
Nov 02 20:03:32.319586 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:32.319490    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4fc2355-c8a3-4c51-9302-f93ced5c343e-workload-socket\") pod \"e2e-inf-a-838bc1-openshift-default-5fbf6db645-sx9sd\" (UID: \"c4fc2355-c8a3-4c51-9302-f93ced5c343e\") " pod="openshift-ingress/e2e-inf-a-838bc1-openshift-default-5fbf6db645-sx9sd"
Nov 02 20:03:32.319586 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:32.319574    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4fc2355-c8a3-4c51-9302-f93ced5c343e-workload-certs\") pod \"e2e-inf-a-838bc1-openshift-default-5fbf6db645-sx9sd\" (UID: \"c4fc2355-c8a3-4c51-9302-f93ced5c343e\") " pod="openshift-ingress/e2e-inf-a-838bc1-openshift-default-5fbf6db645-sx9sd"
Nov 02 20:03:32.319816 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:32.319653    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4fc2355-c8a3-4c51-9302-f93ced5c343e-istio-token\") pod \"e2e-inf-a-838bc1-openshift-default-5fbf6db645-sx9sd\" (UID: \"c4fc2355-c8a3-4c51-9302-f93ced5c343e\") " pod="openshift-ingress/e2e-inf-a-838bc1-openshift-default-5fbf6db645-sx9sd"
Nov 02 20:03:32.319816 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:32.319762    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4fc2355-c8a3-4c51-9302-f93ced5c343e-istio-podinfo\") pod \"e2e-inf-a-838bc1-openshift-default-5fbf6db645-sx9sd\" (UID: \"c4fc2355-c8a3-4c51-9302-f93ced5c343e\") " pod="openshift-ingress/e2e-inf-a-838bc1-openshift-default-5fbf6db645-sx9sd"
Nov 02 20:03:32.319816 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:32.319796    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4fc2355-c8a3-4c51-9302-f93ced5c343e-istiod-ca-cert\") pod \"e2e-inf-a-838bc1-openshift-default-5fbf6db645-sx9sd\" (UID: \"c4fc2355-c8a3-4c51-9302-f93ced5c343e\") " pod="openshift-ingress/e2e-inf-a-838bc1-openshift-default-5fbf6db645-sx9sd"
Nov 02 20:03:32.319969 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:32.319867    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4fc2355-c8a3-4c51-9302-f93ced5c343e-istio-data\") pod \"e2e-inf-a-838bc1-openshift-default-5fbf6db645-sx9sd\" (UID: \"c4fc2355-c8a3-4c51-9302-f93ced5c343e\") " pod="openshift-ingress/e2e-inf-a-838bc1-openshift-default-5fbf6db645-sx9sd"
Nov 02 20:03:32.319969 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:32.319948    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4fc2355-c8a3-4c51-9302-f93ced5c343e-workload-socket\") pod \"e2e-inf-a-838bc1-openshift-default-5fbf6db645-sx9sd\" (UID: \"c4fc2355-c8a3-4c51-9302-f93ced5c343e\") " pod="openshift-ingress/e2e-inf-a-838bc1-openshift-default-5fbf6db645-sx9sd"
Nov 02 20:03:32.320100 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:32.319958    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4fc2355-c8a3-4c51-9302-f93ced5c343e-workload-certs\") pod \"e2e-inf-a-838bc1-openshift-default-5fbf6db645-sx9sd\" (UID: \"c4fc2355-c8a3-4c51-9302-f93ced5c343e\") " pod="openshift-ingress/e2e-inf-a-838bc1-openshift-default-5fbf6db645-sx9sd"
Nov 02 20:03:32.320100 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:32.320048    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9zbq\" (UniqueName: \"kubernetes.io/projected/c4fc2355-c8a3-4c51-9302-f93ced5c343e-kube-api-access-d9zbq\") pod \"e2e-inf-a-838bc1-openshift-default-5fbf6db645-sx9sd\" (UID: \"c4fc2355-c8a3-4c51-9302-f93ced5c343e\") " pod="openshift-ingress/e2e-inf-a-838bc1-openshift-default-5fbf6db645-sx9sd"
Nov 02 20:03:32.320100 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:32.320086    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4fc2355-c8a3-4c51-9302-f93ced5c343e-credential-socket\") pod \"e2e-inf-a-838bc1-openshift-default-5fbf6db645-sx9sd\" (UID: \"c4fc2355-c8a3-4c51-9302-f93ced5c343e\") " pod="openshift-ingress/e2e-inf-a-838bc1-openshift-default-5fbf6db645-sx9sd"
Nov 02 20:03:32.320256 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:32.320127    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4fc2355-c8a3-4c51-9302-f93ced5c343e-istio-envoy\") pod \"e2e-inf-a-838bc1-openshift-default-5fbf6db645-sx9sd\" (UID: \"c4fc2355-c8a3-4c51-9302-f93ced5c343e\") " pod="openshift-ingress/e2e-inf-a-838bc1-openshift-default-5fbf6db645-sx9sd"
Nov 02 20:03:32.320357 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:32.320338    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4fc2355-c8a3-4c51-9302-f93ced5c343e-istio-data\") pod \"e2e-inf-a-838bc1-openshift-default-5fbf6db645-sx9sd\" (UID: \"c4fc2355-c8a3-4c51-9302-f93ced5c343e\") " pod="openshift-ingress/e2e-inf-a-838bc1-openshift-default-5fbf6db645-sx9sd"
Nov 02 20:03:32.320774 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:32.320748    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4fc2355-c8a3-4c51-9302-f93ced5c343e-credential-socket\") pod \"e2e-inf-a-838bc1-openshift-default-5fbf6db645-sx9sd\" (UID: \"c4fc2355-c8a3-4c51-9302-f93ced5c343e\") " pod="openshift-ingress/e2e-inf-a-838bc1-openshift-default-5fbf6db645-sx9sd"
Nov 02 20:03:32.321007 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:32.320984    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4fc2355-c8a3-4c51-9302-f93ced5c343e-istiod-ca-cert\") pod \"e2e-inf-a-838bc1-openshift-default-5fbf6db645-sx9sd\" (UID: \"c4fc2355-c8a3-4c51-9302-f93ced5c343e\") " pod="openshift-ingress/e2e-inf-a-838bc1-openshift-default-5fbf6db645-sx9sd"
Nov 02 20:03:32.322795 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:32.322774    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4fc2355-c8a3-4c51-9302-f93ced5c343e-istio-envoy\") pod \"e2e-inf-a-838bc1-openshift-default-5fbf6db645-sx9sd\" (UID: \"c4fc2355-c8a3-4c51-9302-f93ced5c343e\") " pod="openshift-ingress/e2e-inf-a-838bc1-openshift-default-5fbf6db645-sx9sd"
Nov 02 20:03:32.323234 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:32.323214    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4fc2355-c8a3-4c51-9302-f93ced5c343e-istio-podinfo\") pod \"e2e-inf-a-838bc1-openshift-default-5fbf6db645-sx9sd\" (UID: \"c4fc2355-c8a3-4c51-9302-f93ced5c343e\") " pod="openshift-ingress/e2e-inf-a-838bc1-openshift-default-5fbf6db645-sx9sd"
Nov 02 20:03:32.373665 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:32.373632    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4fc2355-c8a3-4c51-9302-f93ced5c343e-istio-token\") pod \"e2e-inf-a-838bc1-openshift-default-5fbf6db645-sx9sd\" (UID: \"c4fc2355-c8a3-4c51-9302-f93ced5c343e\") " pod="openshift-ingress/e2e-inf-a-838bc1-openshift-default-5fbf6db645-sx9sd"
Nov 02 20:03:32.373844 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:32.373714    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9zbq\" (UniqueName: \"kubernetes.io/projected/c4fc2355-c8a3-4c51-9302-f93ced5c343e-kube-api-access-d9zbq\") pod \"e2e-inf-a-838bc1-openshift-default-5fbf6db645-sx9sd\" (UID: \"c4fc2355-c8a3-4c51-9302-f93ced5c343e\") " pod="openshift-ingress/e2e-inf-a-838bc1-openshift-default-5fbf6db645-sx9sd"
Nov 02 20:03:32.616785 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:32.616706    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-838bc1-openshift-default-5fbf6db645-sx9sd"
Nov 02 20:03:33.036293 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:33.036257    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-838bc1-openshift-default-5fbf6db645-sx9sd"]
Nov 02 20:03:33.037138 ip-10-0-133-9 kubenswrapper[4463]: W1102 20:03:33.037112    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4fc2355_c8a3_4c51_9302_f93ced5c343e.slice/crio-6a4b48808c0645a45e4180e7a6bf8048b74b23e25dcb647e9b2730073c3d5815 WatchSource:0}: Error finding container 6a4b48808c0645a45e4180e7a6bf8048b74b23e25dcb647e9b2730073c3d5815: Status 404 returned error can't find the container with id 6a4b48808c0645a45e4180e7a6bf8048b74b23e25dcb647e9b2730073c3d5815
Nov 02 20:03:33.039911 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:33.039877    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:03:33.040004 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:33.039942    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:03:33.040004 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:33.039968    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:03:33.925073 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:33.925038    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-838bc1-openshift-default-5fbf6db645-sx9sd" event={"ID":"c4fc2355-c8a3-4c51-9302-f93ced5c343e","Type":"ContainerStarted","Data":"b285fc201b6ce043240d90cf63f62f54f1dc7b558f56265afe4c032c16bc603d"}
Nov 02 20:03:33.925254 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:33.925091    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-838bc1-openshift-default-5fbf6db645-sx9sd" event={"ID":"c4fc2355-c8a3-4c51-9302-f93ced5c343e","Type":"ContainerStarted","Data":"6a4b48808c0645a45e4180e7a6bf8048b74b23e25dcb647e9b2730073c3d5815"}
Nov 02 20:03:34.065045 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:34.064750    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-838bc1-openshift-default-5fbf6db645-sx9sd" podStartSLOduration=3.064729102 podStartE2EDuration="3.064729102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:03:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:03:34.050158287 +0000 UTC m=+2120.657192056" watchObservedRunningTime="2026-11-02 20:03:34.064729102 +0000 UTC m=+2120.671762873"
Nov 02 20:03:34.295262 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:03:34.295178    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 20:03:34.617706 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:34.617634    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-838bc1-openshift-default-5fbf6db645-sx9sd"
Nov 02 20:03:34.624109 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:34.623912    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-838bc1-openshift-default-5fbf6db645-sx9sd"
Nov 02 20:03:34.789834 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:34.789808    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9f8f23-openshift-default-5dd8c74c6f-n75dv"
Nov 02 20:03:34.863745 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:34.863717    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e054dacd-930b-430d-8719-3e22a12ce72f-kube-api-access-2gzvk\" (UniqueName: \"kubernetes.io/projected/e054dacd-930b-430d-8719-3e22a12ce72f-kube-api-access-2gzvk\") pod \"e054dacd-930b-430d-8719-3e22a12ce72f\" (UID: \"e054dacd-930b-430d-8719-3e22a12ce72f\") "
Nov 02 20:03:34.863907 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:34.863757    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e054dacd-930b-430d-8719-3e22a12ce72f-istio-token\" (UniqueName: \"kubernetes.io/projected/e054dacd-930b-430d-8719-3e22a12ce72f-istio-token\") pod \"e054dacd-930b-430d-8719-3e22a12ce72f\" (UID: \"e054dacd-930b-430d-8719-3e22a12ce72f\") "
Nov 02 20:03:34.863907 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:34.863782    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e054dacd-930b-430d-8719-3e22a12ce72f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e054dacd-930b-430d-8719-3e22a12ce72f-credential-socket\") pod \"e054dacd-930b-430d-8719-3e22a12ce72f\" (UID: \"e054dacd-930b-430d-8719-3e22a12ce72f\") "
Nov 02 20:03:34.863907 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:34.863886    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e054dacd-930b-430d-8719-3e22a12ce72f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e054dacd-930b-430d-8719-3e22a12ce72f-istiod-ca-cert\") pod \"e054dacd-930b-430d-8719-3e22a12ce72f\" (UID: \"e054dacd-930b-430d-8719-3e22a12ce72f\") "
Nov 02 20:03:34.864148 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:34.863973    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e054dacd-930b-430d-8719-3e22a12ce72f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e054dacd-930b-430d-8719-3e22a12ce72f-istio-envoy\") pod \"e054dacd-930b-430d-8719-3e22a12ce72f\" (UID: \"e054dacd-930b-430d-8719-3e22a12ce72f\") "
Nov 02 20:03:34.864148 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:34.864046    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e054dacd-930b-430d-8719-3e22a12ce72f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e054dacd-930b-430d-8719-3e22a12ce72f-istio-data\") pod \"e054dacd-930b-430d-8719-3e22a12ce72f\" (UID: \"e054dacd-930b-430d-8719-3e22a12ce72f\") "
Nov 02 20:03:34.864148 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:34.864120    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e054dacd-930b-430d-8719-3e22a12ce72f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e054dacd-930b-430d-8719-3e22a12ce72f-workload-socket\") pod \"e054dacd-930b-430d-8719-3e22a12ce72f\" (UID: \"e054dacd-930b-430d-8719-3e22a12ce72f\") "
Nov 02 20:03:34.864318 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:34.864145    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e054dacd-930b-430d-8719-3e22a12ce72f-credential-socket" pod "e054dacd-930b-430d-8719-3e22a12ce72f" (UID: "e054dacd-930b-430d-8719-3e22a12ce72f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:03:34.864318 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:34.864174    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e054dacd-930b-430d-8719-3e22a12ce72f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e054dacd-930b-430d-8719-3e22a12ce72f-istio-podinfo\") pod \"e054dacd-930b-430d-8719-3e22a12ce72f\" (UID: \"e054dacd-930b-430d-8719-3e22a12ce72f\") "
Nov 02 20:03:34.864318 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:34.864214    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e054dacd-930b-430d-8719-3e22a12ce72f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e054dacd-930b-430d-8719-3e22a12ce72f-workload-certs\") pod \"e054dacd-930b-430d-8719-3e22a12ce72f\" (UID: \"e054dacd-930b-430d-8719-3e22a12ce72f\") "
Nov 02 20:03:34.864318 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:34.864280    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e054dacd-930b-430d-8719-3e22a12ce72f-istiod-ca-cert" pod "e054dacd-930b-430d-8719-3e22a12ce72f" (UID: "e054dacd-930b-430d-8719-3e22a12ce72f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:03:34.864516 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:34.864396    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e054dacd-930b-430d-8719-3e22a12ce72f-istio-data" pod "e054dacd-930b-430d-8719-3e22a12ce72f" (UID: "e054dacd-930b-430d-8719-3e22a12ce72f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:03:34.864516 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:34.864416    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e054dacd-930b-430d-8719-3e22a12ce72f-workload-socket" pod "e054dacd-930b-430d-8719-3e22a12ce72f" (UID: "e054dacd-930b-430d-8719-3e22a12ce72f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:03:34.864615 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:34.864581    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e054dacd-930b-430d-8719-3e22a12ce72f-workload-certs" pod "e054dacd-930b-430d-8719-3e22a12ce72f" (UID: "e054dacd-930b-430d-8719-3e22a12ce72f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:03:34.864908 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:34.864885    4463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e054dacd-930b-430d-8719-3e22a12ce72f-workload-socket\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:03:34.865001 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:34.864912    4463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e054dacd-930b-430d-8719-3e22a12ce72f-workload-certs\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:03:34.865001 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:34.864929    4463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e054dacd-930b-430d-8719-3e22a12ce72f-credential-socket\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:03:34.865001 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:34.864943    4463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e054dacd-930b-430d-8719-3e22a12ce72f-istiod-ca-cert\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:03:34.865001 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:34.864956    4463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e054dacd-930b-430d-8719-3e22a12ce72f-istio-data\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:03:34.866149 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:34.866123    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e054dacd-930b-430d-8719-3e22a12ce72f-istio-envoy" pod "e054dacd-930b-430d-8719-3e22a12ce72f" (UID: "e054dacd-930b-430d-8719-3e22a12ce72f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:03:34.866288 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:34.866272    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e054dacd-930b-430d-8719-3e22a12ce72f-kube-api-access-2gzvk" pod "e054dacd-930b-430d-8719-3e22a12ce72f" (UID: "e054dacd-930b-430d-8719-3e22a12ce72f"). InnerVolumeSpecName "kube-api-access-2gzvk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:03:34.866360 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:34.866324    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e054dacd-930b-430d-8719-3e22a12ce72f-istio-podinfo" pod "e054dacd-930b-430d-8719-3e22a12ce72f" (UID: "e054dacd-930b-430d-8719-3e22a12ce72f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:03:34.866497 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:34.866473    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e054dacd-930b-430d-8719-3e22a12ce72f-istio-token" pod "e054dacd-930b-430d-8719-3e22a12ce72f" (UID: "e054dacd-930b-430d-8719-3e22a12ce72f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:03:34.935105 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:34.935063    4463 generic.go:356] "Generic (PLEG): container finished" podID="e054dacd-930b-430d-8719-3e22a12ce72f" containerID="4c3ce16179c1418f5628c810b9784add7afa7f26f147540968e796ed8e35ee07" exitCode=0
Nov 02 20:03:34.935280 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:34.935126    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9f8f23-openshift-default-5dd8c74c6f-n75dv"
Nov 02 20:03:34.935280 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:34.935147    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9f8f23-openshift-default-5dd8c74c6f-n75dv" event={"ID":"e054dacd-930b-430d-8719-3e22a12ce72f","Type":"ContainerDied","Data":"4c3ce16179c1418f5628c810b9784add7afa7f26f147540968e796ed8e35ee07"}
Nov 02 20:03:34.935280 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:34.935208    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9f8f23-openshift-default-5dd8c74c6f-n75dv" event={"ID":"e054dacd-930b-430d-8719-3e22a12ce72f","Type":"ContainerDied","Data":"40e0bcda198591038254056420805466c3e479d08246882cb7bd8537bc189c5f"}
Nov 02 20:03:34.935280 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:34.935238    4463 scope.go:122] "RemoveContainer" containerID="4c3ce16179c1418f5628c810b9784add7afa7f26f147540968e796ed8e35ee07"
Nov 02 20:03:34.935596 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:34.935576    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-838bc1-openshift-default-5fbf6db645-sx9sd"
Nov 02 20:03:34.936649 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:34.936630    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-838bc1-openshift-default-5fbf6db645-sx9sd"
Nov 02 20:03:34.948249 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:34.948226    4463 scope.go:122] "RemoveContainer" containerID="6a0ec9645fe5619f3279385f6b436203289097e22f54c65f82cefec790674ba9"
Nov 02 20:03:34.958517 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:34.958500    4463 scope.go:122] "RemoveContainer" containerID="4c3ce16179c1418f5628c810b9784add7afa7f26f147540968e796ed8e35ee07"
Nov 02 20:03:34.958792 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:03:34.958768    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c3ce16179c1418f5628c810b9784add7afa7f26f147540968e796ed8e35ee07\": container with ID starting with 4c3ce16179c1418f5628c810b9784add7afa7f26f147540968e796ed8e35ee07 not found: ID does not exist" containerID="4c3ce16179c1418f5628c810b9784add7afa7f26f147540968e796ed8e35ee07"
Nov 02 20:03:34.958877 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:34.958804    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c3ce16179c1418f5628c810b9784add7afa7f26f147540968e796ed8e35ee07"} err="failed to get container status \"4c3ce16179c1418f5628c810b9784add7afa7f26f147540968e796ed8e35ee07\": rpc error: code = NotFound desc = could not find container \"4c3ce16179c1418f5628c810b9784add7afa7f26f147540968e796ed8e35ee07\": container with ID starting with 4c3ce16179c1418f5628c810b9784add7afa7f26f147540968e796ed8e35ee07 not found: ID does not exist"
Nov 02 20:03:34.958877 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:34.958821    4463 scope.go:122] "RemoveContainer" containerID="6a0ec9645fe5619f3279385f6b436203289097e22f54c65f82cefec790674ba9"
Nov 02 20:03:34.959097 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:03:34.959079    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a0ec9645fe5619f3279385f6b436203289097e22f54c65f82cefec790674ba9\": container with ID starting with 6a0ec9645fe5619f3279385f6b436203289097e22f54c65f82cefec790674ba9 not found: ID does not exist" containerID="6a0ec9645fe5619f3279385f6b436203289097e22f54c65f82cefec790674ba9"
Nov 02 20:03:34.959139 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:34.959101    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a0ec9645fe5619f3279385f6b436203289097e22f54c65f82cefec790674ba9"} err="failed to get container status \"6a0ec9645fe5619f3279385f6b436203289097e22f54c65f82cefec790674ba9\": rpc error: code = NotFound desc = could not find container \"6a0ec9645fe5619f3279385f6b436203289097e22f54c65f82cefec790674ba9\": container with ID starting with 6a0ec9645fe5619f3279385f6b436203289097e22f54c65f82cefec790674ba9 not found: ID does not exist"
Nov 02 20:03:34.966312 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:34.966294    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gzvk\" (UniqueName: \"kubernetes.io/projected/e054dacd-930b-430d-8719-3e22a12ce72f-kube-api-access-2gzvk\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:03:34.966405 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:34.966316    4463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e054dacd-930b-430d-8719-3e22a12ce72f-istio-token\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:03:34.966405 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:34.966328    4463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e054dacd-930b-430d-8719-3e22a12ce72f-istio-envoy\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:03:34.966405 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:34.966337    4463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e054dacd-930b-430d-8719-3e22a12ce72f-istio-podinfo\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:03:35.132082 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:35.126109    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-9f8f23-openshift-default-5dd8c74c6f-n75dv"]
Nov 02 20:03:35.141596 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:35.139486    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-9f8f23-openshift-default-5dd8c74c6f-n75dv"]
Nov 02 20:03:35.588109 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:35.588077    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e054dacd-930b-430d-8719-3e22a12ce72f" path="/var/lib/kubelet/pods/e054dacd-930b-430d-8719-3e22a12ce72f/volumes"
Nov 02 20:03:40.675784 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:40.675744    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-838bc1-869f9bc899-68j9k"]
Nov 02 20:03:40.676834 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:40.676813    4463 cpu_manager.go:414] "containerMap: removing container" podUID="e054dacd-930b-430d-8719-3e22a12ce72f" containerName="istio-proxy"
Nov 02 20:03:40.676896 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:40.676837    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e054dacd-930b-430d-8719-3e22a12ce72f" containerName="istio-proxy"
Nov 02 20:03:40.676896 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:40.676880    4463 cpu_manager.go:414] "containerMap: removing container" podUID="e054dacd-930b-430d-8719-3e22a12ce72f" containerName="istio-proxy"
Nov 02 20:03:40.676896 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:40.676888    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e054dacd-930b-430d-8719-3e22a12ce72f" containerName="istio-proxy"
Nov 02 20:03:40.677059 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:40.677046    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="e054dacd-930b-430d-8719-3e22a12ce72f" containerName="istio-proxy"
Nov 02 20:03:40.680586 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:40.680565    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-838bc1-869f9bc899-68j9k"
Nov 02 20:03:40.734487 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:40.734451    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-838bc1-869f9bc899-68j9k"]
Nov 02 20:03:40.800836 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:40.786415    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-838bc1\""
Nov 02 20:03:40.800836 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:40.787749    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-838bc1-dockercfg-z8wqk\""
Nov 02 20:03:40.803222 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:40.803182    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba35ca07-1e6a-4098-9d52-afa7a68c8805-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-838bc1-869f9bc899-68j9k\" (UID: \"ba35ca07-1e6a-4098-9d52-afa7a68c8805\") " pod="openshift-ingress/payload-processing-e2e-inf-a-838bc1-869f9bc899-68j9k"
Nov 02 20:03:40.803373 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:40.803286    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhzwc\" (UniqueName: \"kubernetes.io/projected/ba35ca07-1e6a-4098-9d52-afa7a68c8805-kube-api-access-mhzwc\") pod \"payload-processing-e2e-inf-a-838bc1-869f9bc899-68j9k\" (UID: \"ba35ca07-1e6a-4098-9d52-afa7a68c8805\") " pod="openshift-ingress/payload-processing-e2e-inf-a-838bc1-869f9bc899-68j9k"
Nov 02 20:03:40.914034 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:40.913408    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba35ca07-1e6a-4098-9d52-afa7a68c8805-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-838bc1-869f9bc899-68j9k\" (UID: \"ba35ca07-1e6a-4098-9d52-afa7a68c8805\") " pod="openshift-ingress/payload-processing-e2e-inf-a-838bc1-869f9bc899-68j9k"
Nov 02 20:03:40.914034 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:40.913516    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhzwc\" (UniqueName: \"kubernetes.io/projected/ba35ca07-1e6a-4098-9d52-afa7a68c8805-kube-api-access-mhzwc\") pod \"payload-processing-e2e-inf-a-838bc1-869f9bc899-68j9k\" (UID: \"ba35ca07-1e6a-4098-9d52-afa7a68c8805\") " pod="openshift-ingress/payload-processing-e2e-inf-a-838bc1-869f9bc899-68j9k"
Nov 02 20:03:40.934142 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:40.917084    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba35ca07-1e6a-4098-9d52-afa7a68c8805-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-838bc1-869f9bc899-68j9k\" (UID: \"ba35ca07-1e6a-4098-9d52-afa7a68c8805\") " pod="openshift-ingress/payload-processing-e2e-inf-a-838bc1-869f9bc899-68j9k"
Nov 02 20:03:40.969557 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:40.969519    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhzwc\" (UniqueName: \"kubernetes.io/projected/ba35ca07-1e6a-4098-9d52-afa7a68c8805-kube-api-access-mhzwc\") pod \"payload-processing-e2e-inf-a-838bc1-869f9bc899-68j9k\" (UID: \"ba35ca07-1e6a-4098-9d52-afa7a68c8805\") " pod="openshift-ingress/payload-processing-e2e-inf-a-838bc1-869f9bc899-68j9k"
Nov 02 20:03:40.996049 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:40.996000    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-838bc1-869f9bc899-68j9k"
Nov 02 20:03:41.373580 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:41.373551    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-838bc1-869f9bc899-68j9k"]
Nov 02 20:03:41.375229 ip-10-0-133-9 kubenswrapper[4463]: W1102 20:03:41.375193    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba35ca07_1e6a_4098_9d52_afa7a68c8805.slice/crio-3dae4a43e7c2e692ff386d260ccecebc55dc7700192983f9d279efe5ad62ef5b WatchSource:0}: Error finding container 3dae4a43e7c2e692ff386d260ccecebc55dc7700192983f9d279efe5ad62ef5b: Status 404 returned error can't find the container with id 3dae4a43e7c2e692ff386d260ccecebc55dc7700192983f9d279efe5ad62ef5b
Nov 02 20:03:41.984592 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:41.984556    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-838bc1-869f9bc899-68j9k" event={"ID":"ba35ca07-1e6a-4098-9d52-afa7a68c8805","Type":"ContainerStarted","Data":"240b70899f3e0c96151de7500f3153164940275f2794474ef6c6d09442074946"}
Nov 02 20:03:41.985024 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:41.984619    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-838bc1-869f9bc899-68j9k" event={"ID":"ba35ca07-1e6a-4098-9d52-afa7a68c8805","Type":"ContainerStarted","Data":"3dae4a43e7c2e692ff386d260ccecebc55dc7700192983f9d279efe5ad62ef5b"}
Nov 02 20:03:41.985024 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:41.984754    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-838bc1-869f9bc899-68j9k"
Nov 02 20:03:42.094174 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:42.094103    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-838bc1-869f9bc899-68j9k" podStartSLOduration=2.094065336 podStartE2EDuration="2.094065336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:03:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:03:42.089328267 +0000 UTC m=+2128.696362036" watchObservedRunningTime="2026-11-02 20:03:42.094065336 +0000 UTC m=+2128.701099118"
Nov 02 20:03:42.310198 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:42.310115    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-4fd0d5-openshift-default-8566cd5547-tvdjk"]
Nov 02 20:03:42.310520 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:42.310346    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-4fd0d5-openshift-default-8566cd5547-tvdjk" podUID="c6d73c91-e211-41f4-a929-d3e45bc82366" containerName="istio-proxy" containerID="cri-o://40104457ee4e5fee6490909a12c8470adb9202518cc38a948753eb994ebdf514" gracePeriod=30
Nov 02 20:03:42.920233 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:42.920204    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-a5b64d-59fd995f49-4g9cm"
Nov 02 20:03:43.970249 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:43.970218    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b0cc24-75c69845d6-zhjrj"]
Nov 02 20:03:43.970629 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:43.970397    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b0cc24-75c69845d6-zhjrj" podUID="ef952ef1-649a-4551-b974-d0b00d733f7e" containerName="maas-api" containerID="cri-o://bb685e19cfa709de8c031dc6f63169978cb76015291cf349d0ca4117bef01039" gracePeriod=30
Nov 02 20:03:44.234115 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:44.234052    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b0cc24-75c69845d6-zhjrj"
Nov 02 20:03:44.280326 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:44.280294    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef952ef1-649a-4551-b974-d0b00d733f7e-kube-api-access-j6rbq\" (UniqueName: \"kubernetes.io/projected/ef952ef1-649a-4551-b974-d0b00d733f7e-kube-api-access-j6rbq\") pod \"ef952ef1-649a-4551-b974-d0b00d733f7e\" (UID: \"ef952ef1-649a-4551-b974-d0b00d733f7e\") "
Nov 02 20:03:44.280503 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:44.280463    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef952ef1-649a-4551-b974-d0b00d733f7e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef952ef1-649a-4551-b974-d0b00d733f7e-openshift-service-ca\") pod \"ef952ef1-649a-4551-b974-d0b00d733f7e\" (UID: \"ef952ef1-649a-4551-b974-d0b00d733f7e\") "
Nov 02 20:03:44.280558 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:44.280536    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef952ef1-649a-4551-b974-d0b00d733f7e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef952ef1-649a-4551-b974-d0b00d733f7e-maas-api-tls\") pod \"ef952ef1-649a-4551-b974-d0b00d733f7e\" (UID: \"ef952ef1-649a-4551-b974-d0b00d733f7e\") "
Nov 02 20:03:44.280625 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:44.280599    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef952ef1-649a-4551-b974-d0b00d733f7e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef952ef1-649a-4551-b974-d0b00d733f7e-metrics-certs\") pod \"ef952ef1-649a-4551-b974-d0b00d733f7e\" (UID: \"ef952ef1-649a-4551-b974-d0b00d733f7e\") "
Nov 02 20:03:44.280917 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:44.280891    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef952ef1-649a-4551-b974-d0b00d733f7e-openshift-service-ca" pod "ef952ef1-649a-4551-b974-d0b00d733f7e" (UID: "ef952ef1-649a-4551-b974-d0b00d733f7e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:03:44.281368 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:44.281343    4463 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef952ef1-649a-4551-b974-d0b00d733f7e-openshift-service-ca\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:03:44.282864 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:44.282823    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef952ef1-649a-4551-b974-d0b00d733f7e-maas-api-tls" pod "ef952ef1-649a-4551-b974-d0b00d733f7e" (UID: "ef952ef1-649a-4551-b974-d0b00d733f7e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:03:44.282864 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:44.282846    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef952ef1-649a-4551-b974-d0b00d733f7e-metrics-certs" pod "ef952ef1-649a-4551-b974-d0b00d733f7e" (UID: "ef952ef1-649a-4551-b974-d0b00d733f7e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:03:44.283054 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:44.282949    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef952ef1-649a-4551-b974-d0b00d733f7e-kube-api-access-j6rbq" pod "ef952ef1-649a-4551-b974-d0b00d733f7e" (UID: "ef952ef1-649a-4551-b974-d0b00d733f7e"). InnerVolumeSpecName "kube-api-access-j6rbq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:03:44.370234 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:44.370196    4463 patch_prober.go:29] interesting pod/e2e-shared-a-w5-4fd0d5-openshift-default-8566cd5547-tvdjk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 20:03:44.370410 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:44.370256    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w5-4fd0d5-openshift-default-8566cd5547-tvdjk" podUID="c6d73c91-e211-41f4-a929-d3e45bc82366" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 20:03:44.383287 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:44.383252    4463 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef952ef1-649a-4551-b974-d0b00d733f7e-maas-api-tls\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:03:44.383287 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:44.383286    4463 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef952ef1-649a-4551-b974-d0b00d733f7e-metrics-certs\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:03:44.383458 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:44.383304    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6rbq\" (UniqueName: \"kubernetes.io/projected/ef952ef1-649a-4551-b974-d0b00d733f7e-kube-api-access-j6rbq\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:03:45.064035 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:45.057443    4463 generic.go:356] "Generic (PLEG): container finished" podID="ef952ef1-649a-4551-b974-d0b00d733f7e" containerID="bb685e19cfa709de8c031dc6f63169978cb76015291cf349d0ca4117bef01039" exitCode=0
Nov 02 20:03:45.064035 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:45.057591    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b0cc24-75c69845d6-zhjrj" event={"ID":"ef952ef1-649a-4551-b974-d0b00d733f7e","Type":"ContainerDied","Data":"bb685e19cfa709de8c031dc6f63169978cb76015291cf349d0ca4117bef01039"}
Nov 02 20:03:45.064035 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:45.057645    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b0cc24-75c69845d6-zhjrj" event={"ID":"ef952ef1-649a-4551-b974-d0b00d733f7e","Type":"ContainerDied","Data":"39ad1c16cc195b816f08159ea39596a79a7b618d6bdb342132f54044134404ef"}
Nov 02 20:03:45.064035 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:45.057677    4463 scope.go:122] "RemoveContainer" containerID="bb685e19cfa709de8c031dc6f63169978cb76015291cf349d0ca4117bef01039"
Nov 02 20:03:45.064035 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:45.057846    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b0cc24-75c69845d6-zhjrj"
Nov 02 20:03:45.107802 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:45.107775    4463 scope.go:122] "RemoveContainer" containerID="bb685e19cfa709de8c031dc6f63169978cb76015291cf349d0ca4117bef01039"
Nov 02 20:03:45.108239 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:03:45.108211    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb685e19cfa709de8c031dc6f63169978cb76015291cf349d0ca4117bef01039\": container with ID starting with bb685e19cfa709de8c031dc6f63169978cb76015291cf349d0ca4117bef01039 not found: ID does not exist" containerID="bb685e19cfa709de8c031dc6f63169978cb76015291cf349d0ca4117bef01039"
Nov 02 20:03:45.108372 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:45.108245    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb685e19cfa709de8c031dc6f63169978cb76015291cf349d0ca4117bef01039"} err="failed to get container status \"bb685e19cfa709de8c031dc6f63169978cb76015291cf349d0ca4117bef01039\": rpc error: code = NotFound desc = could not find container \"bb685e19cfa709de8c031dc6f63169978cb76015291cf349d0ca4117bef01039\": container with ID starting with bb685e19cfa709de8c031dc6f63169978cb76015291cf349d0ca4117bef01039 not found: ID does not exist"
Nov 02 20:03:45.220231 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:45.220199    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b0cc24-75c69845d6-zhjrj"]
Nov 02 20:03:45.265718 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:45.265689    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b0cc24-75c69845d6-zhjrj"]
Nov 02 20:03:45.586998 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:45.586965    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef952ef1-649a-4551-b974-d0b00d733f7e" path="/var/lib/kubelet/pods/ef952ef1-649a-4551-b974-d0b00d733f7e/volumes"
Nov 02 20:03:47.591572 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:47.591509    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-4fd0d5-openshift-default-8566cd5547-tvdjk"
Nov 02 20:03:47.643696 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:47.643665    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c6d73c91-e211-41f4-a929-d3e45bc82366-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6d73c91-e211-41f4-a929-d3e45bc82366-istio-podinfo\") pod \"c6d73c91-e211-41f4-a929-d3e45bc82366\" (UID: \"c6d73c91-e211-41f4-a929-d3e45bc82366\") "
Nov 02 20:03:47.643855 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:47.643722    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6d73c91-e211-41f4-a929-d3e45bc82366-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6d73c91-e211-41f4-a929-d3e45bc82366-workload-socket\") pod \"c6d73c91-e211-41f4-a929-d3e45bc82366\" (UID: \"c6d73c91-e211-41f4-a929-d3e45bc82366\") "
Nov 02 20:03:47.643855 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:47.643744    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6d73c91-e211-41f4-a929-d3e45bc82366-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6d73c91-e211-41f4-a929-d3e45bc82366-istio-envoy\") pod \"c6d73c91-e211-41f4-a929-d3e45bc82366\" (UID: \"c6d73c91-e211-41f4-a929-d3e45bc82366\") "
Nov 02 20:03:47.643855 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:47.643785    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6d73c91-e211-41f4-a929-d3e45bc82366-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6d73c91-e211-41f4-a929-d3e45bc82366-workload-certs\") pod \"c6d73c91-e211-41f4-a929-d3e45bc82366\" (UID: \"c6d73c91-e211-41f4-a929-d3e45bc82366\") "
Nov 02 20:03:47.644026 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:47.643976    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6d73c91-e211-41f4-a929-d3e45bc82366-kube-api-access-r6znv\" (UniqueName: \"kubernetes.io/projected/c6d73c91-e211-41f4-a929-d3e45bc82366-kube-api-access-r6znv\") pod \"c6d73c91-e211-41f4-a929-d3e45bc82366\" (UID: \"c6d73c91-e211-41f4-a929-d3e45bc82366\") "
Nov 02 20:03:47.644084 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:47.644043    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6d73c91-e211-41f4-a929-d3e45bc82366-workload-socket" pod "c6d73c91-e211-41f4-a929-d3e45bc82366" (UID: "c6d73c91-e211-41f4-a929-d3e45bc82366"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:03:47.644405 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:47.644359    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6d73c91-e211-41f4-a929-d3e45bc82366-workload-certs" pod "c6d73c91-e211-41f4-a929-d3e45bc82366" (UID: "c6d73c91-e211-41f4-a929-d3e45bc82366"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:03:47.644405 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:47.644384    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6d73c91-e211-41f4-a929-d3e45bc82366-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6d73c91-e211-41f4-a929-d3e45bc82366-credential-socket\") pod \"c6d73c91-e211-41f4-a929-d3e45bc82366\" (UID: \"c6d73c91-e211-41f4-a929-d3e45bc82366\") "
Nov 02 20:03:47.644611 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:47.644460    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6d73c91-e211-41f4-a929-d3e45bc82366-istio-token\" (UniqueName: \"kubernetes.io/projected/c6d73c91-e211-41f4-a929-d3e45bc82366-istio-token\") pod \"c6d73c91-e211-41f4-a929-d3e45bc82366\" (UID: \"c6d73c91-e211-41f4-a929-d3e45bc82366\") "
Nov 02 20:03:47.644611 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:47.644498    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6d73c91-e211-41f4-a929-d3e45bc82366-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6d73c91-e211-41f4-a929-d3e45bc82366-istio-data\") pod \"c6d73c91-e211-41f4-a929-d3e45bc82366\" (UID: \"c6d73c91-e211-41f4-a929-d3e45bc82366\") "
Nov 02 20:03:47.644611 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:47.644572    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6d73c91-e211-41f4-a929-d3e45bc82366-credential-socket" pod "c6d73c91-e211-41f4-a929-d3e45bc82366" (UID: "c6d73c91-e211-41f4-a929-d3e45bc82366"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:03:47.644611 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:47.644597    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6d73c91-e211-41f4-a929-d3e45bc82366-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6d73c91-e211-41f4-a929-d3e45bc82366-istiod-ca-cert\") pod \"c6d73c91-e211-41f4-a929-d3e45bc82366\" (UID: \"c6d73c91-e211-41f4-a929-d3e45bc82366\") "
Nov 02 20:03:47.645134 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:47.645107    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6d73c91-e211-41f4-a929-d3e45bc82366-istio-data" pod "c6d73c91-e211-41f4-a929-d3e45bc82366" (UID: "c6d73c91-e211-41f4-a929-d3e45bc82366"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:03:47.645250 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:47.645169    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6d73c91-e211-41f4-a929-d3e45bc82366-istiod-ca-cert" pod "c6d73c91-e211-41f4-a929-d3e45bc82366" (UID: "c6d73c91-e211-41f4-a929-d3e45bc82366"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:03:47.645687 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:47.645561    4463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6d73c91-e211-41f4-a929-d3e45bc82366-workload-socket\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:03:47.645687 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:47.645585    4463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6d73c91-e211-41f4-a929-d3e45bc82366-workload-certs\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:03:47.645687 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:47.645596    4463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6d73c91-e211-41f4-a929-d3e45bc82366-credential-socket\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:03:47.645687 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:47.645606    4463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6d73c91-e211-41f4-a929-d3e45bc82366-istio-data\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:03:47.645687 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:47.645615    4463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6d73c91-e211-41f4-a929-d3e45bc82366-istiod-ca-cert\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:03:47.646937 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:47.646895    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c6d73c91-e211-41f4-a929-d3e45bc82366-istio-podinfo" pod "c6d73c91-e211-41f4-a929-d3e45bc82366" (UID: "c6d73c91-e211-41f4-a929-d3e45bc82366"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:03:47.647109 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:47.646986    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6d73c91-e211-41f4-a929-d3e45bc82366-istio-envoy" pod "c6d73c91-e211-41f4-a929-d3e45bc82366" (UID: "c6d73c91-e211-41f4-a929-d3e45bc82366"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:03:47.647109 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:47.647046    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6d73c91-e211-41f4-a929-d3e45bc82366-istio-token" pod "c6d73c91-e211-41f4-a929-d3e45bc82366" (UID: "c6d73c91-e211-41f4-a929-d3e45bc82366"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:03:47.647232 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:47.647209    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6d73c91-e211-41f4-a929-d3e45bc82366-kube-api-access-r6znv" pod "c6d73c91-e211-41f4-a929-d3e45bc82366" (UID: "c6d73c91-e211-41f4-a929-d3e45bc82366"). InnerVolumeSpecName "kube-api-access-r6znv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:03:47.746790 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:47.746755    4463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6d73c91-e211-41f4-a929-d3e45bc82366-istio-podinfo\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:03:47.746790 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:47.746789    4463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6d73c91-e211-41f4-a929-d3e45bc82366-istio-envoy\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:03:47.746986 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:47.746809    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6znv\" (UniqueName: \"kubernetes.io/projected/c6d73c91-e211-41f4-a929-d3e45bc82366-kube-api-access-r6znv\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:03:47.746986 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:47.746825    4463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6d73c91-e211-41f4-a929-d3e45bc82366-istio-token\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:03:48.082190 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:48.082153    4463 generic.go:356] "Generic (PLEG): container finished" podID="c6d73c91-e211-41f4-a929-d3e45bc82366" containerID="40104457ee4e5fee6490909a12c8470adb9202518cc38a948753eb994ebdf514" exitCode=0
Nov 02 20:03:48.082334 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:48.082206    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-4fd0d5-openshift-default-8566cd5547-tvdjk"
Nov 02 20:03:48.082334 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:48.082205    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-4fd0d5-openshift-default-8566cd5547-tvdjk" event={"ID":"c6d73c91-e211-41f4-a929-d3e45bc82366","Type":"ContainerDied","Data":"40104457ee4e5fee6490909a12c8470adb9202518cc38a948753eb994ebdf514"}
Nov 02 20:03:48.082334 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:48.082326    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-4fd0d5-openshift-default-8566cd5547-tvdjk" event={"ID":"c6d73c91-e211-41f4-a929-d3e45bc82366","Type":"ContainerDied","Data":"55d20456aecc3eeac092778416a08cad7af3e5d8affa43e9943fa93b8a5bb8cd"}
Nov 02 20:03:48.082462 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:48.082356    4463 scope.go:122] "RemoveContainer" containerID="40104457ee4e5fee6490909a12c8470adb9202518cc38a948753eb994ebdf514"
Nov 02 20:03:48.093087 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:48.093067    4463 scope.go:122] "RemoveContainer" containerID="1bee209fcbd131dde8b21eea904bd5824cfd810296e9501ca02ac775c223f888"
Nov 02 20:03:48.102036 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:48.101980    4463 scope.go:122] "RemoveContainer" containerID="40104457ee4e5fee6490909a12c8470adb9202518cc38a948753eb994ebdf514"
Nov 02 20:03:48.102276 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:03:48.102254    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40104457ee4e5fee6490909a12c8470adb9202518cc38a948753eb994ebdf514\": container with ID starting with 40104457ee4e5fee6490909a12c8470adb9202518cc38a948753eb994ebdf514 not found: ID does not exist" containerID="40104457ee4e5fee6490909a12c8470adb9202518cc38a948753eb994ebdf514"
Nov 02 20:03:48.102353 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:48.102287    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40104457ee4e5fee6490909a12c8470adb9202518cc38a948753eb994ebdf514"} err="failed to get container status \"40104457ee4e5fee6490909a12c8470adb9202518cc38a948753eb994ebdf514\": rpc error: code = NotFound desc = could not find container \"40104457ee4e5fee6490909a12c8470adb9202518cc38a948753eb994ebdf514\": container with ID starting with 40104457ee4e5fee6490909a12c8470adb9202518cc38a948753eb994ebdf514 not found: ID does not exist"
Nov 02 20:03:48.102353 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:48.102312    4463 scope.go:122] "RemoveContainer" containerID="1bee209fcbd131dde8b21eea904bd5824cfd810296e9501ca02ac775c223f888"
Nov 02 20:03:48.102573 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:03:48.102554    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bee209fcbd131dde8b21eea904bd5824cfd810296e9501ca02ac775c223f888\": container with ID starting with 1bee209fcbd131dde8b21eea904bd5824cfd810296e9501ca02ac775c223f888 not found: ID does not exist" containerID="1bee209fcbd131dde8b21eea904bd5824cfd810296e9501ca02ac775c223f888"
Nov 02 20:03:48.102613 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:48.102581    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bee209fcbd131dde8b21eea904bd5824cfd810296e9501ca02ac775c223f888"} err="failed to get container status \"1bee209fcbd131dde8b21eea904bd5824cfd810296e9501ca02ac775c223f888\": rpc error: code = NotFound desc = could not find container \"1bee209fcbd131dde8b21eea904bd5824cfd810296e9501ca02ac775c223f888\": container with ID starting with 1bee209fcbd131dde8b21eea904bd5824cfd810296e9501ca02ac775c223f888 not found: ID does not exist"
Nov 02 20:03:48.235856 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:48.235820    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-4fd0d5-openshift-default-8566cd5547-tvdjk"]
Nov 02 20:03:48.281999 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:48.281968    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-4fd0d5-openshift-default-8566cd5547-tvdjk"]
Nov 02 20:03:49.617042 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:49.613754    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6d73c91-e211-41f4-a929-d3e45bc82366" path="/var/lib/kubelet/pods/c6d73c91-e211-41f4-a929-d3e45bc82366/volumes"
Nov 02 20:03:49.896638 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:49.896549    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7025fa-6fdf99959-qcxbx"]
Nov 02 20:03:49.897639 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:49.897616    4463 cpu_manager.go:414] "containerMap: removing container" podUID="c6d73c91-e211-41f4-a929-d3e45bc82366" containerName="istio-proxy"
Nov 02 20:03:49.897758 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:49.897670    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6d73c91-e211-41f4-a929-d3e45bc82366" containerName="istio-proxy"
Nov 02 20:03:49.897758 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:49.897701    4463 cpu_manager.go:414] "containerMap: removing container" podUID="c6d73c91-e211-41f4-a929-d3e45bc82366" containerName="istio-proxy"
Nov 02 20:03:49.897758 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:49.897707    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6d73c91-e211-41f4-a929-d3e45bc82366" containerName="istio-proxy"
Nov 02 20:03:49.897758 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:49.897723    4463 cpu_manager.go:414] "containerMap: removing container" podUID="ef952ef1-649a-4551-b974-d0b00d733f7e" containerName="maas-api"
Nov 02 20:03:49.897758 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:49.897731    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef952ef1-649a-4551-b974-d0b00d733f7e" containerName="maas-api"
Nov 02 20:03:49.898033 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:49.897877    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6d73c91-e211-41f4-a929-d3e45bc82366" containerName="istio-proxy"
Nov 02 20:03:49.898033 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:49.897893    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6d73c91-e211-41f4-a929-d3e45bc82366" containerName="istio-proxy"
Nov 02 20:03:49.898033 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:49.897907    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef952ef1-649a-4551-b974-d0b00d733f7e" containerName="maas-api"
Nov 02 20:03:49.898033 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:49.897917    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="e054dacd-930b-430d-8719-3e22a12ce72f" containerName="istio-proxy"
Nov 02 20:03:49.901997 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:49.901970    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7025fa-6fdf99959-qcxbx"
Nov 02 20:03:49.952252 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:49.952226    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7025fa-6fdf99959-qcxbx"]
Nov 02 20:03:49.976989 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:49.976952    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbj5p\" (UniqueName: \"kubernetes.io/projected/93875a90-855a-489f-8ea9-971e26ad15eb-kube-api-access-dbj5p\") pod \"payload-pre-processing-e2e-inf-a-7025fa-6fdf99959-qcxbx\" (UID: \"93875a90-855a-489f-8ea9-971e26ad15eb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7025fa-6fdf99959-qcxbx"
Nov 02 20:03:49.977169 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:49.976995    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93875a90-855a-489f-8ea9-971e26ad15eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-7025fa-6fdf99959-qcxbx\" (UID: \"93875a90-855a-489f-8ea9-971e26ad15eb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7025fa-6fdf99959-qcxbx"
Nov 02 20:03:50.006033 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:49.993951    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-7025fa-dockercfg-mvjnq\""
Nov 02 20:03:50.006033 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:49.994064    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-7025fa\""
Nov 02 20:03:50.042876 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:50.042327    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7025fa-745dbc9f5f-l9mrl"]
Nov 02 20:03:50.046795 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:50.046771    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-7025fa-745dbc9f5f-l9mrl"
Nov 02 20:03:50.105041 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:50.100783    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2091abf-b755-49b3-86ad-b4bac6e2e4c5-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-7025fa-745dbc9f5f-l9mrl\" (UID: \"c2091abf-b755-49b3-86ad-b4bac6e2e4c5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7025fa-745dbc9f5f-l9mrl"
Nov 02 20:03:50.105041 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:50.100988    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbj5p\" (UniqueName: \"kubernetes.io/projected/93875a90-855a-489f-8ea9-971e26ad15eb-kube-api-access-dbj5p\") pod \"payload-pre-processing-e2e-inf-a-7025fa-6fdf99959-qcxbx\" (UID: \"93875a90-855a-489f-8ea9-971e26ad15eb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7025fa-6fdf99959-qcxbx"
Nov 02 20:03:50.105041 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:50.101092    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93875a90-855a-489f-8ea9-971e26ad15eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-7025fa-6fdf99959-qcxbx\" (UID: \"93875a90-855a-489f-8ea9-971e26ad15eb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7025fa-6fdf99959-qcxbx"
Nov 02 20:03:50.105041 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:50.101250    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hr2nz\" (UniqueName: \"kubernetes.io/projected/c2091abf-b755-49b3-86ad-b4bac6e2e4c5-kube-api-access-hr2nz\") pod \"payload-processing-e2e-inf-a-7025fa-745dbc9f5f-l9mrl\" (UID: \"c2091abf-b755-49b3-86ad-b4bac6e2e4c5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7025fa-745dbc9f5f-l9mrl"
Nov 02 20:03:50.105041 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:50.102644    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93875a90-855a-489f-8ea9-971e26ad15eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-7025fa-6fdf99959-qcxbx\" (UID: \"93875a90-855a-489f-8ea9-971e26ad15eb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7025fa-6fdf99959-qcxbx"
Nov 02 20:03:50.151124 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:50.151051    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7025fa-745dbc9f5f-l9mrl"]
Nov 02 20:03:50.156381 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:50.156304    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbj5p\" (UniqueName: \"kubernetes.io/projected/93875a90-855a-489f-8ea9-971e26ad15eb-kube-api-access-dbj5p\") pod \"payload-pre-processing-e2e-inf-a-7025fa-6fdf99959-qcxbx\" (UID: \"93875a90-855a-489f-8ea9-971e26ad15eb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7025fa-6fdf99959-qcxbx"
Nov 02 20:03:50.202524 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:50.202474    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hr2nz\" (UniqueName: \"kubernetes.io/projected/c2091abf-b755-49b3-86ad-b4bac6e2e4c5-kube-api-access-hr2nz\") pod \"payload-processing-e2e-inf-a-7025fa-745dbc9f5f-l9mrl\" (UID: \"c2091abf-b755-49b3-86ad-b4bac6e2e4c5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7025fa-745dbc9f5f-l9mrl"
Nov 02 20:03:50.202657 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:50.202640    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2091abf-b755-49b3-86ad-b4bac6e2e4c5-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-7025fa-745dbc9f5f-l9mrl\" (UID: \"c2091abf-b755-49b3-86ad-b4bac6e2e4c5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7025fa-745dbc9f5f-l9mrl"
Nov 02 20:03:50.203346 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:50.203326    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2091abf-b755-49b3-86ad-b4bac6e2e4c5-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-7025fa-745dbc9f5f-l9mrl\" (UID: \"c2091abf-b755-49b3-86ad-b4bac6e2e4c5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7025fa-745dbc9f5f-l9mrl"
Nov 02 20:03:50.217099 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:50.217075    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7025fa-6fdf99959-qcxbx"
Nov 02 20:03:50.255271 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:50.255238    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hr2nz\" (UniqueName: \"kubernetes.io/projected/c2091abf-b755-49b3-86ad-b4bac6e2e4c5-kube-api-access-hr2nz\") pod \"payload-processing-e2e-inf-a-7025fa-745dbc9f5f-l9mrl\" (UID: \"c2091abf-b755-49b3-86ad-b4bac6e2e4c5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7025fa-745dbc9f5f-l9mrl"
Nov 02 20:03:50.441696 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:50.441666    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-7025fa-745dbc9f5f-l9mrl"
Nov 02 20:03:50.585593 ip-10-0-133-9 kubenswrapper[4463]: W1102 20:03:50.585554    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93875a90_855a_489f_8ea9_971e26ad15eb.slice/crio-9f00b584d8e6317ed281f45d9de8faa706c4028e52ab1f4bad719154ffdb6332 WatchSource:0}: Error finding container 9f00b584d8e6317ed281f45d9de8faa706c4028e52ab1f4bad719154ffdb6332: Status 404 returned error can't find the container with id 9f00b584d8e6317ed281f45d9de8faa706c4028e52ab1f4bad719154ffdb6332
Nov 02 20:03:50.586735 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:50.586710    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7025fa-6fdf99959-qcxbx"]
Nov 02 20:03:50.810898 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:50.810867    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7025fa-745dbc9f5f-l9mrl"]
Nov 02 20:03:50.812111 ip-10-0-133-9 kubenswrapper[4463]: W1102 20:03:50.812081    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2091abf_b755_49b3_86ad_b4bac6e2e4c5.slice/crio-37c55330cc54c385c4be101a21725b79d8c720e78cca136f91d9987221557d95 WatchSource:0}: Error finding container 37c55330cc54c385c4be101a21725b79d8c720e78cca136f91d9987221557d95: Status 404 returned error can't find the container with id 37c55330cc54c385c4be101a21725b79d8c720e78cca136f91d9987221557d95
Nov 02 20:03:51.161306 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:51.161219    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-7025fa-745dbc9f5f-l9mrl" event={"ID":"c2091abf-b755-49b3-86ad-b4bac6e2e4c5","Type":"ContainerStarted","Data":"b1b8fe35bd9dce67dd5782a462f0aa4aa76229a6c02a35fd40c9213ea6ab0bcf"}
Nov 02 20:03:51.161306 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:51.161284    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-7025fa-745dbc9f5f-l9mrl" event={"ID":"c2091abf-b755-49b3-86ad-b4bac6e2e4c5","Type":"ContainerStarted","Data":"37c55330cc54c385c4be101a21725b79d8c720e78cca136f91d9987221557d95"}
Nov 02 20:03:51.161546 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:51.161314    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-7025fa-745dbc9f5f-l9mrl"
Nov 02 20:03:51.163107 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:51.163077    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7025fa-6fdf99959-qcxbx" event={"ID":"93875a90-855a-489f-8ea9-971e26ad15eb","Type":"ContainerStarted","Data":"f6f88cbc40faea3f73fd7cbe78f688ad170c9f4317f493cf2fb557451d6511cc"}
Nov 02 20:03:51.163225 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:51.163126    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7025fa-6fdf99959-qcxbx" event={"ID":"93875a90-855a-489f-8ea9-971e26ad15eb","Type":"ContainerStarted","Data":"9f00b584d8e6317ed281f45d9de8faa706c4028e52ab1f4bad719154ffdb6332"}
Nov 02 20:03:51.163225 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:51.163150    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7025fa-6fdf99959-qcxbx"
Nov 02 20:03:51.267351 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:51.267304    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-7025fa-745dbc9f5f-l9mrl" podStartSLOduration=2.267286902 podStartE2EDuration="2.267286902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:03:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:03:51.266053903 +0000 UTC m=+2137.873087672" watchObservedRunningTime="2026-11-02 20:03:51.267286902 +0000 UTC m=+2137.874320672"
Nov 02 20:03:51.372760 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:51.372715    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7025fa-6fdf99959-qcxbx" podStartSLOduration=2.3727019240000002 podStartE2EDuration="2.372701924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:03:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:03:51.369918645 +0000 UTC m=+2137.976952414" watchObservedRunningTime="2026-11-02 20:03:51.372701924 +0000 UTC m=+2137.979735690"
Nov 02 20:03:52.994899 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:52.994872    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-838bc1-869f9bc899-68j9k"
Nov 02 20:03:55.912766 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:55.912734    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 20:03:56.920601 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:56.920562    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-32abda-64b746699-qq2dz"]
Nov 02 20:03:57.039319 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:03:57.039288    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-32abda-9b855d4f9-qb8kc"]
Nov 02 20:04:01.921817 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:01.921760    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-32abda-64b746699-qq2dz" podUID="38b5c5e1-3522-458d-a7c4-950105f94a3e" containerName="payload-processing" containerID="cri-o://bfaf5d1476b8343df33b39d5bddd056880acbdb825e1e0e40b3ee30a35015ab1" gracePeriod=25
Nov 02 20:04:02.039834 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:02.039781    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-32abda-9b855d4f9-qb8kc" podUID="9917b249-a6ac-4b35-babb-fc0ca2ae6a83" containerName="payload-pre-processing" containerID="cri-o://e00434e74a1f24ab8d0e99840da58e904e897fa2d14e63dd959348b952990810" gracePeriod=25
Nov 02 20:04:02.173638 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:02.173569    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7025fa-6fdf99959-qcxbx"
Nov 02 20:04:02.174940 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:02.174188    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-7025fa-745dbc9f5f-l9mrl"
Nov 02 20:04:02.194820 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:02.194795    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-32abda-64b746699-qq2dz"
Nov 02 20:04:02.242863 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:02.242826    4463 generic.go:356] "Generic (PLEG): container finished" podID="38b5c5e1-3522-458d-a7c4-950105f94a3e" containerID="bfaf5d1476b8343df33b39d5bddd056880acbdb825e1e0e40b3ee30a35015ab1" exitCode=0
Nov 02 20:04:02.243072 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:02.242871    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-32abda-64b746699-qq2dz"
Nov 02 20:04:02.243072 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:02.242904    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-32abda-64b746699-qq2dz" event={"ID":"38b5c5e1-3522-458d-a7c4-950105f94a3e","Type":"ContainerDied","Data":"bfaf5d1476b8343df33b39d5bddd056880acbdb825e1e0e40b3ee30a35015ab1"}
Nov 02 20:04:02.243072 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:02.242962    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-32abda-64b746699-qq2dz" event={"ID":"38b5c5e1-3522-458d-a7c4-950105f94a3e","Type":"ContainerDied","Data":"098525b4a88f88acc6d39355c49b7d4906a20d0c587bd034b65ce66ea20ff737"}
Nov 02 20:04:02.243072 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:02.242991    4463 scope.go:122] "RemoveContainer" containerID="bfaf5d1476b8343df33b39d5bddd056880acbdb825e1e0e40b3ee30a35015ab1"
Nov 02 20:04:02.245096 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:02.245073    4463 generic.go:356] "Generic (PLEG): container finished" podID="9917b249-a6ac-4b35-babb-fc0ca2ae6a83" containerID="e00434e74a1f24ab8d0e99840da58e904e897fa2d14e63dd959348b952990810" exitCode=0
Nov 02 20:04:02.245203 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:02.245150    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-32abda-9b855d4f9-qb8kc" event={"ID":"9917b249-a6ac-4b35-babb-fc0ca2ae6a83","Type":"ContainerDied","Data":"e00434e74a1f24ab8d0e99840da58e904e897fa2d14e63dd959348b952990810"}
Nov 02 20:04:02.256707 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:02.256687    4463 scope.go:122] "RemoveContainer" containerID="bfaf5d1476b8343df33b39d5bddd056880acbdb825e1e0e40b3ee30a35015ab1"
Nov 02 20:04:02.256938 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:04:02.256918    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfaf5d1476b8343df33b39d5bddd056880acbdb825e1e0e40b3ee30a35015ab1\": container with ID starting with bfaf5d1476b8343df33b39d5bddd056880acbdb825e1e0e40b3ee30a35015ab1 not found: ID does not exist" containerID="bfaf5d1476b8343df33b39d5bddd056880acbdb825e1e0e40b3ee30a35015ab1"
Nov 02 20:04:02.257039 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:02.256948    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfaf5d1476b8343df33b39d5bddd056880acbdb825e1e0e40b3ee30a35015ab1"} err="failed to get container status \"bfaf5d1476b8343df33b39d5bddd056880acbdb825e1e0e40b3ee30a35015ab1\": rpc error: code = NotFound desc = could not find container \"bfaf5d1476b8343df33b39d5bddd056880acbdb825e1e0e40b3ee30a35015ab1\": container with ID starting with bfaf5d1476b8343df33b39d5bddd056880acbdb825e1e0e40b3ee30a35015ab1 not found: ID does not exist"
Nov 02 20:04:02.277330 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:02.277307    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38b5c5e1-3522-458d-a7c4-950105f94a3e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38b5c5e1-3522-458d-a7c4-950105f94a3e-plugins-config-volume\") pod \"38b5c5e1-3522-458d-a7c4-950105f94a3e\" (UID: \"38b5c5e1-3522-458d-a7c4-950105f94a3e\") "
Nov 02 20:04:02.277451 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:02.277345    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38b5c5e1-3522-458d-a7c4-950105f94a3e-kube-api-access-gt6fd\" (UniqueName: \"kubernetes.io/projected/38b5c5e1-3522-458d-a7c4-950105f94a3e-kube-api-access-gt6fd\") pod \"38b5c5e1-3522-458d-a7c4-950105f94a3e\" (UID: \"38b5c5e1-3522-458d-a7c4-950105f94a3e\") "
Nov 02 20:04:02.277758 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:02.277733    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38b5c5e1-3522-458d-a7c4-950105f94a3e-plugins-config-volume" pod "38b5c5e1-3522-458d-a7c4-950105f94a3e" (UID: "38b5c5e1-3522-458d-a7c4-950105f94a3e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:04:02.278171 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:02.278150    4463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38b5c5e1-3522-458d-a7c4-950105f94a3e-plugins-config-volume\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:04:02.289051 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:02.281256    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38b5c5e1-3522-458d-a7c4-950105f94a3e-kube-api-access-gt6fd" pod "38b5c5e1-3522-458d-a7c4-950105f94a3e" (UID: "38b5c5e1-3522-458d-a7c4-950105f94a3e"). InnerVolumeSpecName "kube-api-access-gt6fd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:04:02.320443 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:02.320422    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-32abda-9b855d4f9-qb8kc"
Nov 02 20:04:02.379559 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:02.379534    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9917b249-a6ac-4b35-babb-fc0ca2ae6a83-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9917b249-a6ac-4b35-babb-fc0ca2ae6a83-plugins-config-volume\") pod \"9917b249-a6ac-4b35-babb-fc0ca2ae6a83\" (UID: \"9917b249-a6ac-4b35-babb-fc0ca2ae6a83\") "
Nov 02 20:04:02.379703 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:02.379623    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9917b249-a6ac-4b35-babb-fc0ca2ae6a83-kube-api-access-jwzrz\" (UniqueName: \"kubernetes.io/projected/9917b249-a6ac-4b35-babb-fc0ca2ae6a83-kube-api-access-jwzrz\") pod \"9917b249-a6ac-4b35-babb-fc0ca2ae6a83\" (UID: \"9917b249-a6ac-4b35-babb-fc0ca2ae6a83\") "
Nov 02 20:04:02.379967 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:02.379940    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9917b249-a6ac-4b35-babb-fc0ca2ae6a83-plugins-config-volume" pod "9917b249-a6ac-4b35-babb-fc0ca2ae6a83" (UID: "9917b249-a6ac-4b35-babb-fc0ca2ae6a83"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:04:02.380071 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:02.380057    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gt6fd\" (UniqueName: \"kubernetes.io/projected/38b5c5e1-3522-458d-a7c4-950105f94a3e-kube-api-access-gt6fd\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:04:02.381663 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:02.381646    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9917b249-a6ac-4b35-babb-fc0ca2ae6a83-kube-api-access-jwzrz" pod "9917b249-a6ac-4b35-babb-fc0ca2ae6a83" (UID: "9917b249-a6ac-4b35-babb-fc0ca2ae6a83"). InnerVolumeSpecName "kube-api-access-jwzrz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:04:02.481403 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:02.481332    4463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9917b249-a6ac-4b35-babb-fc0ca2ae6a83-plugins-config-volume\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:04:02.481403 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:02.481357    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwzrz\" (UniqueName: \"kubernetes.io/projected/9917b249-a6ac-4b35-babb-fc0ca2ae6a83-kube-api-access-jwzrz\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:04:02.704785 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:02.704746    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-32abda-64b746699-qq2dz"]
Nov 02 20:04:02.708574 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:02.708543    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-32abda-64b746699-qq2dz"]
Nov 02 20:04:03.252771 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:03.252699    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-32abda-9b855d4f9-qb8kc"
Nov 02 20:04:03.252771 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:03.252742    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-32abda-9b855d4f9-qb8kc" event={"ID":"9917b249-a6ac-4b35-babb-fc0ca2ae6a83","Type":"ContainerDied","Data":"f0351acae1dcac8a6098ea9b28083ab97558c043ddf03b6ed7452ff2712f1f77"}
Nov 02 20:04:03.253235 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:03.252804    4463 scope.go:122] "RemoveContainer" containerID="e00434e74a1f24ab8d0e99840da58e904e897fa2d14e63dd959348b952990810"
Nov 02 20:04:03.409066 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:03.409030    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-32abda-9b855d4f9-qb8kc"]
Nov 02 20:04:03.456606 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:03.456569    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-32abda-9b855d4f9-qb8kc"]
Nov 02 20:04:03.585396 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:03.585314    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38b5c5e1-3522-458d-a7c4-950105f94a3e" path="/var/lib/kubelet/pods/38b5c5e1-3522-458d-a7c4-950105f94a3e/volumes"
Nov 02 20:04:03.585803 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:03.585781    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9917b249-a6ac-4b35-babb-fc0ca2ae6a83" path="/var/lib/kubelet/pods/9917b249-a6ac-4b35-babb-fc0ca2ae6a83/volumes"
Nov 02 20:04:04.295173 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:04:04.295137    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 20:04:10.012783 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:10.012745    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5bbdh"]
Nov 02 20:04:10.013211 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:10.012963    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5bbdh" podUID="6b600454-98b7-49b2-a129-6b65406c4d9c" containerName="istio-proxy" containerID="cri-o://0da393673fc95d9a09c11a20ca0ebf71d39f7d1ff7395aef4ff48346a0a341f1" gracePeriod=30
Nov 02 20:04:10.059521 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:10.059481    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-4c5vf"]
Nov 02 20:04:10.060397 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:10.059691    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-4c5vf" podUID="630de598-0bde-4b3e-af80-b014ca9db567" containerName="istio-proxy" containerID="cri-o://f63a3558901bb7ac91366545060140770db4cd271caad3969940a232c422d1fc" gracePeriod=30
Nov 02 20:04:14.483262 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:14.483182    4463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-5bbdh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 20:04:14.483657 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:14.483255    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5bbdh" podUID="6b600454-98b7-49b2-a129-6b65406c4d9c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 20:04:14.643213 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:14.643175    4463 patch_prober.go:29] interesting pod/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-4c5vf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 20:04:14.643377 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:14.643229    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-4c5vf" podUID="630de598-0bde-4b3e-af80-b014ca9db567" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 20:04:15.336584 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:15.336553    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5bbdh"
Nov 02 20:04:15.350637 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:15.350610    4463 generic.go:356] "Generic (PLEG): container finished" podID="630de598-0bde-4b3e-af80-b014ca9db567" containerID="f63a3558901bb7ac91366545060140770db4cd271caad3969940a232c422d1fc" exitCode=0
Nov 02 20:04:15.350780 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:15.350692    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-4c5vf" event={"ID":"630de598-0bde-4b3e-af80-b014ca9db567","Type":"ContainerDied","Data":"f63a3558901bb7ac91366545060140770db4cd271caad3969940a232c422d1fc"}
Nov 02 20:04:15.352063 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:15.352038    4463 generic.go:356] "Generic (PLEG): container finished" podID="6b600454-98b7-49b2-a129-6b65406c4d9c" containerID="0da393673fc95d9a09c11a20ca0ebf71d39f7d1ff7395aef4ff48346a0a341f1" exitCode=0
Nov 02 20:04:15.352167 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:15.352100    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5bbdh"
Nov 02 20:04:15.352167 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:15.352138    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5bbdh" event={"ID":"6b600454-98b7-49b2-a129-6b65406c4d9c","Type":"ContainerDied","Data":"0da393673fc95d9a09c11a20ca0ebf71d39f7d1ff7395aef4ff48346a0a341f1"}
Nov 02 20:04:15.352246 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:15.352200    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5bbdh" event={"ID":"6b600454-98b7-49b2-a129-6b65406c4d9c","Type":"ContainerDied","Data":"7a476d4de6ea333903df129c65af4badc21e1e047e6ea4871267fafcd8efe223"}
Nov 02 20:04:15.352246 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:15.352233    4463 scope.go:122] "RemoveContainer" containerID="0da393673fc95d9a09c11a20ca0ebf71d39f7d1ff7395aef4ff48346a0a341f1"
Nov 02 20:04:15.362583 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:15.362565    4463 scope.go:122] "RemoveContainer" containerID="0da393673fc95d9a09c11a20ca0ebf71d39f7d1ff7395aef4ff48346a0a341f1"
Nov 02 20:04:15.362890 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:04:15.362868    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0da393673fc95d9a09c11a20ca0ebf71d39f7d1ff7395aef4ff48346a0a341f1\": container with ID starting with 0da393673fc95d9a09c11a20ca0ebf71d39f7d1ff7395aef4ff48346a0a341f1 not found: ID does not exist" containerID="0da393673fc95d9a09c11a20ca0ebf71d39f7d1ff7395aef4ff48346a0a341f1"
Nov 02 20:04:15.362948 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:15.362897    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0da393673fc95d9a09c11a20ca0ebf71d39f7d1ff7395aef4ff48346a0a341f1"} err="failed to get container status \"0da393673fc95d9a09c11a20ca0ebf71d39f7d1ff7395aef4ff48346a0a341f1\": rpc error: code = NotFound desc = could not find container \"0da393673fc95d9a09c11a20ca0ebf71d39f7d1ff7395aef4ff48346a0a341f1\": container with ID starting with 0da393673fc95d9a09c11a20ca0ebf71d39f7d1ff7395aef4ff48346a0a341f1 not found: ID does not exist"
Nov 02 20:04:15.383266 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:15.383235    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-4c5vf"
Nov 02 20:04:15.459998 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:15.459964    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/630de598-0bde-4b3e-af80-b014ca9db567-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/630de598-0bde-4b3e-af80-b014ca9db567-workload-socket\") pod \"630de598-0bde-4b3e-af80-b014ca9db567\" (UID: \"630de598-0bde-4b3e-af80-b014ca9db567\") "
Nov 02 20:04:15.460180 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:15.460024    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/630de598-0bde-4b3e-af80-b014ca9db567-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/630de598-0bde-4b3e-af80-b014ca9db567-istio-podinfo\") pod \"630de598-0bde-4b3e-af80-b014ca9db567\" (UID: \"630de598-0bde-4b3e-af80-b014ca9db567\") "
Nov 02 20:04:15.460180 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:15.460054    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/630de598-0bde-4b3e-af80-b014ca9db567-istio-token\" (UniqueName: \"kubernetes.io/projected/630de598-0bde-4b3e-af80-b014ca9db567-istio-token\") pod \"630de598-0bde-4b3e-af80-b014ca9db567\" (UID: \"630de598-0bde-4b3e-af80-b014ca9db567\") "
Nov 02 20:04:15.460180 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:15.460080    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b600454-98b7-49b2-a129-6b65406c4d9c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b600454-98b7-49b2-a129-6b65406c4d9c-workload-certs\") pod \"6b600454-98b7-49b2-a129-6b65406c4d9c\" (UID: \"6b600454-98b7-49b2-a129-6b65406c4d9c\") "
Nov 02 20:04:15.460371 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:15.460229    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6b600454-98b7-49b2-a129-6b65406c4d9c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b600454-98b7-49b2-a129-6b65406c4d9c-istio-podinfo\") pod \"6b600454-98b7-49b2-a129-6b65406c4d9c\" (UID: \"6b600454-98b7-49b2-a129-6b65406c4d9c\") "
Nov 02 20:04:15.460371 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:15.460247    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/630de598-0bde-4b3e-af80-b014ca9db567-workload-socket" pod "630de598-0bde-4b3e-af80-b014ca9db567" (UID: "630de598-0bde-4b3e-af80-b014ca9db567"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:04:15.460371 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:15.460307    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/630de598-0bde-4b3e-af80-b014ca9db567-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/630de598-0bde-4b3e-af80-b014ca9db567-istiod-ca-cert\") pod \"630de598-0bde-4b3e-af80-b014ca9db567\" (UID: \"630de598-0bde-4b3e-af80-b014ca9db567\") "
Nov 02 20:04:15.460371 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:15.460367    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b600454-98b7-49b2-a129-6b65406c4d9c-kube-api-access-npr8j\" (UniqueName: \"kubernetes.io/projected/6b600454-98b7-49b2-a129-6b65406c4d9c-kube-api-access-npr8j\") pod \"6b600454-98b7-49b2-a129-6b65406c4d9c\" (UID: \"6b600454-98b7-49b2-a129-6b65406c4d9c\") "
Nov 02 20:04:15.460587 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:15.460389    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b600454-98b7-49b2-a129-6b65406c4d9c-workload-certs" pod "6b600454-98b7-49b2-a129-6b65406c4d9c" (UID: "6b600454-98b7-49b2-a129-6b65406c4d9c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:04:15.460587 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:15.460450    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/630de598-0bde-4b3e-af80-b014ca9db567-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/630de598-0bde-4b3e-af80-b014ca9db567-workload-certs\") pod \"630de598-0bde-4b3e-af80-b014ca9db567\" (UID: \"630de598-0bde-4b3e-af80-b014ca9db567\") "
Nov 02 20:04:15.460587 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:15.460505    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b600454-98b7-49b2-a129-6b65406c4d9c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b600454-98b7-49b2-a129-6b65406c4d9c-istio-data\") pod \"6b600454-98b7-49b2-a129-6b65406c4d9c\" (UID: \"6b600454-98b7-49b2-a129-6b65406c4d9c\") "
Nov 02 20:04:15.460587 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:15.460541    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/630de598-0bde-4b3e-af80-b014ca9db567-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/630de598-0bde-4b3e-af80-b014ca9db567-istio-envoy\") pod \"630de598-0bde-4b3e-af80-b014ca9db567\" (UID: \"630de598-0bde-4b3e-af80-b014ca9db567\") "
Nov 02 20:04:15.460587 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:15.460580    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b600454-98b7-49b2-a129-6b65406c4d9c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b600454-98b7-49b2-a129-6b65406c4d9c-istio-envoy\") pod \"6b600454-98b7-49b2-a129-6b65406c4d9c\" (UID: \"6b600454-98b7-49b2-a129-6b65406c4d9c\") "
Nov 02 20:04:15.460843 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:15.460615    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b600454-98b7-49b2-a129-6b65406c4d9c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b600454-98b7-49b2-a129-6b65406c4d9c-workload-socket\") pod \"6b600454-98b7-49b2-a129-6b65406c4d9c\" (UID: \"6b600454-98b7-49b2-a129-6b65406c4d9c\") "
Nov 02 20:04:15.460843 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:15.460668    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b600454-98b7-49b2-a129-6b65406c4d9c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b600454-98b7-49b2-a129-6b65406c4d9c-istiod-ca-cert\") pod \"6b600454-98b7-49b2-a129-6b65406c4d9c\" (UID: \"6b600454-98b7-49b2-a129-6b65406c4d9c\") "
Nov 02 20:04:15.460843 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:15.460683    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/630de598-0bde-4b3e-af80-b014ca9db567-istiod-ca-cert" pod "630de598-0bde-4b3e-af80-b014ca9db567" (UID: "630de598-0bde-4b3e-af80-b014ca9db567"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:04:15.460843 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:15.460749    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b600454-98b7-49b2-a129-6b65406c4d9c-istio-token\" (UniqueName: \"kubernetes.io/projected/6b600454-98b7-49b2-a129-6b65406c4d9c-istio-token\") pod \"6b600454-98b7-49b2-a129-6b65406c4d9c\" (UID: \"6b600454-98b7-49b2-a129-6b65406c4d9c\") "
Nov 02 20:04:15.460843 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:15.460833    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/630de598-0bde-4b3e-af80-b014ca9db567-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/630de598-0bde-4b3e-af80-b014ca9db567-istio-data\") pod \"630de598-0bde-4b3e-af80-b014ca9db567\" (UID: \"630de598-0bde-4b3e-af80-b014ca9db567\") "
Nov 02 20:04:15.461118 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:15.460872    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/630de598-0bde-4b3e-af80-b014ca9db567-kube-api-access-5w68q\" (UniqueName: \"kubernetes.io/projected/630de598-0bde-4b3e-af80-b014ca9db567-kube-api-access-5w68q\") pod \"630de598-0bde-4b3e-af80-b014ca9db567\" (UID: \"630de598-0bde-4b3e-af80-b014ca9db567\") "
Nov 02 20:04:15.461118 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:15.460924    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b600454-98b7-49b2-a129-6b65406c4d9c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b600454-98b7-49b2-a129-6b65406c4d9c-credential-socket\") pod \"6b600454-98b7-49b2-a129-6b65406c4d9c\" (UID: \"6b600454-98b7-49b2-a129-6b65406c4d9c\") "
Nov 02 20:04:15.461118 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:15.460993    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/630de598-0bde-4b3e-af80-b014ca9db567-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/630de598-0bde-4b3e-af80-b014ca9db567-credential-socket\") pod \"630de598-0bde-4b3e-af80-b014ca9db567\" (UID: \"630de598-0bde-4b3e-af80-b014ca9db567\") "
Nov 02 20:04:15.461271 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:15.461235    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b600454-98b7-49b2-a129-6b65406c4d9c-istio-data" pod "6b600454-98b7-49b2-a129-6b65406c4d9c" (UID: "6b600454-98b7-49b2-a129-6b65406c4d9c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:04:15.461271 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:15.461247    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/630de598-0bde-4b3e-af80-b014ca9db567-workload-certs" pod "630de598-0bde-4b3e-af80-b014ca9db567" (UID: "630de598-0bde-4b3e-af80-b014ca9db567"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:04:15.461472 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:15.461446    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/630de598-0bde-4b3e-af80-b014ca9db567-credential-socket" pod "630de598-0bde-4b3e-af80-b014ca9db567" (UID: "630de598-0bde-4b3e-af80-b014ca9db567"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:04:15.461791 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:15.461619    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/630de598-0bde-4b3e-af80-b014ca9db567-istio-data" pod "630de598-0bde-4b3e-af80-b014ca9db567" (UID: "630de598-0bde-4b3e-af80-b014ca9db567"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:04:15.461791 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:15.461674    4463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b600454-98b7-49b2-a129-6b65406c4d9c-istio-data\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:04:15.461791 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:15.461700    4463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/630de598-0bde-4b3e-af80-b014ca9db567-credential-socket\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:04:15.461791 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:15.461716    4463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/630de598-0bde-4b3e-af80-b014ca9db567-workload-socket\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:04:15.461791 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:15.461731    4463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b600454-98b7-49b2-a129-6b65406c4d9c-workload-certs\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:04:15.461791 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:15.461748    4463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/630de598-0bde-4b3e-af80-b014ca9db567-istiod-ca-cert\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:04:15.461791 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:15.461761    4463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/630de598-0bde-4b3e-af80-b014ca9db567-workload-certs\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:04:15.461791 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:15.461678    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b600454-98b7-49b2-a129-6b65406c4d9c-credential-socket" pod "6b600454-98b7-49b2-a129-6b65406c4d9c" (UID: "6b600454-98b7-49b2-a129-6b65406c4d9c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:04:15.462269 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:15.461830    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b600454-98b7-49b2-a129-6b65406c4d9c-workload-socket" pod "6b600454-98b7-49b2-a129-6b65406c4d9c" (UID: "6b600454-98b7-49b2-a129-6b65406c4d9c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:04:15.462269 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:15.461930    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b600454-98b7-49b2-a129-6b65406c4d9c-istiod-ca-cert" pod "6b600454-98b7-49b2-a129-6b65406c4d9c" (UID: "6b600454-98b7-49b2-a129-6b65406c4d9c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:04:15.463212 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:15.463181    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6b600454-98b7-49b2-a129-6b65406c4d9c-istio-podinfo" pod "6b600454-98b7-49b2-a129-6b65406c4d9c" (UID: "6b600454-98b7-49b2-a129-6b65406c4d9c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:04:15.463321 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:15.463259    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/630de598-0bde-4b3e-af80-b014ca9db567-istio-podinfo" pod "630de598-0bde-4b3e-af80-b014ca9db567" (UID: "630de598-0bde-4b3e-af80-b014ca9db567"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:04:15.463393 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:15.463334    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/630de598-0bde-4b3e-af80-b014ca9db567-istio-token" pod "630de598-0bde-4b3e-af80-b014ca9db567" (UID: "630de598-0bde-4b3e-af80-b014ca9db567"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:04:15.463393 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:15.463345    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/630de598-0bde-4b3e-af80-b014ca9db567-istio-envoy" pod "630de598-0bde-4b3e-af80-b014ca9db567" (UID: "630de598-0bde-4b3e-af80-b014ca9db567"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:04:15.463963 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:15.463939    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b600454-98b7-49b2-a129-6b65406c4d9c-kube-api-access-npr8j" pod "6b600454-98b7-49b2-a129-6b65406c4d9c" (UID: "6b600454-98b7-49b2-a129-6b65406c4d9c"). InnerVolumeSpecName "kube-api-access-npr8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:04:15.464136 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:15.464115    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/630de598-0bde-4b3e-af80-b014ca9db567-kube-api-access-5w68q" pod "630de598-0bde-4b3e-af80-b014ca9db567" (UID: "630de598-0bde-4b3e-af80-b014ca9db567"). InnerVolumeSpecName "kube-api-access-5w68q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:04:15.464384 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:15.464364    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b600454-98b7-49b2-a129-6b65406c4d9c-istio-token" pod "6b600454-98b7-49b2-a129-6b65406c4d9c" (UID: "6b600454-98b7-49b2-a129-6b65406c4d9c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:04:15.464590 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:15.464570    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b600454-98b7-49b2-a129-6b65406c4d9c-istio-envoy" pod "6b600454-98b7-49b2-a129-6b65406c4d9c" (UID: "6b600454-98b7-49b2-a129-6b65406c4d9c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:04:15.563683 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:15.563611    4463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b600454-98b7-49b2-a129-6b65406c4d9c-istio-podinfo\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:04:15.563683 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:15.563649    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npr8j\" (UniqueName: \"kubernetes.io/projected/6b600454-98b7-49b2-a129-6b65406c4d9c-kube-api-access-npr8j\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:04:15.563683 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:15.563665    4463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/630de598-0bde-4b3e-af80-b014ca9db567-istio-envoy\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:04:15.563683 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:15.563678    4463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b600454-98b7-49b2-a129-6b65406c4d9c-istio-envoy\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:04:15.564142 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:15.563693    4463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b600454-98b7-49b2-a129-6b65406c4d9c-workload-socket\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:04:15.564142 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:15.563706    4463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b600454-98b7-49b2-a129-6b65406c4d9c-istiod-ca-cert\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:04:15.564142 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:15.563721    4463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b600454-98b7-49b2-a129-6b65406c4d9c-istio-token\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:04:15.564142 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:15.563733    4463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/630de598-0bde-4b3e-af80-b014ca9db567-istio-data\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:04:15.564142 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:15.563745    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5w68q\" (UniqueName: \"kubernetes.io/projected/630de598-0bde-4b3e-af80-b014ca9db567-kube-api-access-5w68q\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:04:15.564142 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:15.563758    4463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b600454-98b7-49b2-a129-6b65406c4d9c-credential-socket\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:04:15.564142 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:15.563772    4463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/630de598-0bde-4b3e-af80-b014ca9db567-istio-podinfo\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:04:15.564142 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:15.563785    4463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/630de598-0bde-4b3e-af80-b014ca9db567-istio-token\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:04:15.760731 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:15.760698    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5bbdh"]
Nov 02 20:04:15.765899 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:15.765870    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5bbdh"]
Nov 02 20:04:16.359606 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:16.359513    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-4c5vf" event={"ID":"630de598-0bde-4b3e-af80-b014ca9db567","Type":"ContainerDied","Data":"f566e9f2747506766595b6f202f0ef3c498a29e5ce1b0ac2eba1d2d3a89d395f"}
Nov 02 20:04:16.359606 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:16.359602    4463 scope.go:122] "RemoveContainer" containerID="f63a3558901bb7ac91366545060140770db4cd271caad3969940a232c422d1fc"
Nov 02 20:04:16.359856 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:16.359533    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-4c5vf"
Nov 02 20:04:16.517021 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:16.516975    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-4c5vf"]
Nov 02 20:04:16.562976 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:16.562943    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-4c5vf"]
Nov 02 20:04:17.588829 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:17.588791    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="630de598-0bde-4b3e-af80-b014ca9db567" path="/var/lib/kubelet/pods/630de598-0bde-4b3e-af80-b014ca9db567/volumes"
Nov 02 20:04:17.589526 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:17.589502    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b600454-98b7-49b2-a129-6b65406c4d9c" path="/var/lib/kubelet/pods/6b600454-98b7-49b2-a129-6b65406c4d9c/volumes"
Nov 02 20:04:34.295706 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:04:34.295669    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 20:04:37.970584 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:37.970550    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 20:04:40.722696 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:40.722664    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a5b64d-64599d48c8-97d8b"]
Nov 02 20:04:40.723528 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:40.723507    4463 cpu_manager.go:414] "containerMap: removing container" podUID="630de598-0bde-4b3e-af80-b014ca9db567" containerName="istio-proxy"
Nov 02 20:04:40.723528 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:40.723526    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="630de598-0bde-4b3e-af80-b014ca9db567" containerName="istio-proxy"
Nov 02 20:04:40.723677 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:40.723551    4463 cpu_manager.go:414] "containerMap: removing container" podUID="38b5c5e1-3522-458d-a7c4-950105f94a3e" containerName="payload-processing"
Nov 02 20:04:40.723677 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:40.723556    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38b5c5e1-3522-458d-a7c4-950105f94a3e" containerName="payload-processing"
Nov 02 20:04:40.723677 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:40.723567    4463 cpu_manager.go:414] "containerMap: removing container" podUID="6b600454-98b7-49b2-a129-6b65406c4d9c" containerName="istio-proxy"
Nov 02 20:04:40.723677 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:40.723571    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b600454-98b7-49b2-a129-6b65406c4d9c" containerName="istio-proxy"
Nov 02 20:04:40.723677 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:40.723583    4463 cpu_manager.go:414] "containerMap: removing container" podUID="9917b249-a6ac-4b35-babb-fc0ca2ae6a83" containerName="payload-pre-processing"
Nov 02 20:04:40.723677 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:40.723587    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9917b249-a6ac-4b35-babb-fc0ca2ae6a83" containerName="payload-pre-processing"
Nov 02 20:04:40.723677 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:40.723669    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="9917b249-a6ac-4b35-babb-fc0ca2ae6a83" containerName="payload-pre-processing"
Nov 02 20:04:40.723882 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:40.723684    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="630de598-0bde-4b3e-af80-b014ca9db567" containerName="istio-proxy"
Nov 02 20:04:40.723882 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:40.723694    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="38b5c5e1-3522-458d-a7c4-950105f94a3e" containerName="payload-processing"
Nov 02 20:04:40.723882 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:40.723701    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b600454-98b7-49b2-a129-6b65406c4d9c" containerName="istio-proxy"
Nov 02 20:04:40.727225 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:40.727200    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a5b64d-64599d48c8-97d8b"
Nov 02 20:04:40.781844 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:40.781801    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a5b64d-64599d48c8-97d8b"]
Nov 02 20:04:40.822715 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:40.822682    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-a5b64d-serving-cert\""
Nov 02 20:04:40.896694 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:40.896662    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ec406bc-f697-42e1-9a76-857c3a28d0d6-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a5b64d-64599d48c8-97d8b\" (UID: \"2ec406bc-f697-42e1-9a76-857c3a28d0d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a5b64d-64599d48c8-97d8b"
Nov 02 20:04:40.896870 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:40.896709    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7kh8\" (UniqueName: \"kubernetes.io/projected/2ec406bc-f697-42e1-9a76-857c3a28d0d6-kube-api-access-t7kh8\") pod \"maas-api-e2e-inf-a-a5b64d-64599d48c8-97d8b\" (UID: \"2ec406bc-f697-42e1-9a76-857c3a28d0d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a5b64d-64599d48c8-97d8b"
Nov 02 20:04:40.896870 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:40.896821    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ec406bc-f697-42e1-9a76-857c3a28d0d6-metrics-certs\") pod \"maas-api-e2e-inf-a-a5b64d-64599d48c8-97d8b\" (UID: \"2ec406bc-f697-42e1-9a76-857c3a28d0d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a5b64d-64599d48c8-97d8b"
Nov 02 20:04:40.896870 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:40.896852    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ec406bc-f697-42e1-9a76-857c3a28d0d6-maas-api-tls\") pod \"maas-api-e2e-inf-a-a5b64d-64599d48c8-97d8b\" (UID: \"2ec406bc-f697-42e1-9a76-857c3a28d0d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a5b64d-64599d48c8-97d8b"
Nov 02 20:04:40.998516 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:40.998421    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ec406bc-f697-42e1-9a76-857c3a28d0d6-metrics-certs\") pod \"maas-api-e2e-inf-a-a5b64d-64599d48c8-97d8b\" (UID: \"2ec406bc-f697-42e1-9a76-857c3a28d0d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a5b64d-64599d48c8-97d8b"
Nov 02 20:04:40.998516 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:40.998486    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ec406bc-f697-42e1-9a76-857c3a28d0d6-maas-api-tls\") pod \"maas-api-e2e-inf-a-a5b64d-64599d48c8-97d8b\" (UID: \"2ec406bc-f697-42e1-9a76-857c3a28d0d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a5b64d-64599d48c8-97d8b"
Nov 02 20:04:40.998788 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:40.998765    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ec406bc-f697-42e1-9a76-857c3a28d0d6-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a5b64d-64599d48c8-97d8b\" (UID: \"2ec406bc-f697-42e1-9a76-857c3a28d0d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a5b64d-64599d48c8-97d8b"
Nov 02 20:04:40.998905 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:40.998822    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7kh8\" (UniqueName: \"kubernetes.io/projected/2ec406bc-f697-42e1-9a76-857c3a28d0d6-kube-api-access-t7kh8\") pod \"maas-api-e2e-inf-a-a5b64d-64599d48c8-97d8b\" (UID: \"2ec406bc-f697-42e1-9a76-857c3a28d0d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a5b64d-64599d48c8-97d8b"
Nov 02 20:04:40.999716 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:04:40.999691    4463 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-a5b64d-serving-cert: secret "maas-api-e2e-inf-a-a5b64d-serving-cert" not found
Nov 02 20:04:40.999914 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:04:40.999898    4463 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ec406bc-f697-42e1-9a76-857c3a28d0d6-maas-api-tls podName:2ec406bc-f697-42e1-9a76-857c3a28d0d6 nodeName:}" failed. No retries permitted until 2026-11-02 20:04:41.499874809 +0000 UTC m=+2188.106908564 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2ec406bc-f697-42e1-9a76-857c3a28d0d6-maas-api-tls") pod "maas-api-e2e-inf-a-a5b64d-64599d48c8-97d8b" (UID: "2ec406bc-f697-42e1-9a76-857c3a28d0d6") : secret "maas-api-e2e-inf-a-a5b64d-serving-cert" not found
Nov 02 20:04:41.000942 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:41.000918    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ec406bc-f697-42e1-9a76-857c3a28d0d6-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a5b64d-64599d48c8-97d8b\" (UID: \"2ec406bc-f697-42e1-9a76-857c3a28d0d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a5b64d-64599d48c8-97d8b"
Nov 02 20:04:41.003184 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:41.003159    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ec406bc-f697-42e1-9a76-857c3a28d0d6-metrics-certs\") pod \"maas-api-e2e-inf-a-a5b64d-64599d48c8-97d8b\" (UID: \"2ec406bc-f697-42e1-9a76-857c3a28d0d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a5b64d-64599d48c8-97d8b"
Nov 02 20:04:41.054231 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:41.054203    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7kh8\" (UniqueName: \"kubernetes.io/projected/2ec406bc-f697-42e1-9a76-857c3a28d0d6-kube-api-access-t7kh8\") pod \"maas-api-e2e-inf-a-a5b64d-64599d48c8-97d8b\" (UID: \"2ec406bc-f697-42e1-9a76-857c3a28d0d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a5b64d-64599d48c8-97d8b"
Nov 02 20:04:41.505561 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:41.505530    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ec406bc-f697-42e1-9a76-857c3a28d0d6-maas-api-tls\") pod \"maas-api-e2e-inf-a-a5b64d-64599d48c8-97d8b\" (UID: \"2ec406bc-f697-42e1-9a76-857c3a28d0d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a5b64d-64599d48c8-97d8b"
Nov 02 20:04:41.507883 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:41.507855    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ec406bc-f697-42e1-9a76-857c3a28d0d6-maas-api-tls\") pod \"maas-api-e2e-inf-a-a5b64d-64599d48c8-97d8b\" (UID: \"2ec406bc-f697-42e1-9a76-857c3a28d0d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a5b64d-64599d48c8-97d8b"
Nov 02 20:04:41.641326 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:41.641296    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a5b64d-64599d48c8-97d8b"
Nov 02 20:04:42.011783 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:42.011752    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a5b64d-64599d48c8-97d8b"]
Nov 02 20:04:42.013027 ip-10-0-133-9 kubenswrapper[4463]: W1102 20:04:42.012988    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ec406bc_f697_42e1_9a76_857c3a28d0d6.slice/crio-e9f0edf517dfc2098d3b34bed2f269f1e0ac5daac5acd06f23c9c9d6023ad516 WatchSource:0}: Error finding container e9f0edf517dfc2098d3b34bed2f269f1e0ac5daac5acd06f23c9c9d6023ad516: Status 404 returned error can't find the container with id e9f0edf517dfc2098d3b34bed2f269f1e0ac5daac5acd06f23c9c9d6023ad516
Nov 02 20:04:42.014488 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:42.014466    4463 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 20:04:42.531487 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:42.531450    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a5b64d-64599d48c8-97d8b" event={"ID":"2ec406bc-f697-42e1-9a76-857c3a28d0d6","Type":"ContainerStarted","Data":"e9f0edf517dfc2098d3b34bed2f269f1e0ac5daac5acd06f23c9c9d6023ad516"}
Nov 02 20:04:43.539305 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:43.539255    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a5b64d-64599d48c8-97d8b" event={"ID":"2ec406bc-f697-42e1-9a76-857c3a28d0d6","Type":"ContainerStarted","Data":"6bf04077320aaa5b1d2b9e3c747afe550dce38b189b9b602c2ed5f501f43b3f1"}
Nov 02 20:04:43.539774 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:43.539335    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a5b64d-64599d48c8-97d8b"
Nov 02 20:04:43.647221 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:43.647165    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a5b64d-64599d48c8-97d8b" podStartSLOduration=2.852441093 podStartE2EDuration="3.647148526s" totalImagesPullingTime="794.707433ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:04:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:04:43.64639835 +0000 UTC m=+2190.253432139" watchObservedRunningTime="2026-11-02 20:04:43.647148526 +0000 UTC m=+2190.254182296"
Nov 02 20:04:49.564274 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:04:49.564243    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a5b64d-64599d48c8-97d8b"
Nov 02 20:05:01.217464 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:01.217430    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7025fa-56795cd64c-j844g"]
Nov 02 20:05:01.222234 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:01.222216    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7025fa-56795cd64c-j844g"
Nov 02 20:05:01.275763 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:01.275735    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7025fa-56795cd64c-j844g"]
Nov 02 20:05:01.317007 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:01.316981    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-7025fa-serving-cert\""
Nov 02 20:05:01.375211 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:01.375174    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a33f6a91-4468-4ea7-a107-c569cc82a306-openshift-service-ca\") pod \"maas-api-e2e-inf-a-7025fa-56795cd64c-j844g\" (UID: \"a33f6a91-4468-4ea7-a107-c569cc82a306\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7025fa-56795cd64c-j844g"
Nov 02 20:05:01.375384 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:01.375236    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a33f6a91-4468-4ea7-a107-c569cc82a306-maas-api-tls\") pod \"maas-api-e2e-inf-a-7025fa-56795cd64c-j844g\" (UID: \"a33f6a91-4468-4ea7-a107-c569cc82a306\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7025fa-56795cd64c-j844g"
Nov 02 20:05:01.375384 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:01.375333    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a33f6a91-4468-4ea7-a107-c569cc82a306-metrics-certs\") pod \"maas-api-e2e-inf-a-7025fa-56795cd64c-j844g\" (UID: \"a33f6a91-4468-4ea7-a107-c569cc82a306\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7025fa-56795cd64c-j844g"
Nov 02 20:05:01.375384 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:01.375360    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bchsk\" (UniqueName: \"kubernetes.io/projected/a33f6a91-4468-4ea7-a107-c569cc82a306-kube-api-access-bchsk\") pod \"maas-api-e2e-inf-a-7025fa-56795cd64c-j844g\" (UID: \"a33f6a91-4468-4ea7-a107-c569cc82a306\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7025fa-56795cd64c-j844g"
Nov 02 20:05:01.477200 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:01.477111    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a33f6a91-4468-4ea7-a107-c569cc82a306-openshift-service-ca\") pod \"maas-api-e2e-inf-a-7025fa-56795cd64c-j844g\" (UID: \"a33f6a91-4468-4ea7-a107-c569cc82a306\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7025fa-56795cd64c-j844g"
Nov 02 20:05:01.477200 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:01.477174    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a33f6a91-4468-4ea7-a107-c569cc82a306-maas-api-tls\") pod \"maas-api-e2e-inf-a-7025fa-56795cd64c-j844g\" (UID: \"a33f6a91-4468-4ea7-a107-c569cc82a306\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7025fa-56795cd64c-j844g"
Nov 02 20:05:01.477444 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:01.477219    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a33f6a91-4468-4ea7-a107-c569cc82a306-metrics-certs\") pod \"maas-api-e2e-inf-a-7025fa-56795cd64c-j844g\" (UID: \"a33f6a91-4468-4ea7-a107-c569cc82a306\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7025fa-56795cd64c-j844g"
Nov 02 20:05:01.477444 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:01.477239    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bchsk\" (UniqueName: \"kubernetes.io/projected/a33f6a91-4468-4ea7-a107-c569cc82a306-kube-api-access-bchsk\") pod \"maas-api-e2e-inf-a-7025fa-56795cd64c-j844g\" (UID: \"a33f6a91-4468-4ea7-a107-c569cc82a306\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7025fa-56795cd64c-j844g"
Nov 02 20:05:01.478458 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:01.478418    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a33f6a91-4468-4ea7-a107-c569cc82a306-openshift-service-ca\") pod \"maas-api-e2e-inf-a-7025fa-56795cd64c-j844g\" (UID: \"a33f6a91-4468-4ea7-a107-c569cc82a306\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7025fa-56795cd64c-j844g"
Nov 02 20:05:01.480477 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:01.480453    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a33f6a91-4468-4ea7-a107-c569cc82a306-maas-api-tls\") pod \"maas-api-e2e-inf-a-7025fa-56795cd64c-j844g\" (UID: \"a33f6a91-4468-4ea7-a107-c569cc82a306\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7025fa-56795cd64c-j844g"
Nov 02 20:05:01.480710 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:01.480674    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a33f6a91-4468-4ea7-a107-c569cc82a306-metrics-certs\") pod \"maas-api-e2e-inf-a-7025fa-56795cd64c-j844g\" (UID: \"a33f6a91-4468-4ea7-a107-c569cc82a306\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7025fa-56795cd64c-j844g"
Nov 02 20:05:01.530718 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:01.530683    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bchsk\" (UniqueName: \"kubernetes.io/projected/a33f6a91-4468-4ea7-a107-c569cc82a306-kube-api-access-bchsk\") pod \"maas-api-e2e-inf-a-7025fa-56795cd64c-j844g\" (UID: \"a33f6a91-4468-4ea7-a107-c569cc82a306\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7025fa-56795cd64c-j844g"
Nov 02 20:05:01.535700 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:01.535676    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7025fa-56795cd64c-j844g"
Nov 02 20:05:01.907801 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:01.907776    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7025fa-56795cd64c-j844g"]
Nov 02 20:05:01.908692 ip-10-0-133-9 kubenswrapper[4463]: W1102 20:05:01.908668    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda33f6a91_4468_4ea7_a107_c569cc82a306.slice/crio-248e1985cc3e2c81ffe1f66db340e3f00077f6cb22a98960b505ad8e2fdd85f0 WatchSource:0}: Error finding container 248e1985cc3e2c81ffe1f66db340e3f00077f6cb22a98960b505ad8e2fdd85f0: Status 404 returned error can't find the container with id 248e1985cc3e2c81ffe1f66db340e3f00077f6cb22a98960b505ad8e2fdd85f0
Nov 02 20:05:02.687816 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:02.687780    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7025fa-56795cd64c-j844g" event={"ID":"a33f6a91-4468-4ea7-a107-c569cc82a306","Type":"ContainerStarted","Data":"248e1985cc3e2c81ffe1f66db340e3f00077f6cb22a98960b505ad8e2fdd85f0"}
Nov 02 20:05:03.696316 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:03.696268    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7025fa-56795cd64c-j844g" event={"ID":"a33f6a91-4468-4ea7-a107-c569cc82a306","Type":"ContainerStarted","Data":"d6000bbd4db2bba4edf22706769b7eb4805147221b75acb81a9d1f770d733999"}
Nov 02 20:05:03.696739 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:03.696353    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7025fa-56795cd64c-j844g"
Nov 02 20:05:03.803141 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:03.802882    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7025fa-56795cd64c-j844g" podStartSLOduration=1.962363348 podStartE2EDuration="2.80286551s" totalImagesPullingTime="840.502162ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:05:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:05:03.802219697 +0000 UTC m=+2210.409253485" watchObservedRunningTime="2026-11-02 20:05:03.80286551 +0000 UTC m=+2210.409899278"
Nov 02 20:05:04.309329 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:05:04.309292    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 20:05:08.188050 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:08.187999    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ffdc4d-openshift-default-6995c74bfc-mt2f2"]
Nov 02 20:05:08.188496 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:08.188220    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-ffdc4d-openshift-default-6995c74bfc-mt2f2" podUID="d5ffe43f-1a80-4126-9cf1-251fdde99de1" containerName="istio-proxy" containerID="cri-o://fe477fa7a8d2626d231a2ee729ee2204560465ab75763a9dba0ec97543d12d40" gracePeriod=30
Nov 02 20:05:08.830737 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:08.830656    4463 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ffdc4d-openshift-default-6995c74bfc-mt2f2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 20:05:08.830737 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:08.830737    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-ffdc4d-openshift-default-6995c74bfc-mt2f2" podUID="d5ffe43f-1a80-4126-9cf1-251fdde99de1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 20:05:09.797559 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:09.797529    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7025fa-56795cd64c-j844g"
Nov 02 20:05:13.497128 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:13.497096    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ffdc4d-openshift-default-6995c74bfc-mt2f2"
Nov 02 20:05:13.534418 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:13.534379    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5ffe43f-1a80-4126-9cf1-251fdde99de1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5ffe43f-1a80-4126-9cf1-251fdde99de1-istio-envoy\") pod \"d5ffe43f-1a80-4126-9cf1-251fdde99de1\" (UID: \"d5ffe43f-1a80-4126-9cf1-251fdde99de1\") "
Nov 02 20:05:13.534610 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:13.534430    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5ffe43f-1a80-4126-9cf1-251fdde99de1-kube-api-access-hj7kt\" (UniqueName: \"kubernetes.io/projected/d5ffe43f-1a80-4126-9cf1-251fdde99de1-kube-api-access-hj7kt\") pod \"d5ffe43f-1a80-4126-9cf1-251fdde99de1\" (UID: \"d5ffe43f-1a80-4126-9cf1-251fdde99de1\") "
Nov 02 20:05:13.534610 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:13.534526    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5ffe43f-1a80-4126-9cf1-251fdde99de1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5ffe43f-1a80-4126-9cf1-251fdde99de1-istio-data\") pod \"d5ffe43f-1a80-4126-9cf1-251fdde99de1\" (UID: \"d5ffe43f-1a80-4126-9cf1-251fdde99de1\") "
Nov 02 20:05:13.534610 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:13.534546    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d5ffe43f-1a80-4126-9cf1-251fdde99de1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5ffe43f-1a80-4126-9cf1-251fdde99de1-istio-podinfo\") pod \"d5ffe43f-1a80-4126-9cf1-251fdde99de1\" (UID: \"d5ffe43f-1a80-4126-9cf1-251fdde99de1\") "
Nov 02 20:05:13.534610 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:13.534595    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5ffe43f-1a80-4126-9cf1-251fdde99de1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5ffe43f-1a80-4126-9cf1-251fdde99de1-istiod-ca-cert\") pod \"d5ffe43f-1a80-4126-9cf1-251fdde99de1\" (UID: \"d5ffe43f-1a80-4126-9cf1-251fdde99de1\") "
Nov 02 20:05:13.534802 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:13.534701    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5ffe43f-1a80-4126-9cf1-251fdde99de1-istio-token\" (UniqueName: \"kubernetes.io/projected/d5ffe43f-1a80-4126-9cf1-251fdde99de1-istio-token\") pod \"d5ffe43f-1a80-4126-9cf1-251fdde99de1\" (UID: \"d5ffe43f-1a80-4126-9cf1-251fdde99de1\") "
Nov 02 20:05:13.534850 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:13.534838    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5ffe43f-1a80-4126-9cf1-251fdde99de1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5ffe43f-1a80-4126-9cf1-251fdde99de1-credential-socket\") pod \"d5ffe43f-1a80-4126-9cf1-251fdde99de1\" (UID: \"d5ffe43f-1a80-4126-9cf1-251fdde99de1\") "
Nov 02 20:05:13.534921 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:13.534890    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5ffe43f-1a80-4126-9cf1-251fdde99de1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5ffe43f-1a80-4126-9cf1-251fdde99de1-workload-socket\") pod \"d5ffe43f-1a80-4126-9cf1-251fdde99de1\" (UID: \"d5ffe43f-1a80-4126-9cf1-251fdde99de1\") "
Nov 02 20:05:13.534971 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:13.534956    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5ffe43f-1a80-4126-9cf1-251fdde99de1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5ffe43f-1a80-4126-9cf1-251fdde99de1-workload-certs\") pod \"d5ffe43f-1a80-4126-9cf1-251fdde99de1\" (UID: \"d5ffe43f-1a80-4126-9cf1-251fdde99de1\") "
Nov 02 20:05:13.535212 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:13.535109    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5ffe43f-1a80-4126-9cf1-251fdde99de1-istiod-ca-cert" pod "d5ffe43f-1a80-4126-9cf1-251fdde99de1" (UID: "d5ffe43f-1a80-4126-9cf1-251fdde99de1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:05:13.535298 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:13.535177    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5ffe43f-1a80-4126-9cf1-251fdde99de1-istio-data" pod "d5ffe43f-1a80-4126-9cf1-251fdde99de1" (UID: "d5ffe43f-1a80-4126-9cf1-251fdde99de1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:05:13.535437 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:13.535411    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5ffe43f-1a80-4126-9cf1-251fdde99de1-credential-socket" pod "d5ffe43f-1a80-4126-9cf1-251fdde99de1" (UID: "d5ffe43f-1a80-4126-9cf1-251fdde99de1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:05:13.535616 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:13.535452    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5ffe43f-1a80-4126-9cf1-251fdde99de1-workload-socket" pod "d5ffe43f-1a80-4126-9cf1-251fdde99de1" (UID: "d5ffe43f-1a80-4126-9cf1-251fdde99de1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:05:13.535616 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:13.535531    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5ffe43f-1a80-4126-9cf1-251fdde99de1-workload-certs" pod "d5ffe43f-1a80-4126-9cf1-251fdde99de1" (UID: "d5ffe43f-1a80-4126-9cf1-251fdde99de1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:05:13.535785 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:13.535717    4463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5ffe43f-1a80-4126-9cf1-251fdde99de1-istio-data\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:05:13.535785 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:13.535740    4463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5ffe43f-1a80-4126-9cf1-251fdde99de1-istiod-ca-cert\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:05:13.535785 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:13.535755    4463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5ffe43f-1a80-4126-9cf1-251fdde99de1-credential-socket\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:05:13.535785 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:13.535773    4463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5ffe43f-1a80-4126-9cf1-251fdde99de1-workload-socket\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:05:13.536838 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:13.536809    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5ffe43f-1a80-4126-9cf1-251fdde99de1-istio-envoy" pod "d5ffe43f-1a80-4126-9cf1-251fdde99de1" (UID: "d5ffe43f-1a80-4126-9cf1-251fdde99de1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:05:13.537392 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:13.537350    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d5ffe43f-1a80-4126-9cf1-251fdde99de1-istio-podinfo" pod "d5ffe43f-1a80-4126-9cf1-251fdde99de1" (UID: "d5ffe43f-1a80-4126-9cf1-251fdde99de1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:05:13.537675 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:13.537649    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5ffe43f-1a80-4126-9cf1-251fdde99de1-kube-api-access-hj7kt" pod "d5ffe43f-1a80-4126-9cf1-251fdde99de1" (UID: "d5ffe43f-1a80-4126-9cf1-251fdde99de1"). InnerVolumeSpecName "kube-api-access-hj7kt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:05:13.538079 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:13.538045    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5ffe43f-1a80-4126-9cf1-251fdde99de1-istio-token" pod "d5ffe43f-1a80-4126-9cf1-251fdde99de1" (UID: "d5ffe43f-1a80-4126-9cf1-251fdde99de1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:05:13.636785 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:13.636746    4463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5ffe43f-1a80-4126-9cf1-251fdde99de1-istio-envoy\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:05:13.636785 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:13.636783    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hj7kt\" (UniqueName: \"kubernetes.io/projected/d5ffe43f-1a80-4126-9cf1-251fdde99de1-kube-api-access-hj7kt\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:05:13.637063 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:13.636799    4463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5ffe43f-1a80-4126-9cf1-251fdde99de1-istio-podinfo\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:05:13.637063 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:13.636809    4463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5ffe43f-1a80-4126-9cf1-251fdde99de1-istio-token\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:05:13.637063 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:13.636818    4463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5ffe43f-1a80-4126-9cf1-251fdde99de1-workload-certs\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:05:13.835395 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:13.835294    4463 generic.go:356] "Generic (PLEG): container finished" podID="d5ffe43f-1a80-4126-9cf1-251fdde99de1" containerID="fe477fa7a8d2626d231a2ee729ee2204560465ab75763a9dba0ec97543d12d40" exitCode=0
Nov 02 20:05:13.835395 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:13.835376    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ffdc4d-openshift-default-6995c74bfc-mt2f2"
Nov 02 20:05:13.835600 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:13.835393    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ffdc4d-openshift-default-6995c74bfc-mt2f2" event={"ID":"d5ffe43f-1a80-4126-9cf1-251fdde99de1","Type":"ContainerDied","Data":"fe477fa7a8d2626d231a2ee729ee2204560465ab75763a9dba0ec97543d12d40"}
Nov 02 20:05:13.835600 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:13.835496    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ffdc4d-openshift-default-6995c74bfc-mt2f2" event={"ID":"d5ffe43f-1a80-4126-9cf1-251fdde99de1","Type":"ContainerDied","Data":"2140b4dd69f01ef7bdc2dfdf6af35b9ef88a44c9f97bdacc41f146bf78f20d92"}
Nov 02 20:05:13.835600 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:13.835530    4463 scope.go:122] "RemoveContainer" containerID="fe477fa7a8d2626d231a2ee729ee2204560465ab75763a9dba0ec97543d12d40"
Nov 02 20:05:13.846744 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:13.846676    4463 scope.go:122] "RemoveContainer" containerID="ef65e7eca4ab2cbb96b24402b81accd94d6be216c8f7eb7166b44e3d7db88333"
Nov 02 20:05:13.856467 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:13.856436    4463 scope.go:122] "RemoveContainer" containerID="fe477fa7a8d2626d231a2ee729ee2204560465ab75763a9dba0ec97543d12d40"
Nov 02 20:05:13.856767 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:05:13.856743    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe477fa7a8d2626d231a2ee729ee2204560465ab75763a9dba0ec97543d12d40\": container with ID starting with fe477fa7a8d2626d231a2ee729ee2204560465ab75763a9dba0ec97543d12d40 not found: ID does not exist" containerID="fe477fa7a8d2626d231a2ee729ee2204560465ab75763a9dba0ec97543d12d40"
Nov 02 20:05:13.856859 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:13.856775    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe477fa7a8d2626d231a2ee729ee2204560465ab75763a9dba0ec97543d12d40"} err="failed to get container status \"fe477fa7a8d2626d231a2ee729ee2204560465ab75763a9dba0ec97543d12d40\": rpc error: code = NotFound desc = could not find container \"fe477fa7a8d2626d231a2ee729ee2204560465ab75763a9dba0ec97543d12d40\": container with ID starting with fe477fa7a8d2626d231a2ee729ee2204560465ab75763a9dba0ec97543d12d40 not found: ID does not exist"
Nov 02 20:05:13.856859 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:13.856793    4463 scope.go:122] "RemoveContainer" containerID="ef65e7eca4ab2cbb96b24402b81accd94d6be216c8f7eb7166b44e3d7db88333"
Nov 02 20:05:13.857065 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:05:13.857042    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef65e7eca4ab2cbb96b24402b81accd94d6be216c8f7eb7166b44e3d7db88333\": container with ID starting with ef65e7eca4ab2cbb96b24402b81accd94d6be216c8f7eb7166b44e3d7db88333 not found: ID does not exist" containerID="ef65e7eca4ab2cbb96b24402b81accd94d6be216c8f7eb7166b44e3d7db88333"
Nov 02 20:05:13.857144 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:13.857073    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef65e7eca4ab2cbb96b24402b81accd94d6be216c8f7eb7166b44e3d7db88333"} err="failed to get container status \"ef65e7eca4ab2cbb96b24402b81accd94d6be216c8f7eb7166b44e3d7db88333\": rpc error: code = NotFound desc = could not find container \"ef65e7eca4ab2cbb96b24402b81accd94d6be216c8f7eb7166b44e3d7db88333\": container with ID starting with ef65e7eca4ab2cbb96b24402b81accd94d6be216c8f7eb7166b44e3d7db88333 not found: ID does not exist"
Nov 02 20:05:13.989710 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:13.989677    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ffdc4d-openshift-default-6995c74bfc-mt2f2"]
Nov 02 20:05:13.996048 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:13.996021    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ffdc4d-openshift-default-6995c74bfc-mt2f2"]
Nov 02 20:05:15.588920 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:15.588885    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5ffe43f-1a80-4126-9cf1-251fdde99de1" path="/var/lib/kubelet/pods/d5ffe43f-1a80-4126-9cf1-251fdde99de1/volumes"
Nov 02 20:05:15.851512 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:15.851448    4463 generic.go:356] "Generic (PLEG): container finished" podID="8709c9fb-f685-43fd-a3d5-9cb73cf8329d" containerID="0435c58e164777f9c6652c868d54e806e906fb0b1f35f6619af491e1c538d23b" exitCode=0
Nov 02 20:05:15.851648 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:15.851529    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a5b64d-openshift-default-6f8c4599cf-vztxx" event={"ID":"8709c9fb-f685-43fd-a3d5-9cb73cf8329d","Type":"ContainerDied","Data":"0435c58e164777f9c6652c868d54e806e906fb0b1f35f6619af491e1c538d23b"}
Nov 02 20:05:15.851950 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:15.851935    4463 scope.go:122] "RemoveContainer" containerID="0435c58e164777f9c6652c868d54e806e906fb0b1f35f6619af491e1c538d23b"
Nov 02 20:05:15.853596 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:15.853569    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:05:15.853655 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:15.853640    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:05:15.853693 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:15.853670    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:05:16.882990 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:16.882943    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a5b64d-openshift-default-6f8c4599cf-vztxx" event={"ID":"8709c9fb-f685-43fd-a3d5-9cb73cf8329d","Type":"ContainerStarted","Data":"f18194eded39ac8ef5d038ab615bf6fa11617a4e745b907f6a601b037338b040"}
Nov 02 20:05:17.745910 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:17.745871    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a5b64d-openshift-default-6f8c4599cf-vztxx"
Nov 02 20:05:17.746135 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:17.746068    4463 patch_prober.go:29] interesting pod/e2e-inf-a-a5b64d-openshift-default-6f8c4599cf-vztxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Nov 02 20:05:17.746212 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:17.746125    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5b64d-openshift-default-6f8c4599cf-vztxx" podUID="8709c9fb-f685-43fd-a3d5-9cb73cf8329d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Nov 02 20:05:18.746629 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:18.746594    4463 patch_prober.go:29] interesting pod/e2e-inf-a-a5b64d-openshift-default-6f8c4599cf-vztxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Nov 02 20:05:18.747056 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:18.746655    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5b64d-openshift-default-6f8c4599cf-vztxx" podUID="8709c9fb-f685-43fd-a3d5-9cb73cf8329d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Nov 02 20:05:19.745918 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:19.745883    4463 patch_prober.go:29] interesting pod/e2e-inf-a-a5b64d-openshift-default-6f8c4599cf-vztxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Nov 02 20:05:19.746147 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:19.745939    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5b64d-openshift-default-6f8c4599cf-vztxx" podUID="8709c9fb-f685-43fd-a3d5-9cb73cf8329d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Nov 02 20:05:20.746937 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:20.746899    4463 patch_prober.go:29] interesting pod/e2e-inf-a-a5b64d-openshift-default-6f8c4599cf-vztxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Nov 02 20:05:20.747438 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:20.746964    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5b64d-openshift-default-6f8c4599cf-vztxx" podUID="8709c9fb-f685-43fd-a3d5-9cb73cf8329d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Nov 02 20:05:21.746378 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:21.746345    4463 patch_prober.go:29] interesting pod/e2e-inf-a-a5b64d-openshift-default-6f8c4599cf-vztxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Nov 02 20:05:21.746552 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:21.746405    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5b64d-openshift-default-6f8c4599cf-vztxx" podUID="8709c9fb-f685-43fd-a3d5-9cb73cf8329d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Nov 02 20:05:22.750873 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:22.750841    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a5b64d-openshift-default-6f8c4599cf-vztxx"
Nov 02 20:05:22.751377 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:22.751196    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a5b64d-openshift-default-6f8c4599cf-vztxx"
Nov 02 20:05:22.751936 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:22.751905    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a5b64d-openshift-default-6f8c4599cf-vztxx"
Nov 02 20:05:25.523264 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:25.523228    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-4b0a12-openshift-default-8677ff4fb5-6css2"]
Nov 02 20:05:25.523756 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:25.523429    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-4b0a12-openshift-default-8677ff4fb5-6css2" podUID="277fb120-e642-4215-bc3d-1b9c08c57e81" containerName="istio-proxy" containerID="cri-o://0ef0bbf558b30756c144f89346b9e29b5c6e55ffaf486fbc717d4fdd6d245ed0" gracePeriod=30
Nov 02 20:05:29.468612 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:29.468578    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-1d869e/distinct-2-w5-1d869e-kserve-574f55db46-4cv47"]
Nov 02 20:05:29.516552 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:29.516521    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-1d869e/premium-w5-1d869e-kserve-7745799ff8-pv2th"]
Nov 02 20:05:30.787695 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:30.787641    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4b0a12-openshift-default-8677ff4fb5-6css2"
Nov 02 20:05:30.839469 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:30.839441    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/277fb120-e642-4215-bc3d-1b9c08c57e81-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/277fb120-e642-4215-bc3d-1b9c08c57e81-credential-socket\") pod \"277fb120-e642-4215-bc3d-1b9c08c57e81\" (UID: \"277fb120-e642-4215-bc3d-1b9c08c57e81\") "
Nov 02 20:05:30.839603 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:30.839504    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/277fb120-e642-4215-bc3d-1b9c08c57e81-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/277fb120-e642-4215-bc3d-1b9c08c57e81-istio-podinfo\") pod \"277fb120-e642-4215-bc3d-1b9c08c57e81\" (UID: \"277fb120-e642-4215-bc3d-1b9c08c57e81\") "
Nov 02 20:05:30.839603 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:30.839594    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/277fb120-e642-4215-bc3d-1b9c08c57e81-kube-api-access-rqqkj\" (UniqueName: \"kubernetes.io/projected/277fb120-e642-4215-bc3d-1b9c08c57e81-kube-api-access-rqqkj\") pod \"277fb120-e642-4215-bc3d-1b9c08c57e81\" (UID: \"277fb120-e642-4215-bc3d-1b9c08c57e81\") "
Nov 02 20:05:30.839691 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:30.839663    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/277fb120-e642-4215-bc3d-1b9c08c57e81-istio-token\" (UniqueName: \"kubernetes.io/projected/277fb120-e642-4215-bc3d-1b9c08c57e81-istio-token\") pod \"277fb120-e642-4215-bc3d-1b9c08c57e81\" (UID: \"277fb120-e642-4215-bc3d-1b9c08c57e81\") "
Nov 02 20:05:30.839746 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:30.839688    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/277fb120-e642-4215-bc3d-1b9c08c57e81-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/277fb120-e642-4215-bc3d-1b9c08c57e81-istiod-ca-cert\") pod \"277fb120-e642-4215-bc3d-1b9c08c57e81\" (UID: \"277fb120-e642-4215-bc3d-1b9c08c57e81\") "
Nov 02 20:05:30.839746 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:30.839717    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/277fb120-e642-4215-bc3d-1b9c08c57e81-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/277fb120-e642-4215-bc3d-1b9c08c57e81-istio-data\") pod \"277fb120-e642-4215-bc3d-1b9c08c57e81\" (UID: \"277fb120-e642-4215-bc3d-1b9c08c57e81\") "
Nov 02 20:05:30.839920 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:30.839740    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/277fb120-e642-4215-bc3d-1b9c08c57e81-credential-socket" pod "277fb120-e642-4215-bc3d-1b9c08c57e81" (UID: "277fb120-e642-4215-bc3d-1b9c08c57e81"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:05:30.839920 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:30.839805    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/277fb120-e642-4215-bc3d-1b9c08c57e81-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/277fb120-e642-4215-bc3d-1b9c08c57e81-workload-socket\") pod \"277fb120-e642-4215-bc3d-1b9c08c57e81\" (UID: \"277fb120-e642-4215-bc3d-1b9c08c57e81\") "
Nov 02 20:05:30.839920 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:30.839864    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/277fb120-e642-4215-bc3d-1b9c08c57e81-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/277fb120-e642-4215-bc3d-1b9c08c57e81-workload-certs\") pod \"277fb120-e642-4215-bc3d-1b9c08c57e81\" (UID: \"277fb120-e642-4215-bc3d-1b9c08c57e81\") "
Nov 02 20:05:30.839920 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:30.839902    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/277fb120-e642-4215-bc3d-1b9c08c57e81-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/277fb120-e642-4215-bc3d-1b9c08c57e81-istio-envoy\") pod \"277fb120-e642-4215-bc3d-1b9c08c57e81\" (UID: \"277fb120-e642-4215-bc3d-1b9c08c57e81\") "
Nov 02 20:05:30.840187 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:30.840111    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/277fb120-e642-4215-bc3d-1b9c08c57e81-istio-data" pod "277fb120-e642-4215-bc3d-1b9c08c57e81" (UID: "277fb120-e642-4215-bc3d-1b9c08c57e81"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:05:30.840187 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:30.840124    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/277fb120-e642-4215-bc3d-1b9c08c57e81-workload-certs" pod "277fb120-e642-4215-bc3d-1b9c08c57e81" (UID: "277fb120-e642-4215-bc3d-1b9c08c57e81"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:05:30.840187 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:30.840139    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/277fb120-e642-4215-bc3d-1b9c08c57e81-workload-socket" pod "277fb120-e642-4215-bc3d-1b9c08c57e81" (UID: "277fb120-e642-4215-bc3d-1b9c08c57e81"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:05:30.840187 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:30.840176    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/277fb120-e642-4215-bc3d-1b9c08c57e81-istiod-ca-cert" pod "277fb120-e642-4215-bc3d-1b9c08c57e81" (UID: "277fb120-e642-4215-bc3d-1b9c08c57e81"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:05:30.840545 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:30.840526    4463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/277fb120-e642-4215-bc3d-1b9c08c57e81-workload-socket\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:05:30.840545 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:30.840545    4463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/277fb120-e642-4215-bc3d-1b9c08c57e81-workload-certs\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:05:30.840698 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:30.840554    4463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/277fb120-e642-4215-bc3d-1b9c08c57e81-credential-socket\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:05:30.840698 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:30.840564    4463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/277fb120-e642-4215-bc3d-1b9c08c57e81-istiod-ca-cert\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:05:30.840698 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:30.840577    4463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/277fb120-e642-4215-bc3d-1b9c08c57e81-istio-data\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:05:30.841801 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:30.841779    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/277fb120-e642-4215-bc3d-1b9c08c57e81-istio-podinfo" pod "277fb120-e642-4215-bc3d-1b9c08c57e81" (UID: "277fb120-e642-4215-bc3d-1b9c08c57e81"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:05:30.841958 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:30.841937    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/277fb120-e642-4215-bc3d-1b9c08c57e81-istio-token" pod "277fb120-e642-4215-bc3d-1b9c08c57e81" (UID: "277fb120-e642-4215-bc3d-1b9c08c57e81"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:05:30.842328 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:30.842310    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/277fb120-e642-4215-bc3d-1b9c08c57e81-istio-envoy" pod "277fb120-e642-4215-bc3d-1b9c08c57e81" (UID: "277fb120-e642-4215-bc3d-1b9c08c57e81"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:05:30.842407 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:30.842312    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/277fb120-e642-4215-bc3d-1b9c08c57e81-kube-api-access-rqqkj" pod "277fb120-e642-4215-bc3d-1b9c08c57e81" (UID: "277fb120-e642-4215-bc3d-1b9c08c57e81"). InnerVolumeSpecName "kube-api-access-rqqkj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:05:30.941419 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:30.941388    4463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/277fb120-e642-4215-bc3d-1b9c08c57e81-istio-envoy\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:05:30.941419 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:30.941414    4463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/277fb120-e642-4215-bc3d-1b9c08c57e81-istio-podinfo\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:05:30.941419 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:30.941424    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqqkj\" (UniqueName: \"kubernetes.io/projected/277fb120-e642-4215-bc3d-1b9c08c57e81-kube-api-access-rqqkj\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:05:30.941682 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:30.941433    4463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/277fb120-e642-4215-bc3d-1b9c08c57e81-istio-token\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:05:30.986978 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:30.986945    4463 generic.go:356] "Generic (PLEG): container finished" podID="277fb120-e642-4215-bc3d-1b9c08c57e81" containerID="0ef0bbf558b30756c144f89346b9e29b5c6e55ffaf486fbc717d4fdd6d245ed0" exitCode=0
Nov 02 20:05:30.987172 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:30.986996    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4b0a12-openshift-default-8677ff4fb5-6css2"
Nov 02 20:05:30.987172 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:30.986998    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4b0a12-openshift-default-8677ff4fb5-6css2" event={"ID":"277fb120-e642-4215-bc3d-1b9c08c57e81","Type":"ContainerDied","Data":"0ef0bbf558b30756c144f89346b9e29b5c6e55ffaf486fbc717d4fdd6d245ed0"}
Nov 02 20:05:30.987172 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:30.987152    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4b0a12-openshift-default-8677ff4fb5-6css2" event={"ID":"277fb120-e642-4215-bc3d-1b9c08c57e81","Type":"ContainerDied","Data":"5a3144b31cf58ab5df4104e6f83ea28974dd85ebef97eab70d2f77250079bfe4"}
Nov 02 20:05:30.987172 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:30.987174    4463 scope.go:122] "RemoveContainer" containerID="0ef0bbf558b30756c144f89346b9e29b5c6e55ffaf486fbc717d4fdd6d245ed0"
Nov 02 20:05:30.998056 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:30.998040    4463 scope.go:122] "RemoveContainer" containerID="985dd2c51fbde5438cae0f74b3691c5bca1361a42cb033dc5bc62ed5c386ae25"
Nov 02 20:05:31.006875 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:31.006856    4463 scope.go:122] "RemoveContainer" containerID="0ef0bbf558b30756c144f89346b9e29b5c6e55ffaf486fbc717d4fdd6d245ed0"
Nov 02 20:05:31.007214 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:05:31.007194    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ef0bbf558b30756c144f89346b9e29b5c6e55ffaf486fbc717d4fdd6d245ed0\": container with ID starting with 0ef0bbf558b30756c144f89346b9e29b5c6e55ffaf486fbc717d4fdd6d245ed0 not found: ID does not exist" containerID="0ef0bbf558b30756c144f89346b9e29b5c6e55ffaf486fbc717d4fdd6d245ed0"
Nov 02 20:05:31.007282 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:31.007222    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ef0bbf558b30756c144f89346b9e29b5c6e55ffaf486fbc717d4fdd6d245ed0"} err="failed to get container status \"0ef0bbf558b30756c144f89346b9e29b5c6e55ffaf486fbc717d4fdd6d245ed0\": rpc error: code = NotFound desc = could not find container \"0ef0bbf558b30756c144f89346b9e29b5c6e55ffaf486fbc717d4fdd6d245ed0\": container with ID starting with 0ef0bbf558b30756c144f89346b9e29b5c6e55ffaf486fbc717d4fdd6d245ed0 not found: ID does not exist"
Nov 02 20:05:31.007282 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:31.007239    4463 scope.go:122] "RemoveContainer" containerID="985dd2c51fbde5438cae0f74b3691c5bca1361a42cb033dc5bc62ed5c386ae25"
Nov 02 20:05:31.007466 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:05:31.007454    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"985dd2c51fbde5438cae0f74b3691c5bca1361a42cb033dc5bc62ed5c386ae25\": container with ID starting with 985dd2c51fbde5438cae0f74b3691c5bca1361a42cb033dc5bc62ed5c386ae25 not found: ID does not exist" containerID="985dd2c51fbde5438cae0f74b3691c5bca1361a42cb033dc5bc62ed5c386ae25"
Nov 02 20:05:31.007511 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:31.007469    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"985dd2c51fbde5438cae0f74b3691c5bca1361a42cb033dc5bc62ed5c386ae25"} err="failed to get container status \"985dd2c51fbde5438cae0f74b3691c5bca1361a42cb033dc5bc62ed5c386ae25\": rpc error: code = NotFound desc = could not find container \"985dd2c51fbde5438cae0f74b3691c5bca1361a42cb033dc5bc62ed5c386ae25\": container with ID starting with 985dd2c51fbde5438cae0f74b3691c5bca1361a42cb033dc5bc62ed5c386ae25 not found: ID does not exist"
Nov 02 20:05:31.144512 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:31.144437    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-4b0a12-openshift-default-8677ff4fb5-6css2"]
Nov 02 20:05:31.152623 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:31.152595    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-4b0a12-openshift-default-8677ff4fb5-6css2"]
Nov 02 20:05:31.586379 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:31.586339    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="277fb120-e642-4215-bc3d-1b9c08c57e81" path="/var/lib/kubelet/pods/277fb120-e642-4215-bc3d-1b9c08c57e81/volumes"
Nov 02 20:05:34.295296 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:05:34.295263    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 20:05:36.781475 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:36.781440    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-1d869e-7d8f7ccf9f-nhdnm"]
Nov 02 20:05:41.781860 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:41.781817    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-1d869e-7d8f7ccf9f-nhdnm" podUID="7ebcf427-e9e4-4445-86d8-7d4a32398724" containerName="payload-pre-processing" containerID="cri-o://99b3a6aa46bebea585ad09fe6e48386f4bafd9387866b0af60885be803e2ee75" gracePeriod=25
Nov 02 20:05:44.495517 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:44.495458    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-1d869e/distinct-2-w5-1d869e-kserve-574f55db46-4cv47" podUID="7e236108-328b-4170-be20-13f16b56c4fc" containerName="main" containerID="cri-o://710ec917c9760293985d1402b9aa8aac118aeb304ae914e8f9a310db1485730b" gracePeriod=45
Nov 02 20:05:44.536572 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:44.536535    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-1d869e/premium-w5-1d869e-kserve-7745799ff8-pv2th" podUID="1d784b10-99e4-4561-bfb7-6614877d2b6d" containerName="main" containerID="cri-o://fc910b085bc6f56193e79977d9dd7b30867c6e9144953196d6a6bb57a4ce2058" gracePeriod=45
Nov 02 20:05:44.837344 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:44.837317    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-1d869e/distinct-2-w5-1d869e-kserve-574f55db46-4cv47"
Nov 02 20:05:44.840638 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:44.840618    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-1d869e/premium-w5-1d869e-kserve-7745799ff8-pv2th"
Nov 02 20:05:44.927679 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:44.927652    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e236108-328b-4170-be20-13f16b56c4fc-tls-certs\" (UniqueName: \"kubernetes.io/secret/7e236108-328b-4170-be20-13f16b56c4fc-tls-certs\") pod \"7e236108-328b-4170-be20-13f16b56c4fc\" (UID: \"7e236108-328b-4170-be20-13f16b56c4fc\") "
Nov 02 20:05:44.927826 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:44.927711    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d784b10-99e4-4561-bfb7-6614877d2b6d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d784b10-99e4-4561-bfb7-6614877d2b6d-tmp-dir\") pod \"1d784b10-99e4-4561-bfb7-6614877d2b6d\" (UID: \"1d784b10-99e4-4561-bfb7-6614877d2b6d\") "
Nov 02 20:05:44.927826 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:44.927759    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d784b10-99e4-4561-bfb7-6614877d2b6d-home\" (UniqueName: \"kubernetes.io/empty-dir/1d784b10-99e4-4561-bfb7-6614877d2b6d-home\") pod \"1d784b10-99e4-4561-bfb7-6614877d2b6d\" (UID: \"1d784b10-99e4-4561-bfb7-6614877d2b6d\") "
Nov 02 20:05:44.927826 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:44.927779    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d784b10-99e4-4561-bfb7-6614877d2b6d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d784b10-99e4-4561-bfb7-6614877d2b6d-model-cache\") pod \"1d784b10-99e4-4561-bfb7-6614877d2b6d\" (UID: \"1d784b10-99e4-4561-bfb7-6614877d2b6d\") "
Nov 02 20:05:44.927826 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:44.927824    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e236108-328b-4170-be20-13f16b56c4fc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e236108-328b-4170-be20-13f16b56c4fc-tmp-dir\") pod \"7e236108-328b-4170-be20-13f16b56c4fc\" (UID: \"7e236108-328b-4170-be20-13f16b56c4fc\") "
Nov 02 20:05:44.928173 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:44.927851    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d784b10-99e4-4561-bfb7-6614877d2b6d-tls-certs\" (UniqueName: \"kubernetes.io/secret/1d784b10-99e4-4561-bfb7-6614877d2b6d-tls-certs\") pod \"1d784b10-99e4-4561-bfb7-6614877d2b6d\" (UID: \"1d784b10-99e4-4561-bfb7-6614877d2b6d\") "
Nov 02 20:05:44.928173 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:44.927873    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e236108-328b-4170-be20-13f16b56c4fc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e236108-328b-4170-be20-13f16b56c4fc-model-cache\") pod \"7e236108-328b-4170-be20-13f16b56c4fc\" (UID: \"7e236108-328b-4170-be20-13f16b56c4fc\") "
Nov 02 20:05:44.928173 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:44.927904    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d784b10-99e4-4561-bfb7-6614877d2b6d-kube-api-access-p268k\" (UniqueName: \"kubernetes.io/projected/1d784b10-99e4-4561-bfb7-6614877d2b6d-kube-api-access-p268k\") pod \"1d784b10-99e4-4561-bfb7-6614877d2b6d\" (UID: \"1d784b10-99e4-4561-bfb7-6614877d2b6d\") "
Nov 02 20:05:44.928173 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:44.927923    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e236108-328b-4170-be20-13f16b56c4fc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e236108-328b-4170-be20-13f16b56c4fc-dshm\") pod \"7e236108-328b-4170-be20-13f16b56c4fc\" (UID: \"7e236108-328b-4170-be20-13f16b56c4fc\") "
Nov 02 20:05:44.928173 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:44.928055    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d784b10-99e4-4561-bfb7-6614877d2b6d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d784b10-99e4-4561-bfb7-6614877d2b6d-dshm\") pod \"1d784b10-99e4-4561-bfb7-6614877d2b6d\" (UID: \"1d784b10-99e4-4561-bfb7-6614877d2b6d\") "
Nov 02 20:05:44.928173 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:44.928075    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d784b10-99e4-4561-bfb7-6614877d2b6d-model-cache" pod "1d784b10-99e4-4561-bfb7-6614877d2b6d" (UID: "1d784b10-99e4-4561-bfb7-6614877d2b6d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:05:44.928173 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:44.928095    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e236108-328b-4170-be20-13f16b56c4fc-home\" (UniqueName: \"kubernetes.io/empty-dir/7e236108-328b-4170-be20-13f16b56c4fc-home\") pod \"7e236108-328b-4170-be20-13f16b56c4fc\" (UID: \"7e236108-328b-4170-be20-13f16b56c4fc\") "
Nov 02 20:05:44.928549 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:44.928201    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e236108-328b-4170-be20-13f16b56c4fc-kube-api-access-zcqwf\" (UniqueName: \"kubernetes.io/projected/7e236108-328b-4170-be20-13f16b56c4fc-kube-api-access-zcqwf\") pod \"7e236108-328b-4170-be20-13f16b56c4fc\" (UID: \"7e236108-328b-4170-be20-13f16b56c4fc\") "
Nov 02 20:05:44.928549 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:44.928202    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e236108-328b-4170-be20-13f16b56c4fc-model-cache" pod "7e236108-328b-4170-be20-13f16b56c4fc" (UID: "7e236108-328b-4170-be20-13f16b56c4fc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:05:44.928549 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:44.928218    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e236108-328b-4170-be20-13f16b56c4fc-tmp-dir" pod "7e236108-328b-4170-be20-13f16b56c4fc" (UID: "7e236108-328b-4170-be20-13f16b56c4fc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:05:44.928549 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:44.928275    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d784b10-99e4-4561-bfb7-6614877d2b6d-tmp-dir" pod "1d784b10-99e4-4561-bfb7-6614877d2b6d" (UID: "1d784b10-99e4-4561-bfb7-6614877d2b6d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:05:44.928549 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:44.928465    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d784b10-99e4-4561-bfb7-6614877d2b6d-home" pod "1d784b10-99e4-4561-bfb7-6614877d2b6d" (UID: "1d784b10-99e4-4561-bfb7-6614877d2b6d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:05:44.928791 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:44.928557    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e236108-328b-4170-be20-13f16b56c4fc-home" pod "7e236108-328b-4170-be20-13f16b56c4fc" (UID: "7e236108-328b-4170-be20-13f16b56c4fc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:05:44.929054 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:44.928960    4463 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d784b10-99e4-4561-bfb7-6614877d2b6d-tmp-dir\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:05:44.929054 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:44.928983    4463 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d784b10-99e4-4561-bfb7-6614877d2b6d-home\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:05:44.929054 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:44.928998    4463 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d784b10-99e4-4561-bfb7-6614877d2b6d-model-cache\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:05:44.929054 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:44.929028    4463 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e236108-328b-4170-be20-13f16b56c4fc-tmp-dir\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:05:44.929054 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:44.929043    4463 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e236108-328b-4170-be20-13f16b56c4fc-model-cache\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:05:44.929054 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:44.929055    4463 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e236108-328b-4170-be20-13f16b56c4fc-home\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:05:44.930445 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:44.930420    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d784b10-99e4-4561-bfb7-6614877d2b6d-tls-certs" pod "1d784b10-99e4-4561-bfb7-6614877d2b6d" (UID: "1d784b10-99e4-4561-bfb7-6614877d2b6d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:05:44.930561 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:44.930436    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e236108-328b-4170-be20-13f16b56c4fc-tls-certs" pod "7e236108-328b-4170-be20-13f16b56c4fc" (UID: "7e236108-328b-4170-be20-13f16b56c4fc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:05:44.930840 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:44.930816    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e236108-328b-4170-be20-13f16b56c4fc-dshm" pod "7e236108-328b-4170-be20-13f16b56c4fc" (UID: "7e236108-328b-4170-be20-13f16b56c4fc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:05:44.930940 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:44.930877    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d784b10-99e4-4561-bfb7-6614877d2b6d-dshm" pod "1d784b10-99e4-4561-bfb7-6614877d2b6d" (UID: "1d784b10-99e4-4561-bfb7-6614877d2b6d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:05:44.931056 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:44.931032    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d784b10-99e4-4561-bfb7-6614877d2b6d-kube-api-access-p268k" pod "1d784b10-99e4-4561-bfb7-6614877d2b6d" (UID: "1d784b10-99e4-4561-bfb7-6614877d2b6d"). InnerVolumeSpecName "kube-api-access-p268k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:05:44.931129 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:44.931062    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e236108-328b-4170-be20-13f16b56c4fc-kube-api-access-zcqwf" pod "7e236108-328b-4170-be20-13f16b56c4fc" (UID: "7e236108-328b-4170-be20-13f16b56c4fc"). InnerVolumeSpecName "kube-api-access-zcqwf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:05:45.030046 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:45.029989    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcqwf\" (UniqueName: \"kubernetes.io/projected/7e236108-328b-4170-be20-13f16b56c4fc-kube-api-access-zcqwf\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:05:45.030046 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:45.030033    4463 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e236108-328b-4170-be20-13f16b56c4fc-tls-certs\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:05:45.030046 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:45.030048    4463 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d784b10-99e4-4561-bfb7-6614877d2b6d-tls-certs\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:05:45.030266 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:45.030057    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p268k\" (UniqueName: \"kubernetes.io/projected/1d784b10-99e4-4561-bfb7-6614877d2b6d-kube-api-access-p268k\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:05:45.030266 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:45.030067    4463 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e236108-328b-4170-be20-13f16b56c4fc-dshm\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:05:45.030266 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:45.030076    4463 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d784b10-99e4-4561-bfb7-6614877d2b6d-dshm\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:05:45.087192 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:45.087104    4463 generic.go:356] "Generic (PLEG): container finished" podID="7e236108-328b-4170-be20-13f16b56c4fc" containerID="710ec917c9760293985d1402b9aa8aac118aeb304ae914e8f9a310db1485730b" exitCode=0
Nov 02 20:05:45.087354 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:45.087231    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-1d869e/distinct-2-w5-1d869e-kserve-574f55db46-4cv47"
Nov 02 20:05:45.087354 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:45.087273    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-1d869e/distinct-2-w5-1d869e-kserve-574f55db46-4cv47" event={"ID":"7e236108-328b-4170-be20-13f16b56c4fc","Type":"ContainerDied","Data":"710ec917c9760293985d1402b9aa8aac118aeb304ae914e8f9a310db1485730b"}
Nov 02 20:05:45.087354 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:45.087345    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-1d869e/distinct-2-w5-1d869e-kserve-574f55db46-4cv47" event={"ID":"7e236108-328b-4170-be20-13f16b56c4fc","Type":"ContainerDied","Data":"74100345d41ddc8241a0ff8ab3b6a61a0fb4167e0136f5d72ca8f7c27b59b740"}
Nov 02 20:05:45.087524 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:45.087374    4463 scope.go:122] "RemoveContainer" containerID="710ec917c9760293985d1402b9aa8aac118aeb304ae914e8f9a310db1485730b"
Nov 02 20:05:45.093397 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:45.093356    4463 generic.go:356] "Generic (PLEG): container finished" podID="1d784b10-99e4-4561-bfb7-6614877d2b6d" containerID="fc910b085bc6f56193e79977d9dd7b30867c6e9144953196d6a6bb57a4ce2058" exitCode=0
Nov 02 20:05:45.093525 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:45.093435    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-1d869e/premium-w5-1d869e-kserve-7745799ff8-pv2th" event={"ID":"1d784b10-99e4-4561-bfb7-6614877d2b6d","Type":"ContainerDied","Data":"fc910b085bc6f56193e79977d9dd7b30867c6e9144953196d6a6bb57a4ce2058"}
Nov 02 20:05:45.093525 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:45.093476    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-1d869e/premium-w5-1d869e-kserve-7745799ff8-pv2th" event={"ID":"1d784b10-99e4-4561-bfb7-6614877d2b6d","Type":"ContainerDied","Data":"9ddf0236e7ca264b47e545a93ebf3896fcfe6d049342579712d2f064e4a5b340"}
Nov 02 20:05:45.093791 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:45.093761    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-1d869e/premium-w5-1d869e-kserve-7745799ff8-pv2th"
Nov 02 20:05:45.103420 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:45.103405    4463 scope.go:122] "RemoveContainer" containerID="710ec917c9760293985d1402b9aa8aac118aeb304ae914e8f9a310db1485730b"
Nov 02 20:05:45.103730 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:05:45.103709    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"710ec917c9760293985d1402b9aa8aac118aeb304ae914e8f9a310db1485730b\": container with ID starting with 710ec917c9760293985d1402b9aa8aac118aeb304ae914e8f9a310db1485730b not found: ID does not exist" containerID="710ec917c9760293985d1402b9aa8aac118aeb304ae914e8f9a310db1485730b"
Nov 02 20:05:45.103797 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:45.103737    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"710ec917c9760293985d1402b9aa8aac118aeb304ae914e8f9a310db1485730b"} err="failed to get container status \"710ec917c9760293985d1402b9aa8aac118aeb304ae914e8f9a310db1485730b\": rpc error: code = NotFound desc = could not find container \"710ec917c9760293985d1402b9aa8aac118aeb304ae914e8f9a310db1485730b\": container with ID starting with 710ec917c9760293985d1402b9aa8aac118aeb304ae914e8f9a310db1485730b not found: ID does not exist"
Nov 02 20:05:45.103797 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:45.103752    4463 scope.go:122] "RemoveContainer" containerID="fc910b085bc6f56193e79977d9dd7b30867c6e9144953196d6a6bb57a4ce2058"
Nov 02 20:05:45.118688 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:45.118663    4463 scope.go:122] "RemoveContainer" containerID="fc910b085bc6f56193e79977d9dd7b30867c6e9144953196d6a6bb57a4ce2058"
Nov 02 20:05:45.119048 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:05:45.119007    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc910b085bc6f56193e79977d9dd7b30867c6e9144953196d6a6bb57a4ce2058\": container with ID starting with fc910b085bc6f56193e79977d9dd7b30867c6e9144953196d6a6bb57a4ce2058 not found: ID does not exist" containerID="fc910b085bc6f56193e79977d9dd7b30867c6e9144953196d6a6bb57a4ce2058"
Nov 02 20:05:45.119151 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:45.119059    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc910b085bc6f56193e79977d9dd7b30867c6e9144953196d6a6bb57a4ce2058"} err="failed to get container status \"fc910b085bc6f56193e79977d9dd7b30867c6e9144953196d6a6bb57a4ce2058\": rpc error: code = NotFound desc = could not find container \"fc910b085bc6f56193e79977d9dd7b30867c6e9144953196d6a6bb57a4ce2058\": container with ID starting with fc910b085bc6f56193e79977d9dd7b30867c6e9144953196d6a6bb57a4ce2058 not found: ID does not exist"
Nov 02 20:05:45.240556 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:45.240523    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-1d869e/distinct-2-w5-1d869e-kserve-574f55db46-4cv47"]
Nov 02 20:05:45.244489 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:45.244463    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-1d869e/distinct-2-w5-1d869e-kserve-574f55db46-4cv47"]
Nov 02 20:05:45.344424 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:45.344339    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-1d869e/premium-w5-1d869e-kserve-7745799ff8-pv2th"]
Nov 02 20:05:45.347511 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:45.347488    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-1d869e/premium-w5-1d869e-kserve-7745799ff8-pv2th"]
Nov 02 20:05:45.586329 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:45.586299    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d784b10-99e4-4561-bfb7-6614877d2b6d" path="/var/lib/kubelet/pods/1d784b10-99e4-4561-bfb7-6614877d2b6d/volumes"
Nov 02 20:05:45.586827 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:45.586807    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e236108-328b-4170-be20-13f16b56c4fc" path="/var/lib/kubelet/pods/7e236108-328b-4170-be20-13f16b56c4fc/volumes"
Nov 02 20:05:45.749186 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:45.749146    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d869e/distinct-2-w5-1d869e-kserve-574f55db46-4cv47" podUID="7e236108-328b-4170-be20-13f16b56c4fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:05:47.209049 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:47.201154    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-1d869e-7d8f7ccf9f-nhdnm" podUID="7ebcf427-e9e4-4445-86d8-7d4a32398724" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 20:05:47.425879 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:47.425835    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 20:05:57.200793 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:57.200745    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-1d869e-7d8f7ccf9f-nhdnm" podUID="7ebcf427-e9e4-4445-86d8-7d4a32398724" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 20:05:58.211207 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:58.211172    4463 generic.go:356] "Generic (PLEG): container finished" podID="c4fc2355-c8a3-4c51-9302-f93ced5c343e" containerID="b285fc201b6ce043240d90cf63f62f54f1dc7b558f56265afe4c032c16bc603d" exitCode=0
Nov 02 20:05:58.211596 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:58.211247    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-838bc1-openshift-default-5fbf6db645-sx9sd" event={"ID":"c4fc2355-c8a3-4c51-9302-f93ced5c343e","Type":"ContainerDied","Data":"b285fc201b6ce043240d90cf63f62f54f1dc7b558f56265afe4c032c16bc603d"}
Nov 02 20:05:58.211703 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:58.211681    4463 scope.go:122] "RemoveContainer" containerID="b285fc201b6ce043240d90cf63f62f54f1dc7b558f56265afe4c032c16bc603d"
Nov 02 20:05:58.213461 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:58.213431    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:05:58.213543 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:58.213503    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:05:58.213543 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:58.213534    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:05:59.220806 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:59.220769    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-838bc1-openshift-default-5fbf6db645-sx9sd" event={"ID":"c4fc2355-c8a3-4c51-9302-f93ced5c343e","Type":"ContainerStarted","Data":"4e6c435aa00e1cf2706c79d8aa6e774bcb5f347e7c468887b72060ef279e4924"}
Nov 02 20:05:59.617818 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:59.617731    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-838bc1-openshift-default-5fbf6db645-sx9sd"
Nov 02 20:05:59.617963 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:59.617921    4463 patch_prober.go:29] interesting pod/e2e-inf-a-838bc1-openshift-default-5fbf6db645-sx9sd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Nov 02 20:05:59.618030 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:05:59.617988    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-838bc1-openshift-default-5fbf6db645-sx9sd" podUID="c4fc2355-c8a3-4c51-9302-f93ced5c343e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Nov 02 20:06:00.617277 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:00.617242    4463 patch_prober.go:29] interesting pod/e2e-inf-a-838bc1-openshift-default-5fbf6db645-sx9sd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Nov 02 20:06:00.617667 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:00.617299    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-838bc1-openshift-default-5fbf6db645-sx9sd" podUID="c4fc2355-c8a3-4c51-9302-f93ced5c343e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Nov 02 20:06:01.617209 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:01.617169    4463 patch_prober.go:29] interesting pod/e2e-inf-a-838bc1-openshift-default-5fbf6db645-sx9sd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Nov 02 20:06:01.617379 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:01.617234    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-838bc1-openshift-default-5fbf6db645-sx9sd" podUID="c4fc2355-c8a3-4c51-9302-f93ced5c343e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Nov 02 20:06:02.119840 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:02.119813    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-1d869e-7d8f7ccf9f-nhdnm"
Nov 02 20:06:02.212968 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:02.212934    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ebcf427-e9e4-4445-86d8-7d4a32398724-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ebcf427-e9e4-4445-86d8-7d4a32398724-plugins-config-volume\") pod \"7ebcf427-e9e4-4445-86d8-7d4a32398724\" (UID: \"7ebcf427-e9e4-4445-86d8-7d4a32398724\") "
Nov 02 20:06:02.213319 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:02.213298    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ebcf427-e9e4-4445-86d8-7d4a32398724-kube-api-access-p5gns\" (UniqueName: \"kubernetes.io/projected/7ebcf427-e9e4-4445-86d8-7d4a32398724-kube-api-access-p5gns\") pod \"7ebcf427-e9e4-4445-86d8-7d4a32398724\" (UID: \"7ebcf427-e9e4-4445-86d8-7d4a32398724\") "
Nov 02 20:06:02.214801 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:02.214777    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ebcf427-e9e4-4445-86d8-7d4a32398724-plugins-config-volume" pod "7ebcf427-e9e4-4445-86d8-7d4a32398724" (UID: "7ebcf427-e9e4-4445-86d8-7d4a32398724"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:06:02.217670 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:02.217636    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ebcf427-e9e4-4445-86d8-7d4a32398724-kube-api-access-p5gns" pod "7ebcf427-e9e4-4445-86d8-7d4a32398724" (UID: "7ebcf427-e9e4-4445-86d8-7d4a32398724"). InnerVolumeSpecName "kube-api-access-p5gns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:06:02.240916 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:02.240886    4463 generic.go:356] "Generic (PLEG): container finished" podID="7ebcf427-e9e4-4445-86d8-7d4a32398724" containerID="99b3a6aa46bebea585ad09fe6e48386f4bafd9387866b0af60885be803e2ee75" exitCode=0
Nov 02 20:06:02.241037 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:02.240942    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-1d869e-7d8f7ccf9f-nhdnm"
Nov 02 20:06:02.241091 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:02.240992    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-1d869e-7d8f7ccf9f-nhdnm" event={"ID":"7ebcf427-e9e4-4445-86d8-7d4a32398724","Type":"ContainerDied","Data":"99b3a6aa46bebea585ad09fe6e48386f4bafd9387866b0af60885be803e2ee75"}
Nov 02 20:06:02.241165 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:02.241145    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-1d869e-7d8f7ccf9f-nhdnm" event={"ID":"7ebcf427-e9e4-4445-86d8-7d4a32398724","Type":"ContainerDied","Data":"48b3ba40a95d0743447c6952c8131723282a95002e66912ce2c718b04fbaecdd"}
Nov 02 20:06:02.241213 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:02.241200    4463 scope.go:122] "RemoveContainer" containerID="99b3a6aa46bebea585ad09fe6e48386f4bafd9387866b0af60885be803e2ee75"
Nov 02 20:06:02.280963 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:02.280939    4463 scope.go:122] "RemoveContainer" containerID="99b3a6aa46bebea585ad09fe6e48386f4bafd9387866b0af60885be803e2ee75"
Nov 02 20:06:02.281300 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:06:02.281278    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99b3a6aa46bebea585ad09fe6e48386f4bafd9387866b0af60885be803e2ee75\": container with ID starting with 99b3a6aa46bebea585ad09fe6e48386f4bafd9387866b0af60885be803e2ee75 not found: ID does not exist" containerID="99b3a6aa46bebea585ad09fe6e48386f4bafd9387866b0af60885be803e2ee75"
Nov 02 20:06:02.281386 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:02.281311    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99b3a6aa46bebea585ad09fe6e48386f4bafd9387866b0af60885be803e2ee75"} err="failed to get container status \"99b3a6aa46bebea585ad09fe6e48386f4bafd9387866b0af60885be803e2ee75\": rpc error: code = NotFound desc = could not find container \"99b3a6aa46bebea585ad09fe6e48386f4bafd9387866b0af60885be803e2ee75\": container with ID starting with 99b3a6aa46bebea585ad09fe6e48386f4bafd9387866b0af60885be803e2ee75 not found: ID does not exist"
Nov 02 20:06:02.315286 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:02.315262    4463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ebcf427-e9e4-4445-86d8-7d4a32398724-plugins-config-volume\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:06:02.315286 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:02.315287    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5gns\" (UniqueName: \"kubernetes.io/projected/7ebcf427-e9e4-4445-86d8-7d4a32398724-kube-api-access-p5gns\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:06:02.394497 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:02.394432    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-1d869e-7d8f7ccf9f-nhdnm"]
Nov 02 20:06:02.397485 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:02.397457    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-1d869e-7d8f7ccf9f-nhdnm"]
Nov 02 20:06:02.617618 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:02.617531    4463 patch_prober.go:29] interesting pod/e2e-inf-a-838bc1-openshift-default-5fbf6db645-sx9sd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Nov 02 20:06:02.617618 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:02.617588    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-838bc1-openshift-default-5fbf6db645-sx9sd" podUID="c4fc2355-c8a3-4c51-9302-f93ced5c343e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Nov 02 20:06:03.587590 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:03.587555    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ebcf427-e9e4-4445-86d8-7d4a32398724" path="/var/lib/kubelet/pods/7ebcf427-e9e4-4445-86d8-7d4a32398724/volumes"
Nov 02 20:06:03.617285 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:03.617252    4463 patch_prober.go:29] interesting pod/e2e-inf-a-838bc1-openshift-default-5fbf6db645-sx9sd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Nov 02 20:06:03.617436 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:03.617303    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-838bc1-openshift-default-5fbf6db645-sx9sd" podUID="c4fc2355-c8a3-4c51-9302-f93ced5c343e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Nov 02 20:06:04.296132 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:06:04.296088    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 20:06:04.621334 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:04.621256    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-838bc1-openshift-default-5fbf6db645-sx9sd"
Nov 02 20:06:04.621724 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:04.621699    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-838bc1-openshift-default-5fbf6db645-sx9sd"
Nov 02 20:06:04.622366 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:04.622344    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-838bc1-openshift-default-5fbf6db645-sx9sd"
Nov 02 20:06:09.127298 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:09.127261    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-c60a0d-openshift-default-6464975799-g9tvr"]
Nov 02 20:06:09.127719 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:09.127475    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-c60a0d-openshift-default-6464975799-g9tvr" podUID="618bcdd7-16ca-40c3-9a41-feed4d5f7161" containerName="istio-proxy" containerID="cri-o://8d1069229b6457b0b97e70fd618dd7fcfca991676d6fe15413782732e24db6d9" gracePeriod=30
Nov 02 20:06:10.505395 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:10.505350    4463 patch_prober.go:29] interesting pod/e2e-api-b-c60a0d-openshift-default-6464975799-g9tvr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 20:06:10.506000 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:10.505967    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-c60a0d-openshift-default-6464975799-g9tvr" podUID="618bcdd7-16ca-40c3-9a41-feed4d5f7161" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 20:06:14.362674 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:14.362628    4463 generic.go:356] "Generic (PLEG): container finished" podID="618bcdd7-16ca-40c3-9a41-feed4d5f7161" containerID="8d1069229b6457b0b97e70fd618dd7fcfca991676d6fe15413782732e24db6d9" exitCode=0
Nov 02 20:06:14.363440 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:14.363409    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c60a0d-openshift-default-6464975799-g9tvr" event={"ID":"618bcdd7-16ca-40c3-9a41-feed4d5f7161","Type":"ContainerDied","Data":"8d1069229b6457b0b97e70fd618dd7fcfca991676d6fe15413782732e24db6d9"}
Nov 02 20:06:14.363554 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:14.363489    4463 scope.go:122] "RemoveContainer" containerID="5479f80afbc79dbc92d7fdcbb032f1942e61a73148b8910c12f9330b98b69962"
Nov 02 20:06:14.423571 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:14.423548    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c60a0d-openshift-default-6464975799-g9tvr"
Nov 02 20:06:14.517731 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:14.517699    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/618bcdd7-16ca-40c3-9a41-feed4d5f7161-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/618bcdd7-16ca-40c3-9a41-feed4d5f7161-istiod-ca-cert\") pod \"618bcdd7-16ca-40c3-9a41-feed4d5f7161\" (UID: \"618bcdd7-16ca-40c3-9a41-feed4d5f7161\") "
Nov 02 20:06:14.517886 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:14.517748    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/618bcdd7-16ca-40c3-9a41-feed4d5f7161-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/618bcdd7-16ca-40c3-9a41-feed4d5f7161-workload-certs\") pod \"618bcdd7-16ca-40c3-9a41-feed4d5f7161\" (UID: \"618bcdd7-16ca-40c3-9a41-feed4d5f7161\") "
Nov 02 20:06:14.517886 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:14.517806    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/618bcdd7-16ca-40c3-9a41-feed4d5f7161-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/618bcdd7-16ca-40c3-9a41-feed4d5f7161-workload-socket\") pod \"618bcdd7-16ca-40c3-9a41-feed4d5f7161\" (UID: \"618bcdd7-16ca-40c3-9a41-feed4d5f7161\") "
Nov 02 20:06:14.517886 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:14.517843    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/618bcdd7-16ca-40c3-9a41-feed4d5f7161-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/618bcdd7-16ca-40c3-9a41-feed4d5f7161-istio-podinfo\") pod \"618bcdd7-16ca-40c3-9a41-feed4d5f7161\" (UID: \"618bcdd7-16ca-40c3-9a41-feed4d5f7161\") "
Nov 02 20:06:14.518092 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:14.517921    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/618bcdd7-16ca-40c3-9a41-feed4d5f7161-kube-api-access-8zj5k\" (UniqueName: \"kubernetes.io/projected/618bcdd7-16ca-40c3-9a41-feed4d5f7161-kube-api-access-8zj5k\") pod \"618bcdd7-16ca-40c3-9a41-feed4d5f7161\" (UID: \"618bcdd7-16ca-40c3-9a41-feed4d5f7161\") "
Nov 02 20:06:14.518154 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:14.518101    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/618bcdd7-16ca-40c3-9a41-feed4d5f7161-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/618bcdd7-16ca-40c3-9a41-feed4d5f7161-istio-envoy\") pod \"618bcdd7-16ca-40c3-9a41-feed4d5f7161\" (UID: \"618bcdd7-16ca-40c3-9a41-feed4d5f7161\") "
Nov 02 20:06:14.518154 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:14.518141    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/618bcdd7-16ca-40c3-9a41-feed4d5f7161-istio-token\" (UniqueName: \"kubernetes.io/projected/618bcdd7-16ca-40c3-9a41-feed4d5f7161-istio-token\") pod \"618bcdd7-16ca-40c3-9a41-feed4d5f7161\" (UID: \"618bcdd7-16ca-40c3-9a41-feed4d5f7161\") "
Nov 02 20:06:14.518271 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:14.518175    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/618bcdd7-16ca-40c3-9a41-feed4d5f7161-workload-certs" pod "618bcdd7-16ca-40c3-9a41-feed4d5f7161" (UID: "618bcdd7-16ca-40c3-9a41-feed4d5f7161"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:06:14.518271 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:14.518192    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/618bcdd7-16ca-40c3-9a41-feed4d5f7161-workload-socket" pod "618bcdd7-16ca-40c3-9a41-feed4d5f7161" (UID: "618bcdd7-16ca-40c3-9a41-feed4d5f7161"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:06:14.518271 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:14.518220    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/618bcdd7-16ca-40c3-9a41-feed4d5f7161-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/618bcdd7-16ca-40c3-9a41-feed4d5f7161-credential-socket\") pod \"618bcdd7-16ca-40c3-9a41-feed4d5f7161\" (UID: \"618bcdd7-16ca-40c3-9a41-feed4d5f7161\") "
Nov 02 20:06:14.518271 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:14.518196    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/618bcdd7-16ca-40c3-9a41-feed4d5f7161-istiod-ca-cert" pod "618bcdd7-16ca-40c3-9a41-feed4d5f7161" (UID: "618bcdd7-16ca-40c3-9a41-feed4d5f7161"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:06:14.518271 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:14.518256    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/618bcdd7-16ca-40c3-9a41-feed4d5f7161-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/618bcdd7-16ca-40c3-9a41-feed4d5f7161-istio-data\") pod \"618bcdd7-16ca-40c3-9a41-feed4d5f7161\" (UID: \"618bcdd7-16ca-40c3-9a41-feed4d5f7161\") "
Nov 02 20:06:14.518906 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:14.518751    4463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/618bcdd7-16ca-40c3-9a41-feed4d5f7161-istiod-ca-cert\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:06:14.518906 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:14.518772    4463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/618bcdd7-16ca-40c3-9a41-feed4d5f7161-workload-certs\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:06:14.518906 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:14.518789    4463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/618bcdd7-16ca-40c3-9a41-feed4d5f7161-workload-socket\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:06:14.519526 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:14.519048    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/618bcdd7-16ca-40c3-9a41-feed4d5f7161-istio-data" pod "618bcdd7-16ca-40c3-9a41-feed4d5f7161" (UID: "618bcdd7-16ca-40c3-9a41-feed4d5f7161"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:06:14.519526 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:14.519260    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/618bcdd7-16ca-40c3-9a41-feed4d5f7161-credential-socket" pod "618bcdd7-16ca-40c3-9a41-feed4d5f7161" (UID: "618bcdd7-16ca-40c3-9a41-feed4d5f7161"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:06:14.521564 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:14.521238    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/618bcdd7-16ca-40c3-9a41-feed4d5f7161-kube-api-access-8zj5k" pod "618bcdd7-16ca-40c3-9a41-feed4d5f7161" (UID: "618bcdd7-16ca-40c3-9a41-feed4d5f7161"). InnerVolumeSpecName "kube-api-access-8zj5k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:06:14.521865 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:14.521784    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/618bcdd7-16ca-40c3-9a41-feed4d5f7161-istio-token" pod "618bcdd7-16ca-40c3-9a41-feed4d5f7161" (UID: "618bcdd7-16ca-40c3-9a41-feed4d5f7161"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:06:14.522485 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:14.522453    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/618bcdd7-16ca-40c3-9a41-feed4d5f7161-istio-envoy" pod "618bcdd7-16ca-40c3-9a41-feed4d5f7161" (UID: "618bcdd7-16ca-40c3-9a41-feed4d5f7161"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:06:14.522573 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:14.522449    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/618bcdd7-16ca-40c3-9a41-feed4d5f7161-istio-podinfo" pod "618bcdd7-16ca-40c3-9a41-feed4d5f7161" (UID: "618bcdd7-16ca-40c3-9a41-feed4d5f7161"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:06:14.620369 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:14.620333    4463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/618bcdd7-16ca-40c3-9a41-feed4d5f7161-istio-podinfo\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:06:14.620369 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:14.620364    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zj5k\" (UniqueName: \"kubernetes.io/projected/618bcdd7-16ca-40c3-9a41-feed4d5f7161-kube-api-access-8zj5k\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:06:14.620570 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:14.620379    4463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/618bcdd7-16ca-40c3-9a41-feed4d5f7161-istio-envoy\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:06:14.620570 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:14.620393    4463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/618bcdd7-16ca-40c3-9a41-feed4d5f7161-istio-token\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:06:14.620570 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:14.620405    4463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/618bcdd7-16ca-40c3-9a41-feed4d5f7161-credential-socket\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:06:14.620570 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:14.620416    4463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/618bcdd7-16ca-40c3-9a41-feed4d5f7161-istio-data\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:06:14.644053 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:14.643999    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8795bf/distinct-2-w4-8795bf-kserve-69c8b6b885-2chhn"]
Nov 02 20:06:14.649136 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:14.648812    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8795bf/distinct-w4-8795bf-kserve-64b4796df4-8qgqh"]
Nov 02 20:06:15.370633 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:15.370603    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c60a0d-openshift-default-6464975799-g9tvr"
Nov 02 20:06:15.371119 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:15.370624    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c60a0d-openshift-default-6464975799-g9tvr" event={"ID":"618bcdd7-16ca-40c3-9a41-feed4d5f7161","Type":"ContainerDied","Data":"4708f7fbf0f67d0c675a8f43ada375205811bfd8c4096de071ec614d56908890"}
Nov 02 20:06:15.371119 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:15.370724    4463 scope.go:122] "RemoveContainer" containerID="8d1069229b6457b0b97e70fd618dd7fcfca991676d6fe15413782732e24db6d9"
Nov 02 20:06:15.524436 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:15.524405    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-c60a0d-openshift-default-6464975799-g9tvr"]
Nov 02 20:06:15.571025 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:15.570977    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-c60a0d-openshift-default-6464975799-g9tvr"]
Nov 02 20:06:15.586641 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:15.586613    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="618bcdd7-16ca-40c3-9a41-feed4d5f7161" path="/var/lib/kubelet/pods/618bcdd7-16ca-40c3-9a41-feed4d5f7161/volumes"
Nov 02 20:06:20.327572 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:20.327538    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-28fa24-6d779499-gvdmm"]
Nov 02 20:06:20.328160 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:20.328146    4463 cpu_manager.go:414] "containerMap: removing container" podUID="7e236108-328b-4170-be20-13f16b56c4fc" containerName="main"
Nov 02 20:06:20.328160 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:20.328161    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e236108-328b-4170-be20-13f16b56c4fc" containerName="main"
Nov 02 20:06:20.328249 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:20.328172    4463 cpu_manager.go:414] "containerMap: removing container" podUID="7ebcf427-e9e4-4445-86d8-7d4a32398724" containerName="payload-pre-processing"
Nov 02 20:06:20.328249 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:20.328179    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ebcf427-e9e4-4445-86d8-7d4a32398724" containerName="payload-pre-processing"
Nov 02 20:06:20.328249 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:20.328187    4463 cpu_manager.go:414] "containerMap: removing container" podUID="618bcdd7-16ca-40c3-9a41-feed4d5f7161" containerName="istio-proxy"
Nov 02 20:06:20.328249 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:20.328192    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="618bcdd7-16ca-40c3-9a41-feed4d5f7161" containerName="istio-proxy"
Nov 02 20:06:20.328249 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:20.328220    4463 cpu_manager.go:414] "containerMap: removing container" podUID="277fb120-e642-4215-bc3d-1b9c08c57e81" containerName="istio-proxy"
Nov 02 20:06:20.328249 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:20.328225    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="277fb120-e642-4215-bc3d-1b9c08c57e81" containerName="istio-proxy"
Nov 02 20:06:20.328249 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:20.328232    4463 cpu_manager.go:414] "containerMap: removing container" podUID="d5ffe43f-1a80-4126-9cf1-251fdde99de1" containerName="istio-proxy"
Nov 02 20:06:20.328249 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:20.328237    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5ffe43f-1a80-4126-9cf1-251fdde99de1" containerName="istio-proxy"
Nov 02 20:06:20.328249 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:20.328244    4463 cpu_manager.go:414] "containerMap: removing container" podUID="d5ffe43f-1a80-4126-9cf1-251fdde99de1" containerName="istio-proxy"
Nov 02 20:06:20.328249 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:20.328249    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5ffe43f-1a80-4126-9cf1-251fdde99de1" containerName="istio-proxy"
Nov 02 20:06:20.328557 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:20.328259    4463 cpu_manager.go:414] "containerMap: removing container" podUID="277fb120-e642-4215-bc3d-1b9c08c57e81" containerName="istio-proxy"
Nov 02 20:06:20.328557 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:20.328264    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="277fb120-e642-4215-bc3d-1b9c08c57e81" containerName="istio-proxy"
Nov 02 20:06:20.328557 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:20.328273    4463 cpu_manager.go:414] "containerMap: removing container" podUID="1d784b10-99e4-4561-bfb7-6614877d2b6d" containerName="main"
Nov 02 20:06:20.328557 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:20.328277    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d784b10-99e4-4561-bfb7-6614877d2b6d" containerName="main"
Nov 02 20:06:20.328557 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:20.328361    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="277fb120-e642-4215-bc3d-1b9c08c57e81" containerName="istio-proxy"
Nov 02 20:06:20.328557 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:20.328371    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d784b10-99e4-4561-bfb7-6614877d2b6d" containerName="main"
Nov 02 20:06:20.328557 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:20.328379    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e236108-328b-4170-be20-13f16b56c4fc" containerName="main"
Nov 02 20:06:20.328557 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:20.328385    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ebcf427-e9e4-4445-86d8-7d4a32398724" containerName="payload-pre-processing"
Nov 02 20:06:20.328557 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:20.328392    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="618bcdd7-16ca-40c3-9a41-feed4d5f7161" containerName="istio-proxy"
Nov 02 20:06:20.328557 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:20.328402    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5ffe43f-1a80-4126-9cf1-251fdde99de1" containerName="istio-proxy"
Nov 02 20:06:20.328557 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:20.328406    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5ffe43f-1a80-4126-9cf1-251fdde99de1" containerName="istio-proxy"
Nov 02 20:06:20.328557 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:20.328412    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="618bcdd7-16ca-40c3-9a41-feed4d5f7161" containerName="istio-proxy"
Nov 02 20:06:20.331826 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:20.331807    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-28fa24-6d779499-gvdmm"
Nov 02 20:06:20.387443 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:20.387415    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-28fa24-6d779499-gvdmm"]
Nov 02 20:06:20.390363 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:20.390326    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db50c476-4694-4cd0-8fa0-cca68002faa8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-28fa24-6d779499-gvdmm\" (UID: \"db50c476-4694-4cd0-8fa0-cca68002faa8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-28fa24-6d779499-gvdmm"
Nov 02 20:06:20.390509 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:20.390389    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqp82\" (UniqueName: \"kubernetes.io/projected/db50c476-4694-4cd0-8fa0-cca68002faa8-kube-api-access-jqp82\") pod \"payload-pre-processing-e2e-inf-b-28fa24-6d779499-gvdmm\" (UID: \"db50c476-4694-4cd0-8fa0-cca68002faa8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-28fa24-6d779499-gvdmm"
Nov 02 20:06:20.426781 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:20.426749    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-28fa24-dockercfg-dqtc8\""
Nov 02 20:06:20.427027 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:20.426760    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-28fa24\""
Nov 02 20:06:20.494752 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:20.494713    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db50c476-4694-4cd0-8fa0-cca68002faa8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-28fa24-6d779499-gvdmm\" (UID: \"db50c476-4694-4cd0-8fa0-cca68002faa8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-28fa24-6d779499-gvdmm"
Nov 02 20:06:20.506652 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:20.506510    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqp82\" (UniqueName: \"kubernetes.io/projected/db50c476-4694-4cd0-8fa0-cca68002faa8-kube-api-access-jqp82\") pod \"payload-pre-processing-e2e-inf-b-28fa24-6d779499-gvdmm\" (UID: \"db50c476-4694-4cd0-8fa0-cca68002faa8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-28fa24-6d779499-gvdmm"
Nov 02 20:06:20.510198 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:20.510163    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-28fa24-8f6447cc7-x6jsz"]
Nov 02 20:06:20.511666 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:20.511638    4463 cpu_manager.go:414] "containerMap: removing container" podUID="618bcdd7-16ca-40c3-9a41-feed4d5f7161" containerName="istio-proxy"
Nov 02 20:06:20.512456 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:20.512437    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="618bcdd7-16ca-40c3-9a41-feed4d5f7161" containerName="istio-proxy"
Nov 02 20:06:20.512722 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:20.512405    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db50c476-4694-4cd0-8fa0-cca68002faa8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-28fa24-6d779499-gvdmm\" (UID: \"db50c476-4694-4cd0-8fa0-cca68002faa8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-28fa24-6d779499-gvdmm"
Nov 02 20:06:20.512964 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:20.512767    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="277fb120-e642-4215-bc3d-1b9c08c57e81" containerName="istio-proxy"
Nov 02 20:06:20.517670 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:20.517633    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-28fa24-8f6447cc7-x6jsz"
Nov 02 20:06:20.534242 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:20.534204    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-28fa24-8f6447cc7-x6jsz"]
Nov 02 20:06:20.559827 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:20.559800    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqp82\" (UniqueName: \"kubernetes.io/projected/db50c476-4694-4cd0-8fa0-cca68002faa8-kube-api-access-jqp82\") pod \"payload-pre-processing-e2e-inf-b-28fa24-6d779499-gvdmm\" (UID: \"db50c476-4694-4cd0-8fa0-cca68002faa8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-28fa24-6d779499-gvdmm"
Nov 02 20:06:20.608354 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:20.608252    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33f548d7-2602-477f-a2b4-80196795d489-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-28fa24-8f6447cc7-x6jsz\" (UID: \"33f548d7-2602-477f-a2b4-80196795d489\") " pod="openshift-ingress/payload-processing-e2e-inf-b-28fa24-8f6447cc7-x6jsz"
Nov 02 20:06:20.608526 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:20.608496    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfgn4\" (UniqueName: \"kubernetes.io/projected/33f548d7-2602-477f-a2b4-80196795d489-kube-api-access-zfgn4\") pod \"payload-processing-e2e-inf-b-28fa24-8f6447cc7-x6jsz\" (UID: \"33f548d7-2602-477f-a2b4-80196795d489\") " pod="openshift-ingress/payload-processing-e2e-inf-b-28fa24-8f6447cc7-x6jsz"
Nov 02 20:06:20.643585 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:20.643551    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-28fa24-6d779499-gvdmm"
Nov 02 20:06:20.714128 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:20.709893    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfgn4\" (UniqueName: \"kubernetes.io/projected/33f548d7-2602-477f-a2b4-80196795d489-kube-api-access-zfgn4\") pod \"payload-processing-e2e-inf-b-28fa24-8f6447cc7-x6jsz\" (UID: \"33f548d7-2602-477f-a2b4-80196795d489\") " pod="openshift-ingress/payload-processing-e2e-inf-b-28fa24-8f6447cc7-x6jsz"
Nov 02 20:06:20.714128 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:20.710183    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33f548d7-2602-477f-a2b4-80196795d489-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-28fa24-8f6447cc7-x6jsz\" (UID: \"33f548d7-2602-477f-a2b4-80196795d489\") " pod="openshift-ingress/payload-processing-e2e-inf-b-28fa24-8f6447cc7-x6jsz"
Nov 02 20:06:20.714128 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:20.711162    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33f548d7-2602-477f-a2b4-80196795d489-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-28fa24-8f6447cc7-x6jsz\" (UID: \"33f548d7-2602-477f-a2b4-80196795d489\") " pod="openshift-ingress/payload-processing-e2e-inf-b-28fa24-8f6447cc7-x6jsz"
Nov 02 20:06:20.764829 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:20.764793    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfgn4\" (UniqueName: \"kubernetes.io/projected/33f548d7-2602-477f-a2b4-80196795d489-kube-api-access-zfgn4\") pod \"payload-processing-e2e-inf-b-28fa24-8f6447cc7-x6jsz\" (UID: \"33f548d7-2602-477f-a2b4-80196795d489\") " pod="openshift-ingress/payload-processing-e2e-inf-b-28fa24-8f6447cc7-x6jsz"
Nov 02 20:06:20.834742 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:20.834718    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-28fa24-8f6447cc7-x6jsz"
Nov 02 20:06:21.013924 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:21.013896    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-28fa24-6d779499-gvdmm"]
Nov 02 20:06:21.015464 ip-10-0-133-9 kubenswrapper[4463]: W1102 20:06:21.015435    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb50c476_4694_4cd0_8fa0_cca68002faa8.slice/crio-9fd93a2ba4c28d131456023d207373da91b6fc4a6675b08544b7c08d0eaf45a7 WatchSource:0}: Error finding container 9fd93a2ba4c28d131456023d207373da91b6fc4a6675b08544b7c08d0eaf45a7: Status 404 returned error can't find the container with id 9fd93a2ba4c28d131456023d207373da91b6fc4a6675b08544b7c08d0eaf45a7
Nov 02 20:06:21.203763 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:21.203739    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-28fa24-8f6447cc7-x6jsz"]
Nov 02 20:06:21.205530 ip-10-0-133-9 kubenswrapper[4463]: W1102 20:06:21.205501    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33f548d7_2602_477f_a2b4_80196795d489.slice/crio-2f3d901ca3464774b643b22a975853b59ca26d4fd27ee897ddfe952b34ed2ac9 WatchSource:0}: Error finding container 2f3d901ca3464774b643b22a975853b59ca26d4fd27ee897ddfe952b34ed2ac9: Status 404 returned error can't find the container with id 2f3d901ca3464774b643b22a975853b59ca26d4fd27ee897ddfe952b34ed2ac9
Nov 02 20:06:21.279198 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:21.279164    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1db0e4-6b545bbdb7-nkvgn"]
Nov 02 20:06:21.282165 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:21.282145    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1db0e4-6b545bbdb7-nkvgn"
Nov 02 20:06:21.317093 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:21.317065    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc539282-391f-4350-8c34-95084fe87e65-metrics-certs\") pod \"maas-api-e2e-api-a-1db0e4-6b545bbdb7-nkvgn\" (UID: \"dc539282-391f-4350-8c34-95084fe87e65\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1db0e4-6b545bbdb7-nkvgn"
Nov 02 20:06:21.317195 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:21.317119    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44bq8\" (UniqueName: \"kubernetes.io/projected/dc539282-391f-4350-8c34-95084fe87e65-kube-api-access-44bq8\") pod \"maas-api-e2e-api-a-1db0e4-6b545bbdb7-nkvgn\" (UID: \"dc539282-391f-4350-8c34-95084fe87e65\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1db0e4-6b545bbdb7-nkvgn"
Nov 02 20:06:21.317269 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:21.317257    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc539282-391f-4350-8c34-95084fe87e65-maas-api-tls\") pod \"maas-api-e2e-api-a-1db0e4-6b545bbdb7-nkvgn\" (UID: \"dc539282-391f-4350-8c34-95084fe87e65\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1db0e4-6b545bbdb7-nkvgn"
Nov 02 20:06:21.317338 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:21.317322    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc539282-391f-4350-8c34-95084fe87e65-openshift-service-ca\") pod \"maas-api-e2e-api-a-1db0e4-6b545bbdb7-nkvgn\" (UID: \"dc539282-391f-4350-8c34-95084fe87e65\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1db0e4-6b545bbdb7-nkvgn"
Nov 02 20:06:21.331588 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:21.331557    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-1db0e4-serving-cert\""
Nov 02 20:06:21.336663 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:21.336642    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1db0e4-6b545bbdb7-nkvgn"]
Nov 02 20:06:21.410771 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:21.410708    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-28fa24-6d779499-gvdmm" event={"ID":"db50c476-4694-4cd0-8fa0-cca68002faa8","Type":"ContainerStarted","Data":"65949a11aaa2ea0b9de4ade7625c0cdb4d601c7888a8d38378c9b91d5ea7c72a"}
Nov 02 20:06:21.410771 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:21.410769    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-28fa24-6d779499-gvdmm"
Nov 02 20:06:21.410981 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:21.410793    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-28fa24-6d779499-gvdmm" event={"ID":"db50c476-4694-4cd0-8fa0-cca68002faa8","Type":"ContainerStarted","Data":"9fd93a2ba4c28d131456023d207373da91b6fc4a6675b08544b7c08d0eaf45a7"}
Nov 02 20:06:21.418736 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:21.418704    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44bq8\" (UniqueName: \"kubernetes.io/projected/dc539282-391f-4350-8c34-95084fe87e65-kube-api-access-44bq8\") pod \"maas-api-e2e-api-a-1db0e4-6b545bbdb7-nkvgn\" (UID: \"dc539282-391f-4350-8c34-95084fe87e65\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1db0e4-6b545bbdb7-nkvgn"
Nov 02 20:06:21.418878 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:21.418848    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc539282-391f-4350-8c34-95084fe87e65-maas-api-tls\") pod \"maas-api-e2e-api-a-1db0e4-6b545bbdb7-nkvgn\" (UID: \"dc539282-391f-4350-8c34-95084fe87e65\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1db0e4-6b545bbdb7-nkvgn"
Nov 02 20:06:21.418965 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:21.418943    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc539282-391f-4350-8c34-95084fe87e65-openshift-service-ca\") pod \"maas-api-e2e-api-a-1db0e4-6b545bbdb7-nkvgn\" (UID: \"dc539282-391f-4350-8c34-95084fe87e65\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1db0e4-6b545bbdb7-nkvgn"
Nov 02 20:06:21.419040 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:21.419002    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc539282-391f-4350-8c34-95084fe87e65-metrics-certs\") pod \"maas-api-e2e-api-a-1db0e4-6b545bbdb7-nkvgn\" (UID: \"dc539282-391f-4350-8c34-95084fe87e65\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1db0e4-6b545bbdb7-nkvgn"
Nov 02 20:06:21.419702 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:06:21.419205    4463 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-1db0e4-serving-cert: secret "maas-api-e2e-api-a-1db0e4-serving-cert" not found
Nov 02 20:06:21.419702 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:06:21.419275    4463 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc539282-391f-4350-8c34-95084fe87e65-maas-api-tls podName:dc539282-391f-4350-8c34-95084fe87e65 nodeName:}" failed. No retries permitted until 2026-11-02 20:06:21.919254626 +0000 UTC m=+2288.526288392 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/dc539282-391f-4350-8c34-95084fe87e65-maas-api-tls") pod "maas-api-e2e-api-a-1db0e4-6b545bbdb7-nkvgn" (UID: "dc539282-391f-4350-8c34-95084fe87e65") : secret "maas-api-e2e-api-a-1db0e4-serving-cert" not found
Nov 02 20:06:21.420093 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:21.420055    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc539282-391f-4350-8c34-95084fe87e65-openshift-service-ca\") pod \"maas-api-e2e-api-a-1db0e4-6b545bbdb7-nkvgn\" (UID: \"dc539282-391f-4350-8c34-95084fe87e65\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1db0e4-6b545bbdb7-nkvgn"
Nov 02 20:06:21.420178 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:21.420151    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-28fa24-8f6447cc7-x6jsz" event={"ID":"33f548d7-2602-477f-a2b4-80196795d489","Type":"ContainerStarted","Data":"e300f19464c523bb1bc69c95cabd8e341d01938d4b242863a44c2d3626ee0b62"}
Nov 02 20:06:21.420238 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:21.420211    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-28fa24-8f6447cc7-x6jsz" event={"ID":"33f548d7-2602-477f-a2b4-80196795d489","Type":"ContainerStarted","Data":"2f3d901ca3464774b643b22a975853b59ca26d4fd27ee897ddfe952b34ed2ac9"}
Nov 02 20:06:21.421043 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:21.420788    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-28fa24-8f6447cc7-x6jsz"
Nov 02 20:06:21.422104 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:21.422076    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc539282-391f-4350-8c34-95084fe87e65-metrics-certs\") pod \"maas-api-e2e-api-a-1db0e4-6b545bbdb7-nkvgn\" (UID: \"dc539282-391f-4350-8c34-95084fe87e65\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1db0e4-6b545bbdb7-nkvgn"
Nov 02 20:06:21.473093 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:21.472999    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44bq8\" (UniqueName: \"kubernetes.io/projected/dc539282-391f-4350-8c34-95084fe87e65-kube-api-access-44bq8\") pod \"maas-api-e2e-api-a-1db0e4-6b545bbdb7-nkvgn\" (UID: \"dc539282-391f-4350-8c34-95084fe87e65\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1db0e4-6b545bbdb7-nkvgn"
Nov 02 20:06:21.528659 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:21.526061    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-28fa24-6d779499-gvdmm" podStartSLOduration=1.5260443160000001 podStartE2EDuration="1.526044316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:06:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:06:21.520269941 +0000 UTC m=+2288.127303735" watchObservedRunningTime="2026-11-02 20:06:21.526044316 +0000 UTC m=+2288.133078091"
Nov 02 20:06:21.627617 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:21.627570    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-28fa24-8f6447cc7-x6jsz" podStartSLOduration=1.627556507 podStartE2EDuration="1.627556507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:06:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:06:21.625957772 +0000 UTC m=+2288.232991552" watchObservedRunningTime="2026-11-02 20:06:21.627556507 +0000 UTC m=+2288.234590276"
Nov 02 20:06:21.925728 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:21.925673    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc539282-391f-4350-8c34-95084fe87e65-maas-api-tls\") pod \"maas-api-e2e-api-a-1db0e4-6b545bbdb7-nkvgn\" (UID: \"dc539282-391f-4350-8c34-95084fe87e65\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1db0e4-6b545bbdb7-nkvgn"
Nov 02 20:06:21.928176 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:21.928154    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc539282-391f-4350-8c34-95084fe87e65-maas-api-tls\") pod \"maas-api-e2e-api-a-1db0e4-6b545bbdb7-nkvgn\" (UID: \"dc539282-391f-4350-8c34-95084fe87e65\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1db0e4-6b545bbdb7-nkvgn"
Nov 02 20:06:22.194082 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:22.193974    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1db0e4-6b545bbdb7-nkvgn"
Nov 02 20:06:22.566330 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:22.566301    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1db0e4-6b545bbdb7-nkvgn"]
Nov 02 20:06:22.567158 ip-10-0-133-9 kubenswrapper[4463]: W1102 20:06:22.567131    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc539282_391f_4350_8c34_95084fe87e65.slice/crio-db0b7b09bcfcfa1b0173428783d124276c8dfe1a48ed15b9b39f2226cc471baf WatchSource:0}: Error finding container db0b7b09bcfcfa1b0173428783d124276c8dfe1a48ed15b9b39f2226cc471baf: Status 404 returned error can't find the container with id db0b7b09bcfcfa1b0173428783d124276c8dfe1a48ed15b9b39f2226cc471baf
Nov 02 20:06:23.433253 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:23.433213    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1db0e4-6b545bbdb7-nkvgn" event={"ID":"dc539282-391f-4350-8c34-95084fe87e65","Type":"ContainerStarted","Data":"db0b7b09bcfcfa1b0173428783d124276c8dfe1a48ed15b9b39f2226cc471baf"}
Nov 02 20:06:23.586952 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:23.586923    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-8795bf-7bd68d77d9-5xm6w"]
Nov 02 20:06:23.686272 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:23.686195    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-8795bf-66d99fc7c9-nj9gz"]
Nov 02 20:06:28.588186 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:28.588129    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-8795bf-7bd68d77d9-5xm6w" podUID="0368fdf1-1c9f-4ee9-8da1-5dd57731a8fd" containerName="payload-processing" containerID="cri-o://867aca871d89fa6ca408b48822438b55aa9c1e9419e7452ba8baa39d821afb71" gracePeriod=25
Nov 02 20:06:28.687438 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:28.687400    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8795bf-66d99fc7c9-nj9gz" podUID="4553e306-34a0-48cf-b662-843e8de5bb45" containerName="payload-pre-processing" containerID="cri-o://db579ede9b1a5ed8b51f1953fa0cbdd7f646ad0744547b7d5a7fffd6feb07c19" gracePeriod=25
Nov 02 20:06:29.488159 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:29.488116    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1db0e4-6b545bbdb7-nkvgn" event={"ID":"dc539282-391f-4350-8c34-95084fe87e65","Type":"ContainerStarted","Data":"70fcc06568199b064f4c4eb1b0e7a7d1c416fb659df416777c6c5c40e1d69464"}
Nov 02 20:06:29.488356 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:29.488190    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1db0e4-6b545bbdb7-nkvgn"
Nov 02 20:06:29.598513 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:29.598451    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1db0e4-6b545bbdb7-nkvgn" podStartSLOduration=2.713409053 podStartE2EDuration="8.598433681s" totalImagesPullingTime="5.885024628s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:06:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:06:29.595315908 +0000 UTC m=+2296.202349676" watchObservedRunningTime="2026-11-02 20:06:29.598433681 +0000 UTC m=+2296.205467451"
Nov 02 20:06:29.665727 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:29.665675    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-8795bf/distinct-2-w4-8795bf-kserve-69c8b6b885-2chhn" podUID="9f430131-4938-47ca-ba86-b6a236f8cc3b" containerName="main" containerID="cri-o://776b5f17729e872597f4d31a2c6f0b826c106f88742ae67f05214246255cc9a9" gracePeriod=45
Nov 02 20:06:29.669309 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:29.669276    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-8795bf/distinct-w4-8795bf-kserve-64b4796df4-8qgqh" podUID="ee7d28f4-e9c7-43e2-b983-1427589f1785" containerName="main" containerID="cri-o://f52fdb1337b65d4c6774275aa530bc13f1b35b7ff2e505567c9d660d18c7befb" gracePeriod=45
Nov 02 20:06:29.929707 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:29.929682    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8795bf/distinct-2-w4-8795bf-kserve-69c8b6b885-2chhn"
Nov 02 20:06:29.959396 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:29.959373    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8795bf/distinct-w4-8795bf-kserve-64b4796df4-8qgqh"
Nov 02 20:06:30.036869 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:30.036803    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee7d28f4-e9c7-43e2-b983-1427589f1785-home\" (UniqueName: \"kubernetes.io/empty-dir/ee7d28f4-e9c7-43e2-b983-1427589f1785-home\") pod \"ee7d28f4-e9c7-43e2-b983-1427589f1785\" (UID: \"ee7d28f4-e9c7-43e2-b983-1427589f1785\") "
Nov 02 20:06:30.036869 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:30.036837    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f430131-4938-47ca-ba86-b6a236f8cc3b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f430131-4938-47ca-ba86-b6a236f8cc3b-dshm\") pod \"9f430131-4938-47ca-ba86-b6a236f8cc3b\" (UID: \"9f430131-4938-47ca-ba86-b6a236f8cc3b\") "
Nov 02 20:06:30.037128 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:30.036872    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee7d28f4-e9c7-43e2-b983-1427589f1785-tls-certs\" (UniqueName: \"kubernetes.io/secret/ee7d28f4-e9c7-43e2-b983-1427589f1785-tls-certs\") pod \"ee7d28f4-e9c7-43e2-b983-1427589f1785\" (UID: \"ee7d28f4-e9c7-43e2-b983-1427589f1785\") "
Nov 02 20:06:30.037128 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:30.036918    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee7d28f4-e9c7-43e2-b983-1427589f1785-kube-api-access-68xht\" (UniqueName: \"kubernetes.io/projected/ee7d28f4-e9c7-43e2-b983-1427589f1785-kube-api-access-68xht\") pod \"ee7d28f4-e9c7-43e2-b983-1427589f1785\" (UID: \"ee7d28f4-e9c7-43e2-b983-1427589f1785\") "
Nov 02 20:06:30.037128 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:30.036957    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f430131-4938-47ca-ba86-b6a236f8cc3b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f430131-4938-47ca-ba86-b6a236f8cc3b-tmp-dir\") pod \"9f430131-4938-47ca-ba86-b6a236f8cc3b\" (UID: \"9f430131-4938-47ca-ba86-b6a236f8cc3b\") "
Nov 02 20:06:30.037128 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:30.037053    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f430131-4938-47ca-ba86-b6a236f8cc3b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f430131-4938-47ca-ba86-b6a236f8cc3b-model-cache\") pod \"9f430131-4938-47ca-ba86-b6a236f8cc3b\" (UID: \"9f430131-4938-47ca-ba86-b6a236f8cc3b\") "
Nov 02 20:06:30.037423 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:30.037189    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f430131-4938-47ca-ba86-b6a236f8cc3b-kube-api-access-rb9qr\" (UniqueName: \"kubernetes.io/projected/9f430131-4938-47ca-ba86-b6a236f8cc3b-kube-api-access-rb9qr\") pod \"9f430131-4938-47ca-ba86-b6a236f8cc3b\" (UID: \"9f430131-4938-47ca-ba86-b6a236f8cc3b\") "
Nov 02 20:06:30.037423 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:30.037244    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee7d28f4-e9c7-43e2-b983-1427589f1785-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee7d28f4-e9c7-43e2-b983-1427589f1785-tmp-dir\") pod \"ee7d28f4-e9c7-43e2-b983-1427589f1785\" (UID: \"ee7d28f4-e9c7-43e2-b983-1427589f1785\") "
Nov 02 20:06:30.037423 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:30.037297    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee7d28f4-e9c7-43e2-b983-1427589f1785-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee7d28f4-e9c7-43e2-b983-1427589f1785-model-cache\") pod \"ee7d28f4-e9c7-43e2-b983-1427589f1785\" (UID: \"ee7d28f4-e9c7-43e2-b983-1427589f1785\") "
Nov 02 20:06:30.037423 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:30.037323    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee7d28f4-e9c7-43e2-b983-1427589f1785-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee7d28f4-e9c7-43e2-b983-1427589f1785-dshm\") pod \"ee7d28f4-e9c7-43e2-b983-1427589f1785\" (UID: \"ee7d28f4-e9c7-43e2-b983-1427589f1785\") "
Nov 02 20:06:30.037423 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:30.037408    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f430131-4938-47ca-ba86-b6a236f8cc3b-home\" (UniqueName: \"kubernetes.io/empty-dir/9f430131-4938-47ca-ba86-b6a236f8cc3b-home\") pod \"9f430131-4938-47ca-ba86-b6a236f8cc3b\" (UID: \"9f430131-4938-47ca-ba86-b6a236f8cc3b\") "
Nov 02 20:06:30.037691 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:30.037457    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f430131-4938-47ca-ba86-b6a236f8cc3b-tls-certs\" (UniqueName: \"kubernetes.io/secret/9f430131-4938-47ca-ba86-b6a236f8cc3b-tls-certs\") pod \"9f430131-4938-47ca-ba86-b6a236f8cc3b\" (UID: \"9f430131-4938-47ca-ba86-b6a236f8cc3b\") "
Nov 02 20:06:30.037691 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:30.037620    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f430131-4938-47ca-ba86-b6a236f8cc3b-model-cache" pod "9f430131-4938-47ca-ba86-b6a236f8cc3b" (UID: "9f430131-4938-47ca-ba86-b6a236f8cc3b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:06:30.039272 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:30.037861    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee7d28f4-e9c7-43e2-b983-1427589f1785-home" pod "ee7d28f4-e9c7-43e2-b983-1427589f1785" (UID: "ee7d28f4-e9c7-43e2-b983-1427589f1785"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:06:30.039272 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:30.038126    4463 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee7d28f4-e9c7-43e2-b983-1427589f1785-home\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:06:30.039272 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:30.038149    4463 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f430131-4938-47ca-ba86-b6a236f8cc3b-model-cache\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:06:30.039272 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:30.038748    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee7d28f4-e9c7-43e2-b983-1427589f1785-tmp-dir" pod "ee7d28f4-e9c7-43e2-b983-1427589f1785" (UID: "ee7d28f4-e9c7-43e2-b983-1427589f1785"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:06:30.039272 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:30.038955    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f430131-4938-47ca-ba86-b6a236f8cc3b-tmp-dir" pod "9f430131-4938-47ca-ba86-b6a236f8cc3b" (UID: "9f430131-4938-47ca-ba86-b6a236f8cc3b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:06:30.039272 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:30.039040    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f430131-4938-47ca-ba86-b6a236f8cc3b-home" pod "9f430131-4938-47ca-ba86-b6a236f8cc3b" (UID: "9f430131-4938-47ca-ba86-b6a236f8cc3b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:06:30.039272 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:30.039090    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee7d28f4-e9c7-43e2-b983-1427589f1785-model-cache" pod "ee7d28f4-e9c7-43e2-b983-1427589f1785" (UID: "ee7d28f4-e9c7-43e2-b983-1427589f1785"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:06:30.041292 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:30.041264    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee7d28f4-e9c7-43e2-b983-1427589f1785-kube-api-access-68xht" pod "ee7d28f4-e9c7-43e2-b983-1427589f1785" (UID: "ee7d28f4-e9c7-43e2-b983-1427589f1785"). InnerVolumeSpecName "kube-api-access-68xht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:06:30.041409 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:30.041375    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee7d28f4-e9c7-43e2-b983-1427589f1785-tls-certs" pod "ee7d28f4-e9c7-43e2-b983-1427589f1785" (UID: "ee7d28f4-e9c7-43e2-b983-1427589f1785"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:06:30.041856 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:30.041825    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee7d28f4-e9c7-43e2-b983-1427589f1785-dshm" pod "ee7d28f4-e9c7-43e2-b983-1427589f1785" (UID: "ee7d28f4-e9c7-43e2-b983-1427589f1785"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:06:30.042563 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:30.042536    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f430131-4938-47ca-ba86-b6a236f8cc3b-tls-certs" pod "9f430131-4938-47ca-ba86-b6a236f8cc3b" (UID: "9f430131-4938-47ca-ba86-b6a236f8cc3b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:06:30.042843 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:30.042820    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f430131-4938-47ca-ba86-b6a236f8cc3b-kube-api-access-rb9qr" pod "9f430131-4938-47ca-ba86-b6a236f8cc3b" (UID: "9f430131-4938-47ca-ba86-b6a236f8cc3b"). InnerVolumeSpecName "kube-api-access-rb9qr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:06:30.042954 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:30.042842    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f430131-4938-47ca-ba86-b6a236f8cc3b-dshm" pod "9f430131-4938-47ca-ba86-b6a236f8cc3b" (UID: "9f430131-4938-47ca-ba86-b6a236f8cc3b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:06:30.139124 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:30.139083    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68xht\" (UniqueName: \"kubernetes.io/projected/ee7d28f4-e9c7-43e2-b983-1427589f1785-kube-api-access-68xht\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:06:30.139124 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:30.139122    4463 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f430131-4938-47ca-ba86-b6a236f8cc3b-tmp-dir\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:06:30.139124 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:30.139132    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rb9qr\" (UniqueName: \"kubernetes.io/projected/9f430131-4938-47ca-ba86-b6a236f8cc3b-kube-api-access-rb9qr\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:06:30.139349 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:30.139144    4463 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee7d28f4-e9c7-43e2-b983-1427589f1785-tmp-dir\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:06:30.139349 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:30.139152    4463 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee7d28f4-e9c7-43e2-b983-1427589f1785-model-cache\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:06:30.139349 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:30.139160    4463 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee7d28f4-e9c7-43e2-b983-1427589f1785-dshm\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:06:30.139349 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:30.139169    4463 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f430131-4938-47ca-ba86-b6a236f8cc3b-home\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:06:30.139349 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:30.139176    4463 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f430131-4938-47ca-ba86-b6a236f8cc3b-tls-certs\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:06:30.139349 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:30.139183    4463 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f430131-4938-47ca-ba86-b6a236f8cc3b-dshm\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:06:30.139349 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:30.139191    4463 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee7d28f4-e9c7-43e2-b983-1427589f1785-tls-certs\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:06:30.500236 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:30.500187    4463 generic.go:356] "Generic (PLEG): container finished" podID="9f430131-4938-47ca-ba86-b6a236f8cc3b" containerID="776b5f17729e872597f4d31a2c6f0b826c106f88742ae67f05214246255cc9a9" exitCode=0
Nov 02 20:06:30.500445 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:30.500330    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8795bf/distinct-2-w4-8795bf-kserve-69c8b6b885-2chhn" event={"ID":"9f430131-4938-47ca-ba86-b6a236f8cc3b","Type":"ContainerDied","Data":"776b5f17729e872597f4d31a2c6f0b826c106f88742ae67f05214246255cc9a9"}
Nov 02 20:06:30.500445 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:30.500370    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8795bf/distinct-2-w4-8795bf-kserve-69c8b6b885-2chhn" event={"ID":"9f430131-4938-47ca-ba86-b6a236f8cc3b","Type":"ContainerDied","Data":"4226485cc1abdd71f7999c4bc55d2528908345f45083e327bd3ebe012687af61"}
Nov 02 20:06:30.500445 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:30.500390    4463 scope.go:122] "RemoveContainer" containerID="776b5f17729e872597f4d31a2c6f0b826c106f88742ae67f05214246255cc9a9"
Nov 02 20:06:30.500609 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:30.500513    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8795bf/distinct-2-w4-8795bf-kserve-69c8b6b885-2chhn"
Nov 02 20:06:30.502759 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:30.502697    4463 generic.go:356] "Generic (PLEG): container finished" podID="ee7d28f4-e9c7-43e2-b983-1427589f1785" containerID="f52fdb1337b65d4c6774275aa530bc13f1b35b7ff2e505567c9d660d18c7befb" exitCode=0
Nov 02 20:06:30.503565 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:30.503545    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8795bf/distinct-w4-8795bf-kserve-64b4796df4-8qgqh"
Nov 02 20:06:30.508163 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:30.508114    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8795bf/distinct-w4-8795bf-kserve-64b4796df4-8qgqh" event={"ID":"ee7d28f4-e9c7-43e2-b983-1427589f1785","Type":"ContainerDied","Data":"f52fdb1337b65d4c6774275aa530bc13f1b35b7ff2e505567c9d660d18c7befb"}
Nov 02 20:06:30.508276 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:30.508183    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8795bf/distinct-w4-8795bf-kserve-64b4796df4-8qgqh" event={"ID":"ee7d28f4-e9c7-43e2-b983-1427589f1785","Type":"ContainerDied","Data":"00fb291e44ab7e4c63cf3034fd67466e4b190533f68a5ae9fe1ad254240d241f"}
Nov 02 20:06:30.514777 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:30.514664    4463 scope.go:122] "RemoveContainer" containerID="776b5f17729e872597f4d31a2c6f0b826c106f88742ae67f05214246255cc9a9"
Nov 02 20:06:30.514988 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:06:30.514954    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"776b5f17729e872597f4d31a2c6f0b826c106f88742ae67f05214246255cc9a9\": container with ID starting with 776b5f17729e872597f4d31a2c6f0b826c106f88742ae67f05214246255cc9a9 not found: ID does not exist" containerID="776b5f17729e872597f4d31a2c6f0b826c106f88742ae67f05214246255cc9a9"
Nov 02 20:06:30.515106 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:30.514991    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"776b5f17729e872597f4d31a2c6f0b826c106f88742ae67f05214246255cc9a9"} err="failed to get container status \"776b5f17729e872597f4d31a2c6f0b826c106f88742ae67f05214246255cc9a9\": rpc error: code = NotFound desc = could not find container \"776b5f17729e872597f4d31a2c6f0b826c106f88742ae67f05214246255cc9a9\": container with ID starting with 776b5f17729e872597f4d31a2c6f0b826c106f88742ae67f05214246255cc9a9 not found: ID does not exist"
Nov 02 20:06:30.515106 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:30.515043    4463 scope.go:122] "RemoveContainer" containerID="f52fdb1337b65d4c6774275aa530bc13f1b35b7ff2e505567c9d660d18c7befb"
Nov 02 20:06:30.524861 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:30.524840    4463 scope.go:122] "RemoveContainer" containerID="f52fdb1337b65d4c6774275aa530bc13f1b35b7ff2e505567c9d660d18c7befb"
Nov 02 20:06:30.525269 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:06:30.525238    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f52fdb1337b65d4c6774275aa530bc13f1b35b7ff2e505567c9d660d18c7befb\": container with ID starting with f52fdb1337b65d4c6774275aa530bc13f1b35b7ff2e505567c9d660d18c7befb not found: ID does not exist" containerID="f52fdb1337b65d4c6774275aa530bc13f1b35b7ff2e505567c9d660d18c7befb"
Nov 02 20:06:30.525375 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:30.525279    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f52fdb1337b65d4c6774275aa530bc13f1b35b7ff2e505567c9d660d18c7befb"} err="failed to get container status \"f52fdb1337b65d4c6774275aa530bc13f1b35b7ff2e505567c9d660d18c7befb\": rpc error: code = NotFound desc = could not find container \"f52fdb1337b65d4c6774275aa530bc13f1b35b7ff2e505567c9d660d18c7befb\": container with ID starting with f52fdb1337b65d4c6774275aa530bc13f1b35b7ff2e505567c9d660d18c7befb not found: ID does not exist"
Nov 02 20:06:30.653182 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:30.653154    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8795bf/distinct-w4-8795bf-kserve-64b4796df4-8qgqh"]
Nov 02 20:06:30.658604 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:30.658579    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-8795bf/distinct-w4-8795bf-kserve-64b4796df4-8qgqh"]
Nov 02 20:06:30.704199 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:30.704165    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-19d134-5fc6b854f5-k78vk"]
Nov 02 20:06:30.705445 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:30.705417    4463 cpu_manager.go:414] "containerMap: removing container" podUID="ee7d28f4-e9c7-43e2-b983-1427589f1785" containerName="main"
Nov 02 20:06:30.705602 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:30.705586    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee7d28f4-e9c7-43e2-b983-1427589f1785" containerName="main"
Nov 02 20:06:30.705786 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:30.705772    4463 cpu_manager.go:414] "containerMap: removing container" podUID="9f430131-4938-47ca-ba86-b6a236f8cc3b" containerName="main"
Nov 02 20:06:30.705886 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:30.705865    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f430131-4938-47ca-ba86-b6a236f8cc3b" containerName="main"
Nov 02 20:06:30.710859 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:30.706066    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee7d28f4-e9c7-43e2-b983-1427589f1785" containerName="main"
Nov 02 20:06:30.710859 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:30.706093    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f430131-4938-47ca-ba86-b6a236f8cc3b" containerName="main"
Nov 02 20:06:30.710859 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:30.710162    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-19d134-5fc6b854f5-k78vk"
Nov 02 20:06:30.765089 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:30.763102    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-19d134-5fc6b854f5-k78vk"]
Nov 02 20:06:30.765089 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:30.764179    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-19d134\""
Nov 02 20:06:30.765089 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:30.764257    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-19d134-dockercfg-v2xsk\""
Nov 02 20:06:30.773037 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:30.768220    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8795bf/distinct-2-w4-8795bf-kserve-69c8b6b885-2chhn"]
Nov 02 20:06:30.773037 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:30.771796    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-8795bf/distinct-2-w4-8795bf-kserve-69c8b6b885-2chhn"]
Nov 02 20:06:30.849443 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:30.849412    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkcbb\" (UniqueName: \"kubernetes.io/projected/d6bed081-d1c4-426f-9611-81894a7b3e57-kube-api-access-fkcbb\") pod \"payload-processing-e2e-inf-b-19d134-5fc6b854f5-k78vk\" (UID: \"d6bed081-d1c4-426f-9611-81894a7b3e57\") " pod="openshift-ingress/payload-processing-e2e-inf-b-19d134-5fc6b854f5-k78vk"
Nov 02 20:06:30.849602 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:30.849582    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6bed081-d1c4-426f-9611-81894a7b3e57-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-19d134-5fc6b854f5-k78vk\" (UID: \"d6bed081-d1c4-426f-9611-81894a7b3e57\") " pod="openshift-ingress/payload-processing-e2e-inf-b-19d134-5fc6b854f5-k78vk"
Nov 02 20:06:30.870629 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:30.870590    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8795bf/distinct-w4-8795bf-kserve-64b4796df4-8qgqh" podUID="ee7d28f4-e9c7-43e2-b983-1427589f1785" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:06:30.951115 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:30.951079    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6bed081-d1c4-426f-9611-81894a7b3e57-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-19d134-5fc6b854f5-k78vk\" (UID: \"d6bed081-d1c4-426f-9611-81894a7b3e57\") " pod="openshift-ingress/payload-processing-e2e-inf-b-19d134-5fc6b854f5-k78vk"
Nov 02 20:06:30.951265 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:30.951235    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkcbb\" (UniqueName: \"kubernetes.io/projected/d6bed081-d1c4-426f-9611-81894a7b3e57-kube-api-access-fkcbb\") pod \"payload-processing-e2e-inf-b-19d134-5fc6b854f5-k78vk\" (UID: \"d6bed081-d1c4-426f-9611-81894a7b3e57\") " pod="openshift-ingress/payload-processing-e2e-inf-b-19d134-5fc6b854f5-k78vk"
Nov 02 20:06:30.951791 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:30.951765    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6bed081-d1c4-426f-9611-81894a7b3e57-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-19d134-5fc6b854f5-k78vk\" (UID: \"d6bed081-d1c4-426f-9611-81894a7b3e57\") " pod="openshift-ingress/payload-processing-e2e-inf-b-19d134-5fc6b854f5-k78vk"
Nov 02 20:06:31.004434 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:31.004405    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkcbb\" (UniqueName: \"kubernetes.io/projected/d6bed081-d1c4-426f-9611-81894a7b3e57-kube-api-access-fkcbb\") pod \"payload-processing-e2e-inf-b-19d134-5fc6b854f5-k78vk\" (UID: \"d6bed081-d1c4-426f-9611-81894a7b3e57\") " pod="openshift-ingress/payload-processing-e2e-inf-b-19d134-5fc6b854f5-k78vk"
Nov 02 20:06:31.033984 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:31.033929    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-19d134-5fc6b854f5-k78vk"
Nov 02 20:06:31.409273 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:31.409244    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-19d134-5fc6b854f5-k78vk"]
Nov 02 20:06:31.409940 ip-10-0-133-9 kubenswrapper[4463]: W1102 20:06:31.409908    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6bed081_d1c4_426f_9611_81894a7b3e57.slice/crio-7b60d7f276f6b9a37c1c6f820cda5171404257227379e3a6461e36643b106c71 WatchSource:0}: Error finding container 7b60d7f276f6b9a37c1c6f820cda5171404257227379e3a6461e36643b106c71: Status 404 returned error can't find the container with id 7b60d7f276f6b9a37c1c6f820cda5171404257227379e3a6461e36643b106c71
Nov 02 20:06:31.511970 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:31.511927    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-19d134-5fc6b854f5-k78vk" event={"ID":"d6bed081-d1c4-426f-9611-81894a7b3e57","Type":"ContainerStarted","Data":"da44ded62f93416710d44746e0d53d41c55993ae66afd70a42d6c7c243e17b1e"}
Nov 02 20:06:31.512121 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:31.511996    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-19d134-5fc6b854f5-k78vk" event={"ID":"d6bed081-d1c4-426f-9611-81894a7b3e57","Type":"ContainerStarted","Data":"7b60d7f276f6b9a37c1c6f820cda5171404257227379e3a6461e36643b106c71"}
Nov 02 20:06:31.512121 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:31.512051    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-19d134-5fc6b854f5-k78vk"
Nov 02 20:06:31.585411 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:31.585385    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f430131-4938-47ca-ba86-b6a236f8cc3b" path="/var/lib/kubelet/pods/9f430131-4938-47ca-ba86-b6a236f8cc3b/volumes"
Nov 02 20:06:31.585810 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:31.585797    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee7d28f4-e9c7-43e2-b983-1427589f1785" path="/var/lib/kubelet/pods/ee7d28f4-e9c7-43e2-b983-1427589f1785/volumes"
Nov 02 20:06:31.621109 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:31.621067    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-19d134-5fc6b854f5-k78vk" podStartSLOduration=1.621054577 podStartE2EDuration="1.621054577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:06:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:06:31.617239983 +0000 UTC m=+2298.224273754" watchObservedRunningTime="2026-11-02 20:06:31.621054577 +0000 UTC m=+2298.228088346"
Nov 02 20:06:32.430346 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:32.430319    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-28fa24-6d779499-gvdmm"
Nov 02 20:06:32.430727 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:32.430382    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-28fa24-8f6447cc7-x6jsz"
Nov 02 20:06:33.457962 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:33.433266    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-13e739-565886dbbc-wbgfw"]
Nov 02 20:06:33.473034 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:33.464735    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-13e739-565886dbbc-wbgfw"
Nov 02 20:06:33.503038 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:33.497493    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnpmj\" (UniqueName: \"kubernetes.io/projected/e20408d3-f3a6-4ca9-8413-aafe209a1477-kube-api-access-fnpmj\") pod \"payload-processing-e2e-inf-b-13e739-565886dbbc-wbgfw\" (UID: \"e20408d3-f3a6-4ca9-8413-aafe209a1477\") " pod="openshift-ingress/payload-processing-e2e-inf-b-13e739-565886dbbc-wbgfw"
Nov 02 20:06:33.503038 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:33.497752    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e20408d3-f3a6-4ca9-8413-aafe209a1477-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-13e739-565886dbbc-wbgfw\" (UID: \"e20408d3-f3a6-4ca9-8413-aafe209a1477\") " pod="openshift-ingress/payload-processing-e2e-inf-b-13e739-565886dbbc-wbgfw"
Nov 02 20:06:33.514595 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:33.514557    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-13e739\""
Nov 02 20:06:33.515414 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:33.515389    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-13e739-dockercfg-lpmb9\""
Nov 02 20:06:33.519318 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:33.518038    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-13e739-565886dbbc-wbgfw"]
Nov 02 20:06:33.598693 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:33.598661    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e20408d3-f3a6-4ca9-8413-aafe209a1477-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-13e739-565886dbbc-wbgfw\" (UID: \"e20408d3-f3a6-4ca9-8413-aafe209a1477\") " pod="openshift-ingress/payload-processing-e2e-inf-b-13e739-565886dbbc-wbgfw"
Nov 02 20:06:33.598862 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:33.598821    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnpmj\" (UniqueName: \"kubernetes.io/projected/e20408d3-f3a6-4ca9-8413-aafe209a1477-kube-api-access-fnpmj\") pod \"payload-processing-e2e-inf-b-13e739-565886dbbc-wbgfw\" (UID: \"e20408d3-f3a6-4ca9-8413-aafe209a1477\") " pod="openshift-ingress/payload-processing-e2e-inf-b-13e739-565886dbbc-wbgfw"
Nov 02 20:06:33.599470 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:33.599449    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e20408d3-f3a6-4ca9-8413-aafe209a1477-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-13e739-565886dbbc-wbgfw\" (UID: \"e20408d3-f3a6-4ca9-8413-aafe209a1477\") " pod="openshift-ingress/payload-processing-e2e-inf-b-13e739-565886dbbc-wbgfw"
Nov 02 20:06:33.651629 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:33.651605    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnpmj\" (UniqueName: \"kubernetes.io/projected/e20408d3-f3a6-4ca9-8413-aafe209a1477-kube-api-access-fnpmj\") pod \"payload-processing-e2e-inf-b-13e739-565886dbbc-wbgfw\" (UID: \"e20408d3-f3a6-4ca9-8413-aafe209a1477\") " pod="openshift-ingress/payload-processing-e2e-inf-b-13e739-565886dbbc-wbgfw"
Nov 02 20:06:33.801097 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:33.801004    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-13e739-565886dbbc-wbgfw"
Nov 02 20:06:34.172005 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:34.171972    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-13e739-565886dbbc-wbgfw"]
Nov 02 20:06:34.173435 ip-10-0-133-9 kubenswrapper[4463]: W1102 20:06:34.173408    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode20408d3_f3a6_4ca9_8413_aafe209a1477.slice/crio-cbb16f4ce1ca01e239441fd8bf917eeff095802acdae386b5ac6f08658a0f5f1 WatchSource:0}: Error finding container cbb16f4ce1ca01e239441fd8bf917eeff095802acdae386b5ac6f08658a0f5f1: Status 404 returned error can't find the container with id cbb16f4ce1ca01e239441fd8bf917eeff095802acdae386b5ac6f08658a0f5f1
Nov 02 20:06:34.293480 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:06:34.293444    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 20:06:34.540240 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:34.540210    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-13e739-565886dbbc-wbgfw" event={"ID":"e20408d3-f3a6-4ca9-8413-aafe209a1477","Type":"ContainerStarted","Data":"d508d0a4ce5b48c8a62e8c256f746208ea5baff320d3713213b4c4be51d6c014"}
Nov 02 20:06:34.540634 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:34.540257    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-13e739-565886dbbc-wbgfw" event={"ID":"e20408d3-f3a6-4ca9-8413-aafe209a1477","Type":"ContainerStarted","Data":"cbb16f4ce1ca01e239441fd8bf917eeff095802acdae386b5ac6f08658a0f5f1"}
Nov 02 20:06:34.540634 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:34.540351    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-13e739-565886dbbc-wbgfw"
Nov 02 20:06:34.651111 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:34.651063    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-13e739-565886dbbc-wbgfw" podStartSLOduration=1.651049099 podStartE2EDuration="1.651049099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:06:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:06:34.645619411 +0000 UTC m=+2301.252653199" watchObservedRunningTime="2026-11-02 20:06:34.651049099 +0000 UTC m=+2301.258082868"
Nov 02 20:06:35.510091 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:35.509984    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1db0e4-6b545bbdb7-nkvgn"
Nov 02 20:06:37.129940 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:37.129902    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-8795bf-7bd68d77d9-5xm6w" podUID="0368fdf1-1c9f-4ee9-8da1-5dd57731a8fd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 20:06:37.130335 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:37.129915    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8795bf-66d99fc7c9-nj9gz" podUID="4553e306-34a0-48cf-b662-843e8de5bb45" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 20:06:37.704684 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:37.704649    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 20:06:42.522500 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:42.522468    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-19d134-5fc6b854f5-k78vk"
Nov 02 20:06:45.549244 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:45.549216    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-13e739-565886dbbc-wbgfw"
Nov 02 20:06:47.129393 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:47.129349    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8795bf-66d99fc7c9-nj9gz" podUID="4553e306-34a0-48cf-b662-843e8de5bb45" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 20:06:47.129798 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:47.129443    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-8795bf-7bd68d77d9-5xm6w" podUID="0368fdf1-1c9f-4ee9-8da1-5dd57731a8fd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 20:06:48.894991 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:48.894652    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-8795bf-7bd68d77d9-5xm6w"
Nov 02 20:06:48.970365 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:48.970343    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8795bf-66d99fc7c9-nj9gz"
Nov 02 20:06:49.000212 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:49.000189    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0368fdf1-1c9f-4ee9-8da1-5dd57731a8fd-kube-api-access-ps895\" (UniqueName: \"kubernetes.io/projected/0368fdf1-1c9f-4ee9-8da1-5dd57731a8fd-kube-api-access-ps895\") pod \"0368fdf1-1c9f-4ee9-8da1-5dd57731a8fd\" (UID: \"0368fdf1-1c9f-4ee9-8da1-5dd57731a8fd\") "
Nov 02 20:06:49.000849 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:49.000354    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0368fdf1-1c9f-4ee9-8da1-5dd57731a8fd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0368fdf1-1c9f-4ee9-8da1-5dd57731a8fd-plugins-config-volume\") pod \"0368fdf1-1c9f-4ee9-8da1-5dd57731a8fd\" (UID: \"0368fdf1-1c9f-4ee9-8da1-5dd57731a8fd\") "
Nov 02 20:06:49.000849 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:49.000733    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0368fdf1-1c9f-4ee9-8da1-5dd57731a8fd-plugins-config-volume" pod "0368fdf1-1c9f-4ee9-8da1-5dd57731a8fd" (UID: "0368fdf1-1c9f-4ee9-8da1-5dd57731a8fd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:06:49.001368 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:49.001337    4463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0368fdf1-1c9f-4ee9-8da1-5dd57731a8fd-plugins-config-volume\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:06:49.002596 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:49.002576    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0368fdf1-1c9f-4ee9-8da1-5dd57731a8fd-kube-api-access-ps895" pod "0368fdf1-1c9f-4ee9-8da1-5dd57731a8fd" (UID: "0368fdf1-1c9f-4ee9-8da1-5dd57731a8fd"). InnerVolumeSpecName "kube-api-access-ps895". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:06:49.102582 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:49.102552    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4553e306-34a0-48cf-b662-843e8de5bb45-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4553e306-34a0-48cf-b662-843e8de5bb45-plugins-config-volume\") pod \"4553e306-34a0-48cf-b662-843e8de5bb45\" (UID: \"4553e306-34a0-48cf-b662-843e8de5bb45\") "
Nov 02 20:06:49.102768 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:49.102750    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4553e306-34a0-48cf-b662-843e8de5bb45-kube-api-access-xpr6p\" (UniqueName: \"kubernetes.io/projected/4553e306-34a0-48cf-b662-843e8de5bb45-kube-api-access-xpr6p\") pod \"4553e306-34a0-48cf-b662-843e8de5bb45\" (UID: \"4553e306-34a0-48cf-b662-843e8de5bb45\") "
Nov 02 20:06:49.102970 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:49.102920    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4553e306-34a0-48cf-b662-843e8de5bb45-plugins-config-volume" pod "4553e306-34a0-48cf-b662-843e8de5bb45" (UID: "4553e306-34a0-48cf-b662-843e8de5bb45"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:06:49.103256 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:49.103241    4463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4553e306-34a0-48cf-b662-843e8de5bb45-plugins-config-volume\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:06:49.103302 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:49.103260    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ps895\" (UniqueName: \"kubernetes.io/projected/0368fdf1-1c9f-4ee9-8da1-5dd57731a8fd-kube-api-access-ps895\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:06:49.104867 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:49.104842    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4553e306-34a0-48cf-b662-843e8de5bb45-kube-api-access-xpr6p" pod "4553e306-34a0-48cf-b662-843e8de5bb45" (UID: "4553e306-34a0-48cf-b662-843e8de5bb45"). InnerVolumeSpecName "kube-api-access-xpr6p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:06:49.204608 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:49.204573    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpr6p\" (UniqueName: \"kubernetes.io/projected/4553e306-34a0-48cf-b662-843e8de5bb45-kube-api-access-xpr6p\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:06:49.643794 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:49.643766    4463 generic.go:356] "Generic (PLEG): container finished" podID="4553e306-34a0-48cf-b662-843e8de5bb45" containerID="db579ede9b1a5ed8b51f1953fa0cbdd7f646ad0744547b7d5a7fffd6feb07c19" exitCode=0
Nov 02 20:06:49.643906 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:49.643818    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8795bf-66d99fc7c9-nj9gz"
Nov 02 20:06:49.643906 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:49.643844    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8795bf-66d99fc7c9-nj9gz" event={"ID":"4553e306-34a0-48cf-b662-843e8de5bb45","Type":"ContainerDied","Data":"db579ede9b1a5ed8b51f1953fa0cbdd7f646ad0744547b7d5a7fffd6feb07c19"}
Nov 02 20:06:49.643906 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:49.643896    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8795bf-66d99fc7c9-nj9gz" event={"ID":"4553e306-34a0-48cf-b662-843e8de5bb45","Type":"ContainerDied","Data":"37b3a61ce10d160ea639dc86295080e9b40e8c4abf99d9f92aac5f2ce0cd177f"}
Nov 02 20:06:49.644064 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:49.643917    4463 scope.go:122] "RemoveContainer" containerID="db579ede9b1a5ed8b51f1953fa0cbdd7f646ad0744547b7d5a7fffd6feb07c19"
Nov 02 20:06:49.645500 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:49.645474    4463 generic.go:356] "Generic (PLEG): container finished" podID="0368fdf1-1c9f-4ee9-8da1-5dd57731a8fd" containerID="867aca871d89fa6ca408b48822438b55aa9c1e9419e7452ba8baa39d821afb71" exitCode=0
Nov 02 20:06:49.645622 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:49.645519    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-8795bf-7bd68d77d9-5xm6w" event={"ID":"0368fdf1-1c9f-4ee9-8da1-5dd57731a8fd","Type":"ContainerDied","Data":"867aca871d89fa6ca408b48822438b55aa9c1e9419e7452ba8baa39d821afb71"}
Nov 02 20:06:49.645622 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:49.645556    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-8795bf-7bd68d77d9-5xm6w" event={"ID":"0368fdf1-1c9f-4ee9-8da1-5dd57731a8fd","Type":"ContainerDied","Data":"eb36f80e1fb6ac4d1145320fae725da255756b589c216a743d7511ccfe9abe92"}
Nov 02 20:06:49.645622 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:49.645524    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-8795bf-7bd68d77d9-5xm6w"
Nov 02 20:06:49.656391 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:49.656366    4463 scope.go:122] "RemoveContainer" containerID="db579ede9b1a5ed8b51f1953fa0cbdd7f646ad0744547b7d5a7fffd6feb07c19"
Nov 02 20:06:49.656952 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:06:49.656927    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db579ede9b1a5ed8b51f1953fa0cbdd7f646ad0744547b7d5a7fffd6feb07c19\": container with ID starting with db579ede9b1a5ed8b51f1953fa0cbdd7f646ad0744547b7d5a7fffd6feb07c19 not found: ID does not exist" containerID="db579ede9b1a5ed8b51f1953fa0cbdd7f646ad0744547b7d5a7fffd6feb07c19"
Nov 02 20:06:49.657088 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:49.656958    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db579ede9b1a5ed8b51f1953fa0cbdd7f646ad0744547b7d5a7fffd6feb07c19"} err="failed to get container status \"db579ede9b1a5ed8b51f1953fa0cbdd7f646ad0744547b7d5a7fffd6feb07c19\": rpc error: code = NotFound desc = could not find container \"db579ede9b1a5ed8b51f1953fa0cbdd7f646ad0744547b7d5a7fffd6feb07c19\": container with ID starting with db579ede9b1a5ed8b51f1953fa0cbdd7f646ad0744547b7d5a7fffd6feb07c19 not found: ID does not exist"
Nov 02 20:06:49.657088 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:49.656974    4463 scope.go:122] "RemoveContainer" containerID="867aca871d89fa6ca408b48822438b55aa9c1e9419e7452ba8baa39d821afb71"
Nov 02 20:06:49.665428 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:49.665409    4463 scope.go:122] "RemoveContainer" containerID="867aca871d89fa6ca408b48822438b55aa9c1e9419e7452ba8baa39d821afb71"
Nov 02 20:06:49.665687 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:06:49.665668    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"867aca871d89fa6ca408b48822438b55aa9c1e9419e7452ba8baa39d821afb71\": container with ID starting with 867aca871d89fa6ca408b48822438b55aa9c1e9419e7452ba8baa39d821afb71 not found: ID does not exist" containerID="867aca871d89fa6ca408b48822438b55aa9c1e9419e7452ba8baa39d821afb71"
Nov 02 20:06:49.665744 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:49.665696    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"867aca871d89fa6ca408b48822438b55aa9c1e9419e7452ba8baa39d821afb71"} err="failed to get container status \"867aca871d89fa6ca408b48822438b55aa9c1e9419e7452ba8baa39d821afb71\": rpc error: code = NotFound desc = could not find container \"867aca871d89fa6ca408b48822438b55aa9c1e9419e7452ba8baa39d821afb71\": container with ID starting with 867aca871d89fa6ca408b48822438b55aa9c1e9419e7452ba8baa39d821afb71 not found: ID does not exist"
Nov 02 20:06:49.797508 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:49.797477    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-8795bf-66d99fc7c9-nj9gz"]
Nov 02 20:06:49.843575 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:49.843550    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-8795bf-66d99fc7c9-nj9gz"]
Nov 02 20:06:49.945622 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:49.945590    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-8795bf-7bd68d77d9-5xm6w"]
Nov 02 20:06:49.954125 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:49.954094    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-8795bf-7bd68d77d9-5xm6w"]
Nov 02 20:06:51.585321 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:51.585280    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0368fdf1-1c9f-4ee9-8da1-5dd57731a8fd" path="/var/lib/kubelet/pods/0368fdf1-1c9f-4ee9-8da1-5dd57731a8fd/volumes"
Nov 02 20:06:51.585918 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:51.585896    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4553e306-34a0-48cf-b662-843e8de5bb45" path="/var/lib/kubelet/pods/4553e306-34a0-48cf-b662-843e8de5bb45/volumes"
Nov 02 20:06:56.370118 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:56.370083    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-rjhcs"]
Nov 02 20:06:56.370706 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:56.370693    4463 cpu_manager.go:414] "containerMap: removing container" podUID="0368fdf1-1c9f-4ee9-8da1-5dd57731a8fd" containerName="payload-processing"
Nov 02 20:06:56.370763 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:56.370707    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0368fdf1-1c9f-4ee9-8da1-5dd57731a8fd" containerName="payload-processing"
Nov 02 20:06:56.370763 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:56.370718    4463 cpu_manager.go:414] "containerMap: removing container" podUID="4553e306-34a0-48cf-b662-843e8de5bb45" containerName="payload-pre-processing"
Nov 02 20:06:56.370763 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:56.370723    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4553e306-34a0-48cf-b662-843e8de5bb45" containerName="payload-pre-processing"
Nov 02 20:06:56.370860 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:56.370831    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="0368fdf1-1c9f-4ee9-8da1-5dd57731a8fd" containerName="payload-processing"
Nov 02 20:06:56.370860 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:56.370839    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="4553e306-34a0-48cf-b662-843e8de5bb45" containerName="payload-pre-processing"
Nov 02 20:06:56.374680 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:56.374660    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-rjhcs"
Nov 02 20:06:56.382460 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:56.382438    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd10e27d-811c-4488-af13-fa8c3272b54a-workload-certs\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-rjhcs\" (UID: \"dd10e27d-811c-4488-af13-fa8c3272b54a\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-rjhcs"
Nov 02 20:06:56.382565 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:56.382496    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd10e27d-811c-4488-af13-fa8c3272b54a-istio-data\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-rjhcs\" (UID: \"dd10e27d-811c-4488-af13-fa8c3272b54a\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-rjhcs"
Nov 02 20:06:56.382632 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:56.382605    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd10e27d-811c-4488-af13-fa8c3272b54a-istio-podinfo\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-rjhcs\" (UID: \"dd10e27d-811c-4488-af13-fa8c3272b54a\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-rjhcs"
Nov 02 20:06:56.382696 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:56.382659    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd10e27d-811c-4488-af13-fa8c3272b54a-workload-socket\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-rjhcs\" (UID: \"dd10e27d-811c-4488-af13-fa8c3272b54a\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-rjhcs"
Nov 02 20:06:56.382696 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:56.382683    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd10e27d-811c-4488-af13-fa8c3272b54a-credential-socket\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-rjhcs\" (UID: \"dd10e27d-811c-4488-af13-fa8c3272b54a\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-rjhcs"
Nov 02 20:06:56.382843 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:56.382823    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd10e27d-811c-4488-af13-fa8c3272b54a-istio-token\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-rjhcs\" (UID: \"dd10e27d-811c-4488-af13-fa8c3272b54a\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-rjhcs"
Nov 02 20:06:56.382904 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:56.382865    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd10e27d-811c-4488-af13-fa8c3272b54a-istiod-ca-cert\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-rjhcs\" (UID: \"dd10e27d-811c-4488-af13-fa8c3272b54a\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-rjhcs"
Nov 02 20:06:56.382904 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:56.382893    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd10e27d-811c-4488-af13-fa8c3272b54a-istio-envoy\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-rjhcs\" (UID: \"dd10e27d-811c-4488-af13-fa8c3272b54a\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-rjhcs"
Nov 02 20:06:56.383005 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:56.382957    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6w9q\" (UniqueName: \"kubernetes.io/projected/dd10e27d-811c-4488-af13-fa8c3272b54a-kube-api-access-d6w9q\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-rjhcs\" (UID: \"dd10e27d-811c-4488-af13-fa8c3272b54a\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-rjhcs"
Nov 02 20:06:56.416036 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:56.415992    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-88zmb"]
Nov 02 20:06:56.418805 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:56.418791    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-88zmb"
Nov 02 20:06:56.427695 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:56.427673    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-8795bf-openshift-default-dockercfg-s7q56\""
Nov 02 20:06:56.462947 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:56.462911    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-rjhcs"]
Nov 02 20:06:56.476483 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:56.476454    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-88zmb"]
Nov 02 20:06:56.484913 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:56.484837    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd10e27d-811c-4488-af13-fa8c3272b54a-istio-data\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-rjhcs\" (UID: \"dd10e27d-811c-4488-af13-fa8c3272b54a\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-rjhcs"
Nov 02 20:06:56.485726 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:56.485277    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd8684fd-7158-4fa0-8a30-911e86bd53e4-istiod-ca-cert\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-88zmb\" (UID: \"fd8684fd-7158-4fa0-8a30-911e86bd53e4\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-88zmb"
Nov 02 20:06:56.485726 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:56.485392    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd10e27d-811c-4488-af13-fa8c3272b54a-istio-podinfo\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-rjhcs\" (UID: \"dd10e27d-811c-4488-af13-fa8c3272b54a\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-rjhcs"
Nov 02 20:06:56.485726 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:56.485478    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd10e27d-811c-4488-af13-fa8c3272b54a-workload-socket\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-rjhcs\" (UID: \"dd10e27d-811c-4488-af13-fa8c3272b54a\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-rjhcs"
Nov 02 20:06:56.485726 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:56.485562    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd10e27d-811c-4488-af13-fa8c3272b54a-istio-data\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-rjhcs\" (UID: \"dd10e27d-811c-4488-af13-fa8c3272b54a\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-rjhcs"
Nov 02 20:06:56.485726 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:56.485591    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd10e27d-811c-4488-af13-fa8c3272b54a-credential-socket\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-rjhcs\" (UID: \"dd10e27d-811c-4488-af13-fa8c3272b54a\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-rjhcs"
Nov 02 20:06:56.485726 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:56.485689    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd8684fd-7158-4fa0-8a30-911e86bd53e4-istio-podinfo\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-88zmb\" (UID: \"fd8684fd-7158-4fa0-8a30-911e86bd53e4\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-88zmb"
Nov 02 20:06:56.485726 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:56.485719    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd10e27d-811c-4488-af13-fa8c3272b54a-istio-token\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-rjhcs\" (UID: \"dd10e27d-811c-4488-af13-fa8c3272b54a\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-rjhcs"
Nov 02 20:06:56.486263 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:56.485759    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd10e27d-811c-4488-af13-fa8c3272b54a-istiod-ca-cert\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-rjhcs\" (UID: \"dd10e27d-811c-4488-af13-fa8c3272b54a\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-rjhcs"
Nov 02 20:06:56.486263 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:56.485793    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd8684fd-7158-4fa0-8a30-911e86bd53e4-credential-socket\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-88zmb\" (UID: \"fd8684fd-7158-4fa0-8a30-911e86bd53e4\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-88zmb"
Nov 02 20:06:56.486263 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:56.485822    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd10e27d-811c-4488-af13-fa8c3272b54a-istio-envoy\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-rjhcs\" (UID: \"dd10e27d-811c-4488-af13-fa8c3272b54a\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-rjhcs"
Nov 02 20:06:56.486263 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:56.485866    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6w9q\" (UniqueName: \"kubernetes.io/projected/dd10e27d-811c-4488-af13-fa8c3272b54a-kube-api-access-d6w9q\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-rjhcs\" (UID: \"dd10e27d-811c-4488-af13-fa8c3272b54a\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-rjhcs"
Nov 02 20:06:56.486263 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:56.485899    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd8684fd-7158-4fa0-8a30-911e86bd53e4-workload-certs\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-88zmb\" (UID: \"fd8684fd-7158-4fa0-8a30-911e86bd53e4\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-88zmb"
Nov 02 20:06:56.486524 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:56.486332    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd8684fd-7158-4fa0-8a30-911e86bd53e4-workload-socket\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-88zmb\" (UID: \"fd8684fd-7158-4fa0-8a30-911e86bd53e4\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-88zmb"
Nov 02 20:06:56.486641 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:56.486620    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd10e27d-811c-4488-af13-fa8c3272b54a-istiod-ca-cert\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-rjhcs\" (UID: \"dd10e27d-811c-4488-af13-fa8c3272b54a\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-rjhcs"
Nov 02 20:06:56.486752 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:56.486732    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd8684fd-7158-4fa0-8a30-911e86bd53e4-istio-envoy\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-88zmb\" (UID: \"fd8684fd-7158-4fa0-8a30-911e86bd53e4\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-88zmb"
Nov 02 20:06:56.486818 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:56.486783    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd8684fd-7158-4fa0-8a30-911e86bd53e4-istio-token\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-88zmb\" (UID: \"fd8684fd-7158-4fa0-8a30-911e86bd53e4\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-88zmb"
Nov 02 20:06:56.486881 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:56.486820    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd10e27d-811c-4488-af13-fa8c3272b54a-workload-socket\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-rjhcs\" (UID: \"dd10e27d-811c-4488-af13-fa8c3272b54a\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-rjhcs"
Nov 02 20:06:56.486881 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:56.486842    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd10e27d-811c-4488-af13-fa8c3272b54a-workload-certs\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-rjhcs\" (UID: \"dd10e27d-811c-4488-af13-fa8c3272b54a\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-rjhcs"
Nov 02 20:06:56.486985 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:56.486883    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd8684fd-7158-4fa0-8a30-911e86bd53e4-istio-data\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-88zmb\" (UID: \"fd8684fd-7158-4fa0-8a30-911e86bd53e4\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-88zmb"
Nov 02 20:06:56.486985 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:56.486926    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd10e27d-811c-4488-af13-fa8c3272b54a-credential-socket\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-rjhcs\" (UID: \"dd10e27d-811c-4488-af13-fa8c3272b54a\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-rjhcs"
Nov 02 20:06:56.486985 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:56.486966    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zffg6\" (UniqueName: \"kubernetes.io/projected/fd8684fd-7158-4fa0-8a30-911e86bd53e4-kube-api-access-zffg6\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-88zmb\" (UID: \"fd8684fd-7158-4fa0-8a30-911e86bd53e4\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-88zmb"
Nov 02 20:06:56.487193 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:56.487099    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd10e27d-811c-4488-af13-fa8c3272b54a-workload-certs\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-rjhcs\" (UID: \"dd10e27d-811c-4488-af13-fa8c3272b54a\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-rjhcs"
Nov 02 20:06:56.488613 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:56.488580    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd10e27d-811c-4488-af13-fa8c3272b54a-istio-envoy\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-rjhcs\" (UID: \"dd10e27d-811c-4488-af13-fa8c3272b54a\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-rjhcs"
Nov 02 20:06:56.488807 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:56.488788    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd10e27d-811c-4488-af13-fa8c3272b54a-istio-podinfo\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-rjhcs\" (UID: \"dd10e27d-811c-4488-af13-fa8c3272b54a\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-rjhcs"
Nov 02 20:06:56.539068 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:56.539037    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd10e27d-811c-4488-af13-fa8c3272b54a-istio-token\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-rjhcs\" (UID: \"dd10e27d-811c-4488-af13-fa8c3272b54a\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-rjhcs"
Nov 02 20:06:56.539199 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:56.539148    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6w9q\" (UniqueName: \"kubernetes.io/projected/dd10e27d-811c-4488-af13-fa8c3272b54a-kube-api-access-d6w9q\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-rjhcs\" (UID: \"dd10e27d-811c-4488-af13-fa8c3272b54a\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-rjhcs"
Nov 02 20:06:56.588286 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:56.588256    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd8684fd-7158-4fa0-8a30-911e86bd53e4-istio-token\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-88zmb\" (UID: \"fd8684fd-7158-4fa0-8a30-911e86bd53e4\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-88zmb"
Nov 02 20:06:56.588424 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:56.588317    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd8684fd-7158-4fa0-8a30-911e86bd53e4-istio-data\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-88zmb\" (UID: \"fd8684fd-7158-4fa0-8a30-911e86bd53e4\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-88zmb"
Nov 02 20:06:56.588424 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:56.588345    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zffg6\" (UniqueName: \"kubernetes.io/projected/fd8684fd-7158-4fa0-8a30-911e86bd53e4-kube-api-access-zffg6\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-88zmb\" (UID: \"fd8684fd-7158-4fa0-8a30-911e86bd53e4\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-88zmb"
Nov 02 20:06:56.588424 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:56.588380    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd8684fd-7158-4fa0-8a30-911e86bd53e4-istiod-ca-cert\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-88zmb\" (UID: \"fd8684fd-7158-4fa0-8a30-911e86bd53e4\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-88zmb"
Nov 02 20:06:56.588608 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:56.588486    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd8684fd-7158-4fa0-8a30-911e86bd53e4-istio-podinfo\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-88zmb\" (UID: \"fd8684fd-7158-4fa0-8a30-911e86bd53e4\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-88zmb"
Nov 02 20:06:56.588608 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:56.588539    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd8684fd-7158-4fa0-8a30-911e86bd53e4-credential-socket\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-88zmb\" (UID: \"fd8684fd-7158-4fa0-8a30-911e86bd53e4\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-88zmb"
Nov 02 20:06:56.588608 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:56.588597    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd8684fd-7158-4fa0-8a30-911e86bd53e4-workload-certs\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-88zmb\" (UID: \"fd8684fd-7158-4fa0-8a30-911e86bd53e4\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-88zmb"
Nov 02 20:06:56.588766 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:56.588634    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd8684fd-7158-4fa0-8a30-911e86bd53e4-workload-socket\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-88zmb\" (UID: \"fd8684fd-7158-4fa0-8a30-911e86bd53e4\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-88zmb"
Nov 02 20:06:56.588766 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:56.588710    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd8684fd-7158-4fa0-8a30-911e86bd53e4-istio-data\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-88zmb\" (UID: \"fd8684fd-7158-4fa0-8a30-911e86bd53e4\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-88zmb"
Nov 02 20:06:56.588766 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:56.588737    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd8684fd-7158-4fa0-8a30-911e86bd53e4-istio-envoy\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-88zmb\" (UID: \"fd8684fd-7158-4fa0-8a30-911e86bd53e4\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-88zmb"
Nov 02 20:06:56.588969 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:56.588889    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd8684fd-7158-4fa0-8a30-911e86bd53e4-credential-socket\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-88zmb\" (UID: \"fd8684fd-7158-4fa0-8a30-911e86bd53e4\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-88zmb"
Nov 02 20:06:56.589122 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:56.589087    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd8684fd-7158-4fa0-8a30-911e86bd53e4-workload-certs\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-88zmb\" (UID: \"fd8684fd-7158-4fa0-8a30-911e86bd53e4\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-88zmb"
Nov 02 20:06:56.589122 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:56.589105    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd8684fd-7158-4fa0-8a30-911e86bd53e4-workload-socket\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-88zmb\" (UID: \"fd8684fd-7158-4fa0-8a30-911e86bd53e4\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-88zmb"
Nov 02 20:06:56.589226 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:56.589160    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd8684fd-7158-4fa0-8a30-911e86bd53e4-istiod-ca-cert\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-88zmb\" (UID: \"fd8684fd-7158-4fa0-8a30-911e86bd53e4\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-88zmb"
Nov 02 20:06:56.590946 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:56.590922    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd8684fd-7158-4fa0-8a30-911e86bd53e4-istio-envoy\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-88zmb\" (UID: \"fd8684fd-7158-4fa0-8a30-911e86bd53e4\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-88zmb"
Nov 02 20:06:56.591591 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:56.591564    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd8684fd-7158-4fa0-8a30-911e86bd53e4-istio-podinfo\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-88zmb\" (UID: \"fd8684fd-7158-4fa0-8a30-911e86bd53e4\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-88zmb"
Nov 02 20:06:56.640489 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:56.640433    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd8684fd-7158-4fa0-8a30-911e86bd53e4-istio-token\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-88zmb\" (UID: \"fd8684fd-7158-4fa0-8a30-911e86bd53e4\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-88zmb"
Nov 02 20:06:56.640679 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:56.640661    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zffg6\" (UniqueName: \"kubernetes.io/projected/fd8684fd-7158-4fa0-8a30-911e86bd53e4-kube-api-access-zffg6\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-88zmb\" (UID: \"fd8684fd-7158-4fa0-8a30-911e86bd53e4\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-88zmb"
Nov 02 20:06:56.687501 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:56.687475    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-rjhcs"
Nov 02 20:06:56.732428 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:56.732398    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-88zmb"
Nov 02 20:06:57.122387 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:57.122357    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-rjhcs"]
Nov 02 20:06:57.123188 ip-10-0-133-9 kubenswrapper[4463]: W1102 20:06:57.123160    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd10e27d_811c_4488_af13_fa8c3272b54a.slice/crio-2f9f24856e150cd392e22e1a2e59f77166da6700e3499a48e0e2807c97eff2e3 WatchSource:0}: Error finding container 2f9f24856e150cd392e22e1a2e59f77166da6700e3499a48e0e2807c97eff2e3: Status 404 returned error can't find the container with id 2f9f24856e150cd392e22e1a2e59f77166da6700e3499a48e0e2807c97eff2e3
Nov 02 20:06:57.125740 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:57.125706    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:06:57.125816 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:57.125772    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:06:57.125816 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:57.125801    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:06:57.175223 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:57.175197    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-88zmb"]
Nov 02 20:06:57.177308 ip-10-0-133-9 kubenswrapper[4463]: W1102 20:06:57.177284    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd8684fd_7158_4fa0_8a30_911e86bd53e4.slice/crio-25d55892af830f3b5dc9df658faea7bd659b48bc46b4e39696e9e1d6655c9c6c WatchSource:0}: Error finding container 25d55892af830f3b5dc9df658faea7bd659b48bc46b4e39696e9e1d6655c9c6c: Status 404 returned error can't find the container with id 25d55892af830f3b5dc9df658faea7bd659b48bc46b4e39696e9e1d6655c9c6c
Nov 02 20:06:57.182001 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:57.181212    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:06:57.182001 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:57.181377    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:06:57.182001 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:57.181477    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:06:57.720749 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:57.720710    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-rjhcs" event={"ID":"dd10e27d-811c-4488-af13-fa8c3272b54a","Type":"ContainerStarted","Data":"b628122cd389b04510f80477e100142263336e341aea769d9f81138626984901"}
Nov 02 20:06:57.721381 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:57.721351    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-rjhcs" event={"ID":"dd10e27d-811c-4488-af13-fa8c3272b54a","Type":"ContainerStarted","Data":"2f9f24856e150cd392e22e1a2e59f77166da6700e3499a48e0e2807c97eff2e3"}
Nov 02 20:06:57.725404 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:57.725316    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-88zmb" event={"ID":"fd8684fd-7158-4fa0-8a30-911e86bd53e4","Type":"ContainerStarted","Data":"7014485f60d6977f1e77737879c1b24547bb99d19a05be8a9c6f969b76e5dfda"}
Nov 02 20:06:57.727398 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:57.727368    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-88zmb" event={"ID":"fd8684fd-7158-4fa0-8a30-911e86bd53e4","Type":"ContainerStarted","Data":"25d55892af830f3b5dc9df658faea7bd659b48bc46b4e39696e9e1d6655c9c6c"}
Nov 02 20:06:57.737144 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:57.737112    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-88zmb"
Nov 02 20:06:57.832708 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:57.832646    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-rjhcs" podStartSLOduration=1.832628631 podStartE2EDuration="1.832628631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:06:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:06:57.830993842 +0000 UTC m=+2324.438027611" watchObservedRunningTime="2026-11-02 20:06:57.832628631 +0000 UTC m=+2324.439662401"
Nov 02 20:06:57.944044 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:57.943975    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-88zmb" podStartSLOduration=1.94396177 podStartE2EDuration="1.94396177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:06:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:06:57.939249416 +0000 UTC m=+2324.546283185" watchObservedRunningTime="2026-11-02 20:06:57.94396177 +0000 UTC m=+2324.550995540"
Nov 02 20:06:58.691366 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:58.691329    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-rjhcs"
Nov 02 20:06:58.700198 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:58.700163    4463 patch_prober.go:29] interesting pod/e2e-worker-w4-8795bf-openshift-default-c67bc566c-rjhcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Nov 02 20:06:58.700903 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:58.700870    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-rjhcs" podUID="dd10e27d-811c-4488-af13-fa8c3272b54a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Nov 02 20:06:58.734589 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:58.734560    4463 patch_prober.go:29] interesting pod/e2e-worker-w4-8795bf-openshift-default-c67bc566c-88zmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Nov 02 20:06:58.734934 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:58.734604    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-88zmb" podUID="fd8684fd-7158-4fa0-8a30-911e86bd53e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Nov 02 20:06:59.692569 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:59.692530    4463 patch_prober.go:29] interesting pod/e2e-worker-w4-8795bf-openshift-default-c67bc566c-rjhcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Nov 02 20:06:59.692848 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:59.692817    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-rjhcs" podUID="dd10e27d-811c-4488-af13-fa8c3272b54a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Nov 02 20:06:59.732849 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:59.732822    4463 patch_prober.go:29] interesting pod/e2e-worker-w4-8795bf-openshift-default-c67bc566c-88zmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Nov 02 20:06:59.733137 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:06:59.733105    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-88zmb" podUID="fd8684fd-7158-4fa0-8a30-911e86bd53e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Nov 02 20:07:00.688411 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:00.688370    4463 patch_prober.go:29] interesting pod/e2e-worker-w4-8795bf-openshift-default-c67bc566c-rjhcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Nov 02 20:07:00.689163 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:00.689127    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-rjhcs" podUID="dd10e27d-811c-4488-af13-fa8c3272b54a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Nov 02 20:07:00.733595 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:00.733560    4463 patch_prober.go:29] interesting pod/e2e-worker-w4-8795bf-openshift-default-c67bc566c-88zmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Nov 02 20:07:00.733737 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:00.733606    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-88zmb" podUID="fd8684fd-7158-4fa0-8a30-911e86bd53e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Nov 02 20:07:01.331385 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:01.331343    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-28623b-5b7449b6b4-f4ddr"]
Nov 02 20:07:01.331701 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:01.331649    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-28623b-5b7449b6b4-f4ddr" podUID="0fdc6d2c-45e4-48be-bf24-93ed75441397" containerName="maas-api" containerID="cri-o://633e6c13ae459cea676d4e97b12e32b857493ccce177d3c765883662a9e77b8d" gracePeriod=30
Nov 02 20:07:01.734297 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:01.728681    4463 patch_prober.go:29] interesting pod/e2e-worker-w4-8795bf-openshift-default-c67bc566c-rjhcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Nov 02 20:07:01.734297 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:01.728740    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-rjhcs" podUID="dd10e27d-811c-4488-af13-fa8c3272b54a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Nov 02 20:07:01.735462 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:01.735302    4463 patch_prober.go:29] interesting pod/e2e-worker-w4-8795bf-openshift-default-c67bc566c-88zmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Nov 02 20:07:01.735462 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:01.735360    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-88zmb" podUID="fd8684fd-7158-4fa0-8a30-911e86bd53e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Nov 02 20:07:01.803814 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:01.803776    4463 generic.go:356] "Generic (PLEG): container finished" podID="0fdc6d2c-45e4-48be-bf24-93ed75441397" containerID="633e6c13ae459cea676d4e97b12e32b857493ccce177d3c765883662a9e77b8d" exitCode=0
Nov 02 20:07:01.804216 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:01.804190    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-28623b-5b7449b6b4-f4ddr" event={"ID":"0fdc6d2c-45e4-48be-bf24-93ed75441397","Type":"ContainerDied","Data":"633e6c13ae459cea676d4e97b12e32b857493ccce177d3c765883662a9e77b8d"}
Nov 02 20:07:01.920982 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:01.920906    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-28623b-5b7449b6b4-f4ddr"
Nov 02 20:07:02.034411 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:02.034379    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fdc6d2c-45e4-48be-bf24-93ed75441397-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fdc6d2c-45e4-48be-bf24-93ed75441397-metrics-certs\") pod \"0fdc6d2c-45e4-48be-bf24-93ed75441397\" (UID: \"0fdc6d2c-45e4-48be-bf24-93ed75441397\") "
Nov 02 20:07:02.034774 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:02.034753    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fdc6d2c-45e4-48be-bf24-93ed75441397-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0fdc6d2c-45e4-48be-bf24-93ed75441397-maas-api-tls\") pod \"0fdc6d2c-45e4-48be-bf24-93ed75441397\" (UID: \"0fdc6d2c-45e4-48be-bf24-93ed75441397\") "
Nov 02 20:07:02.034919 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:02.034905    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fdc6d2c-45e4-48be-bf24-93ed75441397-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fdc6d2c-45e4-48be-bf24-93ed75441397-openshift-service-ca\") pod \"0fdc6d2c-45e4-48be-bf24-93ed75441397\" (UID: \"0fdc6d2c-45e4-48be-bf24-93ed75441397\") "
Nov 02 20:07:02.035143 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:02.035125    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fdc6d2c-45e4-48be-bf24-93ed75441397-kube-api-access-xrtdq\" (UniqueName: \"kubernetes.io/projected/0fdc6d2c-45e4-48be-bf24-93ed75441397-kube-api-access-xrtdq\") pod \"0fdc6d2c-45e4-48be-bf24-93ed75441397\" (UID: \"0fdc6d2c-45e4-48be-bf24-93ed75441397\") "
Nov 02 20:07:02.036453 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:02.036417    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fdc6d2c-45e4-48be-bf24-93ed75441397-openshift-service-ca" pod "0fdc6d2c-45e4-48be-bf24-93ed75441397" (UID: "0fdc6d2c-45e4-48be-bf24-93ed75441397"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:07:02.038034 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:02.037988    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fdc6d2c-45e4-48be-bf24-93ed75441397-maas-api-tls" pod "0fdc6d2c-45e4-48be-bf24-93ed75441397" (UID: "0fdc6d2c-45e4-48be-bf24-93ed75441397"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:07:02.038977 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:02.038623    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fdc6d2c-45e4-48be-bf24-93ed75441397-kube-api-access-xrtdq" pod "0fdc6d2c-45e4-48be-bf24-93ed75441397" (UID: "0fdc6d2c-45e4-48be-bf24-93ed75441397"). InnerVolumeSpecName "kube-api-access-xrtdq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:07:02.040674 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:02.040642    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fdc6d2c-45e4-48be-bf24-93ed75441397-metrics-certs" pod "0fdc6d2c-45e4-48be-bf24-93ed75441397" (UID: "0fdc6d2c-45e4-48be-bf24-93ed75441397"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:07:02.137294 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:02.137259    4463 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fdc6d2c-45e4-48be-bf24-93ed75441397-metrics-certs\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:07:02.137294 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:02.137288    4463 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0fdc6d2c-45e4-48be-bf24-93ed75441397-maas-api-tls\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:07:02.137485 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:02.137299    4463 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fdc6d2c-45e4-48be-bf24-93ed75441397-openshift-service-ca\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:07:02.137485 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:02.137311    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrtdq\" (UniqueName: \"kubernetes.io/projected/0fdc6d2c-45e4-48be-bf24-93ed75441397-kube-api-access-xrtdq\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:07:02.693622 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:02.693581    4463 patch_prober.go:29] interesting pod/e2e-worker-w4-8795bf-openshift-default-c67bc566c-rjhcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Nov 02 20:07:02.694051 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:02.694000    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-rjhcs" podUID="dd10e27d-811c-4488-af13-fa8c3272b54a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Nov 02 20:07:02.733214 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:02.733179    4463 patch_prober.go:29] interesting pod/e2e-worker-w4-8795bf-openshift-default-c67bc566c-88zmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Nov 02 20:07:02.733332 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:02.733230    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-88zmb" podUID="fd8684fd-7158-4fa0-8a30-911e86bd53e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Nov 02 20:07:02.812174 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:02.812146    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-28623b-5b7449b6b4-f4ddr"
Nov 02 20:07:02.812619 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:02.812593    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-28623b-5b7449b6b4-f4ddr" event={"ID":"0fdc6d2c-45e4-48be-bf24-93ed75441397","Type":"ContainerDied","Data":"fb3aa1e0ab7e4cc49b5305bf96dbdf3b05076cf788eb821c85b4bfc70935b29d"}
Nov 02 20:07:02.812746 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:02.812736    4463 scope.go:122] "RemoveContainer" containerID="633e6c13ae459cea676d4e97b12e32b857493ccce177d3c765883662a9e77b8d"
Nov 02 20:07:02.999942 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:02.999853    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-28623b-5b7449b6b4-f4ddr"]
Nov 02 20:07:03.003960 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:03.003928    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-28623b-5b7449b6b4-f4ddr"]
Nov 02 20:07:03.588459 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:03.588429    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fdc6d2c-45e4-48be-bf24-93ed75441397" path="/var/lib/kubelet/pods/0fdc6d2c-45e4-48be-bf24-93ed75441397/volumes"
Nov 02 20:07:03.687962 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:03.687926    4463 patch_prober.go:29] interesting pod/e2e-worker-w4-8795bf-openshift-default-c67bc566c-rjhcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Nov 02 20:07:03.688151 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:03.687999    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-rjhcs" podUID="dd10e27d-811c-4488-af13-fa8c3272b54a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Nov 02 20:07:03.732772 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:03.732742    4463 patch_prober.go:29] interesting pod/e2e-worker-w4-8795bf-openshift-default-c67bc566c-88zmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Nov 02 20:07:03.732922 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:03.732818    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-88zmb" podUID="fd8684fd-7158-4fa0-8a30-911e86bd53e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Nov 02 20:07:04.310814 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:07:04.310770    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 20:07:04.693346 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:04.693319    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-rjhcs"
Nov 02 20:07:04.693772 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:04.693752    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-rjhcs"
Nov 02 20:07:04.694467 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:04.694444    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-rjhcs"
Nov 02 20:07:04.737743 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:04.737722    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-88zmb"
Nov 02 20:07:04.737982 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:04.737968    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-88zmb"
Nov 02 20:07:04.738524 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:04.738507    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-88zmb"
Nov 02 20:07:07.011840 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:07.011807    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-1d869e-7486c47697-4qtff"]
Nov 02 20:07:07.012268 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:07.011983    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-1d869e-7486c47697-4qtff" podUID="fe88713d-8bbb-4ef3-b106-476db242f9a5" containerName="maas-api" containerID="cri-o://4934e4982e55082b3ebd053d7a9df07a1674225c4e2cc897ea109ff10dc0cef2" gracePeriod=30
Nov 02 20:07:07.292042 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:07.292005    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-1d869e-7486c47697-4qtff"
Nov 02 20:07:07.423946 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:07.423908    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe88713d-8bbb-4ef3-b106-476db242f9a5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe88713d-8bbb-4ef3-b106-476db242f9a5-openshift-service-ca\") pod \"fe88713d-8bbb-4ef3-b106-476db242f9a5\" (UID: \"fe88713d-8bbb-4ef3-b106-476db242f9a5\") "
Nov 02 20:07:07.424205 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:07.424182    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe88713d-8bbb-4ef3-b106-476db242f9a5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe88713d-8bbb-4ef3-b106-476db242f9a5-metrics-certs\") pod \"fe88713d-8bbb-4ef3-b106-476db242f9a5\" (UID: \"fe88713d-8bbb-4ef3-b106-476db242f9a5\") "
Nov 02 20:07:07.424409 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:07.424390    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe88713d-8bbb-4ef3-b106-476db242f9a5-kube-api-access-dvnmc\" (UniqueName: \"kubernetes.io/projected/fe88713d-8bbb-4ef3-b106-476db242f9a5-kube-api-access-dvnmc\") pod \"fe88713d-8bbb-4ef3-b106-476db242f9a5\" (UID: \"fe88713d-8bbb-4ef3-b106-476db242f9a5\") "
Nov 02 20:07:07.424755 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:07.424739    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe88713d-8bbb-4ef3-b106-476db242f9a5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe88713d-8bbb-4ef3-b106-476db242f9a5-maas-api-tls\") pod \"fe88713d-8bbb-4ef3-b106-476db242f9a5\" (UID: \"fe88713d-8bbb-4ef3-b106-476db242f9a5\") "
Nov 02 20:07:07.430366 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:07.430334    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe88713d-8bbb-4ef3-b106-476db242f9a5-openshift-service-ca" pod "fe88713d-8bbb-4ef3-b106-476db242f9a5" (UID: "fe88713d-8bbb-4ef3-b106-476db242f9a5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:07:07.453487 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:07.452699    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe88713d-8bbb-4ef3-b106-476db242f9a5-metrics-certs" pod "fe88713d-8bbb-4ef3-b106-476db242f9a5" (UID: "fe88713d-8bbb-4ef3-b106-476db242f9a5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:07:07.472534 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:07.472500    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe88713d-8bbb-4ef3-b106-476db242f9a5-kube-api-access-dvnmc" pod "fe88713d-8bbb-4ef3-b106-476db242f9a5" (UID: "fe88713d-8bbb-4ef3-b106-476db242f9a5"). InnerVolumeSpecName "kube-api-access-dvnmc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:07:07.473626 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:07.473601    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe88713d-8bbb-4ef3-b106-476db242f9a5-maas-api-tls" pod "fe88713d-8bbb-4ef3-b106-476db242f9a5" (UID: "fe88713d-8bbb-4ef3-b106-476db242f9a5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:07:07.526730 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:07.526671    4463 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe88713d-8bbb-4ef3-b106-476db242f9a5-maas-api-tls\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:07:07.526730 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:07.526695    4463 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe88713d-8bbb-4ef3-b106-476db242f9a5-openshift-service-ca\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:07:07.526730 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:07.526707    4463 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe88713d-8bbb-4ef3-b106-476db242f9a5-metrics-certs\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:07:07.526730 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:07.526716    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvnmc\" (UniqueName: \"kubernetes.io/projected/fe88713d-8bbb-4ef3-b106-476db242f9a5-kube-api-access-dvnmc\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:07:07.845505 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:07.845428    4463 generic.go:356] "Generic (PLEG): container finished" podID="fe88713d-8bbb-4ef3-b106-476db242f9a5" containerID="4934e4982e55082b3ebd053d7a9df07a1674225c4e2cc897ea109ff10dc0cef2" exitCode=0
Nov 02 20:07:07.845505 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:07.845479    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-1d869e-7486c47697-4qtff"
Nov 02 20:07:07.845714 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:07.845514    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-1d869e-7486c47697-4qtff" event={"ID":"fe88713d-8bbb-4ef3-b106-476db242f9a5","Type":"ContainerDied","Data":"4934e4982e55082b3ebd053d7a9df07a1674225c4e2cc897ea109ff10dc0cef2"}
Nov 02 20:07:07.845714 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:07.845568    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-1d869e-7486c47697-4qtff" event={"ID":"fe88713d-8bbb-4ef3-b106-476db242f9a5","Type":"ContainerDied","Data":"6d7b35f3ed2765d7924b6202a63a92178aafc7bd01dc2199ab0ac4fe869f4238"}
Nov 02 20:07:07.845714 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:07.845596    4463 scope.go:122] "RemoveContainer" containerID="4934e4982e55082b3ebd053d7a9df07a1674225c4e2cc897ea109ff10dc0cef2"
Nov 02 20:07:07.855473 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:07.855455    4463 scope.go:122] "RemoveContainer" containerID="4934e4982e55082b3ebd053d7a9df07a1674225c4e2cc897ea109ff10dc0cef2"
Nov 02 20:07:07.855731 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:07:07.855711    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4934e4982e55082b3ebd053d7a9df07a1674225c4e2cc897ea109ff10dc0cef2\": container with ID starting with 4934e4982e55082b3ebd053d7a9df07a1674225c4e2cc897ea109ff10dc0cef2 not found: ID does not exist" containerID="4934e4982e55082b3ebd053d7a9df07a1674225c4e2cc897ea109ff10dc0cef2"
Nov 02 20:07:07.855781 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:07.855735    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4934e4982e55082b3ebd053d7a9df07a1674225c4e2cc897ea109ff10dc0cef2"} err="failed to get container status \"4934e4982e55082b3ebd053d7a9df07a1674225c4e2cc897ea109ff10dc0cef2\": rpc error: code = NotFound desc = could not find container \"4934e4982e55082b3ebd053d7a9df07a1674225c4e2cc897ea109ff10dc0cef2\": container with ID starting with 4934e4982e55082b3ebd053d7a9df07a1674225c4e2cc897ea109ff10dc0cef2 not found: ID does not exist"
Nov 02 20:07:08.002811 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:08.002765    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-1d869e-7486c47697-4qtff"]
Nov 02 20:07:08.008503 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:08.008462    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-1d869e-7486c47697-4qtff"]
Nov 02 20:07:09.586304 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:09.586270    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe88713d-8bbb-4ef3-b106-476db242f9a5" path="/var/lib/kubelet/pods/fe88713d-8bbb-4ef3-b106-476db242f9a5/volumes"
Nov 02 20:07:09.895375 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:09.895296    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-74ft9"]
Nov 02 20:07:09.895539 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:09.895500    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-74ft9" podUID="1cb6e664-e3c4-498a-978c-6a65ae7ffad9" containerName="istio-proxy" containerID="cri-o://4876240eca1c749c2da9ad82e84f0ef726287db80bd656f4a9aa676cfce5de52" gracePeriod=30
Nov 02 20:07:11.491424 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:11.491384    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-mhd5c"]
Nov 02 20:07:11.492268 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:11.492246    4463 cpu_manager.go:414] "containerMap: removing container" podUID="0fdc6d2c-45e4-48be-bf24-93ed75441397" containerName="maas-api"
Nov 02 20:07:11.492268 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:11.492267    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fdc6d2c-45e4-48be-bf24-93ed75441397" containerName="maas-api"
Nov 02 20:07:11.492453 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:11.492303    4463 cpu_manager.go:414] "containerMap: removing container" podUID="fe88713d-8bbb-4ef3-b106-476db242f9a5" containerName="maas-api"
Nov 02 20:07:11.492453 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:11.492311    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe88713d-8bbb-4ef3-b106-476db242f9a5" containerName="maas-api"
Nov 02 20:07:11.492558 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:11.492456    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fdc6d2c-45e4-48be-bf24-93ed75441397" containerName="maas-api"
Nov 02 20:07:11.492558 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:11.492472    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe88713d-8bbb-4ef3-b106-476db242f9a5" containerName="maas-api"
Nov 02 20:07:11.495977 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:11.495957    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-mhd5c"
Nov 02 20:07:11.569501 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:11.569472    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/721ba0b9-7dec-46b8-9578-f5bc796ff129-credential-socket\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-mhd5c\" (UID: \"721ba0b9-7dec-46b8-9578-f5bc796ff129\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-mhd5c"
Nov 02 20:07:11.569628 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:11.569507    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/721ba0b9-7dec-46b8-9578-f5bc796ff129-istio-token\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-mhd5c\" (UID: \"721ba0b9-7dec-46b8-9578-f5bc796ff129\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-mhd5c"
Nov 02 20:07:11.569628 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:11.569537    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/721ba0b9-7dec-46b8-9578-f5bc796ff129-istio-data\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-mhd5c\" (UID: \"721ba0b9-7dec-46b8-9578-f5bc796ff129\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-mhd5c"
Nov 02 20:07:11.569708 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:11.569643    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/721ba0b9-7dec-46b8-9578-f5bc796ff129-workload-socket\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-mhd5c\" (UID: \"721ba0b9-7dec-46b8-9578-f5bc796ff129\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-mhd5c"
Nov 02 20:07:11.569828 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:11.569808    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6d84\" (UniqueName: \"kubernetes.io/projected/721ba0b9-7dec-46b8-9578-f5bc796ff129-kube-api-access-l6d84\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-mhd5c\" (UID: \"721ba0b9-7dec-46b8-9578-f5bc796ff129\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-mhd5c"
Nov 02 20:07:11.569863 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:11.569843    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/721ba0b9-7dec-46b8-9578-f5bc796ff129-istiod-ca-cert\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-mhd5c\" (UID: \"721ba0b9-7dec-46b8-9578-f5bc796ff129\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-mhd5c"
Nov 02 20:07:11.569900 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:11.569874    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/721ba0b9-7dec-46b8-9578-f5bc796ff129-workload-certs\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-mhd5c\" (UID: \"721ba0b9-7dec-46b8-9578-f5bc796ff129\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-mhd5c"
Nov 02 20:07:11.570038 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:11.569991    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/721ba0b9-7dec-46b8-9578-f5bc796ff129-istio-envoy\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-mhd5c\" (UID: \"721ba0b9-7dec-46b8-9578-f5bc796ff129\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-mhd5c"
Nov 02 20:07:11.570079 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:11.570049    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/721ba0b9-7dec-46b8-9578-f5bc796ff129-istio-podinfo\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-mhd5c\" (UID: \"721ba0b9-7dec-46b8-9578-f5bc796ff129\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-mhd5c"
Nov 02 20:07:11.588175 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:11.588150    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-mhd5c"]
Nov 02 20:07:11.671086 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:11.671045    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/721ba0b9-7dec-46b8-9578-f5bc796ff129-istio-envoy\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-mhd5c\" (UID: \"721ba0b9-7dec-46b8-9578-f5bc796ff129\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-mhd5c"
Nov 02 20:07:11.671086 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:11.671085    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/721ba0b9-7dec-46b8-9578-f5bc796ff129-istio-podinfo\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-mhd5c\" (UID: \"721ba0b9-7dec-46b8-9578-f5bc796ff129\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-mhd5c"
Nov 02 20:07:11.671290 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:11.671121    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/721ba0b9-7dec-46b8-9578-f5bc796ff129-credential-socket\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-mhd5c\" (UID: \"721ba0b9-7dec-46b8-9578-f5bc796ff129\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-mhd5c"
Nov 02 20:07:11.671290 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:11.671141    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/721ba0b9-7dec-46b8-9578-f5bc796ff129-istio-token\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-mhd5c\" (UID: \"721ba0b9-7dec-46b8-9578-f5bc796ff129\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-mhd5c"
Nov 02 20:07:11.671290 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:11.671173    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/721ba0b9-7dec-46b8-9578-f5bc796ff129-istio-data\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-mhd5c\" (UID: \"721ba0b9-7dec-46b8-9578-f5bc796ff129\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-mhd5c"
Nov 02 20:07:11.671290 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:11.671207    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/721ba0b9-7dec-46b8-9578-f5bc796ff129-workload-socket\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-mhd5c\" (UID: \"721ba0b9-7dec-46b8-9578-f5bc796ff129\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-mhd5c"
Nov 02 20:07:11.671523 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:11.671428    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6d84\" (UniqueName: \"kubernetes.io/projected/721ba0b9-7dec-46b8-9578-f5bc796ff129-kube-api-access-l6d84\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-mhd5c\" (UID: \"721ba0b9-7dec-46b8-9578-f5bc796ff129\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-mhd5c"
Nov 02 20:07:11.671523 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:11.671478    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/721ba0b9-7dec-46b8-9578-f5bc796ff129-istiod-ca-cert\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-mhd5c\" (UID: \"721ba0b9-7dec-46b8-9578-f5bc796ff129\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-mhd5c"
Nov 02 20:07:11.671626 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:11.671528    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/721ba0b9-7dec-46b8-9578-f5bc796ff129-workload-certs\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-mhd5c\" (UID: \"721ba0b9-7dec-46b8-9578-f5bc796ff129\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-mhd5c"
Nov 02 20:07:11.671626 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:11.671537    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/721ba0b9-7dec-46b8-9578-f5bc796ff129-credential-socket\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-mhd5c\" (UID: \"721ba0b9-7dec-46b8-9578-f5bc796ff129\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-mhd5c"
Nov 02 20:07:11.671626 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:11.671566    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/721ba0b9-7dec-46b8-9578-f5bc796ff129-istio-data\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-mhd5c\" (UID: \"721ba0b9-7dec-46b8-9578-f5bc796ff129\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-mhd5c"
Nov 02 20:07:11.671727 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:11.671621    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/721ba0b9-7dec-46b8-9578-f5bc796ff129-workload-socket\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-mhd5c\" (UID: \"721ba0b9-7dec-46b8-9578-f5bc796ff129\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-mhd5c"
Nov 02 20:07:11.671899 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:11.671880    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/721ba0b9-7dec-46b8-9578-f5bc796ff129-workload-certs\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-mhd5c\" (UID: \"721ba0b9-7dec-46b8-9578-f5bc796ff129\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-mhd5c"
Nov 02 20:07:11.672150 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:11.672134    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/721ba0b9-7dec-46b8-9578-f5bc796ff129-istiod-ca-cert\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-mhd5c\" (UID: \"721ba0b9-7dec-46b8-9578-f5bc796ff129\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-mhd5c"
Nov 02 20:07:11.673893 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:11.673868    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/721ba0b9-7dec-46b8-9578-f5bc796ff129-istio-envoy\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-mhd5c\" (UID: \"721ba0b9-7dec-46b8-9578-f5bc796ff129\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-mhd5c"
Nov 02 20:07:11.674138 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:11.674116    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/721ba0b9-7dec-46b8-9578-f5bc796ff129-istio-podinfo\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-mhd5c\" (UID: \"721ba0b9-7dec-46b8-9578-f5bc796ff129\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-mhd5c"
Nov 02 20:07:11.723964 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:11.723940    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/721ba0b9-7dec-46b8-9578-f5bc796ff129-istio-token\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-mhd5c\" (UID: \"721ba0b9-7dec-46b8-9578-f5bc796ff129\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-mhd5c"
Nov 02 20:07:11.724116 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:11.724097    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6d84\" (UniqueName: \"kubernetes.io/projected/721ba0b9-7dec-46b8-9578-f5bc796ff129-kube-api-access-l6d84\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-mhd5c\" (UID: \"721ba0b9-7dec-46b8-9578-f5bc796ff129\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-mhd5c"
Nov 02 20:07:11.810250 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:11.810185    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-mhd5c"
Nov 02 20:07:12.227320 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:12.227289    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-mhd5c"]
Nov 02 20:07:12.228246 ip-10-0-133-9 kubenswrapper[4463]: W1102 20:07:12.228222    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod721ba0b9_7dec_46b8_9578_f5bc796ff129.slice/crio-354019136a29c5bd8915020eb74bc387aaa5e0527210e732ec5db26b0e49e545 WatchSource:0}: Error finding container 354019136a29c5bd8915020eb74bc387aaa5e0527210e732ec5db26b0e49e545: Status 404 returned error can't find the container with id 354019136a29c5bd8915020eb74bc387aaa5e0527210e732ec5db26b0e49e545
Nov 02 20:07:12.231026 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:12.230986    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:07:12.231108 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:12.231063    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:07:12.231108 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:12.231093    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:07:12.549513 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:12.549438    4463 patch_prober.go:29] interesting pod/e2e-worker-w2-28623b-openshift-default-5b66f6f775-74ft9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 20:07:12.549513 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:12.549494    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-74ft9" podUID="1cb6e664-e3c4-498a-978c-6a65ae7ffad9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 20:07:12.893750 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:12.893648    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-mhd5c" event={"ID":"721ba0b9-7dec-46b8-9578-f5bc796ff129","Type":"ContainerStarted","Data":"cce9f1e224634b58e5d453a496f3db3d8d4a7f716e28dc267413214dfca5b2f2"}
Nov 02 20:07:12.893750 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:12.893734    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-mhd5c" event={"ID":"721ba0b9-7dec-46b8-9578-f5bc796ff129","Type":"ContainerStarted","Data":"354019136a29c5bd8915020eb74bc387aaa5e0527210e732ec5db26b0e49e545"}
Nov 02 20:07:13.003275 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:13.003222    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-mhd5c" podStartSLOduration=2.003208615 podStartE2EDuration="2.003208615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:07:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:07:13.00083553 +0000 UTC m=+2339.607869322" watchObservedRunningTime="2026-11-02 20:07:13.003208615 +0000 UTC m=+2339.610242385"
Nov 02 20:07:13.811386 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:13.811351    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-mhd5c"
Nov 02 20:07:13.812960 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:13.812934    4463 patch_prober.go:29] interesting pod/e2e-worker-w4-8795bf-openshift-default-c67bc566c-mhd5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Nov 02 20:07:13.813101 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:13.812982    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-mhd5c" podUID="721ba0b9-7dec-46b8-9578-f5bc796ff129" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Nov 02 20:07:14.810465 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:14.810433    4463 patch_prober.go:29] interesting pod/e2e-worker-w4-8795bf-openshift-default-c67bc566c-mhd5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Nov 02 20:07:14.810640 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:14.810497    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-mhd5c" podUID="721ba0b9-7dec-46b8-9578-f5bc796ff129" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Nov 02 20:07:15.208524 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:15.208498    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-74ft9"
Nov 02 20:07:15.322882 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:15.322845    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1cb6e664-e3c4-498a-978c-6a65ae7ffad9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cb6e664-e3c4-498a-978c-6a65ae7ffad9-istiod-ca-cert\") pod \"1cb6e664-e3c4-498a-978c-6a65ae7ffad9\" (UID: \"1cb6e664-e3c4-498a-978c-6a65ae7ffad9\") "
Nov 02 20:07:15.323095 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:15.322896    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cb6e664-e3c4-498a-978c-6a65ae7ffad9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cb6e664-e3c4-498a-978c-6a65ae7ffad9-istio-data\") pod \"1cb6e664-e3c4-498a-978c-6a65ae7ffad9\" (UID: \"1cb6e664-e3c4-498a-978c-6a65ae7ffad9\") "
Nov 02 20:07:15.323095 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:15.323059    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cb6e664-e3c4-498a-978c-6a65ae7ffad9-istio-token\" (UniqueName: \"kubernetes.io/projected/1cb6e664-e3c4-498a-978c-6a65ae7ffad9-istio-token\") pod \"1cb6e664-e3c4-498a-978c-6a65ae7ffad9\" (UID: \"1cb6e664-e3c4-498a-978c-6a65ae7ffad9\") "
Nov 02 20:07:15.323218 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:15.323095    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cb6e664-e3c4-498a-978c-6a65ae7ffad9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cb6e664-e3c4-498a-978c-6a65ae7ffad9-credential-socket\") pod \"1cb6e664-e3c4-498a-978c-6a65ae7ffad9\" (UID: \"1cb6e664-e3c4-498a-978c-6a65ae7ffad9\") "
Nov 02 20:07:15.323218 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:15.323154    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cb6e664-e3c4-498a-978c-6a65ae7ffad9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cb6e664-e3c4-498a-978c-6a65ae7ffad9-istio-envoy\") pod \"1cb6e664-e3c4-498a-978c-6a65ae7ffad9\" (UID: \"1cb6e664-e3c4-498a-978c-6a65ae7ffad9\") "
Nov 02 20:07:15.323218 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:15.323193    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cb6e664-e3c4-498a-978c-6a65ae7ffad9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cb6e664-e3c4-498a-978c-6a65ae7ffad9-workload-certs\") pod \"1cb6e664-e3c4-498a-978c-6a65ae7ffad9\" (UID: \"1cb6e664-e3c4-498a-978c-6a65ae7ffad9\") "
Nov 02 20:07:15.323378 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:15.323205    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cb6e664-e3c4-498a-978c-6a65ae7ffad9-istio-data" pod "1cb6e664-e3c4-498a-978c-6a65ae7ffad9" (UID: "1cb6e664-e3c4-498a-978c-6a65ae7ffad9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:07:15.323378 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:15.323209    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cb6e664-e3c4-498a-978c-6a65ae7ffad9-istiod-ca-cert" pod "1cb6e664-e3c4-498a-978c-6a65ae7ffad9" (UID: "1cb6e664-e3c4-498a-978c-6a65ae7ffad9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:07:15.323378 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:15.323216    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cb6e664-e3c4-498a-978c-6a65ae7ffad9-kube-api-access-tr26s\" (UniqueName: \"kubernetes.io/projected/1cb6e664-e3c4-498a-978c-6a65ae7ffad9-kube-api-access-tr26s\") pod \"1cb6e664-e3c4-498a-978c-6a65ae7ffad9\" (UID: \"1cb6e664-e3c4-498a-978c-6a65ae7ffad9\") "
Nov 02 20:07:15.323378 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:15.323290    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1cb6e664-e3c4-498a-978c-6a65ae7ffad9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cb6e664-e3c4-498a-978c-6a65ae7ffad9-istio-podinfo\") pod \"1cb6e664-e3c4-498a-978c-6a65ae7ffad9\" (UID: \"1cb6e664-e3c4-498a-978c-6a65ae7ffad9\") "
Nov 02 20:07:15.323378 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:15.323348    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cb6e664-e3c4-498a-978c-6a65ae7ffad9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cb6e664-e3c4-498a-978c-6a65ae7ffad9-workload-socket\") pod \"1cb6e664-e3c4-498a-978c-6a65ae7ffad9\" (UID: \"1cb6e664-e3c4-498a-978c-6a65ae7ffad9\") "
Nov 02 20:07:15.323638 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:15.323401    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cb6e664-e3c4-498a-978c-6a65ae7ffad9-workload-certs" pod "1cb6e664-e3c4-498a-978c-6a65ae7ffad9" (UID: "1cb6e664-e3c4-498a-978c-6a65ae7ffad9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:07:15.323638 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:15.323433    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cb6e664-e3c4-498a-978c-6a65ae7ffad9-credential-socket" pod "1cb6e664-e3c4-498a-978c-6a65ae7ffad9" (UID: "1cb6e664-e3c4-498a-978c-6a65ae7ffad9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:07:15.323638 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:15.323580    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cb6e664-e3c4-498a-978c-6a65ae7ffad9-workload-socket" pod "1cb6e664-e3c4-498a-978c-6a65ae7ffad9" (UID: "1cb6e664-e3c4-498a-978c-6a65ae7ffad9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:07:15.323916 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:15.323898    4463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cb6e664-e3c4-498a-978c-6a65ae7ffad9-credential-socket\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:07:15.324001 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:15.323920    4463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cb6e664-e3c4-498a-978c-6a65ae7ffad9-workload-certs\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:07:15.324001 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:15.323935    4463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cb6e664-e3c4-498a-978c-6a65ae7ffad9-workload-socket\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:07:15.324001 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:15.323948    4463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cb6e664-e3c4-498a-978c-6a65ae7ffad9-istiod-ca-cert\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:07:15.324001 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:15.323960    4463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cb6e664-e3c4-498a-978c-6a65ae7ffad9-istio-data\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:07:15.325425 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:15.325397    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cb6e664-e3c4-498a-978c-6a65ae7ffad9-istio-envoy" pod "1cb6e664-e3c4-498a-978c-6a65ae7ffad9" (UID: "1cb6e664-e3c4-498a-978c-6a65ae7ffad9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:07:15.325425 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:15.325412    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cb6e664-e3c4-498a-978c-6a65ae7ffad9-istio-token" pod "1cb6e664-e3c4-498a-978c-6a65ae7ffad9" (UID: "1cb6e664-e3c4-498a-978c-6a65ae7ffad9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:07:15.325793 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:15.325769    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1cb6e664-e3c4-498a-978c-6a65ae7ffad9-istio-podinfo" pod "1cb6e664-e3c4-498a-978c-6a65ae7ffad9" (UID: "1cb6e664-e3c4-498a-978c-6a65ae7ffad9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:07:15.325793 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:15.325778    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cb6e664-e3c4-498a-978c-6a65ae7ffad9-kube-api-access-tr26s" pod "1cb6e664-e3c4-498a-978c-6a65ae7ffad9" (UID: "1cb6e664-e3c4-498a-978c-6a65ae7ffad9"). InnerVolumeSpecName "kube-api-access-tr26s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:07:15.425068 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:15.425033    4463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cb6e664-e3c4-498a-978c-6a65ae7ffad9-istio-token\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:07:15.425068 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:15.425062    4463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cb6e664-e3c4-498a-978c-6a65ae7ffad9-istio-envoy\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:07:15.425068 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:15.425074    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tr26s\" (UniqueName: \"kubernetes.io/projected/1cb6e664-e3c4-498a-978c-6a65ae7ffad9-kube-api-access-tr26s\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:07:15.425298 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:15.425082    4463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cb6e664-e3c4-498a-978c-6a65ae7ffad9-istio-podinfo\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:07:15.810450 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:15.810411    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-94hmq"]
Nov 02 20:07:15.810664 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:15.810608    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-94hmq" podUID="32d7da41-73b0-4ae5-a8b3-69bcb021cf4a" containerName="istio-proxy" containerID="cri-o://79a9f1a468225211f945377c7c63e9d2edb050c28f2cac2924d942cbba60f0a8" gracePeriod=30
Nov 02 20:07:15.810905 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:15.810875    4463 patch_prober.go:29] interesting pod/e2e-worker-w4-8795bf-openshift-default-c67bc566c-mhd5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Nov 02 20:07:15.811008 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:15.810925    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-mhd5c" podUID="721ba0b9-7dec-46b8-9578-f5bc796ff129" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Nov 02 20:07:15.916195 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:15.916118    4463 generic.go:356] "Generic (PLEG): container finished" podID="1cb6e664-e3c4-498a-978c-6a65ae7ffad9" containerID="4876240eca1c749c2da9ad82e84f0ef726287db80bd656f4a9aa676cfce5de52" exitCode=0
Nov 02 20:07:15.916684 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:15.916450    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-74ft9"
Nov 02 20:07:15.917377 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:15.916473    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-74ft9" event={"ID":"1cb6e664-e3c4-498a-978c-6a65ae7ffad9","Type":"ContainerDied","Data":"4876240eca1c749c2da9ad82e84f0ef726287db80bd656f4a9aa676cfce5de52"}
Nov 02 20:07:15.917524 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:15.917406    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-74ft9" event={"ID":"1cb6e664-e3c4-498a-978c-6a65ae7ffad9","Type":"ContainerDied","Data":"d558b5681f8218c8dfbb994c76d69966ef37b9ca72b71093d5d26cb5dc3d3253"}
Nov 02 20:07:15.917524 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:15.917442    4463 scope.go:122] "RemoveContainer" containerID="4876240eca1c749c2da9ad82e84f0ef726287db80bd656f4a9aa676cfce5de52"
Nov 02 20:07:15.931358 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:15.931334    4463 scope.go:122] "RemoveContainer" containerID="4876240eca1c749c2da9ad82e84f0ef726287db80bd656f4a9aa676cfce5de52"
Nov 02 20:07:15.931810 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:07:15.931788    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4876240eca1c749c2da9ad82e84f0ef726287db80bd656f4a9aa676cfce5de52\": container with ID starting with 4876240eca1c749c2da9ad82e84f0ef726287db80bd656f4a9aa676cfce5de52 not found: ID does not exist" containerID="4876240eca1c749c2da9ad82e84f0ef726287db80bd656f4a9aa676cfce5de52"
Nov 02 20:07:15.931960 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:15.931934    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4876240eca1c749c2da9ad82e84f0ef726287db80bd656f4a9aa676cfce5de52"} err="failed to get container status \"4876240eca1c749c2da9ad82e84f0ef726287db80bd656f4a9aa676cfce5de52\": rpc error: code = NotFound desc = could not find container \"4876240eca1c749c2da9ad82e84f0ef726287db80bd656f4a9aa676cfce5de52\": container with ID starting with 4876240eca1c749c2da9ad82e84f0ef726287db80bd656f4a9aa676cfce5de52 not found: ID does not exist"
Nov 02 20:07:16.042552 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:16.042512    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-74ft9"]
Nov 02 20:07:16.046490 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:16.046462    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-74ft9"]
Nov 02 20:07:16.810496 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:16.810460    4463 patch_prober.go:29] interesting pod/e2e-worker-w4-8795bf-openshift-default-c67bc566c-mhd5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Nov 02 20:07:16.810915 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:16.810514    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-mhd5c" podUID="721ba0b9-7dec-46b8-9578-f5bc796ff129" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Nov 02 20:07:17.587086 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:17.587051    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1cb6e664-e3c4-498a-978c-6a65ae7ffad9" path="/var/lib/kubelet/pods/1cb6e664-e3c4-498a-978c-6a65ae7ffad9/volumes"
Nov 02 20:07:17.811080 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:17.811047    4463 patch_prober.go:29] interesting pod/e2e-worker-w4-8795bf-openshift-default-c67bc566c-mhd5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Nov 02 20:07:17.811454 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:17.811109    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-mhd5c" podUID="721ba0b9-7dec-46b8-9578-f5bc796ff129" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Nov 02 20:07:18.194522 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:18.194484    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-28fa24-865f7b8866-9hsz9"]
Nov 02 20:07:18.195161 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:18.195144    4463 cpu_manager.go:414] "containerMap: removing container" podUID="1cb6e664-e3c4-498a-978c-6a65ae7ffad9" containerName="istio-proxy"
Nov 02 20:07:18.195215 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:18.195161    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cb6e664-e3c4-498a-978c-6a65ae7ffad9" containerName="istio-proxy"
Nov 02 20:07:18.195265 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:18.195257    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cb6e664-e3c4-498a-978c-6a65ae7ffad9" containerName="istio-proxy"
Nov 02 20:07:18.200336 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:18.200315    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-28fa24-865f7b8866-9hsz9"
Nov 02 20:07:18.248783 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:18.248753    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-28fa24-serving-cert\""
Nov 02 20:07:18.251066 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:18.251041    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-28fa24-865f7b8866-9hsz9"]
Nov 02 20:07:18.285696 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:18.285669    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b78a889a-2b5d-4ff0-a419-2eb7f7858579-openshift-service-ca\") pod \"maas-api-e2e-inf-b-28fa24-865f7b8866-9hsz9\" (UID: \"b78a889a-2b5d-4ff0-a419-2eb7f7858579\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-28fa24-865f7b8866-9hsz9"
Nov 02 20:07:18.285849 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:18.285740    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b78a889a-2b5d-4ff0-a419-2eb7f7858579-metrics-certs\") pod \"maas-api-e2e-inf-b-28fa24-865f7b8866-9hsz9\" (UID: \"b78a889a-2b5d-4ff0-a419-2eb7f7858579\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-28fa24-865f7b8866-9hsz9"
Nov 02 20:07:18.285959 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:18.285927    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99vvr\" (UniqueName: \"kubernetes.io/projected/b78a889a-2b5d-4ff0-a419-2eb7f7858579-kube-api-access-99vvr\") pod \"maas-api-e2e-inf-b-28fa24-865f7b8866-9hsz9\" (UID: \"b78a889a-2b5d-4ff0-a419-2eb7f7858579\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-28fa24-865f7b8866-9hsz9"
Nov 02 20:07:18.286057 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:18.285988    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b78a889a-2b5d-4ff0-a419-2eb7f7858579-maas-api-tls\") pod \"maas-api-e2e-inf-b-28fa24-865f7b8866-9hsz9\" (UID: \"b78a889a-2b5d-4ff0-a419-2eb7f7858579\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-28fa24-865f7b8866-9hsz9"
Nov 02 20:07:18.387024 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:18.386979    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b78a889a-2b5d-4ff0-a419-2eb7f7858579-openshift-service-ca\") pod \"maas-api-e2e-inf-b-28fa24-865f7b8866-9hsz9\" (UID: \"b78a889a-2b5d-4ff0-a419-2eb7f7858579\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-28fa24-865f7b8866-9hsz9"
Nov 02 20:07:18.387192 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:18.387084    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b78a889a-2b5d-4ff0-a419-2eb7f7858579-metrics-certs\") pod \"maas-api-e2e-inf-b-28fa24-865f7b8866-9hsz9\" (UID: \"b78a889a-2b5d-4ff0-a419-2eb7f7858579\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-28fa24-865f7b8866-9hsz9"
Nov 02 20:07:18.387192 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:18.387159    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99vvr\" (UniqueName: \"kubernetes.io/projected/b78a889a-2b5d-4ff0-a419-2eb7f7858579-kube-api-access-99vvr\") pod \"maas-api-e2e-inf-b-28fa24-865f7b8866-9hsz9\" (UID: \"b78a889a-2b5d-4ff0-a419-2eb7f7858579\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-28fa24-865f7b8866-9hsz9"
Nov 02 20:07:18.387300 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:18.387191    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b78a889a-2b5d-4ff0-a419-2eb7f7858579-maas-api-tls\") pod \"maas-api-e2e-inf-b-28fa24-865f7b8866-9hsz9\" (UID: \"b78a889a-2b5d-4ff0-a419-2eb7f7858579\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-28fa24-865f7b8866-9hsz9"
Nov 02 20:07:18.387379 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:07:18.387364    4463 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-28fa24-serving-cert: secret "maas-api-e2e-inf-b-28fa24-serving-cert" not found
Nov 02 20:07:18.387433 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:07:18.387423    4463 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b78a889a-2b5d-4ff0-a419-2eb7f7858579-maas-api-tls podName:b78a889a-2b5d-4ff0-a419-2eb7f7858579 nodeName:}" failed. No retries permitted until 2026-11-02 20:07:18.887402356 +0000 UTC m=+2345.494436104 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b78a889a-2b5d-4ff0-a419-2eb7f7858579-maas-api-tls") pod "maas-api-e2e-inf-b-28fa24-865f7b8866-9hsz9" (UID: "b78a889a-2b5d-4ff0-a419-2eb7f7858579") : secret "maas-api-e2e-inf-b-28fa24-serving-cert" not found
Nov 02 20:07:18.387744 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:18.387725    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b78a889a-2b5d-4ff0-a419-2eb7f7858579-openshift-service-ca\") pod \"maas-api-e2e-inf-b-28fa24-865f7b8866-9hsz9\" (UID: \"b78a889a-2b5d-4ff0-a419-2eb7f7858579\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-28fa24-865f7b8866-9hsz9"
Nov 02 20:07:18.389713 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:18.389688    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b78a889a-2b5d-4ff0-a419-2eb7f7858579-metrics-certs\") pod \"maas-api-e2e-inf-b-28fa24-865f7b8866-9hsz9\" (UID: \"b78a889a-2b5d-4ff0-a419-2eb7f7858579\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-28fa24-865f7b8866-9hsz9"
Nov 02 20:07:18.445290 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:18.445203    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99vvr\" (UniqueName: \"kubernetes.io/projected/b78a889a-2b5d-4ff0-a419-2eb7f7858579-kube-api-access-99vvr\") pod \"maas-api-e2e-inf-b-28fa24-865f7b8866-9hsz9\" (UID: \"b78a889a-2b5d-4ff0-a419-2eb7f7858579\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-28fa24-865f7b8866-9hsz9"
Nov 02 20:07:18.810559 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:18.810475    4463 patch_prober.go:29] interesting pod/e2e-worker-w4-8795bf-openshift-default-c67bc566c-mhd5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Nov 02 20:07:18.810559 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:18.810531    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-mhd5c" podUID="721ba0b9-7dec-46b8-9578-f5bc796ff129" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Nov 02 20:07:18.894401 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:18.894354    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b78a889a-2b5d-4ff0-a419-2eb7f7858579-maas-api-tls\") pod \"maas-api-e2e-inf-b-28fa24-865f7b8866-9hsz9\" (UID: \"b78a889a-2b5d-4ff0-a419-2eb7f7858579\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-28fa24-865f7b8866-9hsz9"
Nov 02 20:07:18.896721 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:18.896700    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b78a889a-2b5d-4ff0-a419-2eb7f7858579-maas-api-tls\") pod \"maas-api-e2e-inf-b-28fa24-865f7b8866-9hsz9\" (UID: \"b78a889a-2b5d-4ff0-a419-2eb7f7858579\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-28fa24-865f7b8866-9hsz9"
Nov 02 20:07:19.114132 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:19.114053    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-28fa24-865f7b8866-9hsz9"
Nov 02 20:07:19.498746 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:19.498715    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-28fa24-865f7b8866-9hsz9"]
Nov 02 20:07:19.499671 ip-10-0-133-9 kubenswrapper[4463]: W1102 20:07:19.499644    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb78a889a_2b5d_4ff0_a419_2eb7f7858579.slice/crio-c92f8732be89f75f1c8bb924e930c72c65058783bc72bb9c2c30236c289fa1a2 WatchSource:0}: Error finding container c92f8732be89f75f1c8bb924e930c72c65058783bc72bb9c2c30236c289fa1a2: Status 404 returned error can't find the container with id c92f8732be89f75f1c8bb924e930c72c65058783bc72bb9c2c30236c289fa1a2
Nov 02 20:07:19.573332 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:19.573302    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 20:07:19.814668 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:19.814602    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-mhd5c"
Nov 02 20:07:19.815001 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:19.814982    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-mhd5c"
Nov 02 20:07:19.815585 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:19.815567    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-mhd5c"
Nov 02 20:07:19.943422 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:19.943382    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-28fa24-865f7b8866-9hsz9" event={"ID":"b78a889a-2b5d-4ff0-a419-2eb7f7858579","Type":"ContainerStarted","Data":"c92f8732be89f75f1c8bb924e930c72c65058783bc72bb9c2c30236c289fa1a2"}
Nov 02 20:07:20.953429 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:20.953394    4463 generic.go:356] "Generic (PLEG): container finished" podID="32d7da41-73b0-4ae5-a8b3-69bcb021cf4a" containerID="79a9f1a468225211f945377c7c63e9d2edb050c28f2cac2924d942cbba60f0a8" exitCode=0
Nov 02 20:07:20.953790 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:20.953463    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-94hmq" event={"ID":"32d7da41-73b0-4ae5-a8b3-69bcb021cf4a","Type":"ContainerDied","Data":"79a9f1a468225211f945377c7c63e9d2edb050c28f2cac2924d942cbba60f0a8"}
Nov 02 20:07:20.955329 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:20.955305    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-28fa24-865f7b8866-9hsz9" event={"ID":"b78a889a-2b5d-4ff0-a419-2eb7f7858579","Type":"ContainerStarted","Data":"1b07488c539f685213ea7ccb3e91e8e1f877bd0d9605c0920ec0b7ec2951bc82"}
Nov 02 20:07:20.955450 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:20.955374    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-28fa24-865f7b8866-9hsz9"
Nov 02 20:07:21.069320 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:21.069262    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-28fa24-865f7b8866-9hsz9" podStartSLOduration=2.102187095 podStartE2EDuration="3.069242972s" totalImagesPullingTime="967.055877ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:07:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:07:21.063660791 +0000 UTC m=+2347.670694579" watchObservedRunningTime="2026-11-02 20:07:21.069242972 +0000 UTC m=+2347.676276743"
Nov 02 20:07:21.093956 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:21.093901    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-94hmq"
Nov 02 20:07:21.248736 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:21.248699    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32d7da41-73b0-4ae5-a8b3-69bcb021cf4a-kube-api-access-tbnn8\" (UniqueName: \"kubernetes.io/projected/32d7da41-73b0-4ae5-a8b3-69bcb021cf4a-kube-api-access-tbnn8\") pod \"32d7da41-73b0-4ae5-a8b3-69bcb021cf4a\" (UID: \"32d7da41-73b0-4ae5-a8b3-69bcb021cf4a\") "
Nov 02 20:07:21.248948 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:21.248825    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32d7da41-73b0-4ae5-a8b3-69bcb021cf4a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32d7da41-73b0-4ae5-a8b3-69bcb021cf4a-istio-envoy\") pod \"32d7da41-73b0-4ae5-a8b3-69bcb021cf4a\" (UID: \"32d7da41-73b0-4ae5-a8b3-69bcb021cf4a\") "
Nov 02 20:07:21.248948 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:21.248880    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32d7da41-73b0-4ae5-a8b3-69bcb021cf4a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32d7da41-73b0-4ae5-a8b3-69bcb021cf4a-workload-socket\") pod \"32d7da41-73b0-4ae5-a8b3-69bcb021cf4a\" (UID: \"32d7da41-73b0-4ae5-a8b3-69bcb021cf4a\") "
Nov 02 20:07:21.248948 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:21.248911    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32d7da41-73b0-4ae5-a8b3-69bcb021cf4a-istio-token\" (UniqueName: \"kubernetes.io/projected/32d7da41-73b0-4ae5-a8b3-69bcb021cf4a-istio-token\") pod \"32d7da41-73b0-4ae5-a8b3-69bcb021cf4a\" (UID: \"32d7da41-73b0-4ae5-a8b3-69bcb021cf4a\") "
Nov 02 20:07:21.249160 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:21.248983    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32d7da41-73b0-4ae5-a8b3-69bcb021cf4a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32d7da41-73b0-4ae5-a8b3-69bcb021cf4a-istio-data\") pod \"32d7da41-73b0-4ae5-a8b3-69bcb021cf4a\" (UID: \"32d7da41-73b0-4ae5-a8b3-69bcb021cf4a\") "
Nov 02 20:07:21.249160 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:21.249058    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32d7da41-73b0-4ae5-a8b3-69bcb021cf4a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32d7da41-73b0-4ae5-a8b3-69bcb021cf4a-istiod-ca-cert\") pod \"32d7da41-73b0-4ae5-a8b3-69bcb021cf4a\" (UID: \"32d7da41-73b0-4ae5-a8b3-69bcb021cf4a\") "
Nov 02 20:07:21.249160 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:21.249138    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32d7da41-73b0-4ae5-a8b3-69bcb021cf4a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32d7da41-73b0-4ae5-a8b3-69bcb021cf4a-workload-certs\") pod \"32d7da41-73b0-4ae5-a8b3-69bcb021cf4a\" (UID: \"32d7da41-73b0-4ae5-a8b3-69bcb021cf4a\") "
Nov 02 20:07:21.249327 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:21.249186    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32d7da41-73b0-4ae5-a8b3-69bcb021cf4a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32d7da41-73b0-4ae5-a8b3-69bcb021cf4a-credential-socket\") pod \"32d7da41-73b0-4ae5-a8b3-69bcb021cf4a\" (UID: \"32d7da41-73b0-4ae5-a8b3-69bcb021cf4a\") "
Nov 02 20:07:21.249327 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:21.249220    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/32d7da41-73b0-4ae5-a8b3-69bcb021cf4a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32d7da41-73b0-4ae5-a8b3-69bcb021cf4a-istio-podinfo\") pod \"32d7da41-73b0-4ae5-a8b3-69bcb021cf4a\" (UID: \"32d7da41-73b0-4ae5-a8b3-69bcb021cf4a\") "
Nov 02 20:07:21.250974 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:21.250909    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32d7da41-73b0-4ae5-a8b3-69bcb021cf4a-istio-data" pod "32d7da41-73b0-4ae5-a8b3-69bcb021cf4a" (UID: "32d7da41-73b0-4ae5-a8b3-69bcb021cf4a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:07:21.251191 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:21.251160    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32d7da41-73b0-4ae5-a8b3-69bcb021cf4a-workload-socket" pod "32d7da41-73b0-4ae5-a8b3-69bcb021cf4a" (UID: "32d7da41-73b0-4ae5-a8b3-69bcb021cf4a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:07:21.253234 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:21.253196    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32d7da41-73b0-4ae5-a8b3-69bcb021cf4a-workload-certs" pod "32d7da41-73b0-4ae5-a8b3-69bcb021cf4a" (UID: "32d7da41-73b0-4ae5-a8b3-69bcb021cf4a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:07:21.255809 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:21.255599    4463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32d7da41-73b0-4ae5-a8b3-69bcb021cf4a-workload-socket\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:07:21.255809 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:21.255628    4463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32d7da41-73b0-4ae5-a8b3-69bcb021cf4a-istio-data\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:07:21.255809 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:21.255654    4463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32d7da41-73b0-4ae5-a8b3-69bcb021cf4a-workload-certs\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:07:21.255809 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:21.255716    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32d7da41-73b0-4ae5-a8b3-69bcb021cf4a-credential-socket" pod "32d7da41-73b0-4ae5-a8b3-69bcb021cf4a" (UID: "32d7da41-73b0-4ae5-a8b3-69bcb021cf4a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:07:21.259096 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:21.257067    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32d7da41-73b0-4ae5-a8b3-69bcb021cf4a-istio-token" pod "32d7da41-73b0-4ae5-a8b3-69bcb021cf4a" (UID: "32d7da41-73b0-4ae5-a8b3-69bcb021cf4a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:07:21.259096 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:21.257300    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32d7da41-73b0-4ae5-a8b3-69bcb021cf4a-istiod-ca-cert" pod "32d7da41-73b0-4ae5-a8b3-69bcb021cf4a" (UID: "32d7da41-73b0-4ae5-a8b3-69bcb021cf4a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:07:21.262059 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:21.260054    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/32d7da41-73b0-4ae5-a8b3-69bcb021cf4a-istio-podinfo" pod "32d7da41-73b0-4ae5-a8b3-69bcb021cf4a" (UID: "32d7da41-73b0-4ae5-a8b3-69bcb021cf4a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:07:21.269917 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:21.269881    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32d7da41-73b0-4ae5-a8b3-69bcb021cf4a-kube-api-access-tbnn8" pod "32d7da41-73b0-4ae5-a8b3-69bcb021cf4a" (UID: "32d7da41-73b0-4ae5-a8b3-69bcb021cf4a"). InnerVolumeSpecName "kube-api-access-tbnn8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:07:21.275432 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:21.275404    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32d7da41-73b0-4ae5-a8b3-69bcb021cf4a-istio-envoy" pod "32d7da41-73b0-4ae5-a8b3-69bcb021cf4a" (UID: "32d7da41-73b0-4ae5-a8b3-69bcb021cf4a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:07:21.357359 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:21.357295    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbnn8\" (UniqueName: \"kubernetes.io/projected/32d7da41-73b0-4ae5-a8b3-69bcb021cf4a-kube-api-access-tbnn8\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:07:21.357359 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:21.357323    4463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32d7da41-73b0-4ae5-a8b3-69bcb021cf4a-istio-envoy\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:07:21.357359 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:21.357333    4463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32d7da41-73b0-4ae5-a8b3-69bcb021cf4a-istio-token\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:07:21.357359 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:21.357341    4463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32d7da41-73b0-4ae5-a8b3-69bcb021cf4a-istiod-ca-cert\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:07:21.357359 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:21.357351    4463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32d7da41-73b0-4ae5-a8b3-69bcb021cf4a-credential-socket\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:07:21.357359 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:21.357359    4463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32d7da41-73b0-4ae5-a8b3-69bcb021cf4a-istio-podinfo\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:07:21.963539 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:21.963508    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-94hmq"
Nov 02 20:07:21.964054 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:21.963541    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-94hmq" event={"ID":"32d7da41-73b0-4ae5-a8b3-69bcb021cf4a","Type":"ContainerDied","Data":"9b39cc0b486b08cfcade15b1dedc1a95121be7fa76d558f62ae416801405faec"}
Nov 02 20:07:21.964054 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:21.963613    4463 scope.go:122] "RemoveContainer" containerID="79a9f1a468225211f945377c7c63e9d2edb050c28f2cac2924d942cbba60f0a8"
Nov 02 20:07:22.117855 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:22.117825    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-94hmq"]
Nov 02 20:07:22.163919 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:22.163891    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-94hmq"]
Nov 02 20:07:23.585479 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:23.585445    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32d7da41-73b0-4ae5-a8b3-69bcb021cf4a" path="/var/lib/kubelet/pods/32d7da41-73b0-4ae5-a8b3-69bcb021cf4a/volumes"
Nov 02 20:07:26.969792 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:26.969763    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-28fa24-865f7b8866-9hsz9"
Nov 02 20:07:27.196805 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:27.196771    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-19d134-67b47666c7-njhvz"]
Nov 02 20:07:27.198134 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:27.198107    4463 cpu_manager.go:414] "containerMap: removing container" podUID="32d7da41-73b0-4ae5-a8b3-69bcb021cf4a" containerName="istio-proxy"
Nov 02 20:07:27.198134 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:27.198130    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32d7da41-73b0-4ae5-a8b3-69bcb021cf4a" containerName="istio-proxy"
Nov 02 20:07:27.198332 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:27.198274    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="32d7da41-73b0-4ae5-a8b3-69bcb021cf4a" containerName="istio-proxy"
Nov 02 20:07:27.203129 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:27.203107    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19d134-67b47666c7-njhvz"
Nov 02 20:07:27.220091 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:27.220003    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20157f70-df1b-48c9-ab3b-48712bd899fc-maas-api-tls\") pod \"maas-api-e2e-inf-b-19d134-67b47666c7-njhvz\" (UID: \"20157f70-df1b-48c9-ab3b-48712bd899fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19d134-67b47666c7-njhvz"
Nov 02 20:07:27.220091 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:27.220074    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fjvg\" (UniqueName: \"kubernetes.io/projected/20157f70-df1b-48c9-ab3b-48712bd899fc-kube-api-access-7fjvg\") pod \"maas-api-e2e-inf-b-19d134-67b47666c7-njhvz\" (UID: \"20157f70-df1b-48c9-ab3b-48712bd899fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19d134-67b47666c7-njhvz"
Nov 02 20:07:27.220249 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:27.220120    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20157f70-df1b-48c9-ab3b-48712bd899fc-metrics-certs\") pod \"maas-api-e2e-inf-b-19d134-67b47666c7-njhvz\" (UID: \"20157f70-df1b-48c9-ab3b-48712bd899fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19d134-67b47666c7-njhvz"
Nov 02 20:07:27.220317 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:27.220249    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20157f70-df1b-48c9-ab3b-48712bd899fc-openshift-service-ca\") pod \"maas-api-e2e-inf-b-19d134-67b47666c7-njhvz\" (UID: \"20157f70-df1b-48c9-ab3b-48712bd899fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19d134-67b47666c7-njhvz"
Nov 02 20:07:27.251777 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:27.251753    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-19d134-serving-cert\""
Nov 02 20:07:27.255788 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:27.255766    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-19d134-67b47666c7-njhvz"]
Nov 02 20:07:27.321897 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:27.321862    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20157f70-df1b-48c9-ab3b-48712bd899fc-openshift-service-ca\") pod \"maas-api-e2e-inf-b-19d134-67b47666c7-njhvz\" (UID: \"20157f70-df1b-48c9-ab3b-48712bd899fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19d134-67b47666c7-njhvz"
Nov 02 20:07:27.322124 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:27.322082    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20157f70-df1b-48c9-ab3b-48712bd899fc-maas-api-tls\") pod \"maas-api-e2e-inf-b-19d134-67b47666c7-njhvz\" (UID: \"20157f70-df1b-48c9-ab3b-48712bd899fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19d134-67b47666c7-njhvz"
Nov 02 20:07:27.322124 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:27.322114    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fjvg\" (UniqueName: \"kubernetes.io/projected/20157f70-df1b-48c9-ab3b-48712bd899fc-kube-api-access-7fjvg\") pod \"maas-api-e2e-inf-b-19d134-67b47666c7-njhvz\" (UID: \"20157f70-df1b-48c9-ab3b-48712bd899fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19d134-67b47666c7-njhvz"
Nov 02 20:07:27.322291 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:27.322180    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20157f70-df1b-48c9-ab3b-48712bd899fc-metrics-certs\") pod \"maas-api-e2e-inf-b-19d134-67b47666c7-njhvz\" (UID: \"20157f70-df1b-48c9-ab3b-48712bd899fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19d134-67b47666c7-njhvz"
Nov 02 20:07:27.322377 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:07:27.322320    4463 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-19d134-serving-cert: secret "maas-api-e2e-inf-b-19d134-serving-cert" not found
Nov 02 20:07:27.322436 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:07:27.322380    4463 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20157f70-df1b-48c9-ab3b-48712bd899fc-maas-api-tls podName:20157f70-df1b-48c9-ab3b-48712bd899fc nodeName:}" failed. No retries permitted until 2026-11-02 20:07:27.822361576 +0000 UTC m=+2354.429395340 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/20157f70-df1b-48c9-ab3b-48712bd899fc-maas-api-tls") pod "maas-api-e2e-inf-b-19d134-67b47666c7-njhvz" (UID: "20157f70-df1b-48c9-ab3b-48712bd899fc") : secret "maas-api-e2e-inf-b-19d134-serving-cert" not found
Nov 02 20:07:27.323169 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:27.323145    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20157f70-df1b-48c9-ab3b-48712bd899fc-openshift-service-ca\") pod \"maas-api-e2e-inf-b-19d134-67b47666c7-njhvz\" (UID: \"20157f70-df1b-48c9-ab3b-48712bd899fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19d134-67b47666c7-njhvz"
Nov 02 20:07:27.335050 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:27.334542    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20157f70-df1b-48c9-ab3b-48712bd899fc-metrics-certs\") pod \"maas-api-e2e-inf-b-19d134-67b47666c7-njhvz\" (UID: \"20157f70-df1b-48c9-ab3b-48712bd899fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19d134-67b47666c7-njhvz"
Nov 02 20:07:27.375421 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:27.375396    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fjvg\" (UniqueName: \"kubernetes.io/projected/20157f70-df1b-48c9-ab3b-48712bd899fc-kube-api-access-7fjvg\") pod \"maas-api-e2e-inf-b-19d134-67b47666c7-njhvz\" (UID: \"20157f70-df1b-48c9-ab3b-48712bd899fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19d134-67b47666c7-njhvz"
Nov 02 20:07:27.828363 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:27.828332    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20157f70-df1b-48c9-ab3b-48712bd899fc-maas-api-tls\") pod \"maas-api-e2e-inf-b-19d134-67b47666c7-njhvz\" (UID: \"20157f70-df1b-48c9-ab3b-48712bd899fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19d134-67b47666c7-njhvz"
Nov 02 20:07:27.830723 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:27.830696    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20157f70-df1b-48c9-ab3b-48712bd899fc-maas-api-tls\") pod \"maas-api-e2e-inf-b-19d134-67b47666c7-njhvz\" (UID: \"20157f70-df1b-48c9-ab3b-48712bd899fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19d134-67b47666c7-njhvz"
Nov 02 20:07:28.117310 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:28.117236    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19d134-67b47666c7-njhvz"
Nov 02 20:07:28.487657 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:28.487625    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-19d134-67b47666c7-njhvz"]
Nov 02 20:07:28.488319 ip-10-0-133-9 kubenswrapper[4463]: W1102 20:07:28.488283    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20157f70_df1b_48c9_ab3b_48712bd899fc.slice/crio-089377b98835cc9d0e403d78c7c66045a350c92e6b7cfd822472a7c018004041 WatchSource:0}: Error finding container 089377b98835cc9d0e403d78c7c66045a350c92e6b7cfd822472a7c018004041: Status 404 returned error can't find the container with id 089377b98835cc9d0e403d78c7c66045a350c92e6b7cfd822472a7c018004041
Nov 02 20:07:29.007097 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:29.007056    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19d134-67b47666c7-njhvz" event={"ID":"20157f70-df1b-48c9-ab3b-48712bd899fc","Type":"ContainerStarted","Data":"089377b98835cc9d0e403d78c7c66045a350c92e6b7cfd822472a7c018004041"}
Nov 02 20:07:30.020837 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:30.020797    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19d134-67b47666c7-njhvz" event={"ID":"20157f70-df1b-48c9-ab3b-48712bd899fc","Type":"ContainerStarted","Data":"ce622e18ecf0678ed8f5d1a29ced40a24113eb53f9428ca676bb6f69251c3107"}
Nov 02 20:07:30.021943 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:30.021915    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19d134-67b47666c7-njhvz"
Nov 02 20:07:30.129984 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:30.129927    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19d134-67b47666c7-njhvz" podStartSLOduration=2.297316929 podStartE2EDuration="3.129913664s" totalImagesPullingTime="832.596735ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:07:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:07:30.128314427 +0000 UTC m=+2356.735348193" watchObservedRunningTime="2026-11-02 20:07:30.129913664 +0000 UTC m=+2356.736947432"
Nov 02 20:07:34.294348 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:07:34.294315    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 20:07:37.039875 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:37.039843    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19d134-67b47666c7-njhvz"
Nov 02 20:07:43.833596 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:43.833559    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-b45a2b-openshift-default-8cddcbd8c-dq2sb"]
Nov 02 20:07:43.837974 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:43.837957    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b45a2b-openshift-default-8cddcbd8c-dq2sb"
Nov 02 20:07:43.886528 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:43.886488    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-b45a2b-openshift-default-dockercfg-6dgq9\""
Nov 02 20:07:43.921950 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:43.921923    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ab3e969-44bd-4d38-a066-005a66561670-workload-socket\") pod \"e2e-api-b-b45a2b-openshift-default-8cddcbd8c-dq2sb\" (UID: \"7ab3e969-44bd-4d38-a066-005a66561670\") " pod="openshift-ingress/e2e-api-b-b45a2b-openshift-default-8cddcbd8c-dq2sb"
Nov 02 20:07:43.922107 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:43.922047    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ab3e969-44bd-4d38-a066-005a66561670-istio-data\") pod \"e2e-api-b-b45a2b-openshift-default-8cddcbd8c-dq2sb\" (UID: \"7ab3e969-44bd-4d38-a066-005a66561670\") " pod="openshift-ingress/e2e-api-b-b45a2b-openshift-default-8cddcbd8c-dq2sb"
Nov 02 20:07:43.922171 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:43.922149    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ab3e969-44bd-4d38-a066-005a66561670-credential-socket\") pod \"e2e-api-b-b45a2b-openshift-default-8cddcbd8c-dq2sb\" (UID: \"7ab3e969-44bd-4d38-a066-005a66561670\") " pod="openshift-ingress/e2e-api-b-b45a2b-openshift-default-8cddcbd8c-dq2sb"
Nov 02 20:07:43.922214 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:43.922202    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ab3e969-44bd-4d38-a066-005a66561670-istio-podinfo\") pod \"e2e-api-b-b45a2b-openshift-default-8cddcbd8c-dq2sb\" (UID: \"7ab3e969-44bd-4d38-a066-005a66561670\") " pod="openshift-ingress/e2e-api-b-b45a2b-openshift-default-8cddcbd8c-dq2sb"
Nov 02 20:07:43.922261 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:43.922251    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ab3e969-44bd-4d38-a066-005a66561670-istiod-ca-cert\") pod \"e2e-api-b-b45a2b-openshift-default-8cddcbd8c-dq2sb\" (UID: \"7ab3e969-44bd-4d38-a066-005a66561670\") " pod="openshift-ingress/e2e-api-b-b45a2b-openshift-default-8cddcbd8c-dq2sb"
Nov 02 20:07:43.922385 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:43.922369    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmqnx\" (UniqueName: \"kubernetes.io/projected/7ab3e969-44bd-4d38-a066-005a66561670-kube-api-access-tmqnx\") pod \"e2e-api-b-b45a2b-openshift-default-8cddcbd8c-dq2sb\" (UID: \"7ab3e969-44bd-4d38-a066-005a66561670\") " pod="openshift-ingress/e2e-api-b-b45a2b-openshift-default-8cddcbd8c-dq2sb"
Nov 02 20:07:43.922421 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:43.922408    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ab3e969-44bd-4d38-a066-005a66561670-istio-envoy\") pod \"e2e-api-b-b45a2b-openshift-default-8cddcbd8c-dq2sb\" (UID: \"7ab3e969-44bd-4d38-a066-005a66561670\") " pod="openshift-ingress/e2e-api-b-b45a2b-openshift-default-8cddcbd8c-dq2sb"
Nov 02 20:07:43.922502 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:43.922488    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ab3e969-44bd-4d38-a066-005a66561670-workload-certs\") pod \"e2e-api-b-b45a2b-openshift-default-8cddcbd8c-dq2sb\" (UID: \"7ab3e969-44bd-4d38-a066-005a66561670\") " pod="openshift-ingress/e2e-api-b-b45a2b-openshift-default-8cddcbd8c-dq2sb"
Nov 02 20:07:43.922537 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:43.922510    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ab3e969-44bd-4d38-a066-005a66561670-istio-token\") pod \"e2e-api-b-b45a2b-openshift-default-8cddcbd8c-dq2sb\" (UID: \"7ab3e969-44bd-4d38-a066-005a66561670\") " pod="openshift-ingress/e2e-api-b-b45a2b-openshift-default-8cddcbd8c-dq2sb"
Nov 02 20:07:43.936088 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:43.936062    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-b45a2b-openshift-default-8cddcbd8c-dq2sb"]
Nov 02 20:07:44.023870 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:44.023838    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ab3e969-44bd-4d38-a066-005a66561670-istiod-ca-cert\") pod \"e2e-api-b-b45a2b-openshift-default-8cddcbd8c-dq2sb\" (UID: \"7ab3e969-44bd-4d38-a066-005a66561670\") " pod="openshift-ingress/e2e-api-b-b45a2b-openshift-default-8cddcbd8c-dq2sb"
Nov 02 20:07:44.024047 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:44.023897    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmqnx\" (UniqueName: \"kubernetes.io/projected/7ab3e969-44bd-4d38-a066-005a66561670-kube-api-access-tmqnx\") pod \"e2e-api-b-b45a2b-openshift-default-8cddcbd8c-dq2sb\" (UID: \"7ab3e969-44bd-4d38-a066-005a66561670\") " pod="openshift-ingress/e2e-api-b-b45a2b-openshift-default-8cddcbd8c-dq2sb"
Nov 02 20:07:44.024047 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:44.023925    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ab3e969-44bd-4d38-a066-005a66561670-istio-envoy\") pod \"e2e-api-b-b45a2b-openshift-default-8cddcbd8c-dq2sb\" (UID: \"7ab3e969-44bd-4d38-a066-005a66561670\") " pod="openshift-ingress/e2e-api-b-b45a2b-openshift-default-8cddcbd8c-dq2sb"
Nov 02 20:07:44.024047 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:44.023967    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ab3e969-44bd-4d38-a066-005a66561670-workload-certs\") pod \"e2e-api-b-b45a2b-openshift-default-8cddcbd8c-dq2sb\" (UID: \"7ab3e969-44bd-4d38-a066-005a66561670\") " pod="openshift-ingress/e2e-api-b-b45a2b-openshift-default-8cddcbd8c-dq2sb"
Nov 02 20:07:44.024233 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:44.024097    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ab3e969-44bd-4d38-a066-005a66561670-istio-token\") pod \"e2e-api-b-b45a2b-openshift-default-8cddcbd8c-dq2sb\" (UID: \"7ab3e969-44bd-4d38-a066-005a66561670\") " pod="openshift-ingress/e2e-api-b-b45a2b-openshift-default-8cddcbd8c-dq2sb"
Nov 02 20:07:44.024233 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:44.024191    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ab3e969-44bd-4d38-a066-005a66561670-workload-socket\") pod \"e2e-api-b-b45a2b-openshift-default-8cddcbd8c-dq2sb\" (UID: \"7ab3e969-44bd-4d38-a066-005a66561670\") " pod="openshift-ingress/e2e-api-b-b45a2b-openshift-default-8cddcbd8c-dq2sb"
Nov 02 20:07:44.024346 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:44.024255    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ab3e969-44bd-4d38-a066-005a66561670-istio-data\") pod \"e2e-api-b-b45a2b-openshift-default-8cddcbd8c-dq2sb\" (UID: \"7ab3e969-44bd-4d38-a066-005a66561670\") " pod="openshift-ingress/e2e-api-b-b45a2b-openshift-default-8cddcbd8c-dq2sb"
Nov 02 20:07:44.024400 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:44.024348    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ab3e969-44bd-4d38-a066-005a66561670-credential-socket\") pod \"e2e-api-b-b45a2b-openshift-default-8cddcbd8c-dq2sb\" (UID: \"7ab3e969-44bd-4d38-a066-005a66561670\") " pod="openshift-ingress/e2e-api-b-b45a2b-openshift-default-8cddcbd8c-dq2sb"
Nov 02 20:07:44.024461 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:44.024406    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ab3e969-44bd-4d38-a066-005a66561670-istio-podinfo\") pod \"e2e-api-b-b45a2b-openshift-default-8cddcbd8c-dq2sb\" (UID: \"7ab3e969-44bd-4d38-a066-005a66561670\") " pod="openshift-ingress/e2e-api-b-b45a2b-openshift-default-8cddcbd8c-dq2sb"
Nov 02 20:07:44.024520 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:44.024473    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ab3e969-44bd-4d38-a066-005a66561670-workload-certs\") pod \"e2e-api-b-b45a2b-openshift-default-8cddcbd8c-dq2sb\" (UID: \"7ab3e969-44bd-4d38-a066-005a66561670\") " pod="openshift-ingress/e2e-api-b-b45a2b-openshift-default-8cddcbd8c-dq2sb"
Nov 02 20:07:44.024639 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:44.024616    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ab3e969-44bd-4d38-a066-005a66561670-workload-socket\") pod \"e2e-api-b-b45a2b-openshift-default-8cddcbd8c-dq2sb\" (UID: \"7ab3e969-44bd-4d38-a066-005a66561670\") " pod="openshift-ingress/e2e-api-b-b45a2b-openshift-default-8cddcbd8c-dq2sb"
Nov 02 20:07:44.024707 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:44.024654    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ab3e969-44bd-4d38-a066-005a66561670-istiod-ca-cert\") pod \"e2e-api-b-b45a2b-openshift-default-8cddcbd8c-dq2sb\" (UID: \"7ab3e969-44bd-4d38-a066-005a66561670\") " pod="openshift-ingress/e2e-api-b-b45a2b-openshift-default-8cddcbd8c-dq2sb"
Nov 02 20:07:44.024763 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:44.024719    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ab3e969-44bd-4d38-a066-005a66561670-credential-socket\") pod \"e2e-api-b-b45a2b-openshift-default-8cddcbd8c-dq2sb\" (UID: \"7ab3e969-44bd-4d38-a066-005a66561670\") " pod="openshift-ingress/e2e-api-b-b45a2b-openshift-default-8cddcbd8c-dq2sb"
Nov 02 20:07:44.024824 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:44.024785    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ab3e969-44bd-4d38-a066-005a66561670-istio-data\") pod \"e2e-api-b-b45a2b-openshift-default-8cddcbd8c-dq2sb\" (UID: \"7ab3e969-44bd-4d38-a066-005a66561670\") " pod="openshift-ingress/e2e-api-b-b45a2b-openshift-default-8cddcbd8c-dq2sb"
Nov 02 20:07:44.026258 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:44.026240    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ab3e969-44bd-4d38-a066-005a66561670-istio-envoy\") pod \"e2e-api-b-b45a2b-openshift-default-8cddcbd8c-dq2sb\" (UID: \"7ab3e969-44bd-4d38-a066-005a66561670\") " pod="openshift-ingress/e2e-api-b-b45a2b-openshift-default-8cddcbd8c-dq2sb"
Nov 02 20:07:44.026974 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:44.026952    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ab3e969-44bd-4d38-a066-005a66561670-istio-podinfo\") pod \"e2e-api-b-b45a2b-openshift-default-8cddcbd8c-dq2sb\" (UID: \"7ab3e969-44bd-4d38-a066-005a66561670\") " pod="openshift-ingress/e2e-api-b-b45a2b-openshift-default-8cddcbd8c-dq2sb"
Nov 02 20:07:44.084142 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:44.084033    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ab3e969-44bd-4d38-a066-005a66561670-istio-token\") pod \"e2e-api-b-b45a2b-openshift-default-8cddcbd8c-dq2sb\" (UID: \"7ab3e969-44bd-4d38-a066-005a66561670\") " pod="openshift-ingress/e2e-api-b-b45a2b-openshift-default-8cddcbd8c-dq2sb"
Nov 02 20:07:44.084976 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:44.084921    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmqnx\" (UniqueName: \"kubernetes.io/projected/7ab3e969-44bd-4d38-a066-005a66561670-kube-api-access-tmqnx\") pod \"e2e-api-b-b45a2b-openshift-default-8cddcbd8c-dq2sb\" (UID: \"7ab3e969-44bd-4d38-a066-005a66561670\") " pod="openshift-ingress/e2e-api-b-b45a2b-openshift-default-8cddcbd8c-dq2sb"
Nov 02 20:07:44.151534 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:44.151499    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b45a2b-openshift-default-8cddcbd8c-dq2sb"
Nov 02 20:07:44.571112 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:44.571081    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-b45a2b-openshift-default-8cddcbd8c-dq2sb"]
Nov 02 20:07:44.571458 ip-10-0-133-9 kubenswrapper[4463]: W1102 20:07:44.571435    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ab3e969_44bd_4d38_a066_005a66561670.slice/crio-60435dd1700aa96bdfae9518b9e25c3b5c91bfeab4b4a8c5b1e720ce000bd565 WatchSource:0}: Error finding container 60435dd1700aa96bdfae9518b9e25c3b5c91bfeab4b4a8c5b1e720ce000bd565: Status 404 returned error can't find the container with id 60435dd1700aa96bdfae9518b9e25c3b5c91bfeab4b4a8c5b1e720ce000bd565
Nov 02 20:07:44.574206 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:44.574175    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:07:44.574266 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:44.574241    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:07:44.574314 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:44.574276    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:07:45.124518 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:45.124481    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b45a2b-openshift-default-8cddcbd8c-dq2sb" event={"ID":"7ab3e969-44bd-4d38-a066-005a66561670","Type":"ContainerStarted","Data":"cff56eeaf816c2d12a5f41d1891b424270e8974ddf47f80e6cc166ada88cae7a"}
Nov 02 20:07:45.124900 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:45.124565    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b45a2b-openshift-default-8cddcbd8c-dq2sb" event={"ID":"7ab3e969-44bd-4d38-a066-005a66561670","Type":"ContainerStarted","Data":"60435dd1700aa96bdfae9518b9e25c3b5c91bfeab4b4a8c5b1e720ce000bd565"}
Nov 02 20:07:45.151963 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:45.151931    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-b45a2b-openshift-default-8cddcbd8c-dq2sb"
Nov 02 20:07:45.157353 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:45.157334    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-b45a2b-openshift-default-8cddcbd8c-dq2sb"
Nov 02 20:07:45.277771 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:45.277727    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-b45a2b-openshift-default-8cddcbd8c-dq2sb" podStartSLOduration=2.277712255 podStartE2EDuration="2.277712255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:07:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:07:45.230999718 +0000 UTC m=+2371.838033486" watchObservedRunningTime="2026-11-02 20:07:45.277712255 +0000 UTC m=+2371.884746025"
Nov 02 20:07:46.131691 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:46.131656    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-b45a2b-openshift-default-8cddcbd8c-dq2sb"
Nov 02 20:07:46.132505 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:46.132481    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-b45a2b-openshift-default-8cddcbd8c-dq2sb"
Nov 02 20:07:52.824378 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:07:52.824328    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 20:08:04.298947 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:08:04.295963    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 20:08:34.295565 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:08:34.295526    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 20:08:43.069256 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:43.069223    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-mhd5c"]
Nov 02 20:08:43.070045 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:43.069481    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-mhd5c" podUID="721ba0b9-7dec-46b8-9578-f5bc796ff129" containerName="istio-proxy" containerID="cri-o://cce9f1e224634b58e5d453a496f3db3d8d4a7f716e28dc267413214dfca5b2f2" gracePeriod=30
Nov 02 20:08:43.075684 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:43.075655    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-rjhcs"]
Nov 02 20:08:43.075851 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:43.075833    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-rjhcs" podUID="dd10e27d-811c-4488-af13-fa8c3272b54a" containerName="istio-proxy" containerID="cri-o://b628122cd389b04510f80477e100142263336e341aea769d9f81138626984901" gracePeriod=30
Nov 02 20:08:43.097462 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:43.097436    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-88zmb"]
Nov 02 20:08:43.097670 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:43.097650    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-88zmb" podUID="fd8684fd-7158-4fa0-8a30-911e86bd53e4" containerName="istio-proxy" containerID="cri-o://7014485f60d6977f1e77737879c1b24547bb99d19a05be8a9c6f969b76e5dfda" gracePeriod=30
Nov 02 20:08:47.633350 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:47.633310    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 20:08:48.385337 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.384961    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-mhd5c"
Nov 02 20:08:48.397725 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.397641    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-88zmb"
Nov 02 20:08:48.404908 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.404890    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-rjhcs"
Nov 02 20:08:48.444594 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.444573    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/721ba0b9-7dec-46b8-9578-f5bc796ff129-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/721ba0b9-7dec-46b8-9578-f5bc796ff129-istio-data\") pod \"721ba0b9-7dec-46b8-9578-f5bc796ff129\" (UID: \"721ba0b9-7dec-46b8-9578-f5bc796ff129\") "
Nov 02 20:08:48.444692 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.444619    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/721ba0b9-7dec-46b8-9578-f5bc796ff129-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/721ba0b9-7dec-46b8-9578-f5bc796ff129-workload-certs\") pod \"721ba0b9-7dec-46b8-9578-f5bc796ff129\" (UID: \"721ba0b9-7dec-46b8-9578-f5bc796ff129\") "
Nov 02 20:08:48.444740 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.444718    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/721ba0b9-7dec-46b8-9578-f5bc796ff129-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/721ba0b9-7dec-46b8-9578-f5bc796ff129-istiod-ca-cert\") pod \"721ba0b9-7dec-46b8-9578-f5bc796ff129\" (UID: \"721ba0b9-7dec-46b8-9578-f5bc796ff129\") "
Nov 02 20:08:48.444785 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.444771    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/721ba0b9-7dec-46b8-9578-f5bc796ff129-istio-token\" (UniqueName: \"kubernetes.io/projected/721ba0b9-7dec-46b8-9578-f5bc796ff129-istio-token\") pod \"721ba0b9-7dec-46b8-9578-f5bc796ff129\" (UID: \"721ba0b9-7dec-46b8-9578-f5bc796ff129\") "
Nov 02 20:08:48.444838 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.444799    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/721ba0b9-7dec-46b8-9578-f5bc796ff129-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/721ba0b9-7dec-46b8-9578-f5bc796ff129-istio-podinfo\") pod \"721ba0b9-7dec-46b8-9578-f5bc796ff129\" (UID: \"721ba0b9-7dec-46b8-9578-f5bc796ff129\") "
Nov 02 20:08:48.444838 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.444802    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/721ba0b9-7dec-46b8-9578-f5bc796ff129-istio-data" pod "721ba0b9-7dec-46b8-9578-f5bc796ff129" (UID: "721ba0b9-7dec-46b8-9578-f5bc796ff129"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:08:48.444951 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.444845    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/721ba0b9-7dec-46b8-9578-f5bc796ff129-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/721ba0b9-7dec-46b8-9578-f5bc796ff129-istio-envoy\") pod \"721ba0b9-7dec-46b8-9578-f5bc796ff129\" (UID: \"721ba0b9-7dec-46b8-9578-f5bc796ff129\") "
Nov 02 20:08:48.444951 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.444889    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/721ba0b9-7dec-46b8-9578-f5bc796ff129-kube-api-access-l6d84\" (UniqueName: \"kubernetes.io/projected/721ba0b9-7dec-46b8-9578-f5bc796ff129-kube-api-access-l6d84\") pod \"721ba0b9-7dec-46b8-9578-f5bc796ff129\" (UID: \"721ba0b9-7dec-46b8-9578-f5bc796ff129\") "
Nov 02 20:08:48.444951 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.444924    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/721ba0b9-7dec-46b8-9578-f5bc796ff129-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/721ba0b9-7dec-46b8-9578-f5bc796ff129-workload-socket\") pod \"721ba0b9-7dec-46b8-9578-f5bc796ff129\" (UID: \"721ba0b9-7dec-46b8-9578-f5bc796ff129\") "
Nov 02 20:08:48.445146 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.444889    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/721ba0b9-7dec-46b8-9578-f5bc796ff129-workload-certs" pod "721ba0b9-7dec-46b8-9578-f5bc796ff129" (UID: "721ba0b9-7dec-46b8-9578-f5bc796ff129"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:08:48.445146 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.444975    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/721ba0b9-7dec-46b8-9578-f5bc796ff129-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/721ba0b9-7dec-46b8-9578-f5bc796ff129-credential-socket\") pod \"721ba0b9-7dec-46b8-9578-f5bc796ff129\" (UID: \"721ba0b9-7dec-46b8-9578-f5bc796ff129\") "
Nov 02 20:08:48.445286 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.445263    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/721ba0b9-7dec-46b8-9578-f5bc796ff129-istiod-ca-cert" pod "721ba0b9-7dec-46b8-9578-f5bc796ff129" (UID: "721ba0b9-7dec-46b8-9578-f5bc796ff129"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:08:48.445378 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.445354    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/721ba0b9-7dec-46b8-9578-f5bc796ff129-credential-socket" pod "721ba0b9-7dec-46b8-9578-f5bc796ff129" (UID: "721ba0b9-7dec-46b8-9578-f5bc796ff129"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:08:48.445424 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.445371    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/721ba0b9-7dec-46b8-9578-f5bc796ff129-workload-socket" pod "721ba0b9-7dec-46b8-9578-f5bc796ff129" (UID: "721ba0b9-7dec-46b8-9578-f5bc796ff129"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:08:48.445613 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.445599    4463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/721ba0b9-7dec-46b8-9578-f5bc796ff129-istio-data\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:08:48.445668 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.445615    4463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/721ba0b9-7dec-46b8-9578-f5bc796ff129-workload-certs\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:08:48.445668 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.445631    4463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/721ba0b9-7dec-46b8-9578-f5bc796ff129-istiod-ca-cert\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:08:48.445668 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.445641    4463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/721ba0b9-7dec-46b8-9578-f5bc796ff129-workload-socket\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:08:48.445668 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.445650    4463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/721ba0b9-7dec-46b8-9578-f5bc796ff129-credential-socket\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:08:48.447512 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.447487    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/721ba0b9-7dec-46b8-9578-f5bc796ff129-istio-envoy" pod "721ba0b9-7dec-46b8-9578-f5bc796ff129" (UID: "721ba0b9-7dec-46b8-9578-f5bc796ff129"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:08:48.447512 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.447502    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/721ba0b9-7dec-46b8-9578-f5bc796ff129-istio-podinfo" pod "721ba0b9-7dec-46b8-9578-f5bc796ff129" (UID: "721ba0b9-7dec-46b8-9578-f5bc796ff129"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:08:48.447643 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.447535    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/721ba0b9-7dec-46b8-9578-f5bc796ff129-istio-token" pod "721ba0b9-7dec-46b8-9578-f5bc796ff129" (UID: "721ba0b9-7dec-46b8-9578-f5bc796ff129"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:08:48.447643 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.447562    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/721ba0b9-7dec-46b8-9578-f5bc796ff129-kube-api-access-l6d84" pod "721ba0b9-7dec-46b8-9578-f5bc796ff129" (UID: "721ba0b9-7dec-46b8-9578-f5bc796ff129"). InnerVolumeSpecName "kube-api-access-l6d84". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:08:48.532421 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.532384    4463 generic.go:356] "Generic (PLEG): container finished" podID="dd10e27d-811c-4488-af13-fa8c3272b54a" containerID="b628122cd389b04510f80477e100142263336e341aea769d9f81138626984901" exitCode=0
Nov 02 20:08:48.532597 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.532429    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-rjhcs"
Nov 02 20:08:48.532597 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.532467    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-rjhcs" event={"ID":"dd10e27d-811c-4488-af13-fa8c3272b54a","Type":"ContainerDied","Data":"b628122cd389b04510f80477e100142263336e341aea769d9f81138626984901"}
Nov 02 20:08:48.532597 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.532532    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-rjhcs" event={"ID":"dd10e27d-811c-4488-af13-fa8c3272b54a","Type":"ContainerDied","Data":"2f9f24856e150cd392e22e1a2e59f77166da6700e3499a48e0e2807c97eff2e3"}
Nov 02 20:08:48.532597 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.532566    4463 scope.go:122] "RemoveContainer" containerID="b628122cd389b04510f80477e100142263336e341aea769d9f81138626984901"
Nov 02 20:08:48.533868 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.533850    4463 generic.go:356] "Generic (PLEG): container finished" podID="fd8684fd-7158-4fa0-8a30-911e86bd53e4" containerID="7014485f60d6977f1e77737879c1b24547bb99d19a05be8a9c6f969b76e5dfda" exitCode=0
Nov 02 20:08:48.533957 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.533888    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-88zmb"
Nov 02 20:08:48.533957 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.533933    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-88zmb" event={"ID":"fd8684fd-7158-4fa0-8a30-911e86bd53e4","Type":"ContainerDied","Data":"7014485f60d6977f1e77737879c1b24547bb99d19a05be8a9c6f969b76e5dfda"}
Nov 02 20:08:48.534063 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.533980    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-88zmb" event={"ID":"fd8684fd-7158-4fa0-8a30-911e86bd53e4","Type":"ContainerDied","Data":"25d55892af830f3b5dc9df658faea7bd659b48bc46b4e39696e9e1d6655c9c6c"}
Nov 02 20:08:48.535337 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.535315    4463 generic.go:356] "Generic (PLEG): container finished" podID="721ba0b9-7dec-46b8-9578-f5bc796ff129" containerID="cce9f1e224634b58e5d453a496f3db3d8d4a7f716e28dc267413214dfca5b2f2" exitCode=0
Nov 02 20:08:48.535444 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.535349    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-mhd5c"
Nov 02 20:08:48.535444 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.535376    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-mhd5c" event={"ID":"721ba0b9-7dec-46b8-9578-f5bc796ff129","Type":"ContainerDied","Data":"cce9f1e224634b58e5d453a496f3db3d8d4a7f716e28dc267413214dfca5b2f2"}
Nov 02 20:08:48.535444 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.535418    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-mhd5c" event={"ID":"721ba0b9-7dec-46b8-9578-f5bc796ff129","Type":"ContainerDied","Data":"354019136a29c5bd8915020eb74bc387aaa5e0527210e732ec5db26b0e49e545"}
Nov 02 20:08:48.546864 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.546839    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd8684fd-7158-4fa0-8a30-911e86bd53e4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd8684fd-7158-4fa0-8a30-911e86bd53e4-workload-socket\") pod \"fd8684fd-7158-4fa0-8a30-911e86bd53e4\" (UID: \"fd8684fd-7158-4fa0-8a30-911e86bd53e4\") "
Nov 02 20:08:48.546955 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.546906    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd10e27d-811c-4488-af13-fa8c3272b54a-kube-api-access-d6w9q\" (UniqueName: \"kubernetes.io/projected/dd10e27d-811c-4488-af13-fa8c3272b54a-kube-api-access-d6w9q\") pod \"dd10e27d-811c-4488-af13-fa8c3272b54a\" (UID: \"dd10e27d-811c-4488-af13-fa8c3272b54a\") "
Nov 02 20:08:48.546955 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.546941    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dd10e27d-811c-4488-af13-fa8c3272b54a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd10e27d-811c-4488-af13-fa8c3272b54a-istio-podinfo\") pod \"dd10e27d-811c-4488-af13-fa8c3272b54a\" (UID: \"dd10e27d-811c-4488-af13-fa8c3272b54a\") "
Nov 02 20:08:48.547085 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.546978    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd8684fd-7158-4fa0-8a30-911e86bd53e4-istio-token\" (UniqueName: \"kubernetes.io/projected/fd8684fd-7158-4fa0-8a30-911e86bd53e4-istio-token\") pod \"fd8684fd-7158-4fa0-8a30-911e86bd53e4\" (UID: \"fd8684fd-7158-4fa0-8a30-911e86bd53e4\") "
Nov 02 20:08:48.547085 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.547007    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd10e27d-811c-4488-af13-fa8c3272b54a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd10e27d-811c-4488-af13-fa8c3272b54a-istiod-ca-cert\") pod \"dd10e27d-811c-4488-af13-fa8c3272b54a\" (UID: \"dd10e27d-811c-4488-af13-fa8c3272b54a\") "
Nov 02 20:08:48.547085 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.547058    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd8684fd-7158-4fa0-8a30-911e86bd53e4-kube-api-access-zffg6\" (UniqueName: \"kubernetes.io/projected/fd8684fd-7158-4fa0-8a30-911e86bd53e4-kube-api-access-zffg6\") pod \"fd8684fd-7158-4fa0-8a30-911e86bd53e4\" (UID: \"fd8684fd-7158-4fa0-8a30-911e86bd53e4\") "
Nov 02 20:08:48.547259 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.547132    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd10e27d-811c-4488-af13-fa8c3272b54a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd10e27d-811c-4488-af13-fa8c3272b54a-istio-data\") pod \"dd10e27d-811c-4488-af13-fa8c3272b54a\" (UID: \"dd10e27d-811c-4488-af13-fa8c3272b54a\") "
Nov 02 20:08:48.547259 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.547139    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd8684fd-7158-4fa0-8a30-911e86bd53e4-workload-socket" pod "fd8684fd-7158-4fa0-8a30-911e86bd53e4" (UID: "fd8684fd-7158-4fa0-8a30-911e86bd53e4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:08:48.547259 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.547165    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd10e27d-811c-4488-af13-fa8c3272b54a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd10e27d-811c-4488-af13-fa8c3272b54a-workload-certs\") pod \"dd10e27d-811c-4488-af13-fa8c3272b54a\" (UID: \"dd10e27d-811c-4488-af13-fa8c3272b54a\") "
Nov 02 20:08:48.547259 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.547202    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd8684fd-7158-4fa0-8a30-911e86bd53e4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd8684fd-7158-4fa0-8a30-911e86bd53e4-istio-envoy\") pod \"fd8684fd-7158-4fa0-8a30-911e86bd53e4\" (UID: \"fd8684fd-7158-4fa0-8a30-911e86bd53e4\") "
Nov 02 20:08:48.547259 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.547253    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd8684fd-7158-4fa0-8a30-911e86bd53e4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd8684fd-7158-4fa0-8a30-911e86bd53e4-credential-socket\") pod \"fd8684fd-7158-4fa0-8a30-911e86bd53e4\" (UID: \"fd8684fd-7158-4fa0-8a30-911e86bd53e4\") "
Nov 02 20:08:48.547527 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.547327    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd10e27d-811c-4488-af13-fa8c3272b54a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd10e27d-811c-4488-af13-fa8c3272b54a-istio-envoy\") pod \"dd10e27d-811c-4488-af13-fa8c3272b54a\" (UID: \"dd10e27d-811c-4488-af13-fa8c3272b54a\") "
Nov 02 20:08:48.547527 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.547367    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd10e27d-811c-4488-af13-fa8c3272b54a-istio-token\" (UniqueName: \"kubernetes.io/projected/dd10e27d-811c-4488-af13-fa8c3272b54a-istio-token\") pod \"dd10e27d-811c-4488-af13-fa8c3272b54a\" (UID: \"dd10e27d-811c-4488-af13-fa8c3272b54a\") "
Nov 02 20:08:48.547527 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.547454    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd10e27d-811c-4488-af13-fa8c3272b54a-istiod-ca-cert" pod "dd10e27d-811c-4488-af13-fa8c3272b54a" (UID: "dd10e27d-811c-4488-af13-fa8c3272b54a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:08:48.547527 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.547471    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd10e27d-811c-4488-af13-fa8c3272b54a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd10e27d-811c-4488-af13-fa8c3272b54a-credential-socket\") pod \"dd10e27d-811c-4488-af13-fa8c3272b54a\" (UID: \"dd10e27d-811c-4488-af13-fa8c3272b54a\") "
Nov 02 20:08:48.547527 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.547515    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd8684fd-7158-4fa0-8a30-911e86bd53e4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd8684fd-7158-4fa0-8a30-911e86bd53e4-workload-certs\") pod \"fd8684fd-7158-4fa0-8a30-911e86bd53e4\" (UID: \"fd8684fd-7158-4fa0-8a30-911e86bd53e4\") "
Nov 02 20:08:48.548517 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.547556    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fd8684fd-7158-4fa0-8a30-911e86bd53e4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd8684fd-7158-4fa0-8a30-911e86bd53e4-istio-podinfo\") pod \"fd8684fd-7158-4fa0-8a30-911e86bd53e4\" (UID: \"fd8684fd-7158-4fa0-8a30-911e86bd53e4\") "
Nov 02 20:08:48.548517 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.547590    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd8684fd-7158-4fa0-8a30-911e86bd53e4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd8684fd-7158-4fa0-8a30-911e86bd53e4-istiod-ca-cert\") pod \"fd8684fd-7158-4fa0-8a30-911e86bd53e4\" (UID: \"fd8684fd-7158-4fa0-8a30-911e86bd53e4\") "
Nov 02 20:08:48.548517 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.547646    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd10e27d-811c-4488-af13-fa8c3272b54a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd10e27d-811c-4488-af13-fa8c3272b54a-workload-socket\") pod \"dd10e27d-811c-4488-af13-fa8c3272b54a\" (UID: \"dd10e27d-811c-4488-af13-fa8c3272b54a\") "
Nov 02 20:08:48.548517 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.547681    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd8684fd-7158-4fa0-8a30-911e86bd53e4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd8684fd-7158-4fa0-8a30-911e86bd53e4-istio-data\") pod \"fd8684fd-7158-4fa0-8a30-911e86bd53e4\" (UID: \"fd8684fd-7158-4fa0-8a30-911e86bd53e4\") "
Nov 02 20:08:48.548517 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.548169    4463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/721ba0b9-7dec-46b8-9578-f5bc796ff129-istio-token\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:08:48.548517 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.548184    4463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/721ba0b9-7dec-46b8-9578-f5bc796ff129-istio-podinfo\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:08:48.548517 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.548193    4463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/721ba0b9-7dec-46b8-9578-f5bc796ff129-istio-envoy\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:08:48.548517 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.548202    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6d84\" (UniqueName: \"kubernetes.io/projected/721ba0b9-7dec-46b8-9578-f5bc796ff129-kube-api-access-l6d84\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:08:48.548517 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.548213    4463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd8684fd-7158-4fa0-8a30-911e86bd53e4-workload-socket\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:08:48.548517 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.548225    4463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd10e27d-811c-4488-af13-fa8c3272b54a-istiod-ca-cert\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:08:48.549106 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.548824    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd8684fd-7158-4fa0-8a30-911e86bd53e4-workload-certs" pod "fd8684fd-7158-4fa0-8a30-911e86bd53e4" (UID: "fd8684fd-7158-4fa0-8a30-911e86bd53e4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:08:48.549106 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.549063    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd10e27d-811c-4488-af13-fa8c3272b54a-credential-socket" pod "dd10e27d-811c-4488-af13-fa8c3272b54a" (UID: "dd10e27d-811c-4488-af13-fa8c3272b54a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:08:48.549307 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.549284    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd10e27d-811c-4488-af13-fa8c3272b54a-workload-certs" pod "dd10e27d-811c-4488-af13-fa8c3272b54a" (UID: "dd10e27d-811c-4488-af13-fa8c3272b54a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:08:48.549584 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.549560    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dd10e27d-811c-4488-af13-fa8c3272b54a-istio-podinfo" pod "dd10e27d-811c-4488-af13-fa8c3272b54a" (UID: "dd10e27d-811c-4488-af13-fa8c3272b54a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:08:48.549672 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.549609    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd8684fd-7158-4fa0-8a30-911e86bd53e4-istio-data" pod "fd8684fd-7158-4fa0-8a30-911e86bd53e4" (UID: "fd8684fd-7158-4fa0-8a30-911e86bd53e4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:08:48.549878 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.549851    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd8684fd-7158-4fa0-8a30-911e86bd53e4-istio-token" pod "fd8684fd-7158-4fa0-8a30-911e86bd53e4" (UID: "fd8684fd-7158-4fa0-8a30-911e86bd53e4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:08:48.550206 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.550184    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd8684fd-7158-4fa0-8a30-911e86bd53e4-istiod-ca-cert" pod "fd8684fd-7158-4fa0-8a30-911e86bd53e4" (UID: "fd8684fd-7158-4fa0-8a30-911e86bd53e4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:08:48.550444 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.550422    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd10e27d-811c-4488-af13-fa8c3272b54a-workload-socket" pod "dd10e27d-811c-4488-af13-fa8c3272b54a" (UID: "dd10e27d-811c-4488-af13-fa8c3272b54a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:08:48.550631 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.550611    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd8684fd-7158-4fa0-8a30-911e86bd53e4-credential-socket" pod "fd8684fd-7158-4fa0-8a30-911e86bd53e4" (UID: "fd8684fd-7158-4fa0-8a30-911e86bd53e4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:08:48.550814 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.550794    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd10e27d-811c-4488-af13-fa8c3272b54a-istio-data" pod "dd10e27d-811c-4488-af13-fa8c3272b54a" (UID: "dd10e27d-811c-4488-af13-fa8c3272b54a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:08:48.551576 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.551546    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd10e27d-811c-4488-af13-fa8c3272b54a-istio-envoy" pod "dd10e27d-811c-4488-af13-fa8c3272b54a" (UID: "dd10e27d-811c-4488-af13-fa8c3272b54a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:08:48.551917 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.551896    4463 scope.go:122] "RemoveContainer" containerID="b628122cd389b04510f80477e100142263336e341aea769d9f81138626984901"
Nov 02 20:08:48.551917 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.551905    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd8684fd-7158-4fa0-8a30-911e86bd53e4-istio-envoy" pod "fd8684fd-7158-4fa0-8a30-911e86bd53e4" (UID: "fd8684fd-7158-4fa0-8a30-911e86bd53e4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:08:48.552095 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.552007    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd10e27d-811c-4488-af13-fa8c3272b54a-kube-api-access-d6w9q" pod "dd10e27d-811c-4488-af13-fa8c3272b54a" (UID: "dd10e27d-811c-4488-af13-fa8c3272b54a"). InnerVolumeSpecName "kube-api-access-d6w9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:08:48.552229 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.552207    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd8684fd-7158-4fa0-8a30-911e86bd53e4-kube-api-access-zffg6" pod "fd8684fd-7158-4fa0-8a30-911e86bd53e4" (UID: "fd8684fd-7158-4fa0-8a30-911e86bd53e4"). InnerVolumeSpecName "kube-api-access-zffg6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:08:48.552349 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.552232    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fd8684fd-7158-4fa0-8a30-911e86bd53e4-istio-podinfo" pod "fd8684fd-7158-4fa0-8a30-911e86bd53e4" (UID: "fd8684fd-7158-4fa0-8a30-911e86bd53e4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:08:48.552548 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:08:48.552517    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b628122cd389b04510f80477e100142263336e341aea769d9f81138626984901\": container with ID starting with b628122cd389b04510f80477e100142263336e341aea769d9f81138626984901 not found: ID does not exist" containerID="b628122cd389b04510f80477e100142263336e341aea769d9f81138626984901"
Nov 02 20:08:48.552644 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.552558    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b628122cd389b04510f80477e100142263336e341aea769d9f81138626984901"} err="failed to get container status \"b628122cd389b04510f80477e100142263336e341aea769d9f81138626984901\": rpc error: code = NotFound desc = could not find container \"b628122cd389b04510f80477e100142263336e341aea769d9f81138626984901\": container with ID starting with b628122cd389b04510f80477e100142263336e341aea769d9f81138626984901 not found: ID does not exist"
Nov 02 20:08:48.552644 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.552579    4463 scope.go:122] "RemoveContainer" containerID="7014485f60d6977f1e77737879c1b24547bb99d19a05be8a9c6f969b76e5dfda"
Nov 02 20:08:48.562202 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.562159    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd10e27d-811c-4488-af13-fa8c3272b54a-istio-token" pod "dd10e27d-811c-4488-af13-fa8c3272b54a" (UID: "dd10e27d-811c-4488-af13-fa8c3272b54a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:08:48.562979 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.562957    4463 scope.go:122] "RemoveContainer" containerID="7014485f60d6977f1e77737879c1b24547bb99d19a05be8a9c6f969b76e5dfda"
Nov 02 20:08:48.563270 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:08:48.563251    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7014485f60d6977f1e77737879c1b24547bb99d19a05be8a9c6f969b76e5dfda\": container with ID starting with 7014485f60d6977f1e77737879c1b24547bb99d19a05be8a9c6f969b76e5dfda not found: ID does not exist" containerID="7014485f60d6977f1e77737879c1b24547bb99d19a05be8a9c6f969b76e5dfda"
Nov 02 20:08:48.563386 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.563276    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7014485f60d6977f1e77737879c1b24547bb99d19a05be8a9c6f969b76e5dfda"} err="failed to get container status \"7014485f60d6977f1e77737879c1b24547bb99d19a05be8a9c6f969b76e5dfda\": rpc error: code = NotFound desc = could not find container \"7014485f60d6977f1e77737879c1b24547bb99d19a05be8a9c6f969b76e5dfda\": container with ID starting with 7014485f60d6977f1e77737879c1b24547bb99d19a05be8a9c6f969b76e5dfda not found: ID does not exist"
Nov 02 20:08:48.563386 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.563292    4463 scope.go:122] "RemoveContainer" containerID="cce9f1e224634b58e5d453a496f3db3d8d4a7f716e28dc267413214dfca5b2f2"
Nov 02 20:08:48.574359 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.574266    4463 scope.go:122] "RemoveContainer" containerID="cce9f1e224634b58e5d453a496f3db3d8d4a7f716e28dc267413214dfca5b2f2"
Nov 02 20:08:48.574643 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:08:48.574614    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cce9f1e224634b58e5d453a496f3db3d8d4a7f716e28dc267413214dfca5b2f2\": container with ID starting with cce9f1e224634b58e5d453a496f3db3d8d4a7f716e28dc267413214dfca5b2f2 not found: ID does not exist" containerID="cce9f1e224634b58e5d453a496f3db3d8d4a7f716e28dc267413214dfca5b2f2"
Nov 02 20:08:48.574742 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.574648    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cce9f1e224634b58e5d453a496f3db3d8d4a7f716e28dc267413214dfca5b2f2"} err="failed to get container status \"cce9f1e224634b58e5d453a496f3db3d8d4a7f716e28dc267413214dfca5b2f2\": rpc error: code = NotFound desc = could not find container \"cce9f1e224634b58e5d453a496f3db3d8d4a7f716e28dc267413214dfca5b2f2\": container with ID starting with cce9f1e224634b58e5d453a496f3db3d8d4a7f716e28dc267413214dfca5b2f2 not found: ID does not exist"
Nov 02 20:08:48.649328 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.649239    4463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd10e27d-811c-4488-af13-fa8c3272b54a-credential-socket\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:08:48.649328 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.649273    4463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd8684fd-7158-4fa0-8a30-911e86bd53e4-workload-certs\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:08:48.649328 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.649287    4463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd8684fd-7158-4fa0-8a30-911e86bd53e4-istio-podinfo\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:08:48.649328 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.649302    4463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd8684fd-7158-4fa0-8a30-911e86bd53e4-istiod-ca-cert\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:08:48.649328 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.649316    4463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd10e27d-811c-4488-af13-fa8c3272b54a-workload-socket\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:08:48.649328 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.649329    4463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd8684fd-7158-4fa0-8a30-911e86bd53e4-istio-data\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:08:48.649851 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.649341    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6w9q\" (UniqueName: \"kubernetes.io/projected/dd10e27d-811c-4488-af13-fa8c3272b54a-kube-api-access-d6w9q\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:08:48.649851 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.649356    4463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd10e27d-811c-4488-af13-fa8c3272b54a-istio-podinfo\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:08:48.649851 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.649367    4463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd8684fd-7158-4fa0-8a30-911e86bd53e4-istio-token\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:08:48.649851 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.649380    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zffg6\" (UniqueName: \"kubernetes.io/projected/fd8684fd-7158-4fa0-8a30-911e86bd53e4-kube-api-access-zffg6\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:08:48.649851 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.649393    4463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd10e27d-811c-4488-af13-fa8c3272b54a-istio-data\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:08:48.649851 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.649405    4463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd10e27d-811c-4488-af13-fa8c3272b54a-workload-certs\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:08:48.649851 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.649419    4463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd8684fd-7158-4fa0-8a30-911e86bd53e4-istio-envoy\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:08:48.649851 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.649434    4463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd8684fd-7158-4fa0-8a30-911e86bd53e4-credential-socket\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:08:48.649851 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.649447    4463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd10e27d-811c-4488-af13-fa8c3272b54a-istio-envoy\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:08:48.649851 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.649465    4463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd10e27d-811c-4488-af13-fa8c3272b54a-istio-token\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:08:48.915178 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.915112    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-mhd5c"]
Nov 02 20:08:48.920744 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:48.920716    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-mhd5c"]
Nov 02 20:08:49.022131 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:49.022102    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-rjhcs"]
Nov 02 20:08:49.024902 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:49.024881    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-rjhcs"]
Nov 02 20:08:49.122688 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:49.122662    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-88zmb"]
Nov 02 20:08:49.127473 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:49.127451    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-88zmb"]
Nov 02 20:08:49.586511 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:49.586473    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="721ba0b9-7dec-46b8-9578-f5bc796ff129" path="/var/lib/kubelet/pods/721ba0b9-7dec-46b8-9578-f5bc796ff129/volumes"
Nov 02 20:08:49.587094 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:49.587071    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd10e27d-811c-4488-af13-fa8c3272b54a" path="/var/lib/kubelet/pods/dd10e27d-811c-4488-af13-fa8c3272b54a/volumes"
Nov 02 20:08:49.587558 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:49.587537    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd8684fd-7158-4fa0-8a30-911e86bd53e4" path="/var/lib/kubelet/pods/fd8684fd-7158-4fa0-8a30-911e86bd53e4/volumes"
Nov 02 20:08:57.363160 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:57.363125    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-fh467"]
Nov 02 20:08:57.363564 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:57.363347    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-fh467" podUID="0f76ece2-703b-41b2-8833-f909342a8390" containerName="istio-proxy" containerID="cri-o://9c469b9a1882050cec9f8501a6de988758cc55964828388b535c7ff949341a01" gracePeriod=30
Nov 02 20:08:57.910972 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:57.910935    4463 patch_prober.go:29] interesting pod/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-fh467 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 20:08:57.911154 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:08:57.911002    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-fh467" podUID="0f76ece2-703b-41b2-8833-f909342a8390" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 20:09:02.632706 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:02.632672    4463 generic.go:356] "Generic (PLEG): container finished" podID="0f76ece2-703b-41b2-8833-f909342a8390" containerID="9c469b9a1882050cec9f8501a6de988758cc55964828388b535c7ff949341a01" exitCode=0
Nov 02 20:09:02.632706 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:02.632697    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-fh467" event={"ID":"0f76ece2-703b-41b2-8833-f909342a8390","Type":"ContainerDied","Data":"9c469b9a1882050cec9f8501a6de988758cc55964828388b535c7ff949341a01"}
Nov 02 20:09:02.633169 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:02.632759    4463 scope.go:122] "RemoveContainer" containerID="a5850cb61c7e98c613c433a5ac1dc76279be71e87e4c7fa07c31a4c86a55e91d"
Nov 02 20:09:02.663601 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:02.663578    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-fh467"
Nov 02 20:09:02.723095 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:02.723073    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f76ece2-703b-41b2-8833-f909342a8390-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f76ece2-703b-41b2-8833-f909342a8390-istio-envoy\") pod \"0f76ece2-703b-41b2-8833-f909342a8390\" (UID: \"0f76ece2-703b-41b2-8833-f909342a8390\") "
Nov 02 20:09:02.723222 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:02.723121    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f76ece2-703b-41b2-8833-f909342a8390-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f76ece2-703b-41b2-8833-f909342a8390-istio-data\") pod \"0f76ece2-703b-41b2-8833-f909342a8390\" (UID: \"0f76ece2-703b-41b2-8833-f909342a8390\") "
Nov 02 20:09:02.723222 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:02.723192    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f76ece2-703b-41b2-8833-f909342a8390-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f76ece2-703b-41b2-8833-f909342a8390-workload-certs\") pod \"0f76ece2-703b-41b2-8833-f909342a8390\" (UID: \"0f76ece2-703b-41b2-8833-f909342a8390\") "
Nov 02 20:09:02.723222 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:02.723210    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f76ece2-703b-41b2-8833-f909342a8390-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f76ece2-703b-41b2-8833-f909342a8390-workload-socket\") pod \"0f76ece2-703b-41b2-8833-f909342a8390\" (UID: \"0f76ece2-703b-41b2-8833-f909342a8390\") "
Nov 02 20:09:02.723391 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:02.723242    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f76ece2-703b-41b2-8833-f909342a8390-kube-api-access-9xp7d\" (UniqueName: \"kubernetes.io/projected/0f76ece2-703b-41b2-8833-f909342a8390-kube-api-access-9xp7d\") pod \"0f76ece2-703b-41b2-8833-f909342a8390\" (UID: \"0f76ece2-703b-41b2-8833-f909342a8390\") "
Nov 02 20:09:02.723391 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:02.723290    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f76ece2-703b-41b2-8833-f909342a8390-istio-token\" (UniqueName: \"kubernetes.io/projected/0f76ece2-703b-41b2-8833-f909342a8390-istio-token\") pod \"0f76ece2-703b-41b2-8833-f909342a8390\" (UID: \"0f76ece2-703b-41b2-8833-f909342a8390\") "
Nov 02 20:09:02.723391 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:02.723329    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f76ece2-703b-41b2-8833-f909342a8390-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f76ece2-703b-41b2-8833-f909342a8390-istiod-ca-cert\") pod \"0f76ece2-703b-41b2-8833-f909342a8390\" (UID: \"0f76ece2-703b-41b2-8833-f909342a8390\") "
Nov 02 20:09:02.723553 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:02.723462    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f76ece2-703b-41b2-8833-f909342a8390-istio-data" pod "0f76ece2-703b-41b2-8833-f909342a8390" (UID: "0f76ece2-703b-41b2-8833-f909342a8390"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:09:02.723553 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:02.723482    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f76ece2-703b-41b2-8833-f909342a8390-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f76ece2-703b-41b2-8833-f909342a8390-credential-socket\") pod \"0f76ece2-703b-41b2-8833-f909342a8390\" (UID: \"0f76ece2-703b-41b2-8833-f909342a8390\") "
Nov 02 20:09:02.723553 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:02.723489    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f76ece2-703b-41b2-8833-f909342a8390-workload-certs" pod "0f76ece2-703b-41b2-8833-f909342a8390" (UID: "0f76ece2-703b-41b2-8833-f909342a8390"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:09:02.723553 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:02.723538    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0f76ece2-703b-41b2-8833-f909342a8390-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f76ece2-703b-41b2-8833-f909342a8390-istio-podinfo\") pod \"0f76ece2-703b-41b2-8833-f909342a8390\" (UID: \"0f76ece2-703b-41b2-8833-f909342a8390\") "
Nov 02 20:09:02.723766 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:02.723632    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f76ece2-703b-41b2-8833-f909342a8390-workload-socket" pod "0f76ece2-703b-41b2-8833-f909342a8390" (UID: "0f76ece2-703b-41b2-8833-f909342a8390"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:09:02.723826 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:02.723759    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f76ece2-703b-41b2-8833-f909342a8390-credential-socket" pod "0f76ece2-703b-41b2-8833-f909342a8390" (UID: "0f76ece2-703b-41b2-8833-f909342a8390"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:09:02.723955 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:02.723929    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f76ece2-703b-41b2-8833-f909342a8390-istiod-ca-cert" pod "0f76ece2-703b-41b2-8833-f909342a8390" (UID: "0f76ece2-703b-41b2-8833-f909342a8390"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:09:02.724266 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:02.724241    4463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f76ece2-703b-41b2-8833-f909342a8390-istiod-ca-cert\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:09:02.724376 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:02.724272    4463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f76ece2-703b-41b2-8833-f909342a8390-credential-socket\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:09:02.724376 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:02.724296    4463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f76ece2-703b-41b2-8833-f909342a8390-istio-data\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:09:02.724376 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:02.724309    4463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f76ece2-703b-41b2-8833-f909342a8390-workload-certs\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:09:02.724376 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:02.724324    4463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f76ece2-703b-41b2-8833-f909342a8390-workload-socket\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:09:02.725416 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:02.725393    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f76ece2-703b-41b2-8833-f909342a8390-istio-envoy" pod "0f76ece2-703b-41b2-8833-f909342a8390" (UID: "0f76ece2-703b-41b2-8833-f909342a8390"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:09:02.725613 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:02.725592    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f76ece2-703b-41b2-8833-f909342a8390-kube-api-access-9xp7d" pod "0f76ece2-703b-41b2-8833-f909342a8390" (UID: "0f76ece2-703b-41b2-8833-f909342a8390"). InnerVolumeSpecName "kube-api-access-9xp7d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:09:02.725721 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:02.725703    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0f76ece2-703b-41b2-8833-f909342a8390-istio-podinfo" pod "0f76ece2-703b-41b2-8833-f909342a8390" (UID: "0f76ece2-703b-41b2-8833-f909342a8390"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:09:02.725833 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:02.725818    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f76ece2-703b-41b2-8833-f909342a8390-istio-token" pod "0f76ece2-703b-41b2-8833-f909342a8390" (UID: "0f76ece2-703b-41b2-8833-f909342a8390"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:09:02.833558 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:02.827232    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xp7d\" (UniqueName: \"kubernetes.io/projected/0f76ece2-703b-41b2-8833-f909342a8390-kube-api-access-9xp7d\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:09:02.833558 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:02.827273    4463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f76ece2-703b-41b2-8833-f909342a8390-istio-token\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:09:02.833558 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:02.827287    4463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f76ece2-703b-41b2-8833-f909342a8390-istio-podinfo\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:09:02.833558 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:02.827301    4463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f76ece2-703b-41b2-8833-f909342a8390-istio-envoy\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:09:03.639460 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:03.639433    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-fh467"
Nov 02 20:09:03.639460 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:03.639449    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-fh467" event={"ID":"0f76ece2-703b-41b2-8833-f909342a8390","Type":"ContainerDied","Data":"5202a2211b34d4b6cca958f5bf26e17b10fc9b99b6e03f16d66ba3f69f6b5dfb"}
Nov 02 20:09:03.639915 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:03.639511    4463 scope.go:122] "RemoveContainer" containerID="9c469b9a1882050cec9f8501a6de988758cc55964828388b535c7ff949341a01"
Nov 02 20:09:03.838984 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:03.838947    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-fh467"]
Nov 02 20:09:03.845308 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:03.845244    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-fh467"]
Nov 02 20:09:04.294505 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:09:04.294472    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 20:09:05.587045 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:05.586986    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f76ece2-703b-41b2-8833-f909342a8390" path="/var/lib/kubelet/pods/0f76ece2-703b-41b2-8833-f909342a8390/volumes"
Nov 02 20:09:12.183228 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:12.183194    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b45a2b-76df647b98-p92xh"]
Nov 02 20:09:12.183891 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:12.183874    4463 cpu_manager.go:414] "containerMap: removing container" podUID="fd8684fd-7158-4fa0-8a30-911e86bd53e4" containerName="istio-proxy"
Nov 02 20:09:12.183940 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:12.183893    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd8684fd-7158-4fa0-8a30-911e86bd53e4" containerName="istio-proxy"
Nov 02 20:09:12.183940 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:12.183908    4463 cpu_manager.go:414] "containerMap: removing container" podUID="0f76ece2-703b-41b2-8833-f909342a8390" containerName="istio-proxy"
Nov 02 20:09:12.183940 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:12.183914    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f76ece2-703b-41b2-8833-f909342a8390" containerName="istio-proxy"
Nov 02 20:09:12.183940 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:12.183927    4463 cpu_manager.go:414] "containerMap: removing container" podUID="0f76ece2-703b-41b2-8833-f909342a8390" containerName="istio-proxy"
Nov 02 20:09:12.183940 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:12.183932    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f76ece2-703b-41b2-8833-f909342a8390" containerName="istio-proxy"
Nov 02 20:09:12.183940 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:12.183939    4463 cpu_manager.go:414] "containerMap: removing container" podUID="dd10e27d-811c-4488-af13-fa8c3272b54a" containerName="istio-proxy"
Nov 02 20:09:12.184143 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:12.183945    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd10e27d-811c-4488-af13-fa8c3272b54a" containerName="istio-proxy"
Nov 02 20:09:12.184143 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:12.183972    4463 cpu_manager.go:414] "containerMap: removing container" podUID="721ba0b9-7dec-46b8-9578-f5bc796ff129" containerName="istio-proxy"
Nov 02 20:09:12.184143 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:12.183977    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="721ba0b9-7dec-46b8-9578-f5bc796ff129" containerName="istio-proxy"
Nov 02 20:09:12.184143 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:12.184096    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd8684fd-7158-4fa0-8a30-911e86bd53e4" containerName="istio-proxy"
Nov 02 20:09:12.184143 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:12.184106    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f76ece2-703b-41b2-8833-f909342a8390" containerName="istio-proxy"
Nov 02 20:09:12.184143 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:12.184114    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f76ece2-703b-41b2-8833-f909342a8390" containerName="istio-proxy"
Nov 02 20:09:12.184143 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:12.184122    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd10e27d-811c-4488-af13-fa8c3272b54a" containerName="istio-proxy"
Nov 02 20:09:12.184143 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:12.184134    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="721ba0b9-7dec-46b8-9578-f5bc796ff129" containerName="istio-proxy"
Nov 02 20:09:12.187524 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:12.187507    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b45a2b-76df647b98-p92xh"
Nov 02 20:09:12.227210 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:12.227176    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-pgq8d"]
Nov 02 20:09:12.227381 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:12.227361    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-pgq8d" podUID="6ff8e962-f7e6-4d82-a818-261d21bdba26" containerName="istio-proxy" containerID="cri-o://afb37a4cfedbff6c20f0827d8ae976f43b3857e0291ab2f02989bf9ea5f90b3c" gracePeriod=30
Nov 02 20:09:12.238321 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:12.238281    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a52fefb-fb79-4d25-9edf-719bd84cecdd-openshift-service-ca\") pod \"maas-api-e2e-api-b-b45a2b-76df647b98-p92xh\" (UID: \"2a52fefb-fb79-4d25-9edf-719bd84cecdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b45a2b-76df647b98-p92xh"
Nov 02 20:09:12.238593 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:12.238562    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a52fefb-fb79-4d25-9edf-719bd84cecdd-metrics-certs\") pod \"maas-api-e2e-api-b-b45a2b-76df647b98-p92xh\" (UID: \"2a52fefb-fb79-4d25-9edf-719bd84cecdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b45a2b-76df647b98-p92xh"
Nov 02 20:09:12.238710 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:12.238690    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6rrx\" (UniqueName: \"kubernetes.io/projected/2a52fefb-fb79-4d25-9edf-719bd84cecdd-kube-api-access-w6rrx\") pod \"maas-api-e2e-api-b-b45a2b-76df647b98-p92xh\" (UID: \"2a52fefb-fb79-4d25-9edf-719bd84cecdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b45a2b-76df647b98-p92xh"
Nov 02 20:09:12.238997 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:12.238974    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a52fefb-fb79-4d25-9edf-719bd84cecdd-maas-api-tls\") pod \"maas-api-e2e-api-b-b45a2b-76df647b98-p92xh\" (UID: \"2a52fefb-fb79-4d25-9edf-719bd84cecdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b45a2b-76df647b98-p92xh"
Nov 02 20:09:12.317069 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:12.296677    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-b45a2b-serving-cert\""
Nov 02 20:09:12.330690 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:12.330638    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b45a2b-76df647b98-p92xh"]
Nov 02 20:09:12.343105 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:12.340112    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a52fefb-fb79-4d25-9edf-719bd84cecdd-maas-api-tls\") pod \"maas-api-e2e-api-b-b45a2b-76df647b98-p92xh\" (UID: \"2a52fefb-fb79-4d25-9edf-719bd84cecdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b45a2b-76df647b98-p92xh"
Nov 02 20:09:12.343105 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:12.340184    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a52fefb-fb79-4d25-9edf-719bd84cecdd-openshift-service-ca\") pod \"maas-api-e2e-api-b-b45a2b-76df647b98-p92xh\" (UID: \"2a52fefb-fb79-4d25-9edf-719bd84cecdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b45a2b-76df647b98-p92xh"
Nov 02 20:09:12.343105 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:12.340260    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a52fefb-fb79-4d25-9edf-719bd84cecdd-metrics-certs\") pod \"maas-api-e2e-api-b-b45a2b-76df647b98-p92xh\" (UID: \"2a52fefb-fb79-4d25-9edf-719bd84cecdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b45a2b-76df647b98-p92xh"
Nov 02 20:09:12.343105 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:12.340336    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6rrx\" (UniqueName: \"kubernetes.io/projected/2a52fefb-fb79-4d25-9edf-719bd84cecdd-kube-api-access-w6rrx\") pod \"maas-api-e2e-api-b-b45a2b-76df647b98-p92xh\" (UID: \"2a52fefb-fb79-4d25-9edf-719bd84cecdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b45a2b-76df647b98-p92xh"
Nov 02 20:09:12.343105 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:09:12.340840    4463 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-b45a2b-serving-cert: secret "maas-api-e2e-api-b-b45a2b-serving-cert" not found
Nov 02 20:09:12.343105 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:09:12.340908    4463 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a52fefb-fb79-4d25-9edf-719bd84cecdd-maas-api-tls podName:2a52fefb-fb79-4d25-9edf-719bd84cecdd nodeName:}" failed. No retries permitted until 2026-11-02 20:09:12.840888841 +0000 UTC m=+2459.447922605 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2a52fefb-fb79-4d25-9edf-719bd84cecdd-maas-api-tls") pod "maas-api-e2e-api-b-b45a2b-76df647b98-p92xh" (UID: "2a52fefb-fb79-4d25-9edf-719bd84cecdd") : secret "maas-api-e2e-api-b-b45a2b-serving-cert" not found
Nov 02 20:09:12.343105 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:12.341719    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a52fefb-fb79-4d25-9edf-719bd84cecdd-openshift-service-ca\") pod \"maas-api-e2e-api-b-b45a2b-76df647b98-p92xh\" (UID: \"2a52fefb-fb79-4d25-9edf-719bd84cecdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b45a2b-76df647b98-p92xh"
Nov 02 20:09:12.390031 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:12.383183    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a52fefb-fb79-4d25-9edf-719bd84cecdd-metrics-certs\") pod \"maas-api-e2e-api-b-b45a2b-76df647b98-p92xh\" (UID: \"2a52fefb-fb79-4d25-9edf-719bd84cecdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b45a2b-76df647b98-p92xh"
Nov 02 20:09:12.401077 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:12.401045    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6rrx\" (UniqueName: \"kubernetes.io/projected/2a52fefb-fb79-4d25-9edf-719bd84cecdd-kube-api-access-w6rrx\") pod \"maas-api-e2e-api-b-b45a2b-76df647b98-p92xh\" (UID: \"2a52fefb-fb79-4d25-9edf-719bd84cecdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b45a2b-76df647b98-p92xh"
Nov 02 20:09:12.854561 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:12.854521    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a52fefb-fb79-4d25-9edf-719bd84cecdd-maas-api-tls\") pod \"maas-api-e2e-api-b-b45a2b-76df647b98-p92xh\" (UID: \"2a52fefb-fb79-4d25-9edf-719bd84cecdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b45a2b-76df647b98-p92xh"
Nov 02 20:09:12.856889 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:12.856866    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a52fefb-fb79-4d25-9edf-719bd84cecdd-maas-api-tls\") pod \"maas-api-e2e-api-b-b45a2b-76df647b98-p92xh\" (UID: \"2a52fefb-fb79-4d25-9edf-719bd84cecdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b45a2b-76df647b98-p92xh"
Nov 02 20:09:13.101872 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:13.101842    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b45a2b-76df647b98-p92xh"
Nov 02 20:09:13.468302 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:13.468255    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b45a2b-76df647b98-p92xh"]
Nov 02 20:09:13.470428 ip-10-0-133-9 kubenswrapper[4463]: W1102 20:09:13.470403    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a52fefb_fb79_4d25_9edf_719bd84cecdd.slice/crio-d3551f52fd5782088bceaf70ea2557cbc42c81d7ee40a198f2a2a2aa921f06a0 WatchSource:0}: Error finding container d3551f52fd5782088bceaf70ea2557cbc42c81d7ee40a198f2a2a2aa921f06a0: Status 404 returned error can't find the container with id d3551f52fd5782088bceaf70ea2557cbc42c81d7ee40a198f2a2a2aa921f06a0
Nov 02 20:09:13.702733 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:13.702698    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b45a2b-76df647b98-p92xh" event={"ID":"2a52fefb-fb79-4d25-9edf-719bd84cecdd","Type":"ContainerStarted","Data":"d3551f52fd5782088bceaf70ea2557cbc42c81d7ee40a198f2a2a2aa921f06a0"}
Nov 02 20:09:14.420249 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:14.420215    4463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-pgq8d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 20:09:14.420407 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:14.420270    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-pgq8d" podUID="6ff8e962-f7e6-4d82-a818-261d21bdba26" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 20:09:14.712248 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:14.712214    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b45a2b-76df647b98-p92xh" event={"ID":"2a52fefb-fb79-4d25-9edf-719bd84cecdd","Type":"ContainerStarted","Data":"09f91af78557416ce3b155ac24fda055049467f45fc65019cc149937fe0eb1d2"}
Nov 02 20:09:14.712622 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:14.712292    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b45a2b-76df647b98-p92xh"
Nov 02 20:09:14.820622 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:14.820568    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b45a2b-76df647b98-p92xh" podStartSLOduration=1.962092337 podStartE2EDuration="2.820554416s" totalImagesPullingTime="858.462079ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:09:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:09:14.817925296 +0000 UTC m=+2461.424959065" watchObservedRunningTime="2026-11-02 20:09:14.820554416 +0000 UTC m=+2461.427588185"
Nov 02 20:09:17.503154 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:17.503090    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-pgq8d"
Nov 02 20:09:17.616343 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:17.616310    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ff8e962-f7e6-4d82-a818-261d21bdba26-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ff8e962-f7e6-4d82-a818-261d21bdba26-istiod-ca-cert\") pod \"6ff8e962-f7e6-4d82-a818-261d21bdba26\" (UID: \"6ff8e962-f7e6-4d82-a818-261d21bdba26\") "
Nov 02 20:09:17.616526 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:17.616360    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ff8e962-f7e6-4d82-a818-261d21bdba26-kube-api-access-mrmps\" (UniqueName: \"kubernetes.io/projected/6ff8e962-f7e6-4d82-a818-261d21bdba26-kube-api-access-mrmps\") pod \"6ff8e962-f7e6-4d82-a818-261d21bdba26\" (UID: \"6ff8e962-f7e6-4d82-a818-261d21bdba26\") "
Nov 02 20:09:17.616526 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:17.616390    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ff8e962-f7e6-4d82-a818-261d21bdba26-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ff8e962-f7e6-4d82-a818-261d21bdba26-istio-data\") pod \"6ff8e962-f7e6-4d82-a818-261d21bdba26\" (UID: \"6ff8e962-f7e6-4d82-a818-261d21bdba26\") "
Nov 02 20:09:17.616526 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:17.616503    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ff8e962-f7e6-4d82-a818-261d21bdba26-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ff8e962-f7e6-4d82-a818-261d21bdba26-workload-certs\") pod \"6ff8e962-f7e6-4d82-a818-261d21bdba26\" (UID: \"6ff8e962-f7e6-4d82-a818-261d21bdba26\") "
Nov 02 20:09:17.616704 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:17.616598    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ff8e962-f7e6-4d82-a818-261d21bdba26-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ff8e962-f7e6-4d82-a818-261d21bdba26-credential-socket\") pod \"6ff8e962-f7e6-4d82-a818-261d21bdba26\" (UID: \"6ff8e962-f7e6-4d82-a818-261d21bdba26\") "
Nov 02 20:09:17.616704 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:17.616649    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ff8e962-f7e6-4d82-a818-261d21bdba26-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ff8e962-f7e6-4d82-a818-261d21bdba26-istio-envoy\") pod \"6ff8e962-f7e6-4d82-a818-261d21bdba26\" (UID: \"6ff8e962-f7e6-4d82-a818-261d21bdba26\") "
Nov 02 20:09:17.616704 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:17.616679    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ff8e962-f7e6-4d82-a818-261d21bdba26-istio-data" pod "6ff8e962-f7e6-4d82-a818-261d21bdba26" (UID: "6ff8e962-f7e6-4d82-a818-261d21bdba26"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:09:17.616890 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:17.616761    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ff8e962-f7e6-4d82-a818-261d21bdba26-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ff8e962-f7e6-4d82-a818-261d21bdba26-workload-socket\") pod \"6ff8e962-f7e6-4d82-a818-261d21bdba26\" (UID: \"6ff8e962-f7e6-4d82-a818-261d21bdba26\") "
Nov 02 20:09:17.616890 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:17.616809    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ff8e962-f7e6-4d82-a818-261d21bdba26-istio-token\" (UniqueName: \"kubernetes.io/projected/6ff8e962-f7e6-4d82-a818-261d21bdba26-istio-token\") pod \"6ff8e962-f7e6-4d82-a818-261d21bdba26\" (UID: \"6ff8e962-f7e6-4d82-a818-261d21bdba26\") "
Nov 02 20:09:17.616890 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:17.616864    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6ff8e962-f7e6-4d82-a818-261d21bdba26-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ff8e962-f7e6-4d82-a818-261d21bdba26-istio-podinfo\") pod \"6ff8e962-f7e6-4d82-a818-261d21bdba26\" (UID: \"6ff8e962-f7e6-4d82-a818-261d21bdba26\") "
Nov 02 20:09:17.617073 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:17.616886    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ff8e962-f7e6-4d82-a818-261d21bdba26-workload-certs" pod "6ff8e962-f7e6-4d82-a818-261d21bdba26" (UID: "6ff8e962-f7e6-4d82-a818-261d21bdba26"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:09:17.617073 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:17.616896    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ff8e962-f7e6-4d82-a818-261d21bdba26-istiod-ca-cert" pod "6ff8e962-f7e6-4d82-a818-261d21bdba26" (UID: "6ff8e962-f7e6-4d82-a818-261d21bdba26"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:09:17.617182 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:17.617131    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ff8e962-f7e6-4d82-a818-261d21bdba26-workload-socket" pod "6ff8e962-f7e6-4d82-a818-261d21bdba26" (UID: "6ff8e962-f7e6-4d82-a818-261d21bdba26"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:09:17.617329 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:17.617302    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ff8e962-f7e6-4d82-a818-261d21bdba26-credential-socket" pod "6ff8e962-f7e6-4d82-a818-261d21bdba26" (UID: "6ff8e962-f7e6-4d82-a818-261d21bdba26"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:09:17.617464 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:17.617445    4463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ff8e962-f7e6-4d82-a818-261d21bdba26-workload-socket\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:09:17.617536 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:17.617469    4463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ff8e962-f7e6-4d82-a818-261d21bdba26-istiod-ca-cert\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:09:17.617536 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:17.617483    4463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ff8e962-f7e6-4d82-a818-261d21bdba26-istio-data\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:09:17.617536 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:17.617496    4463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ff8e962-f7e6-4d82-a818-261d21bdba26-workload-certs\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:09:17.617536 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:17.617510    4463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ff8e962-f7e6-4d82-a818-261d21bdba26-credential-socket\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:09:17.619359 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:17.619336    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ff8e962-f7e6-4d82-a818-261d21bdba26-istio-envoy" pod "6ff8e962-f7e6-4d82-a818-261d21bdba26" (UID: "6ff8e962-f7e6-4d82-a818-261d21bdba26"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:09:17.620385 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:17.620355    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ff8e962-f7e6-4d82-a818-261d21bdba26-istio-token" pod "6ff8e962-f7e6-4d82-a818-261d21bdba26" (UID: "6ff8e962-f7e6-4d82-a818-261d21bdba26"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:09:17.620503 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:17.620482    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6ff8e962-f7e6-4d82-a818-261d21bdba26-istio-podinfo" pod "6ff8e962-f7e6-4d82-a818-261d21bdba26" (UID: "6ff8e962-f7e6-4d82-a818-261d21bdba26"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:09:17.620704 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:17.620671    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ff8e962-f7e6-4d82-a818-261d21bdba26-kube-api-access-mrmps" pod "6ff8e962-f7e6-4d82-a818-261d21bdba26" (UID: "6ff8e962-f7e6-4d82-a818-261d21bdba26"). InnerVolumeSpecName "kube-api-access-mrmps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:09:17.719565 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:17.719527    4463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ff8e962-f7e6-4d82-a818-261d21bdba26-istio-token\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:09:17.719565 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:17.719566    4463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ff8e962-f7e6-4d82-a818-261d21bdba26-istio-podinfo\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:09:17.719795 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:17.719584    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrmps\" (UniqueName: \"kubernetes.io/projected/6ff8e962-f7e6-4d82-a818-261d21bdba26-kube-api-access-mrmps\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:09:17.719795 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:17.719600    4463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ff8e962-f7e6-4d82-a818-261d21bdba26-istio-envoy\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:09:17.735217 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:17.735190    4463 generic.go:356] "Generic (PLEG): container finished" podID="6ff8e962-f7e6-4d82-a818-261d21bdba26" containerID="afb37a4cfedbff6c20f0827d8ae976f43b3857e0291ab2f02989bf9ea5f90b3c" exitCode=0
Nov 02 20:09:17.735369 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:17.735234    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-pgq8d"
Nov 02 20:09:17.735369 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:17.735241    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-pgq8d" event={"ID":"6ff8e962-f7e6-4d82-a818-261d21bdba26","Type":"ContainerDied","Data":"afb37a4cfedbff6c20f0827d8ae976f43b3857e0291ab2f02989bf9ea5f90b3c"}
Nov 02 20:09:17.735369 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:17.735282    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-pgq8d" event={"ID":"6ff8e962-f7e6-4d82-a818-261d21bdba26","Type":"ContainerDied","Data":"b8831688646578f918979c4f90d63f089bc601477c286e8feadb2fc90bb46215"}
Nov 02 20:09:17.735369 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:17.735304    4463 scope.go:122] "RemoveContainer" containerID="afb37a4cfedbff6c20f0827d8ae976f43b3857e0291ab2f02989bf9ea5f90b3c"
Nov 02 20:09:17.746463 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:17.746445    4463 scope.go:122] "RemoveContainer" containerID="afb37a4cfedbff6c20f0827d8ae976f43b3857e0291ab2f02989bf9ea5f90b3c"
Nov 02 20:09:17.746690 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:09:17.746670    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afb37a4cfedbff6c20f0827d8ae976f43b3857e0291ab2f02989bf9ea5f90b3c\": container with ID starting with afb37a4cfedbff6c20f0827d8ae976f43b3857e0291ab2f02989bf9ea5f90b3c not found: ID does not exist" containerID="afb37a4cfedbff6c20f0827d8ae976f43b3857e0291ab2f02989bf9ea5f90b3c"
Nov 02 20:09:17.746737 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:17.746700    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afb37a4cfedbff6c20f0827d8ae976f43b3857e0291ab2f02989bf9ea5f90b3c"} err="failed to get container status \"afb37a4cfedbff6c20f0827d8ae976f43b3857e0291ab2f02989bf9ea5f90b3c\": rpc error: code = NotFound desc = could not find container \"afb37a4cfedbff6c20f0827d8ae976f43b3857e0291ab2f02989bf9ea5f90b3c\": container with ID starting with afb37a4cfedbff6c20f0827d8ae976f43b3857e0291ab2f02989bf9ea5f90b3c not found: ID does not exist"
Nov 02 20:09:17.889740 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:17.889635    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-pgq8d"]
Nov 02 20:09:17.892945 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:17.892918    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-pgq8d"]
Nov 02 20:09:19.586945 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:19.586910    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ff8e962-f7e6-4d82-a818-261d21bdba26" path="/var/lib/kubelet/pods/6ff8e962-f7e6-4d82-a818-261d21bdba26/volumes"
Nov 02 20:09:20.727802 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:20.727771    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b45a2b-76df647b98-p92xh"
Nov 02 20:09:25.279365 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:25.279327    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 20:09:34.294888 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:09:34.294854    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 20:09:35.852962 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:35.852926    4463 generic.go:356] "Generic (PLEG): container finished" podID="7ab3e969-44bd-4d38-a066-005a66561670" containerID="cff56eeaf816c2d12a5f41d1891b424270e8974ddf47f80e6cc166ada88cae7a" exitCode=0
Nov 02 20:09:35.853506 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:35.853005    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b45a2b-openshift-default-8cddcbd8c-dq2sb" event={"ID":"7ab3e969-44bd-4d38-a066-005a66561670","Type":"ContainerDied","Data":"cff56eeaf816c2d12a5f41d1891b424270e8974ddf47f80e6cc166ada88cae7a"}
Nov 02 20:09:35.853506 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:35.853480    4463 scope.go:122] "RemoveContainer" containerID="cff56eeaf816c2d12a5f41d1891b424270e8974ddf47f80e6cc166ada88cae7a"
Nov 02 20:09:35.855062 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:35.855035    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:09:35.855166 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:35.855100    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:09:35.855166 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:35.855130    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:09:36.881501 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:36.881467    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b45a2b-openshift-default-8cddcbd8c-dq2sb" event={"ID":"7ab3e969-44bd-4d38-a066-005a66561670","Type":"ContainerStarted","Data":"ede9fb98a1670c48449f4e8389f28e31dca6959a76b96837d1aeea5153186455"}
Nov 02 20:09:37.170816 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:37.170726    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-b45a2b-openshift-default-8cddcbd8c-dq2sb"
Nov 02 20:09:37.174371 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:37.174340    4463 patch_prober.go:29] interesting pod/e2e-api-b-b45a2b-openshift-default-8cddcbd8c-dq2sb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Nov 02 20:09:37.174566 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:37.174540    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b45a2b-openshift-default-8cddcbd8c-dq2sb" podUID="7ab3e969-44bd-4d38-a066-005a66561670" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Nov 02 20:09:38.170040 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:38.169327    4463 patch_prober.go:29] interesting pod/e2e-api-b-b45a2b-openshift-default-8cddcbd8c-dq2sb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Nov 02 20:09:38.170040 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:38.169387    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b45a2b-openshift-default-8cddcbd8c-dq2sb" podUID="7ab3e969-44bd-4d38-a066-005a66561670" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Nov 02 20:09:39.167046 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:39.166349    4463 patch_prober.go:29] interesting pod/e2e-api-b-b45a2b-openshift-default-8cddcbd8c-dq2sb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Nov 02 20:09:39.167046 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:39.166419    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b45a2b-openshift-default-8cddcbd8c-dq2sb" podUID="7ab3e969-44bd-4d38-a066-005a66561670" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Nov 02 20:09:40.169038 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:40.164381    4463 patch_prober.go:29] interesting pod/e2e-api-b-b45a2b-openshift-default-8cddcbd8c-dq2sb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Nov 02 20:09:40.169038 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:40.164443    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b45a2b-openshift-default-8cddcbd8c-dq2sb" podUID="7ab3e969-44bd-4d38-a066-005a66561670" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Nov 02 20:09:41.154425 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:41.154385    4463 patch_prober.go:29] interesting pod/e2e-api-b-b45a2b-openshift-default-8cddcbd8c-dq2sb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Nov 02 20:09:41.154709 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:41.154679    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b45a2b-openshift-default-8cddcbd8c-dq2sb" podUID="7ab3e969-44bd-4d38-a066-005a66561670" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Nov 02 20:09:42.157296 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:42.157264    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-b45a2b-openshift-default-8cddcbd8c-dq2sb"
Nov 02 20:09:42.157678 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:42.157596    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-b45a2b-openshift-default-8cddcbd8c-dq2sb"
Nov 02 20:09:42.158290 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:42.158274    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-b45a2b-openshift-default-8cddcbd8c-dq2sb"
Nov 02 20:09:58.999936 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:58.999894    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-13e739/test-model-13e739-kserve-84bd6cf795-5hmft"]
Nov 02 20:09:59.000812 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:59.000778    4463 cpu_manager.go:414] "containerMap: removing container" podUID="6ff8e962-f7e6-4d82-a818-261d21bdba26" containerName="istio-proxy"
Nov 02 20:09:59.000812 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:59.000803    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ff8e962-f7e6-4d82-a818-261d21bdba26" containerName="istio-proxy"
Nov 02 20:09:59.000994 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:59.000968    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ff8e962-f7e6-4d82-a818-261d21bdba26" containerName="istio-proxy"
Nov 02 20:09:59.003357 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:59.003337    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-13e739/test-model-13e739-kserve-84bd6cf795-5hmft"
Nov 02 20:09:59.099381 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:59.099346    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-13e739\"/\"default-dockercfg-c568s\""
Nov 02 20:09:59.099381 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:59.099349    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-13e739\"/\"openshift-service-ca.crt\""
Nov 02 20:09:59.099381 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:59.099383    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-13e739\"/\"test-model-13e739-kserve-self-signed-certs\""
Nov 02 20:09:59.099606 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:59.099429    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-13e739\"/\"kube-root-ca.crt\""
Nov 02 20:09:59.102170 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:59.102149    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-13e739/test-model-13e739-kserve-84bd6cf795-5hmft"]
Nov 02 20:09:59.121218 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:59.121186    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x66wv\" (UniqueName: \"kubernetes.io/projected/3e95fad5-268c-4d06-866e-ed10c87a78d0-kube-api-access-x66wv\") pod \"test-model-13e739-kserve-84bd6cf795-5hmft\" (UID: \"3e95fad5-268c-4d06-866e-ed10c87a78d0\") " pod="ai-tenant-e2e-inf-b-13e739/test-model-13e739-kserve-84bd6cf795-5hmft"
Nov 02 20:09:59.121348 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:59.121268    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e95fad5-268c-4d06-866e-ed10c87a78d0-tmp-dir\") pod \"test-model-13e739-kserve-84bd6cf795-5hmft\" (UID: \"3e95fad5-268c-4d06-866e-ed10c87a78d0\") " pod="ai-tenant-e2e-inf-b-13e739/test-model-13e739-kserve-84bd6cf795-5hmft"
Nov 02 20:09:59.121426 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:59.121405    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e95fad5-268c-4d06-866e-ed10c87a78d0-model-cache\") pod \"test-model-13e739-kserve-84bd6cf795-5hmft\" (UID: \"3e95fad5-268c-4d06-866e-ed10c87a78d0\") " pod="ai-tenant-e2e-inf-b-13e739/test-model-13e739-kserve-84bd6cf795-5hmft"
Nov 02 20:09:59.121464 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:59.121439    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e95fad5-268c-4d06-866e-ed10c87a78d0-home\") pod \"test-model-13e739-kserve-84bd6cf795-5hmft\" (UID: \"3e95fad5-268c-4d06-866e-ed10c87a78d0\") " pod="ai-tenant-e2e-inf-b-13e739/test-model-13e739-kserve-84bd6cf795-5hmft"
Nov 02 20:09:59.121504 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:59.121462    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e95fad5-268c-4d06-866e-ed10c87a78d0-tls-certs\") pod \"test-model-13e739-kserve-84bd6cf795-5hmft\" (UID: \"3e95fad5-268c-4d06-866e-ed10c87a78d0\") " pod="ai-tenant-e2e-inf-b-13e739/test-model-13e739-kserve-84bd6cf795-5hmft"
Nov 02 20:09:59.121568 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:59.121554    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e95fad5-268c-4d06-866e-ed10c87a78d0-dshm\") pod \"test-model-13e739-kserve-84bd6cf795-5hmft\" (UID: \"3e95fad5-268c-4d06-866e-ed10c87a78d0\") " pod="ai-tenant-e2e-inf-b-13e739/test-model-13e739-kserve-84bd6cf795-5hmft"
Nov 02 20:09:59.222796 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:59.222759    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e95fad5-268c-4d06-866e-ed10c87a78d0-model-cache\") pod \"test-model-13e739-kserve-84bd6cf795-5hmft\" (UID: \"3e95fad5-268c-4d06-866e-ed10c87a78d0\") " pod="ai-tenant-e2e-inf-b-13e739/test-model-13e739-kserve-84bd6cf795-5hmft"
Nov 02 20:09:59.222796 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:59.222797    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e95fad5-268c-4d06-866e-ed10c87a78d0-home\") pod \"test-model-13e739-kserve-84bd6cf795-5hmft\" (UID: \"3e95fad5-268c-4d06-866e-ed10c87a78d0\") " pod="ai-tenant-e2e-inf-b-13e739/test-model-13e739-kserve-84bd6cf795-5hmft"
Nov 02 20:09:59.223092 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:59.222817    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e95fad5-268c-4d06-866e-ed10c87a78d0-tls-certs\") pod \"test-model-13e739-kserve-84bd6cf795-5hmft\" (UID: \"3e95fad5-268c-4d06-866e-ed10c87a78d0\") " pod="ai-tenant-e2e-inf-b-13e739/test-model-13e739-kserve-84bd6cf795-5hmft"
Nov 02 20:09:59.223092 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:59.222865    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e95fad5-268c-4d06-866e-ed10c87a78d0-dshm\") pod \"test-model-13e739-kserve-84bd6cf795-5hmft\" (UID: \"3e95fad5-268c-4d06-866e-ed10c87a78d0\") " pod="ai-tenant-e2e-inf-b-13e739/test-model-13e739-kserve-84bd6cf795-5hmft"
Nov 02 20:09:59.223092 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:59.222903    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x66wv\" (UniqueName: \"kubernetes.io/projected/3e95fad5-268c-4d06-866e-ed10c87a78d0-kube-api-access-x66wv\") pod \"test-model-13e739-kserve-84bd6cf795-5hmft\" (UID: \"3e95fad5-268c-4d06-866e-ed10c87a78d0\") " pod="ai-tenant-e2e-inf-b-13e739/test-model-13e739-kserve-84bd6cf795-5hmft"
Nov 02 20:09:59.223092 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:59.222963    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e95fad5-268c-4d06-866e-ed10c87a78d0-tmp-dir\") pod \"test-model-13e739-kserve-84bd6cf795-5hmft\" (UID: \"3e95fad5-268c-4d06-866e-ed10c87a78d0\") " pod="ai-tenant-e2e-inf-b-13e739/test-model-13e739-kserve-84bd6cf795-5hmft"
Nov 02 20:09:59.223321 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:59.223256    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e95fad5-268c-4d06-866e-ed10c87a78d0-home\") pod \"test-model-13e739-kserve-84bd6cf795-5hmft\" (UID: \"3e95fad5-268c-4d06-866e-ed10c87a78d0\") " pod="ai-tenant-e2e-inf-b-13e739/test-model-13e739-kserve-84bd6cf795-5hmft"
Nov 02 20:09:59.223321 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:59.223308    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e95fad5-268c-4d06-866e-ed10c87a78d0-model-cache\") pod \"test-model-13e739-kserve-84bd6cf795-5hmft\" (UID: \"3e95fad5-268c-4d06-866e-ed10c87a78d0\") " pod="ai-tenant-e2e-inf-b-13e739/test-model-13e739-kserve-84bd6cf795-5hmft"
Nov 02 20:09:59.223404 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:59.223333    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e95fad5-268c-4d06-866e-ed10c87a78d0-tmp-dir\") pod \"test-model-13e739-kserve-84bd6cf795-5hmft\" (UID: \"3e95fad5-268c-4d06-866e-ed10c87a78d0\") " pod="ai-tenant-e2e-inf-b-13e739/test-model-13e739-kserve-84bd6cf795-5hmft"
Nov 02 20:09:59.225212 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:59.225192    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e95fad5-268c-4d06-866e-ed10c87a78d0-dshm\") pod \"test-model-13e739-kserve-84bd6cf795-5hmft\" (UID: \"3e95fad5-268c-4d06-866e-ed10c87a78d0\") " pod="ai-tenant-e2e-inf-b-13e739/test-model-13e739-kserve-84bd6cf795-5hmft"
Nov 02 20:09:59.225515 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:59.225495    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e95fad5-268c-4d06-866e-ed10c87a78d0-tls-certs\") pod \"test-model-13e739-kserve-84bd6cf795-5hmft\" (UID: \"3e95fad5-268c-4d06-866e-ed10c87a78d0\") " pod="ai-tenant-e2e-inf-b-13e739/test-model-13e739-kserve-84bd6cf795-5hmft"
Nov 02 20:09:59.275449 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:59.275402    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x66wv\" (UniqueName: \"kubernetes.io/projected/3e95fad5-268c-4d06-866e-ed10c87a78d0-kube-api-access-x66wv\") pod \"test-model-13e739-kserve-84bd6cf795-5hmft\" (UID: \"3e95fad5-268c-4d06-866e-ed10c87a78d0\") " pod="ai-tenant-e2e-inf-b-13e739/test-model-13e739-kserve-84bd6cf795-5hmft"
Nov 02 20:09:59.314901 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:59.314873    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-13e739/test-model-13e739-kserve-84bd6cf795-5hmft"
Nov 02 20:09:59.681209 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:09:59.681183    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-13e739/test-model-13e739-kserve-84bd6cf795-5hmft"]
Nov 02 20:09:59.682282 ip-10-0-133-9 kubenswrapper[4463]: W1102 20:09:59.682249    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3e95fad5_268c_4d06_866e_ed10c87a78d0.slice/crio-b9c5a7562a3f633aa04fc72f71f79532eedecc6e48f4f40ba27bb46cda32eeed WatchSource:0}: Error finding container b9c5a7562a3f633aa04fc72f71f79532eedecc6e48f4f40ba27bb46cda32eeed: Status 404 returned error can't find the container with id b9c5a7562a3f633aa04fc72f71f79532eedecc6e48f4f40ba27bb46cda32eeed
Nov 02 20:10:00.020780 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:10:00.020700    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-13e739/test-model-13e739-kserve-84bd6cf795-5hmft" event={"ID":"3e95fad5-268c-4d06-866e-ed10c87a78d0","Type":"ContainerStarted","Data":"f5460c2f88a7f2a85259edc56efdbaa16162a8c9bf44a578ab2a51cc568b9cbd"}
Nov 02 20:10:00.020780 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:10:00.020741    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-13e739/test-model-13e739-kserve-84bd6cf795-5hmft" event={"ID":"3e95fad5-268c-4d06-866e-ed10c87a78d0","Type":"ContainerStarted","Data":"b9c5a7562a3f633aa04fc72f71f79532eedecc6e48f4f40ba27bb46cda32eeed"}
Nov 02 20:10:00.126607 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:10:00.126559    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-13e739/test-model-13e739-kserve-84bd6cf795-5hmft" podStartSLOduration=2.126545968 podStartE2EDuration="2.126545968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:09:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:10:00.125756663 +0000 UTC m=+2506.732790432" watchObservedRunningTime="2026-11-02 20:10:00.126545968 +0000 UTC m=+2506.733579740"
Nov 02 20:10:00.315578 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:10:00.315499    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-13e739/test-model-13e739-kserve-84bd6cf795-5hmft"
Nov 02 20:10:04.294177 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:10:04.294139    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 20:10:09.315485 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:10:09.315445    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-13e739/test-model-13e739-kserve-84bd6cf795-5hmft"
Nov 02 20:10:09.328257 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:10:09.328231    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-13e739/test-model-13e739-kserve-84bd6cf795-5hmft"
Nov 02 20:10:10.092901 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:10:10.092869    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-13e739/test-model-13e739-kserve-84bd6cf795-5hmft"
Nov 02 20:10:12.860394 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:10:12.860362    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 20:10:34.294677 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:10:34.294638    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 20:11:04.294981 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:11:04.294943    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 20:11:19.357977 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:11:19.357938    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 20:11:34.294171 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:11:34.294132    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 20:11:58.408741 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:11:58.408706    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jzjj4"]
Nov 02 20:11:58.409297 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:11:58.408891    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jzjj4" podUID="73818a40-7ba5-410c-aa9f-f6833bd95623" containerName="istio-proxy" containerID="cri-o://2cd00b7707e73cf906c4db1983024a52404fd53081a00c64c4f7bd1e1872679c" gracePeriod=30
Nov 02 20:12:01.873767 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:12:01.873732    4463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-jzjj4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 20:12:01.874181 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:12:01.873794    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jzjj4" podUID="73818a40-7ba5-410c-aa9f-f6833bd95623" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 20:12:03.688999 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:12:03.688944    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jzjj4"
Nov 02 20:12:03.775701 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:12:03.775667    4463 generic.go:356] "Generic (PLEG): container finished" podID="73818a40-7ba5-410c-aa9f-f6833bd95623" containerID="2cd00b7707e73cf906c4db1983024a52404fd53081a00c64c4f7bd1e1872679c" exitCode=0
Nov 02 20:12:03.775829 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:12:03.775712    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jzjj4"
Nov 02 20:12:03.775829 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:12:03.775751    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jzjj4" event={"ID":"73818a40-7ba5-410c-aa9f-f6833bd95623","Type":"ContainerDied","Data":"2cd00b7707e73cf906c4db1983024a52404fd53081a00c64c4f7bd1e1872679c"}
Nov 02 20:12:03.775829 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:12:03.775804    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jzjj4" event={"ID":"73818a40-7ba5-410c-aa9f-f6833bd95623","Type":"ContainerDied","Data":"730d71192490c729996907aa748d0f4727256fd69814847c6f0afb8ee73ecb3e"}
Nov 02 20:12:03.775944 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:12:03.775830    4463 scope.go:122] "RemoveContainer" containerID="2cd00b7707e73cf906c4db1983024a52404fd53081a00c64c4f7bd1e1872679c"
Nov 02 20:12:03.785261 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:12:03.785241    4463 scope.go:122] "RemoveContainer" containerID="2cd00b7707e73cf906c4db1983024a52404fd53081a00c64c4f7bd1e1872679c"
Nov 02 20:12:03.785518 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:12:03.785501    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cd00b7707e73cf906c4db1983024a52404fd53081a00c64c4f7bd1e1872679c\": container with ID starting with 2cd00b7707e73cf906c4db1983024a52404fd53081a00c64c4f7bd1e1872679c not found: ID does not exist" containerID="2cd00b7707e73cf906c4db1983024a52404fd53081a00c64c4f7bd1e1872679c"
Nov 02 20:12:03.785570 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:12:03.785525    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cd00b7707e73cf906c4db1983024a52404fd53081a00c64c4f7bd1e1872679c"} err="failed to get container status \"2cd00b7707e73cf906c4db1983024a52404fd53081a00c64c4f7bd1e1872679c\": rpc error: code = NotFound desc = could not find container \"2cd00b7707e73cf906c4db1983024a52404fd53081a00c64c4f7bd1e1872679c\": container with ID starting with 2cd00b7707e73cf906c4db1983024a52404fd53081a00c64c4f7bd1e1872679c not found: ID does not exist"
Nov 02 20:12:03.816460 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:12:03.816435    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73818a40-7ba5-410c-aa9f-f6833bd95623-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73818a40-7ba5-410c-aa9f-f6833bd95623-workload-certs\") pod \"73818a40-7ba5-410c-aa9f-f6833bd95623\" (UID: \"73818a40-7ba5-410c-aa9f-f6833bd95623\") "
Nov 02 20:12:03.816561 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:12:03.816482    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/73818a40-7ba5-410c-aa9f-f6833bd95623-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73818a40-7ba5-410c-aa9f-f6833bd95623-istio-podinfo\") pod \"73818a40-7ba5-410c-aa9f-f6833bd95623\" (UID: \"73818a40-7ba5-410c-aa9f-f6833bd95623\") "
Nov 02 20:12:03.816561 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:12:03.816521    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73818a40-7ba5-410c-aa9f-f6833bd95623-kube-api-access-w22c9\" (UniqueName: \"kubernetes.io/projected/73818a40-7ba5-410c-aa9f-f6833bd95623-kube-api-access-w22c9\") pod \"73818a40-7ba5-410c-aa9f-f6833bd95623\" (UID: \"73818a40-7ba5-410c-aa9f-f6833bd95623\") "
Nov 02 20:12:03.816561 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:12:03.816539    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73818a40-7ba5-410c-aa9f-f6833bd95623-istio-token\" (UniqueName: \"kubernetes.io/projected/73818a40-7ba5-410c-aa9f-f6833bd95623-istio-token\") pod \"73818a40-7ba5-410c-aa9f-f6833bd95623\" (UID: \"73818a40-7ba5-410c-aa9f-f6833bd95623\") "
Nov 02 20:12:03.816736 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:12:03.816568    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73818a40-7ba5-410c-aa9f-f6833bd95623-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73818a40-7ba5-410c-aa9f-f6833bd95623-istio-envoy\") pod \"73818a40-7ba5-410c-aa9f-f6833bd95623\" (UID: \"73818a40-7ba5-410c-aa9f-f6833bd95623\") "
Nov 02 20:12:03.816736 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:12:03.816635    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73818a40-7ba5-410c-aa9f-f6833bd95623-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73818a40-7ba5-410c-aa9f-f6833bd95623-credential-socket\") pod \"73818a40-7ba5-410c-aa9f-f6833bd95623\" (UID: \"73818a40-7ba5-410c-aa9f-f6833bd95623\") "
Nov 02 20:12:03.816736 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:12:03.816670    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73818a40-7ba5-410c-aa9f-f6833bd95623-workload-certs" pod "73818a40-7ba5-410c-aa9f-f6833bd95623" (UID: "73818a40-7ba5-410c-aa9f-f6833bd95623"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:12:03.816736 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:12:03.816677    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73818a40-7ba5-410c-aa9f-f6833bd95623-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73818a40-7ba5-410c-aa9f-f6833bd95623-workload-socket\") pod \"73818a40-7ba5-410c-aa9f-f6833bd95623\" (UID: \"73818a40-7ba5-410c-aa9f-f6833bd95623\") "
Nov 02 20:12:03.816947 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:12:03.816806    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73818a40-7ba5-410c-aa9f-f6833bd95623-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73818a40-7ba5-410c-aa9f-f6833bd95623-istio-data\") pod \"73818a40-7ba5-410c-aa9f-f6833bd95623\" (UID: \"73818a40-7ba5-410c-aa9f-f6833bd95623\") "
Nov 02 20:12:03.816947 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:12:03.816923    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73818a40-7ba5-410c-aa9f-f6833bd95623-workload-socket" pod "73818a40-7ba5-410c-aa9f-f6833bd95623" (UID: "73818a40-7ba5-410c-aa9f-f6833bd95623"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:12:03.817100 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:12:03.816943    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73818a40-7ba5-410c-aa9f-f6833bd95623-credential-socket" pod "73818a40-7ba5-410c-aa9f-f6833bd95623" (UID: "73818a40-7ba5-410c-aa9f-f6833bd95623"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:12:03.817100 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:12:03.817081    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73818a40-7ba5-410c-aa9f-f6833bd95623-istio-data" pod "73818a40-7ba5-410c-aa9f-f6833bd95623" (UID: "73818a40-7ba5-410c-aa9f-f6833bd95623"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:12:03.817218 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:12:03.817160    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73818a40-7ba5-410c-aa9f-f6833bd95623-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73818a40-7ba5-410c-aa9f-f6833bd95623-istiod-ca-cert\") pod \"73818a40-7ba5-410c-aa9f-f6833bd95623\" (UID: \"73818a40-7ba5-410c-aa9f-f6833bd95623\") "
Nov 02 20:12:03.817598 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:12:03.817571    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73818a40-7ba5-410c-aa9f-f6833bd95623-istiod-ca-cert" pod "73818a40-7ba5-410c-aa9f-f6833bd95623" (UID: "73818a40-7ba5-410c-aa9f-f6833bd95623"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:12:03.817897 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:12:03.817876    4463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73818a40-7ba5-410c-aa9f-f6833bd95623-istio-data\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:12:03.817977 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:12:03.817903    4463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73818a40-7ba5-410c-aa9f-f6833bd95623-istiod-ca-cert\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:12:03.817977 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:12:03.817921    4463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73818a40-7ba5-410c-aa9f-f6833bd95623-workload-certs\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:12:03.817977 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:12:03.817935    4463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73818a40-7ba5-410c-aa9f-f6833bd95623-credential-socket\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:12:03.817977 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:12:03.817949    4463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73818a40-7ba5-410c-aa9f-f6833bd95623-workload-socket\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:12:03.818741 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:12:03.818720    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73818a40-7ba5-410c-aa9f-f6833bd95623-istio-token" pod "73818a40-7ba5-410c-aa9f-f6833bd95623" (UID: "73818a40-7ba5-410c-aa9f-f6833bd95623"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:12:03.818827 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:12:03.818742    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73818a40-7ba5-410c-aa9f-f6833bd95623-istio-envoy" pod "73818a40-7ba5-410c-aa9f-f6833bd95623" (UID: "73818a40-7ba5-410c-aa9f-f6833bd95623"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:12:03.818867 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:12:03.818832    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73818a40-7ba5-410c-aa9f-f6833bd95623-kube-api-access-w22c9" pod "73818a40-7ba5-410c-aa9f-f6833bd95623" (UID: "73818a40-7ba5-410c-aa9f-f6833bd95623"). InnerVolumeSpecName "kube-api-access-w22c9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:12:03.818867 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:12:03.818860    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/73818a40-7ba5-410c-aa9f-f6833bd95623-istio-podinfo" pod "73818a40-7ba5-410c-aa9f-f6833bd95623" (UID: "73818a40-7ba5-410c-aa9f-f6833bd95623"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:12:03.919076 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:12:03.919049    4463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73818a40-7ba5-410c-aa9f-f6833bd95623-istio-envoy\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:12:03.919076 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:12:03.919073    4463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73818a40-7ba5-410c-aa9f-f6833bd95623-istio-podinfo\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:12:03.919257 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:12:03.919083    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w22c9\" (UniqueName: \"kubernetes.io/projected/73818a40-7ba5-410c-aa9f-f6833bd95623-kube-api-access-w22c9\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:12:03.919257 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:12:03.919093    4463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73818a40-7ba5-410c-aa9f-f6833bd95623-istio-token\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:12:04.194939 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:12:04.194908    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jzjj4"]
Nov 02 20:12:04.197720 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:12:04.197695    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jzjj4"]
Nov 02 20:12:04.298151 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:12:04.298117    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 20:12:05.586781 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:12:05.586747    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73818a40-7ba5-410c-aa9f-f6833bd95623" path="/var/lib/kubelet/pods/73818a40-7ba5-410c-aa9f-f6833bd95623/volumes"
Nov 02 20:12:09.454647 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:12:09.454605    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 20:12:34.264712 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:12:34.264635    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1db0e4-6b545bbdb7-nkvgn"]
Nov 02 20:12:34.265285 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:12:34.264824    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1db0e4-6b545bbdb7-nkvgn" podUID="dc539282-391f-4350-8c34-95084fe87e65" containerName="maas-api" containerID="cri-o://70fcc06568199b064f4c4eb1b0e7a7d1c416fb659df416777c6c5c40e1d69464" gracePeriod=30
Nov 02 20:12:34.295682 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:12:34.295647    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 20:12:34.540377 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:12:34.540351    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1db0e4-6b545bbdb7-nkvgn"
Nov 02 20:12:34.652379 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:12:34.652349    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc539282-391f-4350-8c34-95084fe87e65-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc539282-391f-4350-8c34-95084fe87e65-metrics-certs\") pod \"dc539282-391f-4350-8c34-95084fe87e65\" (UID: \"dc539282-391f-4350-8c34-95084fe87e65\") "
Nov 02 20:12:34.652683 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:12:34.652663    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc539282-391f-4350-8c34-95084fe87e65-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc539282-391f-4350-8c34-95084fe87e65-maas-api-tls\") pod \"dc539282-391f-4350-8c34-95084fe87e65\" (UID: \"dc539282-391f-4350-8c34-95084fe87e65\") "
Nov 02 20:12:34.652863 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:12:34.652846    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc539282-391f-4350-8c34-95084fe87e65-kube-api-access-44bq8\" (UniqueName: \"kubernetes.io/projected/dc539282-391f-4350-8c34-95084fe87e65-kube-api-access-44bq8\") pod \"dc539282-391f-4350-8c34-95084fe87e65\" (UID: \"dc539282-391f-4350-8c34-95084fe87e65\") "
Nov 02 20:12:34.653087 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:12:34.653067    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc539282-391f-4350-8c34-95084fe87e65-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc539282-391f-4350-8c34-95084fe87e65-openshift-service-ca\") pod \"dc539282-391f-4350-8c34-95084fe87e65\" (UID: \"dc539282-391f-4350-8c34-95084fe87e65\") "
Nov 02 20:12:34.653732 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:12:34.653609    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc539282-391f-4350-8c34-95084fe87e65-openshift-service-ca" pod "dc539282-391f-4350-8c34-95084fe87e65" (UID: "dc539282-391f-4350-8c34-95084fe87e65"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:12:34.659717 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:12:34.659690    4463 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc539282-391f-4350-8c34-95084fe87e65-openshift-service-ca\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:12:34.661482 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:12:34.661447    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc539282-391f-4350-8c34-95084fe87e65-kube-api-access-44bq8" pod "dc539282-391f-4350-8c34-95084fe87e65" (UID: "dc539282-391f-4350-8c34-95084fe87e65"). InnerVolumeSpecName "kube-api-access-44bq8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:12:34.670459 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:12:34.670426    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc539282-391f-4350-8c34-95084fe87e65-maas-api-tls" pod "dc539282-391f-4350-8c34-95084fe87e65" (UID: "dc539282-391f-4350-8c34-95084fe87e65"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:12:34.670572 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:12:34.670455    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc539282-391f-4350-8c34-95084fe87e65-metrics-certs" pod "dc539282-391f-4350-8c34-95084fe87e65" (UID: "dc539282-391f-4350-8c34-95084fe87e65"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:12:34.761221 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:12:34.761192    4463 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc539282-391f-4350-8c34-95084fe87e65-metrics-certs\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:12:34.761221 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:12:34.761217    4463 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc539282-391f-4350-8c34-95084fe87e65-maas-api-tls\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:12:34.761392 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:12:34.761227    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44bq8\" (UniqueName: \"kubernetes.io/projected/dc539282-391f-4350-8c34-95084fe87e65-kube-api-access-44bq8\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:12:34.958371 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:12:34.958339    4463 generic.go:356] "Generic (PLEG): container finished" podID="dc539282-391f-4350-8c34-95084fe87e65" containerID="70fcc06568199b064f4c4eb1b0e7a7d1c416fb659df416777c6c5c40e1d69464" exitCode=0
Nov 02 20:12:34.958540 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:12:34.958395    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1db0e4-6b545bbdb7-nkvgn"
Nov 02 20:12:34.958540 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:12:34.958424    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1db0e4-6b545bbdb7-nkvgn" event={"ID":"dc539282-391f-4350-8c34-95084fe87e65","Type":"ContainerDied","Data":"70fcc06568199b064f4c4eb1b0e7a7d1c416fb659df416777c6c5c40e1d69464"}
Nov 02 20:12:34.958540 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:12:34.958478    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1db0e4-6b545bbdb7-nkvgn" event={"ID":"dc539282-391f-4350-8c34-95084fe87e65","Type":"ContainerDied","Data":"db0b7b09bcfcfa1b0173428783d124276c8dfe1a48ed15b9b39f2226cc471baf"}
Nov 02 20:12:34.958540 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:12:34.958501    4463 scope.go:122] "RemoveContainer" containerID="70fcc06568199b064f4c4eb1b0e7a7d1c416fb659df416777c6c5c40e1d69464"
Nov 02 20:12:34.969134 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:12:34.969116    4463 scope.go:122] "RemoveContainer" containerID="70fcc06568199b064f4c4eb1b0e7a7d1c416fb659df416777c6c5c40e1d69464"
Nov 02 20:12:34.969387 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:12:34.969369    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70fcc06568199b064f4c4eb1b0e7a7d1c416fb659df416777c6c5c40e1d69464\": container with ID starting with 70fcc06568199b064f4c4eb1b0e7a7d1c416fb659df416777c6c5c40e1d69464 not found: ID does not exist" containerID="70fcc06568199b064f4c4eb1b0e7a7d1c416fb659df416777c6c5c40e1d69464"
Nov 02 20:12:34.969423 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:12:34.969396    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70fcc06568199b064f4c4eb1b0e7a7d1c416fb659df416777c6c5c40e1d69464"} err="failed to get container status \"70fcc06568199b064f4c4eb1b0e7a7d1c416fb659df416777c6c5c40e1d69464\": rpc error: code = NotFound desc = could not find container \"70fcc06568199b064f4c4eb1b0e7a7d1c416fb659df416777c6c5c40e1d69464\": container with ID starting with 70fcc06568199b064f4c4eb1b0e7a7d1c416fb659df416777c6c5c40e1d69464 not found: ID does not exist"
Nov 02 20:12:35.110643 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:12:35.110612    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1db0e4-6b545bbdb7-nkvgn"]
Nov 02 20:12:35.160492 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:12:35.160443    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1db0e4-6b545bbdb7-nkvgn"]
Nov 02 20:12:35.586465 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:12:35.586435    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc539282-391f-4350-8c34-95084fe87e65" path="/var/lib/kubelet/pods/dc539282-391f-4350-8c34-95084fe87e65/volumes"
Nov 02 20:13:00.690762 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:13:00.690729    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 20:13:04.294401 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:13:04.294364    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 20:13:34.294457 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:13:34.294417    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 20:13:50.439540 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:13:50.439507    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 20:14:04.294483 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:14:04.294448    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 20:14:14.394161 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:14:14.394122    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-hqc7w"]
Nov 02 20:14:14.394566 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:14:14.394310    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-hqc7w" podUID="9f6ba8f5-7ed9-4807-8435-6794197fee7c" containerName="istio-proxy" containerID="cri-o://3ac50a5aa4058ba8746476e950d0eeae572205867e243451c92c523c21a29942" gracePeriod=30
Nov 02 20:14:14.640357 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:14:14.640319    4463 patch_prober.go:29] interesting pod/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-hqc7w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 20:14:14.640521 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:14:14.640386    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-hqc7w" podUID="9f6ba8f5-7ed9-4807-8435-6794197fee7c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 20:14:19.573519 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:14:19.573481    4463 generic.go:356] "Generic (PLEG): container finished" podID="9f6ba8f5-7ed9-4807-8435-6794197fee7c" containerID="3ac50a5aa4058ba8746476e950d0eeae572205867e243451c92c523c21a29942" exitCode=0
Nov 02 20:14:19.573882 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:14:19.573557    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-hqc7w" event={"ID":"9f6ba8f5-7ed9-4807-8435-6794197fee7c","Type":"ContainerDied","Data":"3ac50a5aa4058ba8746476e950d0eeae572205867e243451c92c523c21a29942"}
Nov 02 20:14:19.676813 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:14:19.676792    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-hqc7w"
Nov 02 20:14:19.806552 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:14:19.806522    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f6ba8f5-7ed9-4807-8435-6794197fee7c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f6ba8f5-7ed9-4807-8435-6794197fee7c-istio-envoy\") pod \"9f6ba8f5-7ed9-4807-8435-6794197fee7c\" (UID: \"9f6ba8f5-7ed9-4807-8435-6794197fee7c\") "
Nov 02 20:14:19.806721 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:14:19.806582    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f6ba8f5-7ed9-4807-8435-6794197fee7c-kube-api-access-8dfmc\" (UniqueName: \"kubernetes.io/projected/9f6ba8f5-7ed9-4807-8435-6794197fee7c-kube-api-access-8dfmc\") pod \"9f6ba8f5-7ed9-4807-8435-6794197fee7c\" (UID: \"9f6ba8f5-7ed9-4807-8435-6794197fee7c\") "
Nov 02 20:14:19.806721 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:14:19.806642    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f6ba8f5-7ed9-4807-8435-6794197fee7c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f6ba8f5-7ed9-4807-8435-6794197fee7c-credential-socket\") pod \"9f6ba8f5-7ed9-4807-8435-6794197fee7c\" (UID: \"9f6ba8f5-7ed9-4807-8435-6794197fee7c\") "
Nov 02 20:14:19.806721 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:14:19.806659    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f6ba8f5-7ed9-4807-8435-6794197fee7c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f6ba8f5-7ed9-4807-8435-6794197fee7c-istio-data\") pod \"9f6ba8f5-7ed9-4807-8435-6794197fee7c\" (UID: \"9f6ba8f5-7ed9-4807-8435-6794197fee7c\") "
Nov 02 20:14:19.806721 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:14:19.806711    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9f6ba8f5-7ed9-4807-8435-6794197fee7c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f6ba8f5-7ed9-4807-8435-6794197fee7c-istio-podinfo\") pod \"9f6ba8f5-7ed9-4807-8435-6794197fee7c\" (UID: \"9f6ba8f5-7ed9-4807-8435-6794197fee7c\") "
Nov 02 20:14:19.806943 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:14:19.806755    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f6ba8f5-7ed9-4807-8435-6794197fee7c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f6ba8f5-7ed9-4807-8435-6794197fee7c-istiod-ca-cert\") pod \"9f6ba8f5-7ed9-4807-8435-6794197fee7c\" (UID: \"9f6ba8f5-7ed9-4807-8435-6794197fee7c\") "
Nov 02 20:14:19.806943 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:14:19.806813    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f6ba8f5-7ed9-4807-8435-6794197fee7c-istio-token\" (UniqueName: \"kubernetes.io/projected/9f6ba8f5-7ed9-4807-8435-6794197fee7c-istio-token\") pod \"9f6ba8f5-7ed9-4807-8435-6794197fee7c\" (UID: \"9f6ba8f5-7ed9-4807-8435-6794197fee7c\") "
Nov 02 20:14:19.806943 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:14:19.806849    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f6ba8f5-7ed9-4807-8435-6794197fee7c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f6ba8f5-7ed9-4807-8435-6794197fee7c-workload-socket\") pod \"9f6ba8f5-7ed9-4807-8435-6794197fee7c\" (UID: \"9f6ba8f5-7ed9-4807-8435-6794197fee7c\") "
Nov 02 20:14:19.806943 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:14:19.806878    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f6ba8f5-7ed9-4807-8435-6794197fee7c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f6ba8f5-7ed9-4807-8435-6794197fee7c-workload-certs\") pod \"9f6ba8f5-7ed9-4807-8435-6794197fee7c\" (UID: \"9f6ba8f5-7ed9-4807-8435-6794197fee7c\") "
Nov 02 20:14:19.807189 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:14:19.806955    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f6ba8f5-7ed9-4807-8435-6794197fee7c-credential-socket" pod "9f6ba8f5-7ed9-4807-8435-6794197fee7c" (UID: "9f6ba8f5-7ed9-4807-8435-6794197fee7c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:14:19.807189 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:14:19.807054    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f6ba8f5-7ed9-4807-8435-6794197fee7c-istio-data" pod "9f6ba8f5-7ed9-4807-8435-6794197fee7c" (UID: "9f6ba8f5-7ed9-4807-8435-6794197fee7c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:14:19.807307 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:14:19.807258    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f6ba8f5-7ed9-4807-8435-6794197fee7c-workload-socket" pod "9f6ba8f5-7ed9-4807-8435-6794197fee7c" (UID: "9f6ba8f5-7ed9-4807-8435-6794197fee7c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:14:19.807307 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:14:19.807288    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f6ba8f5-7ed9-4807-8435-6794197fee7c-istiod-ca-cert" pod "9f6ba8f5-7ed9-4807-8435-6794197fee7c" (UID: "9f6ba8f5-7ed9-4807-8435-6794197fee7c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:14:19.807470 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:14:19.807448    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f6ba8f5-7ed9-4807-8435-6794197fee7c-workload-certs" pod "9f6ba8f5-7ed9-4807-8435-6794197fee7c" (UID: "9f6ba8f5-7ed9-4807-8435-6794197fee7c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:14:19.807608 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:14:19.807591    4463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f6ba8f5-7ed9-4807-8435-6794197fee7c-credential-socket\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:14:19.807608 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:14:19.807607    4463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f6ba8f5-7ed9-4807-8435-6794197fee7c-istio-data\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:14:19.807732 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:14:19.807616    4463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f6ba8f5-7ed9-4807-8435-6794197fee7c-istiod-ca-cert\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:14:19.807732 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:14:19.807625    4463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f6ba8f5-7ed9-4807-8435-6794197fee7c-workload-socket\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:14:19.807732 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:14:19.807634    4463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f6ba8f5-7ed9-4807-8435-6794197fee7c-workload-certs\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:14:19.808946 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:14:19.808921    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f6ba8f5-7ed9-4807-8435-6794197fee7c-kube-api-access-8dfmc" pod "9f6ba8f5-7ed9-4807-8435-6794197fee7c" (UID: "9f6ba8f5-7ed9-4807-8435-6794197fee7c"). InnerVolumeSpecName "kube-api-access-8dfmc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:14:19.808946 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:14:19.808931    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f6ba8f5-7ed9-4807-8435-6794197fee7c-istio-envoy" pod "9f6ba8f5-7ed9-4807-8435-6794197fee7c" (UID: "9f6ba8f5-7ed9-4807-8435-6794197fee7c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:14:19.809333 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:14:19.809310    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9f6ba8f5-7ed9-4807-8435-6794197fee7c-istio-podinfo" pod "9f6ba8f5-7ed9-4807-8435-6794197fee7c" (UID: "9f6ba8f5-7ed9-4807-8435-6794197fee7c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:14:19.809415 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:14:19.809333    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f6ba8f5-7ed9-4807-8435-6794197fee7c-istio-token" pod "9f6ba8f5-7ed9-4807-8435-6794197fee7c" (UID: "9f6ba8f5-7ed9-4807-8435-6794197fee7c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:14:19.909135 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:14:19.909104    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dfmc\" (UniqueName: \"kubernetes.io/projected/9f6ba8f5-7ed9-4807-8435-6794197fee7c-kube-api-access-8dfmc\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:14:19.909135 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:14:19.909129    4463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f6ba8f5-7ed9-4807-8435-6794197fee7c-istio-podinfo\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:14:19.909135 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:14:19.909139    4463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f6ba8f5-7ed9-4807-8435-6794197fee7c-istio-token\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:14:19.909400 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:14:19.909148    4463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f6ba8f5-7ed9-4807-8435-6794197fee7c-istio-envoy\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:14:20.582567 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:14:20.582528    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-hqc7w" event={"ID":"9f6ba8f5-7ed9-4807-8435-6794197fee7c","Type":"ContainerDied","Data":"c1f1b84e23a42ad93320f827d3bcc0bfa0315a6032390bf0b0a986a1a918e8a1"}
Nov 02 20:14:20.582988 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:14:20.582596    4463 scope.go:122] "RemoveContainer" containerID="3ac50a5aa4058ba8746476e950d0eeae572205867e243451c92c523c21a29942"
Nov 02 20:14:20.582988 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:14:20.582544    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-hqc7w"
Nov 02 20:14:20.736562 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:14:20.736521    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-hqc7w"]
Nov 02 20:14:20.782182 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:14:20.782149    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-hqc7w"]
Nov 02 20:14:21.585789 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:14:21.585752    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f6ba8f5-7ed9-4807-8435-6794197fee7c" path="/var/lib/kubelet/pods/9f6ba8f5-7ed9-4807-8435-6794197fee7c/volumes"
Nov 02 20:14:33.402693 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:14:33.402654    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-838bc1-869f9bc899-68j9k"]
Nov 02 20:14:34.294766 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:14:34.294730    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 20:14:38.403487 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:14:38.403440    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-838bc1-869f9bc899-68j9k" podUID="ba35ca07-1e6a-4098-9d52-afa7a68c8805" containerName="payload-processing" containerID="cri-o://240b70899f3e0c96151de7500f3153164940275f2794474ef6c6d09442074946" gracePeriod=25
Nov 02 20:14:38.673433 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:14:38.673409    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-838bc1-869f9bc899-68j9k"
Nov 02 20:14:38.683278 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:14:38.683251    4463 generic.go:356] "Generic (PLEG): container finished" podID="ba35ca07-1e6a-4098-9d52-afa7a68c8805" containerID="240b70899f3e0c96151de7500f3153164940275f2794474ef6c6d09442074946" exitCode=0
Nov 02 20:14:38.683418 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:14:38.683290    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-838bc1-869f9bc899-68j9k"
Nov 02 20:14:38.683418 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:14:38.683305    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-838bc1-869f9bc899-68j9k" event={"ID":"ba35ca07-1e6a-4098-9d52-afa7a68c8805","Type":"ContainerDied","Data":"240b70899f3e0c96151de7500f3153164940275f2794474ef6c6d09442074946"}
Nov 02 20:14:38.683418 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:14:38.683345    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-838bc1-869f9bc899-68j9k" event={"ID":"ba35ca07-1e6a-4098-9d52-afa7a68c8805","Type":"ContainerDied","Data":"3dae4a43e7c2e692ff386d260ccecebc55dc7700192983f9d279efe5ad62ef5b"}
Nov 02 20:14:38.683418 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:14:38.683366    4463 scope.go:122] "RemoveContainer" containerID="240b70899f3e0c96151de7500f3153164940275f2794474ef6c6d09442074946"
Nov 02 20:14:38.693003 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:14:38.692981    4463 scope.go:122] "RemoveContainer" containerID="240b70899f3e0c96151de7500f3153164940275f2794474ef6c6d09442074946"
Nov 02 20:14:38.693308 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:14:38.693278    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"240b70899f3e0c96151de7500f3153164940275f2794474ef6c6d09442074946\": container with ID starting with 240b70899f3e0c96151de7500f3153164940275f2794474ef6c6d09442074946 not found: ID does not exist" containerID="240b70899f3e0c96151de7500f3153164940275f2794474ef6c6d09442074946"
Nov 02 20:14:38.693423 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:14:38.693317    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"240b70899f3e0c96151de7500f3153164940275f2794474ef6c6d09442074946"} err="failed to get container status \"240b70899f3e0c96151de7500f3153164940275f2794474ef6c6d09442074946\": rpc error: code = NotFound desc = could not find container \"240b70899f3e0c96151de7500f3153164940275f2794474ef6c6d09442074946\": container with ID starting with 240b70899f3e0c96151de7500f3153164940275f2794474ef6c6d09442074946 not found: ID does not exist"
Nov 02 20:14:38.812925 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:14:38.812891    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba35ca07-1e6a-4098-9d52-afa7a68c8805-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba35ca07-1e6a-4098-9d52-afa7a68c8805-plugins-config-volume\") pod \"ba35ca07-1e6a-4098-9d52-afa7a68c8805\" (UID: \"ba35ca07-1e6a-4098-9d52-afa7a68c8805\") "
Nov 02 20:14:38.813212 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:14:38.813190    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba35ca07-1e6a-4098-9d52-afa7a68c8805-kube-api-access-mhzwc\" (UniqueName: \"kubernetes.io/projected/ba35ca07-1e6a-4098-9d52-afa7a68c8805-kube-api-access-mhzwc\") pod \"ba35ca07-1e6a-4098-9d52-afa7a68c8805\" (UID: \"ba35ca07-1e6a-4098-9d52-afa7a68c8805\") "
Nov 02 20:14:38.813314 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:14:38.813292    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba35ca07-1e6a-4098-9d52-afa7a68c8805-plugins-config-volume" pod "ba35ca07-1e6a-4098-9d52-afa7a68c8805" (UID: "ba35ca07-1e6a-4098-9d52-afa7a68c8805"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:14:38.813682 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:14:38.813643    4463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba35ca07-1e6a-4098-9d52-afa7a68c8805-plugins-config-volume\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:14:38.815418 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:14:38.815393    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba35ca07-1e6a-4098-9d52-afa7a68c8805-kube-api-access-mhzwc" pod "ba35ca07-1e6a-4098-9d52-afa7a68c8805" (UID: "ba35ca07-1e6a-4098-9d52-afa7a68c8805"). InnerVolumeSpecName "kube-api-access-mhzwc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:14:38.914971 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:14:38.914936    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhzwc\" (UniqueName: \"kubernetes.io/projected/ba35ca07-1e6a-4098-9d52-afa7a68c8805-kube-api-access-mhzwc\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:14:39.096680 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:14:39.096641    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-838bc1-869f9bc899-68j9k"]
Nov 02 20:14:39.099433 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:14:39.099406    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-838bc1-869f9bc899-68j9k"]
Nov 02 20:14:39.585525 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:14:39.585496    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba35ca07-1e6a-4098-9d52-afa7a68c8805" path="/var/lib/kubelet/pods/ba35ca07-1e6a-4098-9d52-afa7a68c8805/volumes"
Nov 02 20:14:50.420535 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:14:50.420499    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 20:15:04.294825 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:15:04.294781    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 20:15:09.301168 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:09.301123    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7025fa-745dbc9f5f-l9mrl"]
Nov 02 20:15:09.414265 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:09.414233    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7025fa-6fdf99959-qcxbx"]
Nov 02 20:15:10.638714 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:10.638681    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b45a2b-76df647b98-p92xh"]
Nov 02 20:15:10.639137 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:10.638870    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b45a2b-76df647b98-p92xh" podUID="2a52fefb-fb79-4d25-9edf-719bd84cecdd" containerName="maas-api" containerID="cri-o://09f91af78557416ce3b155ac24fda055049467f45fc65019cc149937fe0eb1d2" gracePeriod=30
Nov 02 20:15:10.723081 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:10.723037    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b45a2b-76df647b98-p92xh" podUID="2a52fefb-fb79-4d25-9edf-719bd84cecdd" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.130:8443/health\": dial tcp 10.133.0.130:8443: connect: connection refused"
Nov 02 20:15:10.903898 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:10.903818    4463 generic.go:356] "Generic (PLEG): container finished" podID="2a52fefb-fb79-4d25-9edf-719bd84cecdd" containerID="09f91af78557416ce3b155ac24fda055049467f45fc65019cc149937fe0eb1d2" exitCode=0
Nov 02 20:15:10.904185 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:10.904165    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b45a2b-76df647b98-p92xh" event={"ID":"2a52fefb-fb79-4d25-9edf-719bd84cecdd","Type":"ContainerDied","Data":"09f91af78557416ce3b155ac24fda055049467f45fc65019cc149937fe0eb1d2"}
Nov 02 20:15:10.928039 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:10.927999    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b45a2b-76df647b98-p92xh"
Nov 02 20:15:11.096100 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:11.096067    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a52fefb-fb79-4d25-9edf-719bd84cecdd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a52fefb-fb79-4d25-9edf-719bd84cecdd-openshift-service-ca\") pod \"2a52fefb-fb79-4d25-9edf-719bd84cecdd\" (UID: \"2a52fefb-fb79-4d25-9edf-719bd84cecdd\") "
Nov 02 20:15:11.096272 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:11.096119    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a52fefb-fb79-4d25-9edf-719bd84cecdd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a52fefb-fb79-4d25-9edf-719bd84cecdd-metrics-certs\") pod \"2a52fefb-fb79-4d25-9edf-719bd84cecdd\" (UID: \"2a52fefb-fb79-4d25-9edf-719bd84cecdd\") "
Nov 02 20:15:11.096272 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:11.096177    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a52fefb-fb79-4d25-9edf-719bd84cecdd-kube-api-access-w6rrx\" (UniqueName: \"kubernetes.io/projected/2a52fefb-fb79-4d25-9edf-719bd84cecdd-kube-api-access-w6rrx\") pod \"2a52fefb-fb79-4d25-9edf-719bd84cecdd\" (UID: \"2a52fefb-fb79-4d25-9edf-719bd84cecdd\") "
Nov 02 20:15:11.096405 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:11.096270    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a52fefb-fb79-4d25-9edf-719bd84cecdd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a52fefb-fb79-4d25-9edf-719bd84cecdd-maas-api-tls\") pod \"2a52fefb-fb79-4d25-9edf-719bd84cecdd\" (UID: \"2a52fefb-fb79-4d25-9edf-719bd84cecdd\") "
Nov 02 20:15:11.096479 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:11.096453    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a52fefb-fb79-4d25-9edf-719bd84cecdd-openshift-service-ca" pod "2a52fefb-fb79-4d25-9edf-719bd84cecdd" (UID: "2a52fefb-fb79-4d25-9edf-719bd84cecdd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:15:11.096880 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:11.096850    4463 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a52fefb-fb79-4d25-9edf-719bd84cecdd-openshift-service-ca\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:11.098874 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:11.098841    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a52fefb-fb79-4d25-9edf-719bd84cecdd-maas-api-tls" pod "2a52fefb-fb79-4d25-9edf-719bd84cecdd" (UID: "2a52fefb-fb79-4d25-9edf-719bd84cecdd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:15:11.098874 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:11.098866    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a52fefb-fb79-4d25-9edf-719bd84cecdd-metrics-certs" pod "2a52fefb-fb79-4d25-9edf-719bd84cecdd" (UID: "2a52fefb-fb79-4d25-9edf-719bd84cecdd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:15:11.099003 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:11.098914    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a52fefb-fb79-4d25-9edf-719bd84cecdd-kube-api-access-w6rrx" pod "2a52fefb-fb79-4d25-9edf-719bd84cecdd" (UID: "2a52fefb-fb79-4d25-9edf-719bd84cecdd"). InnerVolumeSpecName "kube-api-access-w6rrx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:15:11.198252 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:11.198213    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6rrx\" (UniqueName: \"kubernetes.io/projected/2a52fefb-fb79-4d25-9edf-719bd84cecdd-kube-api-access-w6rrx\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:11.198252 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:11.198245    4463 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a52fefb-fb79-4d25-9edf-719bd84cecdd-maas-api-tls\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:11.198252 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:11.198260    4463 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a52fefb-fb79-4d25-9edf-719bd84cecdd-metrics-certs\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:11.911785 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:11.911716    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b45a2b-76df647b98-p92xh"
Nov 02 20:15:11.911785 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:11.911730    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b45a2b-76df647b98-p92xh" event={"ID":"2a52fefb-fb79-4d25-9edf-719bd84cecdd","Type":"ContainerDied","Data":"d3551f52fd5782088bceaf70ea2557cbc42c81d7ee40a198f2a2a2aa921f06a0"}
Nov 02 20:15:11.911785 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:11.911782    4463 scope.go:122] "RemoveContainer" containerID="09f91af78557416ce3b155ac24fda055049467f45fc65019cc149937fe0eb1d2"
Nov 02 20:15:12.065429 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:12.065399    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b45a2b-76df647b98-p92xh"]
Nov 02 20:15:12.111740 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:12.111708    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b45a2b-76df647b98-p92xh"]
Nov 02 20:15:13.589311 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:13.589270    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a52fefb-fb79-4d25-9edf-719bd84cecdd" path="/var/lib/kubelet/pods/2a52fefb-fb79-4d25-9edf-719bd84cecdd/volumes"
Nov 02 20:15:14.302340 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:14.302300    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-7025fa-745dbc9f5f-l9mrl" podUID="c2091abf-b755-49b3-86ad-b4bac6e2e4c5" containerName="payload-processing" containerID="cri-o://b1b8fe35bd9dce67dd5782a462f0aa4aa76229a6c02a35fd40c9213ea6ab0bcf" gracePeriod=25
Nov 02 20:15:14.415485 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:14.415451    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7025fa-6fdf99959-qcxbx" podUID="93875a90-855a-489f-8ea9-971e26ad15eb" containerName="payload-pre-processing" containerID="cri-o://f6f88cbc40faea3f73fd7cbe78f688ad170c9f4317f493cf2fb557451d6511cc" gracePeriod=25
Nov 02 20:15:14.565719 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:14.565696    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-7025fa-745dbc9f5f-l9mrl"
Nov 02 20:15:14.636794 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:14.636768    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2091abf-b755-49b3-86ad-b4bac6e2e4c5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2091abf-b755-49b3-86ad-b4bac6e2e4c5-plugins-config-volume\") pod \"c2091abf-b755-49b3-86ad-b4bac6e2e4c5\" (UID: \"c2091abf-b755-49b3-86ad-b4bac6e2e4c5\") "
Nov 02 20:15:14.637161 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:14.636859    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2091abf-b755-49b3-86ad-b4bac6e2e4c5-kube-api-access-hr2nz\" (UniqueName: \"kubernetes.io/projected/c2091abf-b755-49b3-86ad-b4bac6e2e4c5-kube-api-access-hr2nz\") pod \"c2091abf-b755-49b3-86ad-b4bac6e2e4c5\" (UID: \"c2091abf-b755-49b3-86ad-b4bac6e2e4c5\") "
Nov 02 20:15:14.637161 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:14.637109    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2091abf-b755-49b3-86ad-b4bac6e2e4c5-plugins-config-volume" pod "c2091abf-b755-49b3-86ad-b4bac6e2e4c5" (UID: "c2091abf-b755-49b3-86ad-b4bac6e2e4c5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:15:14.637365 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:14.637345    4463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2091abf-b755-49b3-86ad-b4bac6e2e4c5-plugins-config-volume\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:14.638848 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:14.638828    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2091abf-b755-49b3-86ad-b4bac6e2e4c5-kube-api-access-hr2nz" pod "c2091abf-b755-49b3-86ad-b4bac6e2e4c5" (UID: "c2091abf-b755-49b3-86ad-b4bac6e2e4c5"). InnerVolumeSpecName "kube-api-access-hr2nz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:15:14.738176 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:14.738139    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hr2nz\" (UniqueName: \"kubernetes.io/projected/c2091abf-b755-49b3-86ad-b4bac6e2e4c5-kube-api-access-hr2nz\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:14.932525 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:14.932490    4463 generic.go:356] "Generic (PLEG): container finished" podID="c2091abf-b755-49b3-86ad-b4bac6e2e4c5" containerID="b1b8fe35bd9dce67dd5782a462f0aa4aa76229a6c02a35fd40c9213ea6ab0bcf" exitCode=0
Nov 02 20:15:14.932682 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:14.932548    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-7025fa-745dbc9f5f-l9mrl"
Nov 02 20:15:14.932682 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:14.932610    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-7025fa-745dbc9f5f-l9mrl" event={"ID":"c2091abf-b755-49b3-86ad-b4bac6e2e4c5","Type":"ContainerDied","Data":"b1b8fe35bd9dce67dd5782a462f0aa4aa76229a6c02a35fd40c9213ea6ab0bcf"}
Nov 02 20:15:14.932682 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:14.932671    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-7025fa-745dbc9f5f-l9mrl" event={"ID":"c2091abf-b755-49b3-86ad-b4bac6e2e4c5","Type":"ContainerDied","Data":"37c55330cc54c385c4be101a21725b79d8c720e78cca136f91d9987221557d95"}
Nov 02 20:15:14.932831 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:14.932695    4463 scope.go:122] "RemoveContainer" containerID="b1b8fe35bd9dce67dd5782a462f0aa4aa76229a6c02a35fd40c9213ea6ab0bcf"
Nov 02 20:15:14.943548 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:14.943524    4463 scope.go:122] "RemoveContainer" containerID="b1b8fe35bd9dce67dd5782a462f0aa4aa76229a6c02a35fd40c9213ea6ab0bcf"
Nov 02 20:15:14.943788 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:15:14.943759    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1b8fe35bd9dce67dd5782a462f0aa4aa76229a6c02a35fd40c9213ea6ab0bcf\": container with ID starting with b1b8fe35bd9dce67dd5782a462f0aa4aa76229a6c02a35fd40c9213ea6ab0bcf not found: ID does not exist" containerID="b1b8fe35bd9dce67dd5782a462f0aa4aa76229a6c02a35fd40c9213ea6ab0bcf"
Nov 02 20:15:14.943842 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:14.943788    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1b8fe35bd9dce67dd5782a462f0aa4aa76229a6c02a35fd40c9213ea6ab0bcf"} err="failed to get container status \"b1b8fe35bd9dce67dd5782a462f0aa4aa76229a6c02a35fd40c9213ea6ab0bcf\": rpc error: code = NotFound desc = could not find container \"b1b8fe35bd9dce67dd5782a462f0aa4aa76229a6c02a35fd40c9213ea6ab0bcf\": container with ID starting with b1b8fe35bd9dce67dd5782a462f0aa4aa76229a6c02a35fd40c9213ea6ab0bcf not found: ID does not exist"
Nov 02 20:15:15.086640 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:15.086612    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7025fa-745dbc9f5f-l9mrl"]
Nov 02 20:15:15.091255 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:15.091230    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7025fa-745dbc9f5f-l9mrl"]
Nov 02 20:15:15.584997 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:15.584945    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2091abf-b755-49b3-86ad-b4bac6e2e4c5" path="/var/lib/kubelet/pods/c2091abf-b755-49b3-86ad-b4bac6e2e4c5/volumes"
Nov 02 20:15:19.618901 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:19.618866    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-b45a2b-openshift-default-8cddcbd8c-dq2sb"]
Nov 02 20:15:19.619374 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:19.619106    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-b45a2b-openshift-default-8cddcbd8c-dq2sb" podUID="7ab3e969-44bd-4d38-a066-005a66561670" containerName="istio-proxy" containerID="cri-o://ede9fb98a1670c48449f4e8389f28e31dca6959a76b96837d1aeea5153186455" gracePeriod=30
Nov 02 20:15:22.171114 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:22.171076    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7025fa-6fdf99959-qcxbx" podUID="93875a90-855a-489f-8ea9-971e26ad15eb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 20:15:24.886577 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:24.886514    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b45a2b-openshift-default-8cddcbd8c-dq2sb"
Nov 02 20:15:24.952625 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:24.952592    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ab3e969-44bd-4d38-a066-005a66561670-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ab3e969-44bd-4d38-a066-005a66561670-istio-data\") pod \"7ab3e969-44bd-4d38-a066-005a66561670\" (UID: \"7ab3e969-44bd-4d38-a066-005a66561670\") "
Nov 02 20:15:24.952824 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:24.952653    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ab3e969-44bd-4d38-a066-005a66561670-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ab3e969-44bd-4d38-a066-005a66561670-credential-socket\") pod \"7ab3e969-44bd-4d38-a066-005a66561670\" (UID: \"7ab3e969-44bd-4d38-a066-005a66561670\") "
Nov 02 20:15:24.952824 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:24.952683    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ab3e969-44bd-4d38-a066-005a66561670-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ab3e969-44bd-4d38-a066-005a66561670-workload-certs\") pod \"7ab3e969-44bd-4d38-a066-005a66561670\" (UID: \"7ab3e969-44bd-4d38-a066-005a66561670\") "
Nov 02 20:15:24.952824 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:24.952719    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ab3e969-44bd-4d38-a066-005a66561670-istio-token\" (UniqueName: \"kubernetes.io/projected/7ab3e969-44bd-4d38-a066-005a66561670-istio-token\") pod \"7ab3e969-44bd-4d38-a066-005a66561670\" (UID: \"7ab3e969-44bd-4d38-a066-005a66561670\") "
Nov 02 20:15:24.952824 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:24.952761    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ab3e969-44bd-4d38-a066-005a66561670-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ab3e969-44bd-4d38-a066-005a66561670-workload-socket\") pod \"7ab3e969-44bd-4d38-a066-005a66561670\" (UID: \"7ab3e969-44bd-4d38-a066-005a66561670\") "
Nov 02 20:15:24.953083 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:24.952860    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ab3e969-44bd-4d38-a066-005a66561670-istio-data" pod "7ab3e969-44bd-4d38-a066-005a66561670" (UID: "7ab3e969-44bd-4d38-a066-005a66561670"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:15:24.953083 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:24.952873    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7ab3e969-44bd-4d38-a066-005a66561670-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ab3e969-44bd-4d38-a066-005a66561670-istio-podinfo\") pod \"7ab3e969-44bd-4d38-a066-005a66561670\" (UID: \"7ab3e969-44bd-4d38-a066-005a66561670\") "
Nov 02 20:15:24.953083 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:24.952935    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ab3e969-44bd-4d38-a066-005a66561670-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ab3e969-44bd-4d38-a066-005a66561670-istio-envoy\") pod \"7ab3e969-44bd-4d38-a066-005a66561670\" (UID: \"7ab3e969-44bd-4d38-a066-005a66561670\") "
Nov 02 20:15:24.953083 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:24.952972    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ab3e969-44bd-4d38-a066-005a66561670-kube-api-access-tmqnx\" (UniqueName: \"kubernetes.io/projected/7ab3e969-44bd-4d38-a066-005a66561670-kube-api-access-tmqnx\") pod \"7ab3e969-44bd-4d38-a066-005a66561670\" (UID: \"7ab3e969-44bd-4d38-a066-005a66561670\") "
Nov 02 20:15:24.953083 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:24.952981    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ab3e969-44bd-4d38-a066-005a66561670-credential-socket" pod "7ab3e969-44bd-4d38-a066-005a66561670" (UID: "7ab3e969-44bd-4d38-a066-005a66561670"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:15:24.953083 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:24.953045    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ab3e969-44bd-4d38-a066-005a66561670-workload-certs" pod "7ab3e969-44bd-4d38-a066-005a66561670" (UID: "7ab3e969-44bd-4d38-a066-005a66561670"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:15:24.953400 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:24.953078    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ab3e969-44bd-4d38-a066-005a66561670-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ab3e969-44bd-4d38-a066-005a66561670-istiod-ca-cert\") pod \"7ab3e969-44bd-4d38-a066-005a66561670\" (UID: \"7ab3e969-44bd-4d38-a066-005a66561670\") "
Nov 02 20:15:24.953462 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:24.953427    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ab3e969-44bd-4d38-a066-005a66561670-workload-socket" pod "7ab3e969-44bd-4d38-a066-005a66561670" (UID: "7ab3e969-44bd-4d38-a066-005a66561670"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:15:24.953517 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:24.953477    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ab3e969-44bd-4d38-a066-005a66561670-istiod-ca-cert" pod "7ab3e969-44bd-4d38-a066-005a66561670" (UID: "7ab3e969-44bd-4d38-a066-005a66561670"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:15:24.953633 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:24.953616    4463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ab3e969-44bd-4d38-a066-005a66561670-istiod-ca-cert\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:24.953701 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:24.953638    4463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ab3e969-44bd-4d38-a066-005a66561670-istio-data\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:24.953701 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:24.953653    4463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ab3e969-44bd-4d38-a066-005a66561670-credential-socket\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:24.953701 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:24.953668    4463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ab3e969-44bd-4d38-a066-005a66561670-workload-certs\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:24.953701 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:24.953682    4463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ab3e969-44bd-4d38-a066-005a66561670-workload-socket\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:24.955073 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:24.955052    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ab3e969-44bd-4d38-a066-005a66561670-istio-token" pod "7ab3e969-44bd-4d38-a066-005a66561670" (UID: "7ab3e969-44bd-4d38-a066-005a66561670"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:15:24.955237 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:24.955210    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7ab3e969-44bd-4d38-a066-005a66561670-istio-podinfo" pod "7ab3e969-44bd-4d38-a066-005a66561670" (UID: "7ab3e969-44bd-4d38-a066-005a66561670"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:15:24.955439 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:24.955420    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ab3e969-44bd-4d38-a066-005a66561670-istio-envoy" pod "7ab3e969-44bd-4d38-a066-005a66561670" (UID: "7ab3e969-44bd-4d38-a066-005a66561670"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:15:24.955508 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:24.955456    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ab3e969-44bd-4d38-a066-005a66561670-kube-api-access-tmqnx" pod "7ab3e969-44bd-4d38-a066-005a66561670" (UID: "7ab3e969-44bd-4d38-a066-005a66561670"). InnerVolumeSpecName "kube-api-access-tmqnx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:15:24.993535 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:24.993503    4463 generic.go:356] "Generic (PLEG): container finished" podID="7ab3e969-44bd-4d38-a066-005a66561670" containerID="ede9fb98a1670c48449f4e8389f28e31dca6959a76b96837d1aeea5153186455" exitCode=0
Nov 02 20:15:24.993634 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:24.993547    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b45a2b-openshift-default-8cddcbd8c-dq2sb"
Nov 02 20:15:24.993634 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:24.993591    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b45a2b-openshift-default-8cddcbd8c-dq2sb" event={"ID":"7ab3e969-44bd-4d38-a066-005a66561670","Type":"ContainerDied","Data":"ede9fb98a1670c48449f4e8389f28e31dca6959a76b96837d1aeea5153186455"}
Nov 02 20:15:24.993731 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:24.993646    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b45a2b-openshift-default-8cddcbd8c-dq2sb" event={"ID":"7ab3e969-44bd-4d38-a066-005a66561670","Type":"ContainerDied","Data":"60435dd1700aa96bdfae9518b9e25c3b5c91bfeab4b4a8c5b1e720ce000bd565"}
Nov 02 20:15:24.993731 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:24.993669    4463 scope.go:122] "RemoveContainer" containerID="ede9fb98a1670c48449f4e8389f28e31dca6959a76b96837d1aeea5153186455"
Nov 02 20:15:25.003934 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:25.003914    4463 scope.go:122] "RemoveContainer" containerID="cff56eeaf816c2d12a5f41d1891b424270e8974ddf47f80e6cc166ada88cae7a"
Nov 02 20:15:25.012325 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:25.012307    4463 scope.go:122] "RemoveContainer" containerID="ede9fb98a1670c48449f4e8389f28e31dca6959a76b96837d1aeea5153186455"
Nov 02 20:15:25.012570 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:15:25.012548    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ede9fb98a1670c48449f4e8389f28e31dca6959a76b96837d1aeea5153186455\": container with ID starting with ede9fb98a1670c48449f4e8389f28e31dca6959a76b96837d1aeea5153186455 not found: ID does not exist" containerID="ede9fb98a1670c48449f4e8389f28e31dca6959a76b96837d1aeea5153186455"
Nov 02 20:15:25.012633 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:25.012578    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ede9fb98a1670c48449f4e8389f28e31dca6959a76b96837d1aeea5153186455"} err="failed to get container status \"ede9fb98a1670c48449f4e8389f28e31dca6959a76b96837d1aeea5153186455\": rpc error: code = NotFound desc = could not find container \"ede9fb98a1670c48449f4e8389f28e31dca6959a76b96837d1aeea5153186455\": container with ID starting with ede9fb98a1670c48449f4e8389f28e31dca6959a76b96837d1aeea5153186455 not found: ID does not exist"
Nov 02 20:15:25.012633 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:25.012596    4463 scope.go:122] "RemoveContainer" containerID="cff56eeaf816c2d12a5f41d1891b424270e8974ddf47f80e6cc166ada88cae7a"
Nov 02 20:15:25.012856 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:15:25.012836    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cff56eeaf816c2d12a5f41d1891b424270e8974ddf47f80e6cc166ada88cae7a\": container with ID starting with cff56eeaf816c2d12a5f41d1891b424270e8974ddf47f80e6cc166ada88cae7a not found: ID does not exist" containerID="cff56eeaf816c2d12a5f41d1891b424270e8974ddf47f80e6cc166ada88cae7a"
Nov 02 20:15:25.012903 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:25.012860    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cff56eeaf816c2d12a5f41d1891b424270e8974ddf47f80e6cc166ada88cae7a"} err="failed to get container status \"cff56eeaf816c2d12a5f41d1891b424270e8974ddf47f80e6cc166ada88cae7a\": rpc error: code = NotFound desc = could not find container \"cff56eeaf816c2d12a5f41d1891b424270e8974ddf47f80e6cc166ada88cae7a\": container with ID starting with cff56eeaf816c2d12a5f41d1891b424270e8974ddf47f80e6cc166ada88cae7a not found: ID does not exist"
Nov 02 20:15:25.023896 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:25.023876    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-5c46af/distinct-w3-5c46af-kserve-68b6dff7f5-bp59f"]
Nov 02 20:15:25.054963 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:25.054932    4463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ab3e969-44bd-4d38-a066-005a66561670-istio-token\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:25.054963 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:25.054959    4463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ab3e969-44bd-4d38-a066-005a66561670-istio-podinfo\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:25.055139 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:25.054969    4463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ab3e969-44bd-4d38-a066-005a66561670-istio-envoy\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:25.055139 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:25.054979    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmqnx\" (UniqueName: \"kubernetes.io/projected/7ab3e969-44bd-4d38-a066-005a66561670-kube-api-access-tmqnx\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:25.278064 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:25.278030    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-b45a2b-openshift-default-8cddcbd8c-dq2sb"]
Nov 02 20:15:25.283988 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:25.283955    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-b45a2b-openshift-default-8cddcbd8c-dq2sb"]
Nov 02 20:15:25.585913 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:25.585835    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ab3e969-44bd-4d38-a066-005a66561670" path="/var/lib/kubelet/pods/7ab3e969-44bd-4d38-a066-005a66561670/volumes"
Nov 02 20:15:29.900148 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:29.900116    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h27fz"]
Nov 02 20:15:29.900581 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:29.900321    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h27fz" podUID="b9f30411-f136-4711-9f5d-6f0ef7a24591" containerName="istio-proxy" containerID="cri-o://bba87f664fb4209144c627e4124320f89fa64bebd2f4f86f72349778884f7942" gracePeriod=30
Nov 02 20:15:31.340691 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:31.340651    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7025fa-56795cd64c-j844g"]
Nov 02 20:15:31.341226 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:31.340888    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7025fa-56795cd64c-j844g" podUID="a33f6a91-4468-4ea7-a107-c569cc82a306" containerName="maas-api" containerID="cri-o://d6000bbd4db2bba4edf22706769b7eb4805147221b75acb81a9d1f770d733999" gracePeriod=30
Nov 02 20:15:31.598667 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:31.598603    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7025fa-56795cd64c-j844g"
Nov 02 20:15:31.729128 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:31.729085    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a33f6a91-4468-4ea7-a107-c569cc82a306-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a33f6a91-4468-4ea7-a107-c569cc82a306-openshift-service-ca\") pod \"a33f6a91-4468-4ea7-a107-c569cc82a306\" (UID: \"a33f6a91-4468-4ea7-a107-c569cc82a306\") "
Nov 02 20:15:31.729321 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:31.729189    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a33f6a91-4468-4ea7-a107-c569cc82a306-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a33f6a91-4468-4ea7-a107-c569cc82a306-maas-api-tls\") pod \"a33f6a91-4468-4ea7-a107-c569cc82a306\" (UID: \"a33f6a91-4468-4ea7-a107-c569cc82a306\") "
Nov 02 20:15:31.729321 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:31.729307    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a33f6a91-4468-4ea7-a107-c569cc82a306-kube-api-access-bchsk\" (UniqueName: \"kubernetes.io/projected/a33f6a91-4468-4ea7-a107-c569cc82a306-kube-api-access-bchsk\") pod \"a33f6a91-4468-4ea7-a107-c569cc82a306\" (UID: \"a33f6a91-4468-4ea7-a107-c569cc82a306\") "
Nov 02 20:15:31.729476 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:31.729459    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a33f6a91-4468-4ea7-a107-c569cc82a306-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a33f6a91-4468-4ea7-a107-c569cc82a306-metrics-certs\") pod \"a33f6a91-4468-4ea7-a107-c569cc82a306\" (UID: \"a33f6a91-4468-4ea7-a107-c569cc82a306\") "
Nov 02 20:15:31.729575 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:31.729543    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a33f6a91-4468-4ea7-a107-c569cc82a306-openshift-service-ca" pod "a33f6a91-4468-4ea7-a107-c569cc82a306" (UID: "a33f6a91-4468-4ea7-a107-c569cc82a306"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:15:31.730108 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:31.730084    4463 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a33f6a91-4468-4ea7-a107-c569cc82a306-openshift-service-ca\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:31.732117 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:31.732089    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a33f6a91-4468-4ea7-a107-c569cc82a306-metrics-certs" pod "a33f6a91-4468-4ea7-a107-c569cc82a306" (UID: "a33f6a91-4468-4ea7-a107-c569cc82a306"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:15:31.732225 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:31.732096    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a33f6a91-4468-4ea7-a107-c569cc82a306-kube-api-access-bchsk" pod "a33f6a91-4468-4ea7-a107-c569cc82a306" (UID: "a33f6a91-4468-4ea7-a107-c569cc82a306"). InnerVolumeSpecName "kube-api-access-bchsk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:15:31.732525 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:31.732504    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a33f6a91-4468-4ea7-a107-c569cc82a306-maas-api-tls" pod "a33f6a91-4468-4ea7-a107-c569cc82a306" (UID: "a33f6a91-4468-4ea7-a107-c569cc82a306"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:15:31.831786 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:31.831741    4463 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a33f6a91-4468-4ea7-a107-c569cc82a306-maas-api-tls\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:31.831786 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:31.831774    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bchsk\" (UniqueName: \"kubernetes.io/projected/a33f6a91-4468-4ea7-a107-c569cc82a306-kube-api-access-bchsk\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:31.831786 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:31.831785    4463 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a33f6a91-4468-4ea7-a107-c569cc82a306-metrics-certs\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:32.033834 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:32.033798    4463 generic.go:356] "Generic (PLEG): container finished" podID="a33f6a91-4468-4ea7-a107-c569cc82a306" containerID="d6000bbd4db2bba4edf22706769b7eb4805147221b75acb81a9d1f770d733999" exitCode=0
Nov 02 20:15:32.034008 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:32.033847    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7025fa-56795cd64c-j844g"
Nov 02 20:15:32.034008 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:32.033876    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7025fa-56795cd64c-j844g" event={"ID":"a33f6a91-4468-4ea7-a107-c569cc82a306","Type":"ContainerDied","Data":"d6000bbd4db2bba4edf22706769b7eb4805147221b75acb81a9d1f770d733999"}
Nov 02 20:15:32.034008 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:32.033934    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7025fa-56795cd64c-j844g" event={"ID":"a33f6a91-4468-4ea7-a107-c569cc82a306","Type":"ContainerDied","Data":"248e1985cc3e2c81ffe1f66db340e3f00077f6cb22a98960b505ad8e2fdd85f0"}
Nov 02 20:15:32.034008 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:32.033963    4463 scope.go:122] "RemoveContainer" containerID="d6000bbd4db2bba4edf22706769b7eb4805147221b75acb81a9d1f770d733999"
Nov 02 20:15:32.045334 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:32.045316    4463 scope.go:122] "RemoveContainer" containerID="d6000bbd4db2bba4edf22706769b7eb4805147221b75acb81a9d1f770d733999"
Nov 02 20:15:32.045595 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:15:32.045578    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6000bbd4db2bba4edf22706769b7eb4805147221b75acb81a9d1f770d733999\": container with ID starting with d6000bbd4db2bba4edf22706769b7eb4805147221b75acb81a9d1f770d733999 not found: ID does not exist" containerID="d6000bbd4db2bba4edf22706769b7eb4805147221b75acb81a9d1f770d733999"
Nov 02 20:15:32.045652 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:32.045606    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6000bbd4db2bba4edf22706769b7eb4805147221b75acb81a9d1f770d733999"} err="failed to get container status \"d6000bbd4db2bba4edf22706769b7eb4805147221b75acb81a9d1f770d733999\": rpc error: code = NotFound desc = could not find container \"d6000bbd4db2bba4edf22706769b7eb4805147221b75acb81a9d1f770d733999\": container with ID starting with d6000bbd4db2bba4edf22706769b7eb4805147221b75acb81a9d1f770d733999 not found: ID does not exist"
Nov 02 20:15:32.171390 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:32.171359    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7025fa-6fdf99959-qcxbx" podUID="93875a90-855a-489f-8ea9-971e26ad15eb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 20:15:32.186001 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:32.185308    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-5c46af-7f86f8f5c5-jxhp6"]
Nov 02 20:15:32.194450 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:32.194414    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7025fa-56795cd64c-j844g"]
Nov 02 20:15:32.198874 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:32.198842    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7025fa-56795cd64c-j844g"]
Nov 02 20:15:33.585618 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:33.585583    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a33f6a91-4468-4ea7-a107-c569cc82a306" path="/var/lib/kubelet/pods/a33f6a91-4468-4ea7-a107-c569cc82a306/volumes"
Nov 02 20:15:34.299268 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:15:34.299224    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 20:15:34.699553 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:34.699524    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7025fa-6fdf99959-qcxbx"
Nov 02 20:15:34.866212 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:34.866139    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93875a90-855a-489f-8ea9-971e26ad15eb-kube-api-access-dbj5p\" (UniqueName: \"kubernetes.io/projected/93875a90-855a-489f-8ea9-971e26ad15eb-kube-api-access-dbj5p\") pod \"93875a90-855a-489f-8ea9-971e26ad15eb\" (UID: \"93875a90-855a-489f-8ea9-971e26ad15eb\") "
Nov 02 20:15:34.866212 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:34.866190    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93875a90-855a-489f-8ea9-971e26ad15eb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93875a90-855a-489f-8ea9-971e26ad15eb-plugins-config-volume\") pod \"93875a90-855a-489f-8ea9-971e26ad15eb\" (UID: \"93875a90-855a-489f-8ea9-971e26ad15eb\") "
Nov 02 20:15:34.866642 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:34.866616    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93875a90-855a-489f-8ea9-971e26ad15eb-plugins-config-volume" pod "93875a90-855a-489f-8ea9-971e26ad15eb" (UID: "93875a90-855a-489f-8ea9-971e26ad15eb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:15:34.868290 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:34.868263    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93875a90-855a-489f-8ea9-971e26ad15eb-kube-api-access-dbj5p" pod "93875a90-855a-489f-8ea9-971e26ad15eb" (UID: "93875a90-855a-489f-8ea9-971e26ad15eb"). InnerVolumeSpecName "kube-api-access-dbj5p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:15:34.968115 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:34.968089    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbj5p\" (UniqueName: \"kubernetes.io/projected/93875a90-855a-489f-8ea9-971e26ad15eb-kube-api-access-dbj5p\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:34.968115 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:34.968116    4463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93875a90-855a-489f-8ea9-971e26ad15eb-plugins-config-volume\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:35.053295 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:35.053264    4463 generic.go:356] "Generic (PLEG): container finished" podID="93875a90-855a-489f-8ea9-971e26ad15eb" containerID="f6f88cbc40faea3f73fd7cbe78f688ad170c9f4317f493cf2fb557451d6511cc" exitCode=0
Nov 02 20:15:35.053490 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:35.053312    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7025fa-6fdf99959-qcxbx"
Nov 02 20:15:35.053490 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:35.053350    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7025fa-6fdf99959-qcxbx" event={"ID":"93875a90-855a-489f-8ea9-971e26ad15eb","Type":"ContainerDied","Data":"f6f88cbc40faea3f73fd7cbe78f688ad170c9f4317f493cf2fb557451d6511cc"}
Nov 02 20:15:35.053490 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:35.053419    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7025fa-6fdf99959-qcxbx" event={"ID":"93875a90-855a-489f-8ea9-971e26ad15eb","Type":"ContainerDied","Data":"9f00b584d8e6317ed281f45d9de8faa706c4028e52ab1f4bad719154ffdb6332"}
Nov 02 20:15:35.053490 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:35.053457    4463 scope.go:122] "RemoveContainer" containerID="f6f88cbc40faea3f73fd7cbe78f688ad170c9f4317f493cf2fb557451d6511cc"
Nov 02 20:15:35.055351 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:35.055326    4463 generic.go:356] "Generic (PLEG): container finished" podID="b9f30411-f136-4711-9f5d-6f0ef7a24591" containerID="bba87f664fb4209144c627e4124320f89fa64bebd2f4f86f72349778884f7942" exitCode=0
Nov 02 20:15:35.055462 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:35.055407    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h27fz" event={"ID":"b9f30411-f136-4711-9f5d-6f0ef7a24591","Type":"ContainerDied","Data":"bba87f664fb4209144c627e4124320f89fa64bebd2f4f86f72349778884f7942"}
Nov 02 20:15:35.067169 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:35.064380    4463 scope.go:122] "RemoveContainer" containerID="f6f88cbc40faea3f73fd7cbe78f688ad170c9f4317f493cf2fb557451d6511cc"
Nov 02 20:15:35.067645 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:15:35.067566    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6f88cbc40faea3f73fd7cbe78f688ad170c9f4317f493cf2fb557451d6511cc\": container with ID starting with f6f88cbc40faea3f73fd7cbe78f688ad170c9f4317f493cf2fb557451d6511cc not found: ID does not exist" containerID="f6f88cbc40faea3f73fd7cbe78f688ad170c9f4317f493cf2fb557451d6511cc"
Nov 02 20:15:35.067645 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:35.067603    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6f88cbc40faea3f73fd7cbe78f688ad170c9f4317f493cf2fb557451d6511cc"} err="failed to get container status \"f6f88cbc40faea3f73fd7cbe78f688ad170c9f4317f493cf2fb557451d6511cc\": rpc error: code = NotFound desc = could not find container \"f6f88cbc40faea3f73fd7cbe78f688ad170c9f4317f493cf2fb557451d6511cc\": container with ID starting with f6f88cbc40faea3f73fd7cbe78f688ad170c9f4317f493cf2fb557451d6511cc not found: ID does not exist"
Nov 02 20:15:35.147252 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:35.147228    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h27fz"
Nov 02 20:15:35.206203 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:35.206174    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7025fa-6fdf99959-qcxbx"]
Nov 02 20:15:35.252156 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:35.252128    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7025fa-6fdf99959-qcxbx"]
Nov 02 20:15:35.271717 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:35.271692    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b9f30411-f136-4711-9f5d-6f0ef7a24591-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9f30411-f136-4711-9f5d-6f0ef7a24591-istio-podinfo\") pod \"b9f30411-f136-4711-9f5d-6f0ef7a24591\" (UID: \"b9f30411-f136-4711-9f5d-6f0ef7a24591\") "
Nov 02 20:15:35.271810 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:35.271736    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9f30411-f136-4711-9f5d-6f0ef7a24591-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9f30411-f136-4711-9f5d-6f0ef7a24591-credential-socket\") pod \"b9f30411-f136-4711-9f5d-6f0ef7a24591\" (UID: \"b9f30411-f136-4711-9f5d-6f0ef7a24591\") "
Nov 02 20:15:35.271810 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:35.271771    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9f30411-f136-4711-9f5d-6f0ef7a24591-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9f30411-f136-4711-9f5d-6f0ef7a24591-istio-data\") pod \"b9f30411-f136-4711-9f5d-6f0ef7a24591\" (UID: \"b9f30411-f136-4711-9f5d-6f0ef7a24591\") "
Nov 02 20:15:35.271892 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:35.271846    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9f30411-f136-4711-9f5d-6f0ef7a24591-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9f30411-f136-4711-9f5d-6f0ef7a24591-istiod-ca-cert\") pod \"b9f30411-f136-4711-9f5d-6f0ef7a24591\" (UID: \"b9f30411-f136-4711-9f5d-6f0ef7a24591\") "
Nov 02 20:15:35.271930 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:35.271910    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9f30411-f136-4711-9f5d-6f0ef7a24591-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9f30411-f136-4711-9f5d-6f0ef7a24591-workload-certs\") pod \"b9f30411-f136-4711-9f5d-6f0ef7a24591\" (UID: \"b9f30411-f136-4711-9f5d-6f0ef7a24591\") "
Nov 02 20:15:35.271969 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:35.271939    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9f30411-f136-4711-9f5d-6f0ef7a24591-istio-token\" (UniqueName: \"kubernetes.io/projected/b9f30411-f136-4711-9f5d-6f0ef7a24591-istio-token\") pod \"b9f30411-f136-4711-9f5d-6f0ef7a24591\" (UID: \"b9f30411-f136-4711-9f5d-6f0ef7a24591\") "
Nov 02 20:15:35.272041 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:35.271999    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9f30411-f136-4711-9f5d-6f0ef7a24591-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9f30411-f136-4711-9f5d-6f0ef7a24591-istio-envoy\") pod \"b9f30411-f136-4711-9f5d-6f0ef7a24591\" (UID: \"b9f30411-f136-4711-9f5d-6f0ef7a24591\") "
Nov 02 20:15:35.272204 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:35.272147    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9f30411-f136-4711-9f5d-6f0ef7a24591-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9f30411-f136-4711-9f5d-6f0ef7a24591-workload-socket\") pod \"b9f30411-f136-4711-9f5d-6f0ef7a24591\" (UID: \"b9f30411-f136-4711-9f5d-6f0ef7a24591\") "
Nov 02 20:15:35.272281 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:35.272200    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9f30411-f136-4711-9f5d-6f0ef7a24591-kube-api-access-hfqfb\" (UniqueName: \"kubernetes.io/projected/b9f30411-f136-4711-9f5d-6f0ef7a24591-kube-api-access-hfqfb\") pod \"b9f30411-f136-4711-9f5d-6f0ef7a24591\" (UID: \"b9f30411-f136-4711-9f5d-6f0ef7a24591\") "
Nov 02 20:15:35.272349 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:35.272143    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9f30411-f136-4711-9f5d-6f0ef7a24591-credential-socket" pod "b9f30411-f136-4711-9f5d-6f0ef7a24591" (UID: "b9f30411-f136-4711-9f5d-6f0ef7a24591"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:15:35.272411 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:35.272160    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9f30411-f136-4711-9f5d-6f0ef7a24591-istio-data" pod "b9f30411-f136-4711-9f5d-6f0ef7a24591" (UID: "b9f30411-f136-4711-9f5d-6f0ef7a24591"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:15:35.272411 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:35.272320    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9f30411-f136-4711-9f5d-6f0ef7a24591-workload-certs" pod "b9f30411-f136-4711-9f5d-6f0ef7a24591" (UID: "b9f30411-f136-4711-9f5d-6f0ef7a24591"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:15:35.272411 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:35.272362    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9f30411-f136-4711-9f5d-6f0ef7a24591-istiod-ca-cert" pod "b9f30411-f136-4711-9f5d-6f0ef7a24591" (UID: "b9f30411-f136-4711-9f5d-6f0ef7a24591"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:15:35.272411 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:35.272366    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9f30411-f136-4711-9f5d-6f0ef7a24591-workload-socket" pod "b9f30411-f136-4711-9f5d-6f0ef7a24591" (UID: "b9f30411-f136-4711-9f5d-6f0ef7a24591"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:15:35.272719 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:35.272695    4463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9f30411-f136-4711-9f5d-6f0ef7a24591-workload-socket\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:35.272787 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:35.272725    4463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9f30411-f136-4711-9f5d-6f0ef7a24591-credential-socket\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:35.272787 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:35.272740    4463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9f30411-f136-4711-9f5d-6f0ef7a24591-istio-data\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:35.272787 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:35.272755    4463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9f30411-f136-4711-9f5d-6f0ef7a24591-istiod-ca-cert\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:35.272787 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:35.272769    4463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9f30411-f136-4711-9f5d-6f0ef7a24591-workload-certs\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:35.274145 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:35.274117    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9f30411-f136-4711-9f5d-6f0ef7a24591-istio-token" pod "b9f30411-f136-4711-9f5d-6f0ef7a24591" (UID: "b9f30411-f136-4711-9f5d-6f0ef7a24591"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:15:35.274145 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:35.274132    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b9f30411-f136-4711-9f5d-6f0ef7a24591-istio-podinfo" pod "b9f30411-f136-4711-9f5d-6f0ef7a24591" (UID: "b9f30411-f136-4711-9f5d-6f0ef7a24591"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:15:35.274277 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:35.274223    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9f30411-f136-4711-9f5d-6f0ef7a24591-kube-api-access-hfqfb" pod "b9f30411-f136-4711-9f5d-6f0ef7a24591" (UID: "b9f30411-f136-4711-9f5d-6f0ef7a24591"). InnerVolumeSpecName "kube-api-access-hfqfb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:15:35.274277 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:35.274241    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9f30411-f136-4711-9f5d-6f0ef7a24591-istio-envoy" pod "b9f30411-f136-4711-9f5d-6f0ef7a24591" (UID: "b9f30411-f136-4711-9f5d-6f0ef7a24591"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:15:35.373617 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:35.373591    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfqfb\" (UniqueName: \"kubernetes.io/projected/b9f30411-f136-4711-9f5d-6f0ef7a24591-kube-api-access-hfqfb\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:35.373617 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:35.373618    4463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9f30411-f136-4711-9f5d-6f0ef7a24591-istio-podinfo\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:35.373770 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:35.373628    4463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9f30411-f136-4711-9f5d-6f0ef7a24591-istio-token\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:35.373770 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:35.373637    4463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9f30411-f136-4711-9f5d-6f0ef7a24591-istio-envoy\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:35.585886 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:35.585839    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93875a90-855a-489f-8ea9-971e26ad15eb" path="/var/lib/kubelet/pods/93875a90-855a-489f-8ea9-971e26ad15eb/volumes"
Nov 02 20:15:35.955192 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:35.955153    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 20:15:36.063636 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:36.063612    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h27fz"
Nov 02 20:15:36.063805 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:36.063653    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h27fz" event={"ID":"b9f30411-f136-4711-9f5d-6f0ef7a24591","Type":"ContainerDied","Data":"bb8f364f917c1984379902abffa7a419cf09af0191549f209c013962fc952e19"}
Nov 02 20:15:36.063805 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:36.063713    4463 scope.go:122] "RemoveContainer" containerID="bba87f664fb4209144c627e4124320f89fa64bebd2f4f86f72349778884f7942"
Nov 02 20:15:36.221648 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:36.221571    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h27fz"]
Nov 02 20:15:36.224105 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:36.224083    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h27fz"]
Nov 02 20:15:37.186818 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:37.186774    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5c46af-7f86f8f5c5-jxhp6" podUID="058aa7d8-1404-494f-b25b-70e8b702455b" containerName="payload-pre-processing" containerID="cri-o://58575358d352bd7bd204f26186106be5d4d3bb920881611d0f6cb31f79938748" gracePeriod=25
Nov 02 20:15:37.456637 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:37.456579    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5c46af-7f86f8f5c5-jxhp6"
Nov 02 20:15:37.584917 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:37.584877    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9f30411-f136-4711-9f5d-6f0ef7a24591" path="/var/lib/kubelet/pods/b9f30411-f136-4711-9f5d-6f0ef7a24591/volumes"
Nov 02 20:15:37.604697 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:37.604656    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/058aa7d8-1404-494f-b25b-70e8b702455b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/058aa7d8-1404-494f-b25b-70e8b702455b-plugins-config-volume\") pod \"058aa7d8-1404-494f-b25b-70e8b702455b\" (UID: \"058aa7d8-1404-494f-b25b-70e8b702455b\") "
Nov 02 20:15:37.604906 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:37.604814    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/058aa7d8-1404-494f-b25b-70e8b702455b-kube-api-access-86r4f\" (UniqueName: \"kubernetes.io/projected/058aa7d8-1404-494f-b25b-70e8b702455b-kube-api-access-86r4f\") pod \"058aa7d8-1404-494f-b25b-70e8b702455b\" (UID: \"058aa7d8-1404-494f-b25b-70e8b702455b\") "
Nov 02 20:15:37.605087 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:37.605061    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/058aa7d8-1404-494f-b25b-70e8b702455b-plugins-config-volume" pod "058aa7d8-1404-494f-b25b-70e8b702455b" (UID: "058aa7d8-1404-494f-b25b-70e8b702455b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:15:37.605387 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:37.605361    4463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/058aa7d8-1404-494f-b25b-70e8b702455b-plugins-config-volume\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:37.611186 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:37.611157    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/058aa7d8-1404-494f-b25b-70e8b702455b-kube-api-access-86r4f" pod "058aa7d8-1404-494f-b25b-70e8b702455b" (UID: "058aa7d8-1404-494f-b25b-70e8b702455b"). InnerVolumeSpecName "kube-api-access-86r4f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:15:37.706813 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:37.706719    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86r4f\" (UniqueName: \"kubernetes.io/projected/058aa7d8-1404-494f-b25b-70e8b702455b-kube-api-access-86r4f\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:38.077224 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:38.077134    4463 generic.go:356] "Generic (PLEG): container finished" podID="058aa7d8-1404-494f-b25b-70e8b702455b" containerID="58575358d352bd7bd204f26186106be5d4d3bb920881611d0f6cb31f79938748" exitCode=0
Nov 02 20:15:38.077224 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:38.077181    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5c46af-7f86f8f5c5-jxhp6"
Nov 02 20:15:38.077473 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:38.077217    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5c46af-7f86f8f5c5-jxhp6" event={"ID":"058aa7d8-1404-494f-b25b-70e8b702455b","Type":"ContainerDied","Data":"58575358d352bd7bd204f26186106be5d4d3bb920881611d0f6cb31f79938748"}
Nov 02 20:15:38.077473 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:38.077281    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5c46af-7f86f8f5c5-jxhp6" event={"ID":"058aa7d8-1404-494f-b25b-70e8b702455b","Type":"ContainerDied","Data":"97b8cef001b3280ead2a46ceeb5f371afbe1aae313f380a51f6f36b4704e1efc"}
Nov 02 20:15:38.077473 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:38.077314    4463 scope.go:122] "RemoveContainer" containerID="58575358d352bd7bd204f26186106be5d4d3bb920881611d0f6cb31f79938748"
Nov 02 20:15:38.087492 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:38.087477    4463 scope.go:122] "RemoveContainer" containerID="58575358d352bd7bd204f26186106be5d4d3bb920881611d0f6cb31f79938748"
Nov 02 20:15:38.087758 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:15:38.087738    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58575358d352bd7bd204f26186106be5d4d3bb920881611d0f6cb31f79938748\": container with ID starting with 58575358d352bd7bd204f26186106be5d4d3bb920881611d0f6cb31f79938748 not found: ID does not exist" containerID="58575358d352bd7bd204f26186106be5d4d3bb920881611d0f6cb31f79938748"
Nov 02 20:15:38.087816 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:38.087764    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58575358d352bd7bd204f26186106be5d4d3bb920881611d0f6cb31f79938748"} err="failed to get container status \"58575358d352bd7bd204f26186106be5d4d3bb920881611d0f6cb31f79938748\": rpc error: code = NotFound desc = could not find container \"58575358d352bd7bd204f26186106be5d4d3bb920881611d0f6cb31f79938748\": container with ID starting with 58575358d352bd7bd204f26186106be5d4d3bb920881611d0f6cb31f79938748 not found: ID does not exist"
Nov 02 20:15:38.229286 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:38.229239    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-5c46af-7f86f8f5c5-jxhp6"]
Nov 02 20:15:38.274963 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:38.274934    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-5c46af-7f86f8f5c5-jxhp6"]
Nov 02 20:15:38.846024 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:38.845973    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-838bc1-openshift-default-5fbf6db645-sx9sd"]
Nov 02 20:15:38.846572 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:38.846526    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-838bc1-openshift-default-5fbf6db645-sx9sd" podUID="c4fc2355-c8a3-4c51-9302-f93ced5c343e" containerName="istio-proxy" containerID="cri-o://4e6c435aa00e1cf2706c79d8aa6e774bcb5f347e7c468887b72060ef279e4924" gracePeriod=30
Nov 02 20:15:39.584794 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:39.584756    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="058aa7d8-1404-494f-b25b-70e8b702455b" path="/var/lib/kubelet/pods/058aa7d8-1404-494f-b25b-70e8b702455b/volumes"
Nov 02 20:15:40.043925 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:40.043885    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-5c46af/distinct-w3-5c46af-kserve-68b6dff7f5-bp59f" podUID="8dd6362e-7dbd-489b-b6d8-0d9caab0fa69" containerName="main" containerID="cri-o://84eaf8eee9246dfdc63b511f35cf6cf110b28c0ed17a025bb941243d3ab38b9b" gracePeriod=45
Nov 02 20:15:40.305185 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:40.305119    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5c46af/distinct-w3-5c46af-kserve-68b6dff7f5-bp59f"
Nov 02 20:15:40.333131 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:40.333098    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dd6362e-7dbd-489b-b6d8-0d9caab0fa69-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8dd6362e-7dbd-489b-b6d8-0d9caab0fa69-dshm\") pod \"8dd6362e-7dbd-489b-b6d8-0d9caab0fa69\" (UID: \"8dd6362e-7dbd-489b-b6d8-0d9caab0fa69\") "
Nov 02 20:15:40.333297 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:40.333137    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dd6362e-7dbd-489b-b6d8-0d9caab0fa69-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8dd6362e-7dbd-489b-b6d8-0d9caab0fa69-tmp-dir\") pod \"8dd6362e-7dbd-489b-b6d8-0d9caab0fa69\" (UID: \"8dd6362e-7dbd-489b-b6d8-0d9caab0fa69\") "
Nov 02 20:15:40.333297 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:40.333214    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8dd6362e-7dbd-489b-b6d8-0d9caab0fa69-tls-certs\" (UniqueName: \"kubernetes.io/secret/8dd6362e-7dbd-489b-b6d8-0d9caab0fa69-tls-certs\") pod \"8dd6362e-7dbd-489b-b6d8-0d9caab0fa69\" (UID: \"8dd6362e-7dbd-489b-b6d8-0d9caab0fa69\") "
Nov 02 20:15:40.333297 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:40.333234    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8dd6362e-7dbd-489b-b6d8-0d9caab0fa69-kube-api-access-k7gwp\" (UniqueName: \"kubernetes.io/projected/8dd6362e-7dbd-489b-b6d8-0d9caab0fa69-kube-api-access-k7gwp\") pod \"8dd6362e-7dbd-489b-b6d8-0d9caab0fa69\" (UID: \"8dd6362e-7dbd-489b-b6d8-0d9caab0fa69\") "
Nov 02 20:15:40.333297 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:40.333266    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dd6362e-7dbd-489b-b6d8-0d9caab0fa69-home\" (UniqueName: \"kubernetes.io/empty-dir/8dd6362e-7dbd-489b-b6d8-0d9caab0fa69-home\") pod \"8dd6362e-7dbd-489b-b6d8-0d9caab0fa69\" (UID: \"8dd6362e-7dbd-489b-b6d8-0d9caab0fa69\") "
Nov 02 20:15:40.333525 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:40.333320    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dd6362e-7dbd-489b-b6d8-0d9caab0fa69-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dd6362e-7dbd-489b-b6d8-0d9caab0fa69-model-cache\") pod \"8dd6362e-7dbd-489b-b6d8-0d9caab0fa69\" (UID: \"8dd6362e-7dbd-489b-b6d8-0d9caab0fa69\") "
Nov 02 20:15:40.333525 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:40.333494    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dd6362e-7dbd-489b-b6d8-0d9caab0fa69-tmp-dir" pod "8dd6362e-7dbd-489b-b6d8-0d9caab0fa69" (UID: "8dd6362e-7dbd-489b-b6d8-0d9caab0fa69"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:15:40.333525 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:40.333508    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dd6362e-7dbd-489b-b6d8-0d9caab0fa69-home" pod "8dd6362e-7dbd-489b-b6d8-0d9caab0fa69" (UID: "8dd6362e-7dbd-489b-b6d8-0d9caab0fa69"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:15:40.333786 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:40.333762    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dd6362e-7dbd-489b-b6d8-0d9caab0fa69-model-cache" pod "8dd6362e-7dbd-489b-b6d8-0d9caab0fa69" (UID: "8dd6362e-7dbd-489b-b6d8-0d9caab0fa69"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:15:40.333929 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:40.333913    4463 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8dd6362e-7dbd-489b-b6d8-0d9caab0fa69-home\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:40.333984 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:40.333934    4463 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dd6362e-7dbd-489b-b6d8-0d9caab0fa69-model-cache\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:40.333984 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:40.333948    4463 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8dd6362e-7dbd-489b-b6d8-0d9caab0fa69-tmp-dir\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:40.335500 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:40.335475    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dd6362e-7dbd-489b-b6d8-0d9caab0fa69-kube-api-access-k7gwp" pod "8dd6362e-7dbd-489b-b6d8-0d9caab0fa69" (UID: "8dd6362e-7dbd-489b-b6d8-0d9caab0fa69"). InnerVolumeSpecName "kube-api-access-k7gwp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:15:40.335596 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:40.335495    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8dd6362e-7dbd-489b-b6d8-0d9caab0fa69-tls-certs" pod "8dd6362e-7dbd-489b-b6d8-0d9caab0fa69" (UID: "8dd6362e-7dbd-489b-b6d8-0d9caab0fa69"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:15:40.335596 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:40.335565    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dd6362e-7dbd-489b-b6d8-0d9caab0fa69-dshm" pod "8dd6362e-7dbd-489b-b6d8-0d9caab0fa69" (UID: "8dd6362e-7dbd-489b-b6d8-0d9caab0fa69"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:15:40.439079 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:40.439047    4463 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8dd6362e-7dbd-489b-b6d8-0d9caab0fa69-dshm\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:40.439281 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:40.439265    4463 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dd6362e-7dbd-489b-b6d8-0d9caab0fa69-tls-certs\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:40.439395 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:40.439382    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7gwp\" (UniqueName: \"kubernetes.io/projected/8dd6362e-7dbd-489b-b6d8-0d9caab0fa69-kube-api-access-k7gwp\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:41.097445 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:41.097412    4463 generic.go:356] "Generic (PLEG): container finished" podID="8dd6362e-7dbd-489b-b6d8-0d9caab0fa69" containerID="84eaf8eee9246dfdc63b511f35cf6cf110b28c0ed17a025bb941243d3ab38b9b" exitCode=0
Nov 02 20:15:41.097883 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:41.097454    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5c46af/distinct-w3-5c46af-kserve-68b6dff7f5-bp59f" event={"ID":"8dd6362e-7dbd-489b-b6d8-0d9caab0fa69","Type":"ContainerDied","Data":"84eaf8eee9246dfdc63b511f35cf6cf110b28c0ed17a025bb941243d3ab38b9b"}
Nov 02 20:15:41.097883 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:41.097470    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5c46af/distinct-w3-5c46af-kserve-68b6dff7f5-bp59f"
Nov 02 20:15:41.097883 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:41.097485    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5c46af/distinct-w3-5c46af-kserve-68b6dff7f5-bp59f" event={"ID":"8dd6362e-7dbd-489b-b6d8-0d9caab0fa69","Type":"ContainerDied","Data":"360f3139739280293c1813c029b8b3876efe9f5bf38f82fcfd3ccbaa2d4fc518"}
Nov 02 20:15:41.097883 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:41.097501    4463 scope.go:122] "RemoveContainer" containerID="84eaf8eee9246dfdc63b511f35cf6cf110b28c0ed17a025bb941243d3ab38b9b"
Nov 02 20:15:41.107419 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:41.107391    4463 scope.go:122] "RemoveContainer" containerID="84eaf8eee9246dfdc63b511f35cf6cf110b28c0ed17a025bb941243d3ab38b9b"
Nov 02 20:15:41.107674 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:15:41.107654    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84eaf8eee9246dfdc63b511f35cf6cf110b28c0ed17a025bb941243d3ab38b9b\": container with ID starting with 84eaf8eee9246dfdc63b511f35cf6cf110b28c0ed17a025bb941243d3ab38b9b not found: ID does not exist" containerID="84eaf8eee9246dfdc63b511f35cf6cf110b28c0ed17a025bb941243d3ab38b9b"
Nov 02 20:15:41.107742 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:41.107681    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84eaf8eee9246dfdc63b511f35cf6cf110b28c0ed17a025bb941243d3ab38b9b"} err="failed to get container status \"84eaf8eee9246dfdc63b511f35cf6cf110b28c0ed17a025bb941243d3ab38b9b\": rpc error: code = NotFound desc = could not find container \"84eaf8eee9246dfdc63b511f35cf6cf110b28c0ed17a025bb941243d3ab38b9b\": container with ID starting with 84eaf8eee9246dfdc63b511f35cf6cf110b28c0ed17a025bb941243d3ab38b9b not found: ID does not exist"
Nov 02 20:15:41.249856 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:41.249824    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-5c46af/distinct-w3-5c46af-kserve-68b6dff7f5-bp59f"]
Nov 02 20:15:41.252120 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:41.252093    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-5c46af/distinct-w3-5c46af-kserve-68b6dff7f5-bp59f"]
Nov 02 20:15:41.586692 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:41.586659    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8dd6362e-7dbd-489b-b6d8-0d9caab0fa69" path="/var/lib/kubelet/pods/8dd6362e-7dbd-489b-b6d8-0d9caab0fa69/volumes"
Nov 02 20:15:44.116398 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:44.116368    4463 generic.go:356] "Generic (PLEG): container finished" podID="c4fc2355-c8a3-4c51-9302-f93ced5c343e" containerID="4e6c435aa00e1cf2706c79d8aa6e774bcb5f347e7c468887b72060ef279e4924" exitCode=0
Nov 02 20:15:44.116736 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:44.116408    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-838bc1-openshift-default-5fbf6db645-sx9sd" event={"ID":"c4fc2355-c8a3-4c51-9302-f93ced5c343e","Type":"ContainerDied","Data":"4e6c435aa00e1cf2706c79d8aa6e774bcb5f347e7c468887b72060ef279e4924"}
Nov 02 20:15:44.116736 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:44.116455    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-838bc1-openshift-default-5fbf6db645-sx9sd" event={"ID":"c4fc2355-c8a3-4c51-9302-f93ced5c343e","Type":"ContainerDied","Data":"6a4b48808c0645a45e4180e7a6bf8048b74b23e25dcb647e9b2730073c3d5815"}
Nov 02 20:15:44.116736 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:44.116474    4463 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a4b48808c0645a45e4180e7a6bf8048b74b23e25dcb647e9b2730073c3d5815"
Nov 02 20:15:44.116736 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:44.116489    4463 scope.go:122] "RemoveContainer" containerID="b285fc201b6ce043240d90cf63f62f54f1dc7b558f56265afe4c032c16bc603d"
Nov 02 20:15:44.130761 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:44.130740    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-838bc1-openshift-default-5fbf6db645-sx9sd"
Nov 02 20:15:44.175440 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:44.175413    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4fc2355-c8a3-4c51-9302-f93ced5c343e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4fc2355-c8a3-4c51-9302-f93ced5c343e-istio-envoy\") pod \"c4fc2355-c8a3-4c51-9302-f93ced5c343e\" (UID: \"c4fc2355-c8a3-4c51-9302-f93ced5c343e\") "
Nov 02 20:15:44.175537 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:44.175492    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4fc2355-c8a3-4c51-9302-f93ced5c343e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4fc2355-c8a3-4c51-9302-f93ced5c343e-workload-socket\") pod \"c4fc2355-c8a3-4c51-9302-f93ced5c343e\" (UID: \"c4fc2355-c8a3-4c51-9302-f93ced5c343e\") "
Nov 02 20:15:44.175537 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:44.175522    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4fc2355-c8a3-4c51-9302-f93ced5c343e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4fc2355-c8a3-4c51-9302-f93ced5c343e-credential-socket\") pod \"c4fc2355-c8a3-4c51-9302-f93ced5c343e\" (UID: \"c4fc2355-c8a3-4c51-9302-f93ced5c343e\") "
Nov 02 20:15:44.175619 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:44.175601    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c4fc2355-c8a3-4c51-9302-f93ced5c343e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4fc2355-c8a3-4c51-9302-f93ced5c343e-istio-podinfo\") pod \"c4fc2355-c8a3-4c51-9302-f93ced5c343e\" (UID: \"c4fc2355-c8a3-4c51-9302-f93ced5c343e\") "
Nov 02 20:15:44.175661 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:44.175643    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4fc2355-c8a3-4c51-9302-f93ced5c343e-istio-token\" (UniqueName: \"kubernetes.io/projected/c4fc2355-c8a3-4c51-9302-f93ced5c343e-istio-token\") pod \"c4fc2355-c8a3-4c51-9302-f93ced5c343e\" (UID: \"c4fc2355-c8a3-4c51-9302-f93ced5c343e\") "
Nov 02 20:15:44.175716 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:44.175701    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4fc2355-c8a3-4c51-9302-f93ced5c343e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4fc2355-c8a3-4c51-9302-f93ced5c343e-istiod-ca-cert\") pod \"c4fc2355-c8a3-4c51-9302-f93ced5c343e\" (UID: \"c4fc2355-c8a3-4c51-9302-f93ced5c343e\") "
Nov 02 20:15:44.175761 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:44.175742    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4fc2355-c8a3-4c51-9302-f93ced5c343e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4fc2355-c8a3-4c51-9302-f93ced5c343e-workload-certs\") pod \"c4fc2355-c8a3-4c51-9302-f93ced5c343e\" (UID: \"c4fc2355-c8a3-4c51-9302-f93ced5c343e\") "
Nov 02 20:15:44.175824 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:44.175769    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4fc2355-c8a3-4c51-9302-f93ced5c343e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4fc2355-c8a3-4c51-9302-f93ced5c343e-istio-data\") pod \"c4fc2355-c8a3-4c51-9302-f93ced5c343e\" (UID: \"c4fc2355-c8a3-4c51-9302-f93ced5c343e\") "
Nov 02 20:15:44.175824 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:44.175776    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4fc2355-c8a3-4c51-9302-f93ced5c343e-workload-socket" pod "c4fc2355-c8a3-4c51-9302-f93ced5c343e" (UID: "c4fc2355-c8a3-4c51-9302-f93ced5c343e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:15:44.175930 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:44.175849    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4fc2355-c8a3-4c51-9302-f93ced5c343e-kube-api-access-d9zbq\" (UniqueName: \"kubernetes.io/projected/c4fc2355-c8a3-4c51-9302-f93ced5c343e-kube-api-access-d9zbq\") pod \"c4fc2355-c8a3-4c51-9302-f93ced5c343e\" (UID: \"c4fc2355-c8a3-4c51-9302-f93ced5c343e\") "
Nov 02 20:15:44.175930 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:44.175863    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4fc2355-c8a3-4c51-9302-f93ced5c343e-credential-socket" pod "c4fc2355-c8a3-4c51-9302-f93ced5c343e" (UID: "c4fc2355-c8a3-4c51-9302-f93ced5c343e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:15:44.176070 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:44.176050    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4fc2355-c8a3-4c51-9302-f93ced5c343e-workload-certs" pod "c4fc2355-c8a3-4c51-9302-f93ced5c343e" (UID: "c4fc2355-c8a3-4c51-9302-f93ced5c343e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:15:44.176363 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:44.176112    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4fc2355-c8a3-4c51-9302-f93ced5c343e-istiod-ca-cert" pod "c4fc2355-c8a3-4c51-9302-f93ced5c343e" (UID: "c4fc2355-c8a3-4c51-9302-f93ced5c343e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:15:44.176458 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:44.176130    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4fc2355-c8a3-4c51-9302-f93ced5c343e-istio-data" pod "c4fc2355-c8a3-4c51-9302-f93ced5c343e" (UID: "c4fc2355-c8a3-4c51-9302-f93ced5c343e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:15:44.176458 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:44.176346    4463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4fc2355-c8a3-4c51-9302-f93ced5c343e-workload-socket\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:44.176458 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:44.176402    4463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4fc2355-c8a3-4c51-9302-f93ced5c343e-credential-socket\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:44.177686 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:44.177660    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c4fc2355-c8a3-4c51-9302-f93ced5c343e-istio-podinfo" pod "c4fc2355-c8a3-4c51-9302-f93ced5c343e" (UID: "c4fc2355-c8a3-4c51-9302-f93ced5c343e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:15:44.177796 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:44.177738    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4fc2355-c8a3-4c51-9302-f93ced5c343e-istio-token" pod "c4fc2355-c8a3-4c51-9302-f93ced5c343e" (UID: "c4fc2355-c8a3-4c51-9302-f93ced5c343e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:15:44.177796 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:44.177747    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4fc2355-c8a3-4c51-9302-f93ced5c343e-istio-envoy" pod "c4fc2355-c8a3-4c51-9302-f93ced5c343e" (UID: "c4fc2355-c8a3-4c51-9302-f93ced5c343e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:15:44.177996 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:44.177977    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4fc2355-c8a3-4c51-9302-f93ced5c343e-kube-api-access-d9zbq" pod "c4fc2355-c8a3-4c51-9302-f93ced5c343e" (UID: "c4fc2355-c8a3-4c51-9302-f93ced5c343e"). InnerVolumeSpecName "kube-api-access-d9zbq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:15:44.277574 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:44.277540    4463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4fc2355-c8a3-4c51-9302-f93ced5c343e-istio-envoy\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:44.277574 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:44.277572    4463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4fc2355-c8a3-4c51-9302-f93ced5c343e-istio-podinfo\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:44.277574 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:44.277583    4463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4fc2355-c8a3-4c51-9302-f93ced5c343e-istio-token\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:44.277742 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:44.277593    4463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4fc2355-c8a3-4c51-9302-f93ced5c343e-istiod-ca-cert\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:44.277742 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:44.277602    4463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4fc2355-c8a3-4c51-9302-f93ced5c343e-workload-certs\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:44.277742 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:44.277610    4463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4fc2355-c8a3-4c51-9302-f93ced5c343e-istio-data\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:44.277742 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:44.277619    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9zbq\" (UniqueName: \"kubernetes.io/projected/c4fc2355-c8a3-4c51-9302-f93ced5c343e-kube-api-access-d9zbq\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:45.124376 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:45.124346    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-838bc1-openshift-default-5fbf6db645-sx9sd"
Nov 02 20:15:45.389395 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:45.389307    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-838bc1-openshift-default-5fbf6db645-sx9sd"]
Nov 02 20:15:45.392500 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:45.392470    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-838bc1-openshift-default-5fbf6db645-sx9sd"]
Nov 02 20:15:45.584785 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:45.584751    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4fc2355-c8a3-4c51-9302-f93ced5c343e" path="/var/lib/kubelet/pods/c4fc2355-c8a3-4c51-9302-f93ced5c343e/volumes"
Nov 02 20:15:51.176052 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:51.176002    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-28fa24-8f6447cc7-x6jsz"]
Nov 02 20:15:51.291331 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:51.291297    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-28fa24-6d779499-gvdmm"]
Nov 02 20:15:53.393663 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:53.393629    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-13e739/test-model-13e739-kserve-84bd6cf795-5hmft"]
Nov 02 20:15:56.177241 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:56.177201    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-28fa24-8f6447cc7-x6jsz" podUID="33f548d7-2602-477f-a2b4-80196795d489" containerName="payload-processing" containerID="cri-o://e300f19464c523bb1bc69c95cabd8e341d01938d4b242863a44c2d3626ee0b62" gracePeriod=25
Nov 02 20:15:56.247303 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:56.247273    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-19d134-5fc6b854f5-k78vk"]
Nov 02 20:15:56.292665 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:56.292607    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-28fa24-6d779499-gvdmm" podUID="db50c476-4694-4cd0-8fa0-cca68002faa8" containerName="payload-pre-processing" containerID="cri-o://65949a11aaa2ea0b9de4ade7625c0cdb4d601c7888a8d38378c9b91d5ea7c72a" gracePeriod=25
Nov 02 20:15:56.424777 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:56.424756    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-28fa24-8f6447cc7-x6jsz"
Nov 02 20:15:56.500280 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:56.500200    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33f548d7-2602-477f-a2b4-80196795d489-kube-api-access-zfgn4\" (UniqueName: \"kubernetes.io/projected/33f548d7-2602-477f-a2b4-80196795d489-kube-api-access-zfgn4\") pod \"33f548d7-2602-477f-a2b4-80196795d489\" (UID: \"33f548d7-2602-477f-a2b4-80196795d489\") "
Nov 02 20:15:56.500443 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:56.500292    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33f548d7-2602-477f-a2b4-80196795d489-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33f548d7-2602-477f-a2b4-80196795d489-plugins-config-volume\") pod \"33f548d7-2602-477f-a2b4-80196795d489\" (UID: \"33f548d7-2602-477f-a2b4-80196795d489\") "
Nov 02 20:15:56.500717 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:56.500691    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33f548d7-2602-477f-a2b4-80196795d489-plugins-config-volume" pod "33f548d7-2602-477f-a2b4-80196795d489" (UID: "33f548d7-2602-477f-a2b4-80196795d489"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:15:56.500821 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:56.500807    4463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33f548d7-2602-477f-a2b4-80196795d489-plugins-config-volume\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:56.502347 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:56.502326    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33f548d7-2602-477f-a2b4-80196795d489-kube-api-access-zfgn4" pod "33f548d7-2602-477f-a2b4-80196795d489" (UID: "33f548d7-2602-477f-a2b4-80196795d489"). InnerVolumeSpecName "kube-api-access-zfgn4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:15:56.602039 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:56.601994    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfgn4\" (UniqueName: \"kubernetes.io/projected/33f548d7-2602-477f-a2b4-80196795d489-kube-api-access-zfgn4\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:57.190769 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:57.190731    4463 generic.go:356] "Generic (PLEG): container finished" podID="33f548d7-2602-477f-a2b4-80196795d489" containerID="e300f19464c523bb1bc69c95cabd8e341d01938d4b242863a44c2d3626ee0b62" exitCode=0
Nov 02 20:15:57.191291 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:57.190778    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-28fa24-8f6447cc7-x6jsz"
Nov 02 20:15:57.191291 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:57.190807    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-28fa24-8f6447cc7-x6jsz" event={"ID":"33f548d7-2602-477f-a2b4-80196795d489","Type":"ContainerDied","Data":"e300f19464c523bb1bc69c95cabd8e341d01938d4b242863a44c2d3626ee0b62"}
Nov 02 20:15:57.191291 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:57.190859    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-28fa24-8f6447cc7-x6jsz" event={"ID":"33f548d7-2602-477f-a2b4-80196795d489","Type":"ContainerDied","Data":"2f3d901ca3464774b643b22a975853b59ca26d4fd27ee897ddfe952b34ed2ac9"}
Nov 02 20:15:57.191291 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:57.190881    4463 scope.go:122] "RemoveContainer" containerID="e300f19464c523bb1bc69c95cabd8e341d01938d4b242863a44c2d3626ee0b62"
Nov 02 20:15:57.200446 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:57.200431    4463 scope.go:122] "RemoveContainer" containerID="e300f19464c523bb1bc69c95cabd8e341d01938d4b242863a44c2d3626ee0b62"
Nov 02 20:15:57.200689 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:15:57.200671    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e300f19464c523bb1bc69c95cabd8e341d01938d4b242863a44c2d3626ee0b62\": container with ID starting with e300f19464c523bb1bc69c95cabd8e341d01938d4b242863a44c2d3626ee0b62 not found: ID does not exist" containerID="e300f19464c523bb1bc69c95cabd8e341d01938d4b242863a44c2d3626ee0b62"
Nov 02 20:15:57.200756 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:57.200694    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e300f19464c523bb1bc69c95cabd8e341d01938d4b242863a44c2d3626ee0b62"} err="failed to get container status \"e300f19464c523bb1bc69c95cabd8e341d01938d4b242863a44c2d3626ee0b62\": rpc error: code = NotFound desc = could not find container \"e300f19464c523bb1bc69c95cabd8e341d01938d4b242863a44c2d3626ee0b62\": container with ID starting with e300f19464c523bb1bc69c95cabd8e341d01938d4b242863a44c2d3626ee0b62 not found: ID does not exist"
Nov 02 20:15:57.343697 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:57.343666    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-28fa24-8f6447cc7-x6jsz"]
Nov 02 20:15:57.345777 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:57.345753    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-28fa24-8f6447cc7-x6jsz"]
Nov 02 20:15:57.585418 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:57.585346    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33f548d7-2602-477f-a2b4-80196795d489" path="/var/lib/kubelet/pods/33f548d7-2602-477f-a2b4-80196795d489/volumes"
Nov 02 20:15:58.833040 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:15:58.832993    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a5b64d-59fd995f49-4g9cm"]
Nov 02 20:16:01.248043 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:01.247993    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-19d134-5fc6b854f5-k78vk" podUID="d6bed081-d1c4-426f-9611-81894a7b3e57" containerName="payload-processing" containerID="cri-o://da44ded62f93416710d44746e0d53d41c55993ae66afd70a42d6c7c243e17b1e" gracePeriod=25
Nov 02 20:16:01.524844 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:01.524795    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-19d134-5fc6b854f5-k78vk"
Nov 02 20:16:01.654067 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:01.654037    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6bed081-d1c4-426f-9611-81894a7b3e57-kube-api-access-fkcbb\" (UniqueName: \"kubernetes.io/projected/d6bed081-d1c4-426f-9611-81894a7b3e57-kube-api-access-fkcbb\") pod \"d6bed081-d1c4-426f-9611-81894a7b3e57\" (UID: \"d6bed081-d1c4-426f-9611-81894a7b3e57\") "
Nov 02 20:16:01.654242 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:01.654210    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6bed081-d1c4-426f-9611-81894a7b3e57-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6bed081-d1c4-426f-9611-81894a7b3e57-plugins-config-volume\") pod \"d6bed081-d1c4-426f-9611-81894a7b3e57\" (UID: \"d6bed081-d1c4-426f-9611-81894a7b3e57\") "
Nov 02 20:16:01.654632 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:01.654597    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6bed081-d1c4-426f-9611-81894a7b3e57-plugins-config-volume" pod "d6bed081-d1c4-426f-9611-81894a7b3e57" (UID: "d6bed081-d1c4-426f-9611-81894a7b3e57"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:16:01.656779 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:01.656740    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6bed081-d1c4-426f-9611-81894a7b3e57-kube-api-access-fkcbb" pod "d6bed081-d1c4-426f-9611-81894a7b3e57" (UID: "d6bed081-d1c4-426f-9611-81894a7b3e57"). InnerVolumeSpecName "kube-api-access-fkcbb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:16:01.755735 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:01.755673    4463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6bed081-d1c4-426f-9611-81894a7b3e57-plugins-config-volume\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:16:01.755735 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:01.755697    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkcbb\" (UniqueName: \"kubernetes.io/projected/d6bed081-d1c4-426f-9611-81894a7b3e57-kube-api-access-fkcbb\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:16:02.221543 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:02.221512    4463 generic.go:356] "Generic (PLEG): container finished" podID="d6bed081-d1c4-426f-9611-81894a7b3e57" containerID="da44ded62f93416710d44746e0d53d41c55993ae66afd70a42d6c7c243e17b1e" exitCode=0
Nov 02 20:16:02.221723 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:02.221556    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-19d134-5fc6b854f5-k78vk"
Nov 02 20:16:02.221723 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:02.221597    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-19d134-5fc6b854f5-k78vk" event={"ID":"d6bed081-d1c4-426f-9611-81894a7b3e57","Type":"ContainerDied","Data":"da44ded62f93416710d44746e0d53d41c55993ae66afd70a42d6c7c243e17b1e"}
Nov 02 20:16:02.221723 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:02.221648    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-19d134-5fc6b854f5-k78vk" event={"ID":"d6bed081-d1c4-426f-9611-81894a7b3e57","Type":"ContainerDied","Data":"7b60d7f276f6b9a37c1c6f820cda5171404257227379e3a6461e36643b106c71"}
Nov 02 20:16:02.221723 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:02.221669    4463 scope.go:122] "RemoveContainer" containerID="da44ded62f93416710d44746e0d53d41c55993ae66afd70a42d6c7c243e17b1e"
Nov 02 20:16:02.231366 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:02.231352    4463 scope.go:122] "RemoveContainer" containerID="da44ded62f93416710d44746e0d53d41c55993ae66afd70a42d6c7c243e17b1e"
Nov 02 20:16:02.231627 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:16:02.231612    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da44ded62f93416710d44746e0d53d41c55993ae66afd70a42d6c7c243e17b1e\": container with ID starting with da44ded62f93416710d44746e0d53d41c55993ae66afd70a42d6c7c243e17b1e not found: ID does not exist" containerID="da44ded62f93416710d44746e0d53d41c55993ae66afd70a42d6c7c243e17b1e"
Nov 02 20:16:02.231668 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:02.231634    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da44ded62f93416710d44746e0d53d41c55993ae66afd70a42d6c7c243e17b1e"} err="failed to get container status \"da44ded62f93416710d44746e0d53d41c55993ae66afd70a42d6c7c243e17b1e\": rpc error: code = NotFound desc = could not find container \"da44ded62f93416710d44746e0d53d41c55993ae66afd70a42d6c7c243e17b1e\": container with ID starting with da44ded62f93416710d44746e0d53d41c55993ae66afd70a42d6c7c243e17b1e not found: ID does not exist"
Nov 02 20:16:02.373535 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:02.373505    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-19d134-5fc6b854f5-k78vk"]
Nov 02 20:16:02.419545 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:02.419520    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-19d134-5fc6b854f5-k78vk"]
Nov 02 20:16:02.427851 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:02.427817    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-28fa24-6d779499-gvdmm" podUID="db50c476-4694-4cd0-8fa0-cca68002faa8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 20:16:03.584208 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:03.584176    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6bed081-d1c4-426f-9611-81894a7b3e57" path="/var/lib/kubelet/pods/d6bed081-d1c4-426f-9611-81894a7b3e57/volumes"
Nov 02 20:16:03.833964 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:03.833905    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-a5b64d-59fd995f49-4g9cm" podUID="7f250661-9b39-4d06-af9c-6f067b82f8ca" containerName="payload-processing" containerID="cri-o://a8df302061409f44d31d2fdfe07afae4293717c73ec6ea4b1168d83f289eefe4" gracePeriod=25
Nov 02 20:16:04.295899 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:16:04.295864    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 20:16:08.413399 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:08.413342    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-13e739/test-model-13e739-kserve-84bd6cf795-5hmft" podUID="3e95fad5-268c-4d06-866e-ed10c87a78d0" containerName="main" containerID="cri-o://f5460c2f88a7f2a85259edc56efdbaa16162a8c9bf44a578ab2a51cc568b9cbd" gracePeriod=45
Nov 02 20:16:08.650262 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:08.650231    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 20:16:08.661999 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:08.661975    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-13e739/test-model-13e739-kserve-84bd6cf795-5hmft"
Nov 02 20:16:08.827569 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:08.827539    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e95fad5-268c-4d06-866e-ed10c87a78d0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e95fad5-268c-4d06-866e-ed10c87a78d0-model-cache\") pod \"3e95fad5-268c-4d06-866e-ed10c87a78d0\" (UID: \"3e95fad5-268c-4d06-866e-ed10c87a78d0\") "
Nov 02 20:16:08.827727 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:08.827615    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e95fad5-268c-4d06-866e-ed10c87a78d0-home\" (UniqueName: \"kubernetes.io/empty-dir/3e95fad5-268c-4d06-866e-ed10c87a78d0-home\") pod \"3e95fad5-268c-4d06-866e-ed10c87a78d0\" (UID: \"3e95fad5-268c-4d06-866e-ed10c87a78d0\") "
Nov 02 20:16:08.827727 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:08.827676    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e95fad5-268c-4d06-866e-ed10c87a78d0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e95fad5-268c-4d06-866e-ed10c87a78d0-tmp-dir\") pod \"3e95fad5-268c-4d06-866e-ed10c87a78d0\" (UID: \"3e95fad5-268c-4d06-866e-ed10c87a78d0\") "
Nov 02 20:16:08.827727 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:08.827716    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e95fad5-268c-4d06-866e-ed10c87a78d0-kube-api-access-x66wv\" (UniqueName: \"kubernetes.io/projected/3e95fad5-268c-4d06-866e-ed10c87a78d0-kube-api-access-x66wv\") pod \"3e95fad5-268c-4d06-866e-ed10c87a78d0\" (UID: \"3e95fad5-268c-4d06-866e-ed10c87a78d0\") "
Nov 02 20:16:08.827855 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:08.827768    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e95fad5-268c-4d06-866e-ed10c87a78d0-tls-certs\" (UniqueName: \"kubernetes.io/secret/3e95fad5-268c-4d06-866e-ed10c87a78d0-tls-certs\") pod \"3e95fad5-268c-4d06-866e-ed10c87a78d0\" (UID: \"3e95fad5-268c-4d06-866e-ed10c87a78d0\") "
Nov 02 20:16:08.827855 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:08.827793    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e95fad5-268c-4d06-866e-ed10c87a78d0-model-cache" pod "3e95fad5-268c-4d06-866e-ed10c87a78d0" (UID: "3e95fad5-268c-4d06-866e-ed10c87a78d0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:16:08.827855 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:08.827797    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e95fad5-268c-4d06-866e-ed10c87a78d0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e95fad5-268c-4d06-866e-ed10c87a78d0-dshm\") pod \"3e95fad5-268c-4d06-866e-ed10c87a78d0\" (UID: \"3e95fad5-268c-4d06-866e-ed10c87a78d0\") "
Nov 02 20:16:08.828123 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:08.827876    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e95fad5-268c-4d06-866e-ed10c87a78d0-home" pod "3e95fad5-268c-4d06-866e-ed10c87a78d0" (UID: "3e95fad5-268c-4d06-866e-ed10c87a78d0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:16:08.828123 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:08.827984    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e95fad5-268c-4d06-866e-ed10c87a78d0-tmp-dir" pod "3e95fad5-268c-4d06-866e-ed10c87a78d0" (UID: "3e95fad5-268c-4d06-866e-ed10c87a78d0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:16:08.828276 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:08.828259    4463 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e95fad5-268c-4d06-866e-ed10c87a78d0-tmp-dir\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:16:08.828333 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:08.828282    4463 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e95fad5-268c-4d06-866e-ed10c87a78d0-model-cache\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:16:08.828333 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:08.828303    4463 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e95fad5-268c-4d06-866e-ed10c87a78d0-home\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:16:08.829963 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:08.829940    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e95fad5-268c-4d06-866e-ed10c87a78d0-kube-api-access-x66wv" pod "3e95fad5-268c-4d06-866e-ed10c87a78d0" (UID: "3e95fad5-268c-4d06-866e-ed10c87a78d0"). InnerVolumeSpecName "kube-api-access-x66wv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:16:08.830057 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:08.829945    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e95fad5-268c-4d06-866e-ed10c87a78d0-tls-certs" pod "3e95fad5-268c-4d06-866e-ed10c87a78d0" (UID: "3e95fad5-268c-4d06-866e-ed10c87a78d0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:16:08.830057 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:08.830027    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e95fad5-268c-4d06-866e-ed10c87a78d0-dshm" pod "3e95fad5-268c-4d06-866e-ed10c87a78d0" (UID: "3e95fad5-268c-4d06-866e-ed10c87a78d0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:16:08.929126 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:08.929103    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x66wv\" (UniqueName: \"kubernetes.io/projected/3e95fad5-268c-4d06-866e-ed10c87a78d0-kube-api-access-x66wv\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:16:08.929210 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:08.929127    4463 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e95fad5-268c-4d06-866e-ed10c87a78d0-tls-certs\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:16:08.929210 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:08.929136    4463 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e95fad5-268c-4d06-866e-ed10c87a78d0-dshm\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:16:09.271065 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:09.270996    4463 generic.go:356] "Generic (PLEG): container finished" podID="3e95fad5-268c-4d06-866e-ed10c87a78d0" containerID="f5460c2f88a7f2a85259edc56efdbaa16162a8c9bf44a578ab2a51cc568b9cbd" exitCode=0
Nov 02 20:16:09.271243 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:09.271075    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-13e739/test-model-13e739-kserve-84bd6cf795-5hmft"
Nov 02 20:16:09.271243 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:09.271091    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-13e739/test-model-13e739-kserve-84bd6cf795-5hmft" event={"ID":"3e95fad5-268c-4d06-866e-ed10c87a78d0","Type":"ContainerDied","Data":"f5460c2f88a7f2a85259edc56efdbaa16162a8c9bf44a578ab2a51cc568b9cbd"}
Nov 02 20:16:09.271243 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:09.271130    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-13e739/test-model-13e739-kserve-84bd6cf795-5hmft" event={"ID":"3e95fad5-268c-4d06-866e-ed10c87a78d0","Type":"ContainerDied","Data":"b9c5a7562a3f633aa04fc72f71f79532eedecc6e48f4f40ba27bb46cda32eeed"}
Nov 02 20:16:09.271243 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:09.271148    4463 scope.go:122] "RemoveContainer" containerID="f5460c2f88a7f2a85259edc56efdbaa16162a8c9bf44a578ab2a51cc568b9cbd"
Nov 02 20:16:09.280777 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:09.280756    4463 scope.go:122] "RemoveContainer" containerID="f5460c2f88a7f2a85259edc56efdbaa16162a8c9bf44a578ab2a51cc568b9cbd"
Nov 02 20:16:09.281066 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:16:09.281044    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5460c2f88a7f2a85259edc56efdbaa16162a8c9bf44a578ab2a51cc568b9cbd\": container with ID starting with f5460c2f88a7f2a85259edc56efdbaa16162a8c9bf44a578ab2a51cc568b9cbd not found: ID does not exist" containerID="f5460c2f88a7f2a85259edc56efdbaa16162a8c9bf44a578ab2a51cc568b9cbd"
Nov 02 20:16:09.281159 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:09.281071    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5460c2f88a7f2a85259edc56efdbaa16162a8c9bf44a578ab2a51cc568b9cbd"} err="failed to get container status \"f5460c2f88a7f2a85259edc56efdbaa16162a8c9bf44a578ab2a51cc568b9cbd\": rpc error: code = NotFound desc = could not find container \"f5460c2f88a7f2a85259edc56efdbaa16162a8c9bf44a578ab2a51cc568b9cbd\": container with ID starting with f5460c2f88a7f2a85259edc56efdbaa16162a8c9bf44a578ab2a51cc568b9cbd not found: ID does not exist"
Nov 02 20:16:09.422697 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:09.422653    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-13e739/test-model-13e739-kserve-84bd6cf795-5hmft"]
Nov 02 20:16:09.468758 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:09.468720    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-13e739/test-model-13e739-kserve-84bd6cf795-5hmft"]
Nov 02 20:16:09.586985 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:09.586901    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e95fad5-268c-4d06-866e-ed10c87a78d0" path="/var/lib/kubelet/pods/3e95fad5-268c-4d06-866e-ed10c87a78d0/volumes"
Nov 02 20:16:12.427973 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:12.427927    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-28fa24-6d779499-gvdmm" podUID="db50c476-4694-4cd0-8fa0-cca68002faa8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 20:16:12.918250 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:12.918161    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-a5b64d-59fd995f49-4g9cm" podUID="7f250661-9b39-4d06-af9c-6f067b82f8ca" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 20:16:14.559425 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:14.559395    4463 scope.go:122] "RemoveContainer" containerID="4e6c435aa00e1cf2706c79d8aa6e774bcb5f347e7c468887b72060ef279e4924"
Nov 02 20:16:16.548913 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:16.548853    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-28fa24-6d779499-gvdmm"
Nov 02 20:16:16.709900 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:16.709867    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db50c476-4694-4cd0-8fa0-cca68002faa8-kube-api-access-jqp82\" (UniqueName: \"kubernetes.io/projected/db50c476-4694-4cd0-8fa0-cca68002faa8-kube-api-access-jqp82\") pod \"db50c476-4694-4cd0-8fa0-cca68002faa8\" (UID: \"db50c476-4694-4cd0-8fa0-cca68002faa8\") "
Nov 02 20:16:16.710085 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:16.710044    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db50c476-4694-4cd0-8fa0-cca68002faa8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db50c476-4694-4cd0-8fa0-cca68002faa8-plugins-config-volume\") pod \"db50c476-4694-4cd0-8fa0-cca68002faa8\" (UID: \"db50c476-4694-4cd0-8fa0-cca68002faa8\") "
Nov 02 20:16:16.710405 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:16.710379    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db50c476-4694-4cd0-8fa0-cca68002faa8-plugins-config-volume" pod "db50c476-4694-4cd0-8fa0-cca68002faa8" (UID: "db50c476-4694-4cd0-8fa0-cca68002faa8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:16:16.711910 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:16.711887    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db50c476-4694-4cd0-8fa0-cca68002faa8-kube-api-access-jqp82" pod "db50c476-4694-4cd0-8fa0-cca68002faa8" (UID: "db50c476-4694-4cd0-8fa0-cca68002faa8"). InnerVolumeSpecName "kube-api-access-jqp82". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:16:16.811431 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:16.811359    4463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db50c476-4694-4cd0-8fa0-cca68002faa8-plugins-config-volume\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:16:16.811431 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:16.811386    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqp82\" (UniqueName: \"kubernetes.io/projected/db50c476-4694-4cd0-8fa0-cca68002faa8-kube-api-access-jqp82\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:16:17.324786 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:17.324752    4463 generic.go:356] "Generic (PLEG): container finished" podID="db50c476-4694-4cd0-8fa0-cca68002faa8" containerID="65949a11aaa2ea0b9de4ade7625c0cdb4d601c7888a8d38378c9b91d5ea7c72a" exitCode=0
Nov 02 20:16:17.324952 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:17.324797    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-28fa24-6d779499-gvdmm"
Nov 02 20:16:17.324952 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:17.324802    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-28fa24-6d779499-gvdmm" event={"ID":"db50c476-4694-4cd0-8fa0-cca68002faa8","Type":"ContainerDied","Data":"65949a11aaa2ea0b9de4ade7625c0cdb4d601c7888a8d38378c9b91d5ea7c72a"}
Nov 02 20:16:17.324952 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:17.324901    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-28fa24-6d779499-gvdmm" event={"ID":"db50c476-4694-4cd0-8fa0-cca68002faa8","Type":"ContainerDied","Data":"9fd93a2ba4c28d131456023d207373da91b6fc4a6675b08544b7c08d0eaf45a7"}
Nov 02 20:16:17.324952 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:17.324921    4463 scope.go:122] "RemoveContainer" containerID="65949a11aaa2ea0b9de4ade7625c0cdb4d601c7888a8d38378c9b91d5ea7c72a"
Nov 02 20:16:17.334381 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:17.334363    4463 scope.go:122] "RemoveContainer" containerID="65949a11aaa2ea0b9de4ade7625c0cdb4d601c7888a8d38378c9b91d5ea7c72a"
Nov 02 20:16:17.334626 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:16:17.334609    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65949a11aaa2ea0b9de4ade7625c0cdb4d601c7888a8d38378c9b91d5ea7c72a\": container with ID starting with 65949a11aaa2ea0b9de4ade7625c0cdb4d601c7888a8d38378c9b91d5ea7c72a not found: ID does not exist" containerID="65949a11aaa2ea0b9de4ade7625c0cdb4d601c7888a8d38378c9b91d5ea7c72a"
Nov 02 20:16:17.334670 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:17.334634    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65949a11aaa2ea0b9de4ade7625c0cdb4d601c7888a8d38378c9b91d5ea7c72a"} err="failed to get container status \"65949a11aaa2ea0b9de4ade7625c0cdb4d601c7888a8d38378c9b91d5ea7c72a\": rpc error: code = NotFound desc = could not find container \"65949a11aaa2ea0b9de4ade7625c0cdb4d601c7888a8d38378c9b91d5ea7c72a\": container with ID starting with 65949a11aaa2ea0b9de4ade7625c0cdb4d601c7888a8d38378c9b91d5ea7c72a not found: ID does not exist"
Nov 02 20:16:17.478615 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:17.478578    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-28fa24-6d779499-gvdmm"]
Nov 02 20:16:17.523730 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:17.523695    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-28fa24-6d779499-gvdmm"]
Nov 02 20:16:17.586983 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:17.586907    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db50c476-4694-4cd0-8fa0-cca68002faa8" path="/var/lib/kubelet/pods/db50c476-4694-4cd0-8fa0-cca68002faa8/volumes"
Nov 02 20:16:22.398098 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:22.398063    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a5b64d-64599d48c8-97d8b"]
Nov 02 20:16:22.398505 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:22.398254    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a5b64d-64599d48c8-97d8b" podUID="2ec406bc-f697-42e1-9a76-857c3a28d0d6" containerName="maas-api" containerID="cri-o://6bf04077320aaa5b1d2b9e3c747afe550dce38b189b9b602c2ed5f501f43b3f1" gracePeriod=30
Nov 02 20:16:22.659089 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:22.659031    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a5b64d-64599d48c8-97d8b"
Nov 02 20:16:22.666489 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:22.666467    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ec406bc-f697-42e1-9a76-857c3a28d0d6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ec406bc-f697-42e1-9a76-857c3a28d0d6-maas-api-tls\") pod \"2ec406bc-f697-42e1-9a76-857c3a28d0d6\" (UID: \"2ec406bc-f697-42e1-9a76-857c3a28d0d6\") "
Nov 02 20:16:22.666563 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:22.666551    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ec406bc-f697-42e1-9a76-857c3a28d0d6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ec406bc-f697-42e1-9a76-857c3a28d0d6-metrics-certs\") pod \"2ec406bc-f697-42e1-9a76-857c3a28d0d6\" (UID: \"2ec406bc-f697-42e1-9a76-857c3a28d0d6\") "
Nov 02 20:16:22.666600 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:22.666590    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ec406bc-f697-42e1-9a76-857c3a28d0d6-kube-api-access-t7kh8\" (UniqueName: \"kubernetes.io/projected/2ec406bc-f697-42e1-9a76-857c3a28d0d6-kube-api-access-t7kh8\") pod \"2ec406bc-f697-42e1-9a76-857c3a28d0d6\" (UID: \"2ec406bc-f697-42e1-9a76-857c3a28d0d6\") "
Nov 02 20:16:22.666644 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:22.666627    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ec406bc-f697-42e1-9a76-857c3a28d0d6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ec406bc-f697-42e1-9a76-857c3a28d0d6-openshift-service-ca\") pod \"2ec406bc-f697-42e1-9a76-857c3a28d0d6\" (UID: \"2ec406bc-f697-42e1-9a76-857c3a28d0d6\") "
Nov 02 20:16:22.667058 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:22.667007    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ec406bc-f697-42e1-9a76-857c3a28d0d6-openshift-service-ca" pod "2ec406bc-f697-42e1-9a76-857c3a28d0d6" (UID: "2ec406bc-f697-42e1-9a76-857c3a28d0d6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:16:22.668711 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:22.668679    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ec406bc-f697-42e1-9a76-857c3a28d0d6-metrics-certs" pod "2ec406bc-f697-42e1-9a76-857c3a28d0d6" (UID: "2ec406bc-f697-42e1-9a76-857c3a28d0d6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:16:22.668803 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:22.668725    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ec406bc-f697-42e1-9a76-857c3a28d0d6-maas-api-tls" pod "2ec406bc-f697-42e1-9a76-857c3a28d0d6" (UID: "2ec406bc-f697-42e1-9a76-857c3a28d0d6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:16:22.668841 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:22.668819    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ec406bc-f697-42e1-9a76-857c3a28d0d6-kube-api-access-t7kh8" pod "2ec406bc-f697-42e1-9a76-857c3a28d0d6" (UID: "2ec406bc-f697-42e1-9a76-857c3a28d0d6"). InnerVolumeSpecName "kube-api-access-t7kh8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:16:22.768167 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:22.768133    4463 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ec406bc-f697-42e1-9a76-857c3a28d0d6-metrics-certs\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:16:22.768353 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:22.768175    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7kh8\" (UniqueName: \"kubernetes.io/projected/2ec406bc-f697-42e1-9a76-857c3a28d0d6-kube-api-access-t7kh8\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:16:22.768353 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:22.768190    4463 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ec406bc-f697-42e1-9a76-857c3a28d0d6-openshift-service-ca\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:16:22.768353 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:22.768204    4463 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ec406bc-f697-42e1-9a76-857c3a28d0d6-maas-api-tls\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:16:22.918815 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:22.918731    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-a5b64d-59fd995f49-4g9cm" podUID="7f250661-9b39-4d06-af9c-6f067b82f8ca" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 20:16:23.360568 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:23.360533    4463 generic.go:356] "Generic (PLEG): container finished" podID="2ec406bc-f697-42e1-9a76-857c3a28d0d6" containerID="6bf04077320aaa5b1d2b9e3c747afe550dce38b189b9b602c2ed5f501f43b3f1" exitCode=0
Nov 02 20:16:23.360771 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:23.360584    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a5b64d-64599d48c8-97d8b"
Nov 02 20:16:23.360771 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:23.360617    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a5b64d-64599d48c8-97d8b" event={"ID":"2ec406bc-f697-42e1-9a76-857c3a28d0d6","Type":"ContainerDied","Data":"6bf04077320aaa5b1d2b9e3c747afe550dce38b189b9b602c2ed5f501f43b3f1"}
Nov 02 20:16:23.360771 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:23.360674    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a5b64d-64599d48c8-97d8b" event={"ID":"2ec406bc-f697-42e1-9a76-857c3a28d0d6","Type":"ContainerDied","Data":"e9f0edf517dfc2098d3b34bed2f269f1e0ac5daac5acd06f23c9c9d6023ad516"}
Nov 02 20:16:23.360771 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:23.360698    4463 scope.go:122] "RemoveContainer" containerID="6bf04077320aaa5b1d2b9e3c747afe550dce38b189b9b602c2ed5f501f43b3f1"
Nov 02 20:16:23.370113 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:23.370096    4463 scope.go:122] "RemoveContainer" containerID="6bf04077320aaa5b1d2b9e3c747afe550dce38b189b9b602c2ed5f501f43b3f1"
Nov 02 20:16:23.370360 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:16:23.370339    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bf04077320aaa5b1d2b9e3c747afe550dce38b189b9b602c2ed5f501f43b3f1\": container with ID starting with 6bf04077320aaa5b1d2b9e3c747afe550dce38b189b9b602c2ed5f501f43b3f1 not found: ID does not exist" containerID="6bf04077320aaa5b1d2b9e3c747afe550dce38b189b9b602c2ed5f501f43b3f1"
Nov 02 20:16:23.370404 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:23.370367    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bf04077320aaa5b1d2b9e3c747afe550dce38b189b9b602c2ed5f501f43b3f1"} err="failed to get container status \"6bf04077320aaa5b1d2b9e3c747afe550dce38b189b9b602c2ed5f501f43b3f1\": rpc error: code = NotFound desc = could not find container \"6bf04077320aaa5b1d2b9e3c747afe550dce38b189b9b602c2ed5f501f43b3f1\": container with ID starting with 6bf04077320aaa5b1d2b9e3c747afe550dce38b189b9b602c2ed5f501f43b3f1 not found: ID does not exist"
Nov 02 20:16:23.423891 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:23.423866    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-28fa24-865f7b8866-9hsz9"]
Nov 02 20:16:23.424295 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:23.424052    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-28fa24-865f7b8866-9hsz9" podUID="b78a889a-2b5d-4ff0-a419-2eb7f7858579" containerName="maas-api" containerID="cri-o://1b07488c539f685213ea7ccb3e91e8e1f877bd0d9605c0920ec0b7ec2951bc82" gracePeriod=30
Nov 02 20:16:23.513424 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:23.513371    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a5b64d-64599d48c8-97d8b"]
Nov 02 20:16:23.515689 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:23.515659    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a5b64d-64599d48c8-97d8b"]
Nov 02 20:16:23.585176 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:23.585139    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ec406bc-f697-42e1-9a76-857c3a28d0d6" path="/var/lib/kubelet/pods/2ec406bc-f697-42e1-9a76-857c3a28d0d6/volumes"
Nov 02 20:16:23.681067 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:23.681043    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-28fa24-865f7b8866-9hsz9"
Nov 02 20:16:23.778368 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:23.778334    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b78a889a-2b5d-4ff0-a419-2eb7f7858579-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b78a889a-2b5d-4ff0-a419-2eb7f7858579-openshift-service-ca\") pod \"b78a889a-2b5d-4ff0-a419-2eb7f7858579\" (UID: \"b78a889a-2b5d-4ff0-a419-2eb7f7858579\") "
Nov 02 20:16:23.778573 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:23.778456    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b78a889a-2b5d-4ff0-a419-2eb7f7858579-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b78a889a-2b5d-4ff0-a419-2eb7f7858579-metrics-certs\") pod \"b78a889a-2b5d-4ff0-a419-2eb7f7858579\" (UID: \"b78a889a-2b5d-4ff0-a419-2eb7f7858579\") "
Nov 02 20:16:23.778650 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:23.778583    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b78a889a-2b5d-4ff0-a419-2eb7f7858579-kube-api-access-99vvr\" (UniqueName: \"kubernetes.io/projected/b78a889a-2b5d-4ff0-a419-2eb7f7858579-kube-api-access-99vvr\") pod \"b78a889a-2b5d-4ff0-a419-2eb7f7858579\" (UID: \"b78a889a-2b5d-4ff0-a419-2eb7f7858579\") "
Nov 02 20:16:23.778720 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:23.778658    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b78a889a-2b5d-4ff0-a419-2eb7f7858579-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b78a889a-2b5d-4ff0-a419-2eb7f7858579-maas-api-tls\") pod \"b78a889a-2b5d-4ff0-a419-2eb7f7858579\" (UID: \"b78a889a-2b5d-4ff0-a419-2eb7f7858579\") "
Nov 02 20:16:23.778775 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:23.778740    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b78a889a-2b5d-4ff0-a419-2eb7f7858579-openshift-service-ca" pod "b78a889a-2b5d-4ff0-a419-2eb7f7858579" (UID: "b78a889a-2b5d-4ff0-a419-2eb7f7858579"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:16:23.779007 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:23.778978    4463 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b78a889a-2b5d-4ff0-a419-2eb7f7858579-openshift-service-ca\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:16:23.780883 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:23.780847    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b78a889a-2b5d-4ff0-a419-2eb7f7858579-kube-api-access-99vvr" pod "b78a889a-2b5d-4ff0-a419-2eb7f7858579" (UID: "b78a889a-2b5d-4ff0-a419-2eb7f7858579"). InnerVolumeSpecName "kube-api-access-99vvr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:16:23.781055 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:23.781032    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b78a889a-2b5d-4ff0-a419-2eb7f7858579-metrics-certs" pod "b78a889a-2b5d-4ff0-a419-2eb7f7858579" (UID: "b78a889a-2b5d-4ff0-a419-2eb7f7858579"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:16:23.781309 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:23.781287    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b78a889a-2b5d-4ff0-a419-2eb7f7858579-maas-api-tls" pod "b78a889a-2b5d-4ff0-a419-2eb7f7858579" (UID: "b78a889a-2b5d-4ff0-a419-2eb7f7858579"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:16:23.880546 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:23.880510    4463 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b78a889a-2b5d-4ff0-a419-2eb7f7858579-metrics-certs\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:16:23.880546 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:23.880544    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99vvr\" (UniqueName: \"kubernetes.io/projected/b78a889a-2b5d-4ff0-a419-2eb7f7858579-kube-api-access-99vvr\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:16:23.880748 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:23.880558    4463 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b78a889a-2b5d-4ff0-a419-2eb7f7858579-maas-api-tls\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:16:24.075261 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:24.075236    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a5b64d-59fd995f49-4g9cm"
Nov 02 20:16:24.183146 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:24.183112    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f250661-9b39-4d06-af9c-6f067b82f8ca-kube-api-access-gml7s\" (UniqueName: \"kubernetes.io/projected/7f250661-9b39-4d06-af9c-6f067b82f8ca-kube-api-access-gml7s\") pod \"7f250661-9b39-4d06-af9c-6f067b82f8ca\" (UID: \"7f250661-9b39-4d06-af9c-6f067b82f8ca\") "
Nov 02 20:16:24.183334 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:24.183219    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f250661-9b39-4d06-af9c-6f067b82f8ca-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f250661-9b39-4d06-af9c-6f067b82f8ca-plugins-config-volume\") pod \"7f250661-9b39-4d06-af9c-6f067b82f8ca\" (UID: \"7f250661-9b39-4d06-af9c-6f067b82f8ca\") "
Nov 02 20:16:24.183552 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:24.183531    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f250661-9b39-4d06-af9c-6f067b82f8ca-plugins-config-volume" pod "7f250661-9b39-4d06-af9c-6f067b82f8ca" (UID: "7f250661-9b39-4d06-af9c-6f067b82f8ca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:16:24.186151 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:24.186126    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f250661-9b39-4d06-af9c-6f067b82f8ca-kube-api-access-gml7s" pod "7f250661-9b39-4d06-af9c-6f067b82f8ca" (UID: "7f250661-9b39-4d06-af9c-6f067b82f8ca"). InnerVolumeSpecName "kube-api-access-gml7s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:16:24.284236 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:24.284140    4463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f250661-9b39-4d06-af9c-6f067b82f8ca-plugins-config-volume\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:16:24.284236 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:24.284171    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gml7s\" (UniqueName: \"kubernetes.io/projected/7f250661-9b39-4d06-af9c-6f067b82f8ca-kube-api-access-gml7s\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:16:24.367161 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:24.367127    4463 generic.go:356] "Generic (PLEG): container finished" podID="7f250661-9b39-4d06-af9c-6f067b82f8ca" containerID="a8df302061409f44d31d2fdfe07afae4293717c73ec6ea4b1168d83f289eefe4" exitCode=0
Nov 02 20:16:24.367341 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:24.367187    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a5b64d-59fd995f49-4g9cm"
Nov 02 20:16:24.367341 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:24.367205    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a5b64d-59fd995f49-4g9cm" event={"ID":"7f250661-9b39-4d06-af9c-6f067b82f8ca","Type":"ContainerDied","Data":"a8df302061409f44d31d2fdfe07afae4293717c73ec6ea4b1168d83f289eefe4"}
Nov 02 20:16:24.367341 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:24.367257    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a5b64d-59fd995f49-4g9cm" event={"ID":"7f250661-9b39-4d06-af9c-6f067b82f8ca","Type":"ContainerDied","Data":"37d1ac2dc15c600f8a6a59d962017946a104cfd1dd7aa87d088c7fb5ebc350bf"}
Nov 02 20:16:24.367341 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:24.367287    4463 scope.go:122] "RemoveContainer" containerID="a8df302061409f44d31d2fdfe07afae4293717c73ec6ea4b1168d83f289eefe4"
Nov 02 20:16:24.368646 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:24.368612    4463 generic.go:356] "Generic (PLEG): container finished" podID="b78a889a-2b5d-4ff0-a419-2eb7f7858579" containerID="1b07488c539f685213ea7ccb3e91e8e1f877bd0d9605c0920ec0b7ec2951bc82" exitCode=0
Nov 02 20:16:24.368759 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:24.368652    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-28fa24-865f7b8866-9hsz9"
Nov 02 20:16:24.368759 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:24.368667    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-28fa24-865f7b8866-9hsz9" event={"ID":"b78a889a-2b5d-4ff0-a419-2eb7f7858579","Type":"ContainerDied","Data":"1b07488c539f685213ea7ccb3e91e8e1f877bd0d9605c0920ec0b7ec2951bc82"}
Nov 02 20:16:24.368759 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:24.368718    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-28fa24-865f7b8866-9hsz9" event={"ID":"b78a889a-2b5d-4ff0-a419-2eb7f7858579","Type":"ContainerDied","Data":"c92f8732be89f75f1c8bb924e930c72c65058783bc72bb9c2c30236c289fa1a2"}
Nov 02 20:16:24.376801 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:24.376781    4463 scope.go:122] "RemoveContainer" containerID="a8df302061409f44d31d2fdfe07afae4293717c73ec6ea4b1168d83f289eefe4"
Nov 02 20:16:24.377142 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:16:24.377117    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8df302061409f44d31d2fdfe07afae4293717c73ec6ea4b1168d83f289eefe4\": container with ID starting with a8df302061409f44d31d2fdfe07afae4293717c73ec6ea4b1168d83f289eefe4 not found: ID does not exist" containerID="a8df302061409f44d31d2fdfe07afae4293717c73ec6ea4b1168d83f289eefe4"
Nov 02 20:16:24.377237 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:24.377154    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8df302061409f44d31d2fdfe07afae4293717c73ec6ea4b1168d83f289eefe4"} err="failed to get container status \"a8df302061409f44d31d2fdfe07afae4293717c73ec6ea4b1168d83f289eefe4\": rpc error: code = NotFound desc = could not find container \"a8df302061409f44d31d2fdfe07afae4293717c73ec6ea4b1168d83f289eefe4\": container with ID starting with a8df302061409f44d31d2fdfe07afae4293717c73ec6ea4b1168d83f289eefe4 not found: ID does not exist"
Nov 02 20:16:24.377237 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:24.377178    4463 scope.go:122] "RemoveContainer" containerID="1b07488c539f685213ea7ccb3e91e8e1f877bd0d9605c0920ec0b7ec2951bc82"
Nov 02 20:16:24.386045 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:24.385999    4463 scope.go:122] "RemoveContainer" containerID="1b07488c539f685213ea7ccb3e91e8e1f877bd0d9605c0920ec0b7ec2951bc82"
Nov 02 20:16:24.386338 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:16:24.386317    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b07488c539f685213ea7ccb3e91e8e1f877bd0d9605c0920ec0b7ec2951bc82\": container with ID starting with 1b07488c539f685213ea7ccb3e91e8e1f877bd0d9605c0920ec0b7ec2951bc82 not found: ID does not exist" containerID="1b07488c539f685213ea7ccb3e91e8e1f877bd0d9605c0920ec0b7ec2951bc82"
Nov 02 20:16:24.386396 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:24.386346    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b07488c539f685213ea7ccb3e91e8e1f877bd0d9605c0920ec0b7ec2951bc82"} err="failed to get container status \"1b07488c539f685213ea7ccb3e91e8e1f877bd0d9605c0920ec0b7ec2951bc82\": rpc error: code = NotFound desc = could not find container \"1b07488c539f685213ea7ccb3e91e8e1f877bd0d9605c0920ec0b7ec2951bc82\": container with ID starting with 1b07488c539f685213ea7ccb3e91e8e1f877bd0d9605c0920ec0b7ec2951bc82 not found: ID does not exist"
Nov 02 20:16:24.519250 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:24.519215    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-28fa24-865f7b8866-9hsz9"]
Nov 02 20:16:24.521752 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:24.521725    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-28fa24-865f7b8866-9hsz9"]
Nov 02 20:16:24.621460 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:24.621370    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a5b64d-59fd995f49-4g9cm"]
Nov 02 20:16:24.624989 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:24.624963    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a5b64d-59fd995f49-4g9cm"]
Nov 02 20:16:25.584750 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:25.584704    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f250661-9b39-4d06-af9c-6f067b82f8ca" path="/var/lib/kubelet/pods/7f250661-9b39-4d06-af9c-6f067b82f8ca/volumes"
Nov 02 20:16:25.585318 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:25.585297    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b78a889a-2b5d-4ff0-a419-2eb7f7858579" path="/var/lib/kubelet/pods/b78a889a-2b5d-4ff0-a419-2eb7f7858579/volumes"
Nov 02 20:16:29.184247 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:29.184215    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-19d134-67b47666c7-njhvz"]
Nov 02 20:16:29.184680 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:29.184403    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19d134-67b47666c7-njhvz" podUID="20157f70-df1b-48c9-ab3b-48712bd899fc" containerName="maas-api" containerID="cri-o://ce622e18ecf0678ed8f5d1a29ced40a24113eb53f9428ca676bb6f69251c3107" gracePeriod=30
Nov 02 20:16:29.405793 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:29.405766    4463 generic.go:356] "Generic (PLEG): container finished" podID="20157f70-df1b-48c9-ab3b-48712bd899fc" containerID="ce622e18ecf0678ed8f5d1a29ced40a24113eb53f9428ca676bb6f69251c3107" exitCode=0
Nov 02 20:16:29.405920 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:29.405837    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19d134-67b47666c7-njhvz" event={"ID":"20157f70-df1b-48c9-ab3b-48712bd899fc","Type":"ContainerDied","Data":"ce622e18ecf0678ed8f5d1a29ced40a24113eb53f9428ca676bb6f69251c3107"}
Nov 02 20:16:29.428181 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:29.428154    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19d134-67b47666c7-njhvz"
Nov 02 20:16:29.533885 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:29.533819    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20157f70-df1b-48c9-ab3b-48712bd899fc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/20157f70-df1b-48c9-ab3b-48712bd899fc-metrics-certs\") pod \"20157f70-df1b-48c9-ab3b-48712bd899fc\" (UID: \"20157f70-df1b-48c9-ab3b-48712bd899fc\") "
Nov 02 20:16:29.533885 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:29.533858    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20157f70-df1b-48c9-ab3b-48712bd899fc-kube-api-access-7fjvg\" (UniqueName: \"kubernetes.io/projected/20157f70-df1b-48c9-ab3b-48712bd899fc-kube-api-access-7fjvg\") pod \"20157f70-df1b-48c9-ab3b-48712bd899fc\" (UID: \"20157f70-df1b-48c9-ab3b-48712bd899fc\") "
Nov 02 20:16:29.534217 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:29.534132    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20157f70-df1b-48c9-ab3b-48712bd899fc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20157f70-df1b-48c9-ab3b-48712bd899fc-maas-api-tls\") pod \"20157f70-df1b-48c9-ab3b-48712bd899fc\" (UID: \"20157f70-df1b-48c9-ab3b-48712bd899fc\") "
Nov 02 20:16:29.534217 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:29.534169    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20157f70-df1b-48c9-ab3b-48712bd899fc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20157f70-df1b-48c9-ab3b-48712bd899fc-openshift-service-ca\") pod \"20157f70-df1b-48c9-ab3b-48712bd899fc\" (UID: \"20157f70-df1b-48c9-ab3b-48712bd899fc\") "
Nov 02 20:16:29.535171 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:29.535141    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20157f70-df1b-48c9-ab3b-48712bd899fc-openshift-service-ca" pod "20157f70-df1b-48c9-ab3b-48712bd899fc" (UID: "20157f70-df1b-48c9-ab3b-48712bd899fc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:16:29.536566 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:29.536507    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20157f70-df1b-48c9-ab3b-48712bd899fc-kube-api-access-7fjvg" pod "20157f70-df1b-48c9-ab3b-48712bd899fc" (UID: "20157f70-df1b-48c9-ab3b-48712bd899fc"). InnerVolumeSpecName "kube-api-access-7fjvg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:16:29.536903 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:29.536863    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20157f70-df1b-48c9-ab3b-48712bd899fc-metrics-certs" pod "20157f70-df1b-48c9-ab3b-48712bd899fc" (UID: "20157f70-df1b-48c9-ab3b-48712bd899fc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:16:29.537330 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:29.537280    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20157f70-df1b-48c9-ab3b-48712bd899fc-maas-api-tls" pod "20157f70-df1b-48c9-ab3b-48712bd899fc" (UID: "20157f70-df1b-48c9-ab3b-48712bd899fc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:16:29.635225 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:29.635197    4463 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20157f70-df1b-48c9-ab3b-48712bd899fc-maas-api-tls\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:16:29.635225 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:29.635224    4463 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20157f70-df1b-48c9-ab3b-48712bd899fc-openshift-service-ca\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:16:29.635377 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:29.635235    4463 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20157f70-df1b-48c9-ab3b-48712bd899fc-metrics-certs\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:16:29.635377 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:29.635249    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fjvg\" (UniqueName: \"kubernetes.io/projected/20157f70-df1b-48c9-ab3b-48712bd899fc-kube-api-access-7fjvg\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:16:30.412766 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:30.412735    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19d134-67b47666c7-njhvz"
Nov 02 20:16:30.412766 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:30.412747    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19d134-67b47666c7-njhvz" event={"ID":"20157f70-df1b-48c9-ab3b-48712bd899fc","Type":"ContainerDied","Data":"089377b98835cc9d0e403d78c7c66045a350c92e6b7cfd822472a7c018004041"}
Nov 02 20:16:30.413284 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:30.412814    4463 scope.go:122] "RemoveContainer" containerID="ce622e18ecf0678ed8f5d1a29ced40a24113eb53f9428ca676bb6f69251c3107"
Nov 02 20:16:30.593256 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:30.593228    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-19d134-67b47666c7-njhvz"]
Nov 02 20:16:30.639471 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:30.639435    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-19d134-67b47666c7-njhvz"]
Nov 02 20:16:30.760786 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:30.760747    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a5b64d-openshift-default-6f8c4599cf-vztxx"]
Nov 02 20:16:30.761025 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:30.760985    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-a5b64d-openshift-default-6f8c4599cf-vztxx" podUID="8709c9fb-f685-43fd-a3d5-9cb73cf8329d" containerName="istio-proxy" containerID="cri-o://f18194eded39ac8ef5d038ab615bf6fa11617a4e745b907f6a601b037338b040" gracePeriod=30
Nov 02 20:16:31.584077 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:31.584042    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20157f70-df1b-48c9-ab3b-48712bd899fc" path="/var/lib/kubelet/pods/20157f70-df1b-48c9-ab3b-48712bd899fc/volumes"
Nov 02 20:16:34.293601 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:16:34.293563    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 20:16:36.019669 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:36.019642    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a5b64d-openshift-default-6f8c4599cf-vztxx"
Nov 02 20:16:36.095655 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:36.095595    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8709c9fb-f685-43fd-a3d5-9cb73cf8329d-kube-api-access-tv4f7\" (UniqueName: \"kubernetes.io/projected/8709c9fb-f685-43fd-a3d5-9cb73cf8329d-kube-api-access-tv4f7\") pod \"8709c9fb-f685-43fd-a3d5-9cb73cf8329d\" (UID: \"8709c9fb-f685-43fd-a3d5-9cb73cf8329d\") "
Nov 02 20:16:36.095655 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:36.095634    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8709c9fb-f685-43fd-a3d5-9cb73cf8329d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8709c9fb-f685-43fd-a3d5-9cb73cf8329d-istio-data\") pod \"8709c9fb-f685-43fd-a3d5-9cb73cf8329d\" (UID: \"8709c9fb-f685-43fd-a3d5-9cb73cf8329d\") "
Nov 02 20:16:36.095912 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:36.095674    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8709c9fb-f685-43fd-a3d5-9cb73cf8329d-istio-token\" (UniqueName: \"kubernetes.io/projected/8709c9fb-f685-43fd-a3d5-9cb73cf8329d-istio-token\") pod \"8709c9fb-f685-43fd-a3d5-9cb73cf8329d\" (UID: \"8709c9fb-f685-43fd-a3d5-9cb73cf8329d\") "
Nov 02 20:16:36.095912 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:36.095711    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8709c9fb-f685-43fd-a3d5-9cb73cf8329d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8709c9fb-f685-43fd-a3d5-9cb73cf8329d-istio-podinfo\") pod \"8709c9fb-f685-43fd-a3d5-9cb73cf8329d\" (UID: \"8709c9fb-f685-43fd-a3d5-9cb73cf8329d\") "
Nov 02 20:16:36.095912 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:36.095782    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8709c9fb-f685-43fd-a3d5-9cb73cf8329d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8709c9fb-f685-43fd-a3d5-9cb73cf8329d-workload-certs\") pod \"8709c9fb-f685-43fd-a3d5-9cb73cf8329d\" (UID: \"8709c9fb-f685-43fd-a3d5-9cb73cf8329d\") "
Nov 02 20:16:36.095912 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:36.095818    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8709c9fb-f685-43fd-a3d5-9cb73cf8329d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8709c9fb-f685-43fd-a3d5-9cb73cf8329d-istiod-ca-cert\") pod \"8709c9fb-f685-43fd-a3d5-9cb73cf8329d\" (UID: \"8709c9fb-f685-43fd-a3d5-9cb73cf8329d\") "
Nov 02 20:16:36.095912 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:36.095875    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8709c9fb-f685-43fd-a3d5-9cb73cf8329d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8709c9fb-f685-43fd-a3d5-9cb73cf8329d-workload-socket\") pod \"8709c9fb-f685-43fd-a3d5-9cb73cf8329d\" (UID: \"8709c9fb-f685-43fd-a3d5-9cb73cf8329d\") "
Nov 02 20:16:36.096227 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:36.095914    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8709c9fb-f685-43fd-a3d5-9cb73cf8329d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8709c9fb-f685-43fd-a3d5-9cb73cf8329d-credential-socket\") pod \"8709c9fb-f685-43fd-a3d5-9cb73cf8329d\" (UID: \"8709c9fb-f685-43fd-a3d5-9cb73cf8329d\") "
Nov 02 20:16:36.096227 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:36.095948    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8709c9fb-f685-43fd-a3d5-9cb73cf8329d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8709c9fb-f685-43fd-a3d5-9cb73cf8329d-istio-envoy\") pod \"8709c9fb-f685-43fd-a3d5-9cb73cf8329d\" (UID: \"8709c9fb-f685-43fd-a3d5-9cb73cf8329d\") "
Nov 02 20:16:36.096227 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:36.095991    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8709c9fb-f685-43fd-a3d5-9cb73cf8329d-istio-data" pod "8709c9fb-f685-43fd-a3d5-9cb73cf8329d" (UID: "8709c9fb-f685-43fd-a3d5-9cb73cf8329d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:16:36.096227 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:36.096107    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8709c9fb-f685-43fd-a3d5-9cb73cf8329d-workload-certs" pod "8709c9fb-f685-43fd-a3d5-9cb73cf8329d" (UID: "8709c9fb-f685-43fd-a3d5-9cb73cf8329d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:16:36.096447 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:36.096238    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8709c9fb-f685-43fd-a3d5-9cb73cf8329d-workload-socket" pod "8709c9fb-f685-43fd-a3d5-9cb73cf8329d" (UID: "8709c9fb-f685-43fd-a3d5-9cb73cf8329d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:16:36.096447 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:36.096263    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8709c9fb-f685-43fd-a3d5-9cb73cf8329d-istiod-ca-cert" pod "8709c9fb-f685-43fd-a3d5-9cb73cf8329d" (UID: "8709c9fb-f685-43fd-a3d5-9cb73cf8329d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:16:36.096525 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:36.096453    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8709c9fb-f685-43fd-a3d5-9cb73cf8329d-credential-socket" pod "8709c9fb-f685-43fd-a3d5-9cb73cf8329d" (UID: "8709c9fb-f685-43fd-a3d5-9cb73cf8329d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:16:36.096561 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:36.096534    4463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8709c9fb-f685-43fd-a3d5-9cb73cf8329d-workload-certs\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:16:36.096593 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:36.096557    4463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8709c9fb-f685-43fd-a3d5-9cb73cf8329d-istiod-ca-cert\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:16:36.096593 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:36.096572    4463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8709c9fb-f685-43fd-a3d5-9cb73cf8329d-workload-socket\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:16:36.096593 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:36.096588    4463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8709c9fb-f685-43fd-a3d5-9cb73cf8329d-istio-data\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:16:36.098041 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:36.098006    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8709c9fb-f685-43fd-a3d5-9cb73cf8329d-istio-envoy" pod "8709c9fb-f685-43fd-a3d5-9cb73cf8329d" (UID: "8709c9fb-f685-43fd-a3d5-9cb73cf8329d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:16:36.098138 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:36.098042    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8709c9fb-f685-43fd-a3d5-9cb73cf8329d-istio-podinfo" pod "8709c9fb-f685-43fd-a3d5-9cb73cf8329d" (UID: "8709c9fb-f685-43fd-a3d5-9cb73cf8329d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:16:36.098262 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:36.098242    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8709c9fb-f685-43fd-a3d5-9cb73cf8329d-istio-token" pod "8709c9fb-f685-43fd-a3d5-9cb73cf8329d" (UID: "8709c9fb-f685-43fd-a3d5-9cb73cf8329d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:16:36.098321 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:36.098248    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8709c9fb-f685-43fd-a3d5-9cb73cf8329d-kube-api-access-tv4f7" pod "8709c9fb-f685-43fd-a3d5-9cb73cf8329d" (UID: "8709c9fb-f685-43fd-a3d5-9cb73cf8329d"). InnerVolumeSpecName "kube-api-access-tv4f7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:16:36.197647 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:36.197610    4463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8709c9fb-f685-43fd-a3d5-9cb73cf8329d-credential-socket\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:16:36.197647 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:36.197647    4463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8709c9fb-f685-43fd-a3d5-9cb73cf8329d-istio-envoy\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:16:36.197813 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:36.197663    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tv4f7\" (UniqueName: \"kubernetes.io/projected/8709c9fb-f685-43fd-a3d5-9cb73cf8329d-kube-api-access-tv4f7\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:16:36.197813 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:36.197676    4463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8709c9fb-f685-43fd-a3d5-9cb73cf8329d-istio-token\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:16:36.197813 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:36.197690    4463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8709c9fb-f685-43fd-a3d5-9cb73cf8329d-istio-podinfo\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:16:36.446604 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:36.446554    4463 generic.go:356] "Generic (PLEG): container finished" podID="8709c9fb-f685-43fd-a3d5-9cb73cf8329d" containerID="f18194eded39ac8ef5d038ab615bf6fa11617a4e745b907f6a601b037338b040" exitCode=0
Nov 02 20:16:36.446757 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:36.446627    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a5b64d-openshift-default-6f8c4599cf-vztxx"
Nov 02 20:16:36.446757 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:36.446642    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a5b64d-openshift-default-6f8c4599cf-vztxx" event={"ID":"8709c9fb-f685-43fd-a3d5-9cb73cf8329d","Type":"ContainerDied","Data":"f18194eded39ac8ef5d038ab615bf6fa11617a4e745b907f6a601b037338b040"}
Nov 02 20:16:36.446757 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:36.446691    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a5b64d-openshift-default-6f8c4599cf-vztxx" event={"ID":"8709c9fb-f685-43fd-a3d5-9cb73cf8329d","Type":"ContainerDied","Data":"90e2c6101e009b31b950e182901d59a43a2acd9c64cf590f90ca42949d01c4b8"}
Nov 02 20:16:36.446757 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:36.446713    4463 scope.go:122] "RemoveContainer" containerID="f18194eded39ac8ef5d038ab615bf6fa11617a4e745b907f6a601b037338b040"
Nov 02 20:16:36.458314 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:36.458297    4463 scope.go:122] "RemoveContainer" containerID="0435c58e164777f9c6652c868d54e806e906fb0b1f35f6619af491e1c538d23b"
Nov 02 20:16:36.466686 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:36.466668    4463 scope.go:122] "RemoveContainer" containerID="f18194eded39ac8ef5d038ab615bf6fa11617a4e745b907f6a601b037338b040"
Nov 02 20:16:36.466955 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:16:36.466934    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f18194eded39ac8ef5d038ab615bf6fa11617a4e745b907f6a601b037338b040\": container with ID starting with f18194eded39ac8ef5d038ab615bf6fa11617a4e745b907f6a601b037338b040 not found: ID does not exist" containerID="f18194eded39ac8ef5d038ab615bf6fa11617a4e745b907f6a601b037338b040"
Nov 02 20:16:36.467006 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:36.466966    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f18194eded39ac8ef5d038ab615bf6fa11617a4e745b907f6a601b037338b040"} err="failed to get container status \"f18194eded39ac8ef5d038ab615bf6fa11617a4e745b907f6a601b037338b040\": rpc error: code = NotFound desc = could not find container \"f18194eded39ac8ef5d038ab615bf6fa11617a4e745b907f6a601b037338b040\": container with ID starting with f18194eded39ac8ef5d038ab615bf6fa11617a4e745b907f6a601b037338b040 not found: ID does not exist"
Nov 02 20:16:36.467006 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:36.466983    4463 scope.go:122] "RemoveContainer" containerID="0435c58e164777f9c6652c868d54e806e906fb0b1f35f6619af491e1c538d23b"
Nov 02 20:16:36.467255 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:16:36.467240    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0435c58e164777f9c6652c868d54e806e906fb0b1f35f6619af491e1c538d23b\": container with ID starting with 0435c58e164777f9c6652c868d54e806e906fb0b1f35f6619af491e1c538d23b not found: ID does not exist" containerID="0435c58e164777f9c6652c868d54e806e906fb0b1f35f6619af491e1c538d23b"
Nov 02 20:16:36.467295 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:36.467259    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0435c58e164777f9c6652c868d54e806e906fb0b1f35f6619af491e1c538d23b"} err="failed to get container status \"0435c58e164777f9c6652c868d54e806e906fb0b1f35f6619af491e1c538d23b\": rpc error: code = NotFound desc = could not find container \"0435c58e164777f9c6652c868d54e806e906fb0b1f35f6619af491e1c538d23b\": container with ID starting with 0435c58e164777f9c6652c868d54e806e906fb0b1f35f6619af491e1c538d23b not found: ID does not exist"
Nov 02 20:16:36.598736 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:36.598704    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a5b64d-openshift-default-6f8c4599cf-vztxx"]
Nov 02 20:16:36.644143 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:36.644118    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-a5b64d-openshift-default-6f8c4599cf-vztxx"]
Nov 02 20:16:37.583497 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:37.583462    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8709c9fb-f685-43fd-a3d5-9cb73cf8329d" path="/var/lib/kubelet/pods/8709c9fb-f685-43fd-a3d5-9cb73cf8329d/volumes"
Nov 02 20:16:38.422890 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:38.422836    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-43ceaa/premium-w0-43ceaa-kserve-67b55659b7-rckfd"]
Nov 02 20:16:43.258585 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:43.258550    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-13e739-565886dbbc-wbgfw"]
Nov 02 20:16:45.981748 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:45.981702    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-43ceaa-7bf8fd68b7-nstb8"]
Nov 02 20:16:48.259348 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:48.259306    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-13e739-565886dbbc-wbgfw" podUID="e20408d3-f3a6-4ca9-8413-aafe209a1477" containerName="payload-processing" containerID="cri-o://d508d0a4ce5b48c8a62e8c256f746208ea5baff320d3713213b4c4be51d6c014" gracePeriod=25
Nov 02 20:16:50.414049 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:50.413999    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-d1aaed-786fbbccc-zth6x"]
Nov 02 20:16:50.531563 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:50.531528    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-d1aaed-79985b544c-c8zc9"]
Nov 02 20:16:50.982989 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:50.982949    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-43ceaa-7bf8fd68b7-nstb8" podUID="30d2bf45-98f1-4394-957b-18c5b768f902" containerName="payload-pre-processing" containerID="cri-o://0bba31894ee453d16c367d4a3d9a2205e6c826e728fc317058583a603bc2fb0d" gracePeriod=25
Nov 02 20:16:51.231165 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:51.231142    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-43ceaa-7bf8fd68b7-nstb8"
Nov 02 20:16:51.343076 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:51.342988    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30d2bf45-98f1-4394-957b-18c5b768f902-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30d2bf45-98f1-4394-957b-18c5b768f902-plugins-config-volume\") pod \"30d2bf45-98f1-4394-957b-18c5b768f902\" (UID: \"30d2bf45-98f1-4394-957b-18c5b768f902\") "
Nov 02 20:16:51.343076 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:51.343053    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30d2bf45-98f1-4394-957b-18c5b768f902-kube-api-access-zrccq\" (UniqueName: \"kubernetes.io/projected/30d2bf45-98f1-4394-957b-18c5b768f902-kube-api-access-zrccq\") pod \"30d2bf45-98f1-4394-957b-18c5b768f902\" (UID: \"30d2bf45-98f1-4394-957b-18c5b768f902\") "
Nov 02 20:16:51.343478 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:51.343451    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30d2bf45-98f1-4394-957b-18c5b768f902-plugins-config-volume" pod "30d2bf45-98f1-4394-957b-18c5b768f902" (UID: "30d2bf45-98f1-4394-957b-18c5b768f902"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:16:51.353148 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:51.353123    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30d2bf45-98f1-4394-957b-18c5b768f902-kube-api-access-zrccq" pod "30d2bf45-98f1-4394-957b-18c5b768f902" (UID: "30d2bf45-98f1-4394-957b-18c5b768f902"). InnerVolumeSpecName "kube-api-access-zrccq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:16:51.444846 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:51.444825    4463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30d2bf45-98f1-4394-957b-18c5b768f902-plugins-config-volume\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:16:51.444846 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:51.444848    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrccq\" (UniqueName: \"kubernetes.io/projected/30d2bf45-98f1-4394-957b-18c5b768f902-kube-api-access-zrccq\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:16:51.543224 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:51.543189    4463 generic.go:356] "Generic (PLEG): container finished" podID="30d2bf45-98f1-4394-957b-18c5b768f902" containerID="0bba31894ee453d16c367d4a3d9a2205e6c826e728fc317058583a603bc2fb0d" exitCode=0
Nov 02 20:16:51.543338 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:51.543247    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-43ceaa-7bf8fd68b7-nstb8" event={"ID":"30d2bf45-98f1-4394-957b-18c5b768f902","Type":"ContainerDied","Data":"0bba31894ee453d16c367d4a3d9a2205e6c826e728fc317058583a603bc2fb0d"}
Nov 02 20:16:51.543338 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:51.543258    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-43ceaa-7bf8fd68b7-nstb8"
Nov 02 20:16:51.543338 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:51.543309    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-43ceaa-7bf8fd68b7-nstb8" event={"ID":"30d2bf45-98f1-4394-957b-18c5b768f902","Type":"ContainerDied","Data":"735c5718a2a39ebbf8259997da6a4c7489245bb77141a61ff10e8f0462a1223c"}
Nov 02 20:16:51.543475 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:51.543367    4463 scope.go:122] "RemoveContainer" containerID="0bba31894ee453d16c367d4a3d9a2205e6c826e728fc317058583a603bc2fb0d"
Nov 02 20:16:51.552871 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:51.552849    4463 scope.go:122] "RemoveContainer" containerID="0bba31894ee453d16c367d4a3d9a2205e6c826e728fc317058583a603bc2fb0d"
Nov 02 20:16:51.553182 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:16:51.553154    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bba31894ee453d16c367d4a3d9a2205e6c826e728fc317058583a603bc2fb0d\": container with ID starting with 0bba31894ee453d16c367d4a3d9a2205e6c826e728fc317058583a603bc2fb0d not found: ID does not exist" containerID="0bba31894ee453d16c367d4a3d9a2205e6c826e728fc317058583a603bc2fb0d"
Nov 02 20:16:51.553296 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:51.553188    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bba31894ee453d16c367d4a3d9a2205e6c826e728fc317058583a603bc2fb0d"} err="failed to get container status \"0bba31894ee453d16c367d4a3d9a2205e6c826e728fc317058583a603bc2fb0d\": rpc error: code = NotFound desc = could not find container \"0bba31894ee453d16c367d4a3d9a2205e6c826e728fc317058583a603bc2fb0d\": container with ID starting with 0bba31894ee453d16c367d4a3d9a2205e6c826e728fc317058583a603bc2fb0d not found: ID does not exist"
Nov 02 20:16:51.694583 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:51.694553    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-43ceaa-7bf8fd68b7-nstb8"]
Nov 02 20:16:51.740603 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:51.740570    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-43ceaa-7bf8fd68b7-nstb8"]
Nov 02 20:16:51.907710 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:51.907675    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 20:16:53.442952 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:53.442916    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-43ceaa/premium-w0-43ceaa-kserve-67b55659b7-rckfd" podUID="7be32df1-5049-4f79-be23-a827f3f0a85f" containerName="main" containerID="cri-o://d5da31da419a75e0a447ef9fc079d2874efb0b85c4fbaf0a04237941165a7af7" gracePeriod=45
Nov 02 20:16:53.446674 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:53.446642    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-43ceaa/premium-w0-43ceaa-kserve-67b55659b7-rckfd" podUID="7be32df1-5049-4f79-be23-a827f3f0a85f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": dial tcp 10.133.0.46:8000: connect: connection refused"
Nov 02 20:16:53.557688 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:53.557661    4463 generic.go:356] "Generic (PLEG): container finished" podID="7be32df1-5049-4f79-be23-a827f3f0a85f" containerID="d5da31da419a75e0a447ef9fc079d2874efb0b85c4fbaf0a04237941165a7af7" exitCode=0
Nov 02 20:16:53.557793 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:53.557692    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-43ceaa/premium-w0-43ceaa-kserve-67b55659b7-rckfd" event={"ID":"7be32df1-5049-4f79-be23-a827f3f0a85f","Type":"ContainerDied","Data":"d5da31da419a75e0a447ef9fc079d2874efb0b85c4fbaf0a04237941165a7af7"}
Nov 02 20:16:53.585113 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:53.585082    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30d2bf45-98f1-4394-957b-18c5b768f902" path="/var/lib/kubelet/pods/30d2bf45-98f1-4394-957b-18c5b768f902/volumes"
Nov 02 20:16:53.691243 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:53.691215    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-43ceaa/premium-w0-43ceaa-kserve-67b55659b7-rckfd"
Nov 02 20:16:53.867927 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:53.867846    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7be32df1-5049-4f79-be23-a827f3f0a85f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7be32df1-5049-4f79-be23-a827f3f0a85f-dshm\") pod \"7be32df1-5049-4f79-be23-a827f3f0a85f\" (UID: \"7be32df1-5049-4f79-be23-a827f3f0a85f\") "
Nov 02 20:16:53.867927 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:53.867906    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7be32df1-5049-4f79-be23-a827f3f0a85f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7be32df1-5049-4f79-be23-a827f3f0a85f-tmp-dir\") pod \"7be32df1-5049-4f79-be23-a827f3f0a85f\" (UID: \"7be32df1-5049-4f79-be23-a827f3f0a85f\") "
Nov 02 20:16:53.868182 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:53.868000    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7be32df1-5049-4f79-be23-a827f3f0a85f-kube-api-access-swpf7\" (UniqueName: \"kubernetes.io/projected/7be32df1-5049-4f79-be23-a827f3f0a85f-kube-api-access-swpf7\") pod \"7be32df1-5049-4f79-be23-a827f3f0a85f\" (UID: \"7be32df1-5049-4f79-be23-a827f3f0a85f\") "
Nov 02 20:16:53.868182 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:53.868071    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7be32df1-5049-4f79-be23-a827f3f0a85f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7be32df1-5049-4f79-be23-a827f3f0a85f-model-cache\") pod \"7be32df1-5049-4f79-be23-a827f3f0a85f\" (UID: \"7be32df1-5049-4f79-be23-a827f3f0a85f\") "
Nov 02 20:16:53.868182 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:53.868111    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7be32df1-5049-4f79-be23-a827f3f0a85f-home\" (UniqueName: \"kubernetes.io/empty-dir/7be32df1-5049-4f79-be23-a827f3f0a85f-home\") pod \"7be32df1-5049-4f79-be23-a827f3f0a85f\" (UID: \"7be32df1-5049-4f79-be23-a827f3f0a85f\") "
Nov 02 20:16:53.868356 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:53.868309    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7be32df1-5049-4f79-be23-a827f3f0a85f-tmp-dir" pod "7be32df1-5049-4f79-be23-a827f3f0a85f" (UID: "7be32df1-5049-4f79-be23-a827f3f0a85f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:16:53.868410 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:53.868371    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7be32df1-5049-4f79-be23-a827f3f0a85f-model-cache" pod "7be32df1-5049-4f79-be23-a827f3f0a85f" (UID: "7be32df1-5049-4f79-be23-a827f3f0a85f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:16:53.868410 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:53.868388    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7be32df1-5049-4f79-be23-a827f3f0a85f-home" pod "7be32df1-5049-4f79-be23-a827f3f0a85f" (UID: "7be32df1-5049-4f79-be23-a827f3f0a85f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:16:53.868512 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:53.868433    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7be32df1-5049-4f79-be23-a827f3f0a85f-tls-certs\" (UniqueName: \"kubernetes.io/secret/7be32df1-5049-4f79-be23-a827f3f0a85f-tls-certs\") pod \"7be32df1-5049-4f79-be23-a827f3f0a85f\" (UID: \"7be32df1-5049-4f79-be23-a827f3f0a85f\") "
Nov 02 20:16:53.868812 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:53.868789    4463 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7be32df1-5049-4f79-be23-a827f3f0a85f-model-cache\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:16:53.868812 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:53.868813    4463 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7be32df1-5049-4f79-be23-a827f3f0a85f-home\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:16:53.869008 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:53.868829    4463 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7be32df1-5049-4f79-be23-a827f3f0a85f-tmp-dir\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:16:53.870457 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:53.870434    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7be32df1-5049-4f79-be23-a827f3f0a85f-tls-certs" pod "7be32df1-5049-4f79-be23-a827f3f0a85f" (UID: "7be32df1-5049-4f79-be23-a827f3f0a85f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:16:53.870538 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:53.870448    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7be32df1-5049-4f79-be23-a827f3f0a85f-dshm" pod "7be32df1-5049-4f79-be23-a827f3f0a85f" (UID: "7be32df1-5049-4f79-be23-a827f3f0a85f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:16:53.870672 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:53.870650    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7be32df1-5049-4f79-be23-a827f3f0a85f-kube-api-access-swpf7" pod "7be32df1-5049-4f79-be23-a827f3f0a85f" (UID: "7be32df1-5049-4f79-be23-a827f3f0a85f"). InnerVolumeSpecName "kube-api-access-swpf7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:16:53.969938 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:53.969911    4463 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7be32df1-5049-4f79-be23-a827f3f0a85f-dshm\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:16:53.969938 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:53.969938    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swpf7\" (UniqueName: \"kubernetes.io/projected/7be32df1-5049-4f79-be23-a827f3f0a85f-kube-api-access-swpf7\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:16:53.970135 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:53.969952    4463 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7be32df1-5049-4f79-be23-a827f3f0a85f-tls-certs\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:16:54.563573 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:54.563544    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-43ceaa/premium-w0-43ceaa-kserve-67b55659b7-rckfd"
Nov 02 20:16:54.563573 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:54.563563    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-43ceaa/premium-w0-43ceaa-kserve-67b55659b7-rckfd" event={"ID":"7be32df1-5049-4f79-be23-a827f3f0a85f","Type":"ContainerDied","Data":"5f576ba99d213d2d4757daa276d6d8c96b9456ec757e916fe8b4eea0667dbbce"}
Nov 02 20:16:54.564081 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:54.563605    4463 scope.go:122] "RemoveContainer" containerID="d5da31da419a75e0a447ef9fc079d2874efb0b85c4fbaf0a04237941165a7af7"
Nov 02 20:16:54.715403 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:54.715376    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-43ceaa/premium-w0-43ceaa-kserve-67b55659b7-rckfd"]
Nov 02 20:16:54.761744 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:54.761717    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-43ceaa/premium-w0-43ceaa-kserve-67b55659b7-rckfd"]
Nov 02 20:16:55.415057 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:55.414993    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-d1aaed-786fbbccc-zth6x" podUID="86b84714-1e20-47b0-9a7b-f289c2589ffc" containerName="payload-processing" containerID="cri-o://face98f556aa2e473144f6751cb1769deeffe4b5c50c1eab576f0303fc1956d6" gracePeriod=25
Nov 02 20:16:55.532073 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:55.531990    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-d1aaed-79985b544c-c8zc9" podUID="9fbc861b-be1c-460c-833d-478346a9b840" containerName="payload-pre-processing" containerID="cri-o://c1b4faf6d65ba71bc0e1393d32414f0ad9fb529abad71d9c238f73cd168a67c9" gracePeriod=25
Nov 02 20:16:55.547766 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:55.547739    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-13e739-565886dbbc-wbgfw" podUID="e20408d3-f3a6-4ca9-8413-aafe209a1477" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 20:16:55.573126 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:55.573088    4463 generic.go:356] "Generic (PLEG): container finished" podID="86b84714-1e20-47b0-9a7b-f289c2589ffc" containerID="face98f556aa2e473144f6751cb1769deeffe4b5c50c1eab576f0303fc1956d6" exitCode=0
Nov 02 20:16:55.573521 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:55.573129    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-d1aaed-786fbbccc-zth6x" event={"ID":"86b84714-1e20-47b0-9a7b-f289c2589ffc","Type":"ContainerDied","Data":"face98f556aa2e473144f6751cb1769deeffe4b5c50c1eab576f0303fc1956d6"}
Nov 02 20:16:55.585472 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:55.585441    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7be32df1-5049-4f79-be23-a827f3f0a85f" path="/var/lib/kubelet/pods/7be32df1-5049-4f79-be23-a827f3f0a85f/volumes"
Nov 02 20:16:55.679803 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:55.679774    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-d1aaed-786fbbccc-zth6x"
Nov 02 20:16:55.778054 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:55.778029    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-d1aaed-79985b544c-c8zc9"
Nov 02 20:16:55.787295 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:55.787268    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86b84714-1e20-47b0-9a7b-f289c2589ffc-kube-api-access-66mft\" (UniqueName: \"kubernetes.io/projected/86b84714-1e20-47b0-9a7b-f289c2589ffc-kube-api-access-66mft\") pod \"86b84714-1e20-47b0-9a7b-f289c2589ffc\" (UID: \"86b84714-1e20-47b0-9a7b-f289c2589ffc\") "
Nov 02 20:16:55.787533 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:55.787503    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86b84714-1e20-47b0-9a7b-f289c2589ffc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86b84714-1e20-47b0-9a7b-f289c2589ffc-plugins-config-volume\") pod \"86b84714-1e20-47b0-9a7b-f289c2589ffc\" (UID: \"86b84714-1e20-47b0-9a7b-f289c2589ffc\") "
Nov 02 20:16:55.788116 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:55.788086    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86b84714-1e20-47b0-9a7b-f289c2589ffc-plugins-config-volume" pod "86b84714-1e20-47b0-9a7b-f289c2589ffc" (UID: "86b84714-1e20-47b0-9a7b-f289c2589ffc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:16:55.789931 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:55.789907    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86b84714-1e20-47b0-9a7b-f289c2589ffc-kube-api-access-66mft" pod "86b84714-1e20-47b0-9a7b-f289c2589ffc" (UID: "86b84714-1e20-47b0-9a7b-f289c2589ffc"). InnerVolumeSpecName "kube-api-access-66mft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:16:55.888896 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:55.888829    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fbc861b-be1c-460c-833d-478346a9b840-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fbc861b-be1c-460c-833d-478346a9b840-plugins-config-volume\") pod \"9fbc861b-be1c-460c-833d-478346a9b840\" (UID: \"9fbc861b-be1c-460c-833d-478346a9b840\") "
Nov 02 20:16:55.888896 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:55.888887    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fbc861b-be1c-460c-833d-478346a9b840-kube-api-access-gfkfj\" (UniqueName: \"kubernetes.io/projected/9fbc861b-be1c-460c-833d-478346a9b840-kube-api-access-gfkfj\") pod \"9fbc861b-be1c-460c-833d-478346a9b840\" (UID: \"9fbc861b-be1c-460c-833d-478346a9b840\") "
Nov 02 20:16:55.889215 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:55.889194    4463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86b84714-1e20-47b0-9a7b-f289c2589ffc-plugins-config-volume\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:16:55.889293 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:55.889220    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66mft\" (UniqueName: \"kubernetes.io/projected/86b84714-1e20-47b0-9a7b-f289c2589ffc-kube-api-access-66mft\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:16:55.889293 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:55.889251    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fbc861b-be1c-460c-833d-478346a9b840-plugins-config-volume" pod "9fbc861b-be1c-460c-833d-478346a9b840" (UID: "9fbc861b-be1c-460c-833d-478346a9b840"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:16:55.890906 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:55.890887    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fbc861b-be1c-460c-833d-478346a9b840-kube-api-access-gfkfj" pod "9fbc861b-be1c-460c-833d-478346a9b840" (UID: "9fbc861b-be1c-460c-833d-478346a9b840"). InnerVolumeSpecName "kube-api-access-gfkfj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:16:55.989954 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:55.989888    4463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fbc861b-be1c-460c-833d-478346a9b840-plugins-config-volume\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:16:55.989954 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:55.989915    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfkfj\" (UniqueName: \"kubernetes.io/projected/9fbc861b-be1c-460c-833d-478346a9b840-kube-api-access-gfkfj\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:16:56.580160 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:56.580127    4463 generic.go:356] "Generic (PLEG): container finished" podID="9fbc861b-be1c-460c-833d-478346a9b840" containerID="c1b4faf6d65ba71bc0e1393d32414f0ad9fb529abad71d9c238f73cd168a67c9" exitCode=0
Nov 02 20:16:56.580651 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:56.580176    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-d1aaed-79985b544c-c8zc9"
Nov 02 20:16:56.580651 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:56.580217    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-d1aaed-79985b544c-c8zc9" event={"ID":"9fbc861b-be1c-460c-833d-478346a9b840","Type":"ContainerDied","Data":"c1b4faf6d65ba71bc0e1393d32414f0ad9fb529abad71d9c238f73cd168a67c9"}
Nov 02 20:16:56.580651 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:56.580267    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-d1aaed-79985b544c-c8zc9" event={"ID":"9fbc861b-be1c-460c-833d-478346a9b840","Type":"ContainerDied","Data":"441503211c7add3123959b3b80c44c8e699a642071e63f5b255c90c7e1355837"}
Nov 02 20:16:56.580651 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:56.580292    4463 scope.go:122] "RemoveContainer" containerID="c1b4faf6d65ba71bc0e1393d32414f0ad9fb529abad71d9c238f73cd168a67c9"
Nov 02 20:16:56.581781 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:56.581758    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-d1aaed-786fbbccc-zth6x"
Nov 02 20:16:56.582001 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:56.581972    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-d1aaed-786fbbccc-zth6x" event={"ID":"86b84714-1e20-47b0-9a7b-f289c2589ffc","Type":"ContainerDied","Data":"1994751112eb6e39a559f0ba6b6c28483c6b9eb3bbf23e227ac329b1a94b6ad1"}
Nov 02 20:16:56.589572 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:56.589552    4463 scope.go:122] "RemoveContainer" containerID="c1b4faf6d65ba71bc0e1393d32414f0ad9fb529abad71d9c238f73cd168a67c9"
Nov 02 20:16:56.589823 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:16:56.589801    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1b4faf6d65ba71bc0e1393d32414f0ad9fb529abad71d9c238f73cd168a67c9\": container with ID starting with c1b4faf6d65ba71bc0e1393d32414f0ad9fb529abad71d9c238f73cd168a67c9 not found: ID does not exist" containerID="c1b4faf6d65ba71bc0e1393d32414f0ad9fb529abad71d9c238f73cd168a67c9"
Nov 02 20:16:56.589913 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:56.589833    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1b4faf6d65ba71bc0e1393d32414f0ad9fb529abad71d9c238f73cd168a67c9"} err="failed to get container status \"c1b4faf6d65ba71bc0e1393d32414f0ad9fb529abad71d9c238f73cd168a67c9\": rpc error: code = NotFound desc = could not find container \"c1b4faf6d65ba71bc0e1393d32414f0ad9fb529abad71d9c238f73cd168a67c9\": container with ID starting with c1b4faf6d65ba71bc0e1393d32414f0ad9fb529abad71d9c238f73cd168a67c9 not found: ID does not exist"
Nov 02 20:16:56.589913 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:56.589857    4463 scope.go:122] "RemoveContainer" containerID="face98f556aa2e473144f6751cb1769deeffe4b5c50c1eab576f0303fc1956d6"
Nov 02 20:16:56.732529 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:56.732452    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-d1aaed-79985b544c-c8zc9"]
Nov 02 20:16:56.778641 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:56.778612    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-d1aaed-79985b544c-c8zc9"]
Nov 02 20:16:56.838191 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:56.838155    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-d1aaed-786fbbccc-zth6x"]
Nov 02 20:16:56.841311 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:56.841281    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-d1aaed-786fbbccc-zth6x"]
Nov 02 20:16:57.583463 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:57.583422    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86b84714-1e20-47b0-9a7b-f289c2589ffc" path="/var/lib/kubelet/pods/86b84714-1e20-47b0-9a7b-f289c2589ffc/volumes"
Nov 02 20:16:57.583976 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:57.583958    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fbc861b-be1c-460c-833d-478346a9b840" path="/var/lib/kubelet/pods/9fbc861b-be1c-460c-833d-478346a9b840/volumes"
Nov 02 20:16:57.866710 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:57.866637    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-z7nwb"]
Nov 02 20:16:57.866856 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:57.866837    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-z7nwb" podUID="f40668dc-8c36-4c34-ae69-c87d8c71e338" containerName="istio-proxy" containerID="cri-o://1753e7925ff7ea4b4e8379df3666fb56c09ea49085d3ee5899a6f36484615ba2" gracePeriod=30
Nov 02 20:16:59.585903 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:59.585865    4463 patch_prober.go:29] interesting pod/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-z7nwb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 20:16:59.586314 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:16:59.585917    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-z7nwb" podUID="f40668dc-8c36-4c34-ae69-c87d8c71e338" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 20:17:03.116515 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:03.116489    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-z7nwb"
Nov 02 20:17:03.155231 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:03.155175    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f40668dc-8c36-4c34-ae69-c87d8c71e338-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f40668dc-8c36-4c34-ae69-c87d8c71e338-credential-socket\") pod \"f40668dc-8c36-4c34-ae69-c87d8c71e338\" (UID: \"f40668dc-8c36-4c34-ae69-c87d8c71e338\") "
Nov 02 20:17:03.155338 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:03.155232    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f40668dc-8c36-4c34-ae69-c87d8c71e338-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f40668dc-8c36-4c34-ae69-c87d8c71e338-istio-data\") pod \"f40668dc-8c36-4c34-ae69-c87d8c71e338\" (UID: \"f40668dc-8c36-4c34-ae69-c87d8c71e338\") "
Nov 02 20:17:03.155338 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:03.155271    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f40668dc-8c36-4c34-ae69-c87d8c71e338-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f40668dc-8c36-4c34-ae69-c87d8c71e338-istiod-ca-cert\") pod \"f40668dc-8c36-4c34-ae69-c87d8c71e338\" (UID: \"f40668dc-8c36-4c34-ae69-c87d8c71e338\") "
Nov 02 20:17:03.155338 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:03.155317    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f40668dc-8c36-4c34-ae69-c87d8c71e338-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f40668dc-8c36-4c34-ae69-c87d8c71e338-istio-podinfo\") pod \"f40668dc-8c36-4c34-ae69-c87d8c71e338\" (UID: \"f40668dc-8c36-4c34-ae69-c87d8c71e338\") "
Nov 02 20:17:03.155458 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:03.155389    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f40668dc-8c36-4c34-ae69-c87d8c71e338-istio-token\" (UniqueName: \"kubernetes.io/projected/f40668dc-8c36-4c34-ae69-c87d8c71e338-istio-token\") pod \"f40668dc-8c36-4c34-ae69-c87d8c71e338\" (UID: \"f40668dc-8c36-4c34-ae69-c87d8c71e338\") "
Nov 02 20:17:03.155500 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:03.155445    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f40668dc-8c36-4c34-ae69-c87d8c71e338-credential-socket" pod "f40668dc-8c36-4c34-ae69-c87d8c71e338" (UID: "f40668dc-8c36-4c34-ae69-c87d8c71e338"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:17:03.155500 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:03.155451    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f40668dc-8c36-4c34-ae69-c87d8c71e338-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f40668dc-8c36-4c34-ae69-c87d8c71e338-workload-socket\") pod \"f40668dc-8c36-4c34-ae69-c87d8c71e338\" (UID: \"f40668dc-8c36-4c34-ae69-c87d8c71e338\") "
Nov 02 20:17:03.155500 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:03.155469    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f40668dc-8c36-4c34-ae69-c87d8c71e338-istio-data" pod "f40668dc-8c36-4c34-ae69-c87d8c71e338" (UID: "f40668dc-8c36-4c34-ae69-c87d8c71e338"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:17:03.155635 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:03.155511    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f40668dc-8c36-4c34-ae69-c87d8c71e338-kube-api-access-68hrw\" (UniqueName: \"kubernetes.io/projected/f40668dc-8c36-4c34-ae69-c87d8c71e338-kube-api-access-68hrw\") pod \"f40668dc-8c36-4c34-ae69-c87d8c71e338\" (UID: \"f40668dc-8c36-4c34-ae69-c87d8c71e338\") "
Nov 02 20:17:03.155635 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:03.155567    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f40668dc-8c36-4c34-ae69-c87d8c71e338-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f40668dc-8c36-4c34-ae69-c87d8c71e338-workload-certs\") pod \"f40668dc-8c36-4c34-ae69-c87d8c71e338\" (UID: \"f40668dc-8c36-4c34-ae69-c87d8c71e338\") "
Nov 02 20:17:03.155741 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:03.155665    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f40668dc-8c36-4c34-ae69-c87d8c71e338-istiod-ca-cert" pod "f40668dc-8c36-4c34-ae69-c87d8c71e338" (UID: "f40668dc-8c36-4c34-ae69-c87d8c71e338"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:17:03.155741 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:03.155670    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f40668dc-8c36-4c34-ae69-c87d8c71e338-workload-socket" pod "f40668dc-8c36-4c34-ae69-c87d8c71e338" (UID: "f40668dc-8c36-4c34-ae69-c87d8c71e338"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:17:03.155741 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:03.155697    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f40668dc-8c36-4c34-ae69-c87d8c71e338-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f40668dc-8c36-4c34-ae69-c87d8c71e338-istio-envoy\") pod \"f40668dc-8c36-4c34-ae69-c87d8c71e338\" (UID: \"f40668dc-8c36-4c34-ae69-c87d8c71e338\") "
Nov 02 20:17:03.155905 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:03.155809    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f40668dc-8c36-4c34-ae69-c87d8c71e338-workload-certs" pod "f40668dc-8c36-4c34-ae69-c87d8c71e338" (UID: "f40668dc-8c36-4c34-ae69-c87d8c71e338"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:17:03.156136 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:03.156113    4463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f40668dc-8c36-4c34-ae69-c87d8c71e338-workload-socket\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:17:03.156260 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:03.156138    4463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f40668dc-8c36-4c34-ae69-c87d8c71e338-workload-certs\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:17:03.156260 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:03.156153    4463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f40668dc-8c36-4c34-ae69-c87d8c71e338-credential-socket\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:17:03.156260 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:03.156165    4463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f40668dc-8c36-4c34-ae69-c87d8c71e338-istio-data\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:17:03.156260 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:03.156179    4463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f40668dc-8c36-4c34-ae69-c87d8c71e338-istiod-ca-cert\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:17:03.157567 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:03.157539    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f40668dc-8c36-4c34-ae69-c87d8c71e338-istio-podinfo" pod "f40668dc-8c36-4c34-ae69-c87d8c71e338" (UID: "f40668dc-8c36-4c34-ae69-c87d8c71e338"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:17:03.157673 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:03.157620    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f40668dc-8c36-4c34-ae69-c87d8c71e338-istio-envoy" pod "f40668dc-8c36-4c34-ae69-c87d8c71e338" (UID: "f40668dc-8c36-4c34-ae69-c87d8c71e338"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:17:03.157975 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:03.157954    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f40668dc-8c36-4c34-ae69-c87d8c71e338-istio-token" pod "f40668dc-8c36-4c34-ae69-c87d8c71e338" (UID: "f40668dc-8c36-4c34-ae69-c87d8c71e338"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:17:03.158057 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:03.157987    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f40668dc-8c36-4c34-ae69-c87d8c71e338-kube-api-access-68hrw" pod "f40668dc-8c36-4c34-ae69-c87d8c71e338" (UID: "f40668dc-8c36-4c34-ae69-c87d8c71e338"). InnerVolumeSpecName "kube-api-access-68hrw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:17:03.256680 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:03.256651    4463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f40668dc-8c36-4c34-ae69-c87d8c71e338-istio-token\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:17:03.256680 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:03.256680    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68hrw\" (UniqueName: \"kubernetes.io/projected/f40668dc-8c36-4c34-ae69-c87d8c71e338-kube-api-access-68hrw\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:17:03.256852 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:03.256690    4463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f40668dc-8c36-4c34-ae69-c87d8c71e338-istio-envoy\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:17:03.256852 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:03.256700    4463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f40668dc-8c36-4c34-ae69-c87d8c71e338-istio-podinfo\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:17:03.617646 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:03.617614    4463 generic.go:356] "Generic (PLEG): container finished" podID="f40668dc-8c36-4c34-ae69-c87d8c71e338" containerID="1753e7925ff7ea4b4e8379df3666fb56c09ea49085d3ee5899a6f36484615ba2" exitCode=0
Nov 02 20:17:03.617760 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:03.617662    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-z7nwb"
Nov 02 20:17:03.617760 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:03.617691    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-z7nwb" event={"ID":"f40668dc-8c36-4c34-ae69-c87d8c71e338","Type":"ContainerDied","Data":"1753e7925ff7ea4b4e8379df3666fb56c09ea49085d3ee5899a6f36484615ba2"}
Nov 02 20:17:03.617760 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:03.617736    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-z7nwb" event={"ID":"f40668dc-8c36-4c34-ae69-c87d8c71e338","Type":"ContainerDied","Data":"f5e99d5d0348109dbb82ffde362d4d12f538bbc335af21e81e7e8db7aac32471"}
Nov 02 20:17:03.617760 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:03.617758    4463 scope.go:122] "RemoveContainer" containerID="1753e7925ff7ea4b4e8379df3666fb56c09ea49085d3ee5899a6f36484615ba2"
Nov 02 20:17:03.626275 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:03.626254    4463 scope.go:122] "RemoveContainer" containerID="1753e7925ff7ea4b4e8379df3666fb56c09ea49085d3ee5899a6f36484615ba2"
Nov 02 20:17:03.626536 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:17:03.626514    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1753e7925ff7ea4b4e8379df3666fb56c09ea49085d3ee5899a6f36484615ba2\": container with ID starting with 1753e7925ff7ea4b4e8379df3666fb56c09ea49085d3ee5899a6f36484615ba2 not found: ID does not exist" containerID="1753e7925ff7ea4b4e8379df3666fb56c09ea49085d3ee5899a6f36484615ba2"
Nov 02 20:17:03.626598 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:03.626548    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1753e7925ff7ea4b4e8379df3666fb56c09ea49085d3ee5899a6f36484615ba2"} err="failed to get container status \"1753e7925ff7ea4b4e8379df3666fb56c09ea49085d3ee5899a6f36484615ba2\": rpc error: code = NotFound desc = could not find container \"1753e7925ff7ea4b4e8379df3666fb56c09ea49085d3ee5899a6f36484615ba2\": container with ID starting with 1753e7925ff7ea4b4e8379df3666fb56c09ea49085d3ee5899a6f36484615ba2 not found: ID does not exist"
Nov 02 20:17:03.769889 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:03.769850    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-z7nwb"]
Nov 02 20:17:03.815821 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:03.815787    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-z7nwb"]
Nov 02 20:17:04.296064 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:17:04.296004    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 20:17:05.546771 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:05.546738    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-13e739-565886dbbc-wbgfw" podUID="e20408d3-f3a6-4ca9-8413-aafe209a1477" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 20:17:05.584263 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:05.584231    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f40668dc-8c36-4c34-ae69-c87d8c71e338" path="/var/lib/kubelet/pods/f40668dc-8c36-4c34-ae69-c87d8c71e338/volumes"
Nov 02 20:17:08.527248 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:08.527186    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-13e739-565886dbbc-wbgfw"
Nov 02 20:17:08.603148 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:08.603122    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e20408d3-f3a6-4ca9-8413-aafe209a1477-kube-api-access-fnpmj\" (UniqueName: \"kubernetes.io/projected/e20408d3-f3a6-4ca9-8413-aafe209a1477-kube-api-access-fnpmj\") pod \"e20408d3-f3a6-4ca9-8413-aafe209a1477\" (UID: \"e20408d3-f3a6-4ca9-8413-aafe209a1477\") "
Nov 02 20:17:08.603304 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:08.603204    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e20408d3-f3a6-4ca9-8413-aafe209a1477-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e20408d3-f3a6-4ca9-8413-aafe209a1477-plugins-config-volume\") pod \"e20408d3-f3a6-4ca9-8413-aafe209a1477\" (UID: \"e20408d3-f3a6-4ca9-8413-aafe209a1477\") "
Nov 02 20:17:08.603575 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:08.603551    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e20408d3-f3a6-4ca9-8413-aafe209a1477-plugins-config-volume" pod "e20408d3-f3a6-4ca9-8413-aafe209a1477" (UID: "e20408d3-f3a6-4ca9-8413-aafe209a1477"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:17:08.605042 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:08.605007    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e20408d3-f3a6-4ca9-8413-aafe209a1477-kube-api-access-fnpmj" pod "e20408d3-f3a6-4ca9-8413-aafe209a1477" (UID: "e20408d3-f3a6-4ca9-8413-aafe209a1477"). InnerVolumeSpecName "kube-api-access-fnpmj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:17:08.648722 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:08.648686    4463 generic.go:356] "Generic (PLEG): container finished" podID="e20408d3-f3a6-4ca9-8413-aafe209a1477" containerID="d508d0a4ce5b48c8a62e8c256f746208ea5baff320d3713213b4c4be51d6c014" exitCode=0
Nov 02 20:17:08.648860 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:08.648733    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-13e739-565886dbbc-wbgfw"
Nov 02 20:17:08.648860 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:08.648767    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-13e739-565886dbbc-wbgfw" event={"ID":"e20408d3-f3a6-4ca9-8413-aafe209a1477","Type":"ContainerDied","Data":"d508d0a4ce5b48c8a62e8c256f746208ea5baff320d3713213b4c4be51d6c014"}
Nov 02 20:17:08.648860 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:08.648820    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-13e739-565886dbbc-wbgfw" event={"ID":"e20408d3-f3a6-4ca9-8413-aafe209a1477","Type":"ContainerDied","Data":"cbb16f4ce1ca01e239441fd8bf917eeff095802acdae386b5ac6f08658a0f5f1"}
Nov 02 20:17:08.648860 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:08.648845    4463 scope.go:122] "RemoveContainer" containerID="d508d0a4ce5b48c8a62e8c256f746208ea5baff320d3713213b4c4be51d6c014"
Nov 02 20:17:08.658627 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:08.658611    4463 scope.go:122] "RemoveContainer" containerID="d508d0a4ce5b48c8a62e8c256f746208ea5baff320d3713213b4c4be51d6c014"
Nov 02 20:17:08.658921 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:17:08.658894    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d508d0a4ce5b48c8a62e8c256f746208ea5baff320d3713213b4c4be51d6c014\": container with ID starting with d508d0a4ce5b48c8a62e8c256f746208ea5baff320d3713213b4c4be51d6c014 not found: ID does not exist" containerID="d508d0a4ce5b48c8a62e8c256f746208ea5baff320d3713213b4c4be51d6c014"
Nov 02 20:17:08.658976 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:08.658936    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d508d0a4ce5b48c8a62e8c256f746208ea5baff320d3713213b4c4be51d6c014"} err="failed to get container status \"d508d0a4ce5b48c8a62e8c256f746208ea5baff320d3713213b4c4be51d6c014\": rpc error: code = NotFound desc = could not find container \"d508d0a4ce5b48c8a62e8c256f746208ea5baff320d3713213b4c4be51d6c014\": container with ID starting with d508d0a4ce5b48c8a62e8c256f746208ea5baff320d3713213b4c4be51d6c014 not found: ID does not exist"
Nov 02 20:17:08.704084 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:08.704063    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnpmj\" (UniqueName: \"kubernetes.io/projected/e20408d3-f3a6-4ca9-8413-aafe209a1477-kube-api-access-fnpmj\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:17:08.704084 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:08.704084    4463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e20408d3-f3a6-4ca9-8413-aafe209a1477-plugins-config-volume\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:17:08.800683 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:08.800624    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-13e739-565886dbbc-wbgfw"]
Nov 02 20:17:08.803494 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:08.803466    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-13e739-565886dbbc-wbgfw"]
Nov 02 20:17:09.582322 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:09.582290    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e20408d3-f3a6-4ca9-8413-aafe209a1477" path="/var/lib/kubelet/pods/e20408d3-f3a6-4ca9-8413-aafe209a1477/volumes"
Nov 02 20:17:14.036901 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:14.036865    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-d1aaed-5449d767fd-tcwj8"]
Nov 02 20:17:14.037296 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:14.037069    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-d1aaed-5449d767fd-tcwj8" podUID="b4de3a3a-cbd8-4488-9d04-4ec03e4d53ce" containerName="maas-api" containerID="cri-o://4f0a21b797b70b88bd750b4d7f055f3da4426454fc2c97e814b4d43de72c0000" gracePeriod=30
Nov 02 20:17:14.279721 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:14.279697    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-d1aaed-5449d767fd-tcwj8"
Nov 02 20:17:14.352464 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:14.352403    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4de3a3a-cbd8-4488-9d04-4ec03e4d53ce-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4de3a3a-cbd8-4488-9d04-4ec03e4d53ce-maas-api-tls\") pod \"b4de3a3a-cbd8-4488-9d04-4ec03e4d53ce\" (UID: \"b4de3a3a-cbd8-4488-9d04-4ec03e4d53ce\") "
Nov 02 20:17:14.352464 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:14.352437    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4de3a3a-cbd8-4488-9d04-4ec03e4d53ce-kube-api-access-p7q6m\" (UniqueName: \"kubernetes.io/projected/b4de3a3a-cbd8-4488-9d04-4ec03e4d53ce-kube-api-access-p7q6m\") pod \"b4de3a3a-cbd8-4488-9d04-4ec03e4d53ce\" (UID: \"b4de3a3a-cbd8-4488-9d04-4ec03e4d53ce\") "
Nov 02 20:17:14.352683 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:14.352574    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4de3a3a-cbd8-4488-9d04-4ec03e4d53ce-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4de3a3a-cbd8-4488-9d04-4ec03e4d53ce-metrics-certs\") pod \"b4de3a3a-cbd8-4488-9d04-4ec03e4d53ce\" (UID: \"b4de3a3a-cbd8-4488-9d04-4ec03e4d53ce\") "
Nov 02 20:17:14.352683 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:14.352597    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4de3a3a-cbd8-4488-9d04-4ec03e4d53ce-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4de3a3a-cbd8-4488-9d04-4ec03e4d53ce-openshift-service-ca\") pod \"b4de3a3a-cbd8-4488-9d04-4ec03e4d53ce\" (UID: \"b4de3a3a-cbd8-4488-9d04-4ec03e4d53ce\") "
Nov 02 20:17:14.353090 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:14.353058    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4de3a3a-cbd8-4488-9d04-4ec03e4d53ce-openshift-service-ca" pod "b4de3a3a-cbd8-4488-9d04-4ec03e4d53ce" (UID: "b4de3a3a-cbd8-4488-9d04-4ec03e4d53ce"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:17:14.354643 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:14.354621    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4de3a3a-cbd8-4488-9d04-4ec03e4d53ce-maas-api-tls" pod "b4de3a3a-cbd8-4488-9d04-4ec03e4d53ce" (UID: "b4de3a3a-cbd8-4488-9d04-4ec03e4d53ce"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:17:14.354717 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:14.354683    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4de3a3a-cbd8-4488-9d04-4ec03e4d53ce-metrics-certs" pod "b4de3a3a-cbd8-4488-9d04-4ec03e4d53ce" (UID: "b4de3a3a-cbd8-4488-9d04-4ec03e4d53ce"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:17:14.354717 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:14.354707    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4de3a3a-cbd8-4488-9d04-4ec03e4d53ce-kube-api-access-p7q6m" pod "b4de3a3a-cbd8-4488-9d04-4ec03e4d53ce" (UID: "b4de3a3a-cbd8-4488-9d04-4ec03e4d53ce"). InnerVolumeSpecName "kube-api-access-p7q6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:17:14.453350 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:14.453320    4463 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4de3a3a-cbd8-4488-9d04-4ec03e4d53ce-metrics-certs\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:17:14.453350 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:14.453346    4463 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4de3a3a-cbd8-4488-9d04-4ec03e4d53ce-openshift-service-ca\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:17:14.453350 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:14.453355    4463 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4de3a3a-cbd8-4488-9d04-4ec03e4d53ce-maas-api-tls\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:17:14.453614 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:14.453364    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7q6m\" (UniqueName: \"kubernetes.io/projected/b4de3a3a-cbd8-4488-9d04-4ec03e4d53ce-kube-api-access-p7q6m\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:17:14.622896 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:14.622821    4463 scope.go:122] "RemoveContainer" containerID="4f0a21b797b70b88bd750b4d7f055f3da4426454fc2c97e814b4d43de72c0000"
Nov 02 20:17:14.677390 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:14.677361    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-d1aaed-5449d767fd-tcwj8" event={"ID":"b4de3a3a-cbd8-4488-9d04-4ec03e4d53ce","Type":"ContainerDied","Data":"4f0a21b797b70b88bd750b4d7f055f3da4426454fc2c97e814b4d43de72c0000"}
Nov 02 20:17:14.677492 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:14.677401    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-d1aaed-5449d767fd-tcwj8"
Nov 02 20:17:14.677492 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:14.677412    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-d1aaed-5449d767fd-tcwj8" event={"ID":"b4de3a3a-cbd8-4488-9d04-4ec03e4d53ce","Type":"ContainerDied","Data":"dc918c9284d4e64654e23b2dd455cfb60c764011084f5a1cb8f0aa14d6c5f5b6"}
Nov 02 20:17:14.886362 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:14.886287    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-d1aaed-5449d767fd-tcwj8"]
Nov 02 20:17:14.891132 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:14.891107    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-d1aaed-5449d767fd-tcwj8"]
Nov 02 20:17:15.584142 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:15.584108    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4de3a3a-cbd8-4488-9d04-4ec03e4d53ce" path="/var/lib/kubelet/pods/b4de3a3a-cbd8-4488-9d04-4ec03e4d53ce/volumes"
Nov 02 20:17:23.992972 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:23.992936    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 20:17:34.293314 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:17:34.293278    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 20:17:44.039215 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:44.039184    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4848d2-565fb9d78c-pd59p"]
Nov 02 20:17:44.039691 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:44.039375    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4848d2-565fb9d78c-pd59p" podUID="aa56df3b-fd52-46b8-a626-0dd7e827ab8c" containerName="maas-api" containerID="cri-o://34aa0a30d4ba98e7b72463224c419a7cad22a36effdbc70329f675d137d678d3" gracePeriod=30
Nov 02 20:17:44.285832 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:44.285809    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4848d2-565fb9d78c-pd59p"
Nov 02 20:17:44.424408 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:44.424375    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa56df3b-fd52-46b8-a626-0dd7e827ab8c-kube-api-access-chq9t\" (UniqueName: \"kubernetes.io/projected/aa56df3b-fd52-46b8-a626-0dd7e827ab8c-kube-api-access-chq9t\") pod \"aa56df3b-fd52-46b8-a626-0dd7e827ab8c\" (UID: \"aa56df3b-fd52-46b8-a626-0dd7e827ab8c\") "
Nov 02 20:17:44.424590 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:44.424436    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa56df3b-fd52-46b8-a626-0dd7e827ab8c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa56df3b-fd52-46b8-a626-0dd7e827ab8c-maas-api-tls\") pod \"aa56df3b-fd52-46b8-a626-0dd7e827ab8c\" (UID: \"aa56df3b-fd52-46b8-a626-0dd7e827ab8c\") "
Nov 02 20:17:44.424590 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:44.424522    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa56df3b-fd52-46b8-a626-0dd7e827ab8c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa56df3b-fd52-46b8-a626-0dd7e827ab8c-openshift-service-ca\") pod \"aa56df3b-fd52-46b8-a626-0dd7e827ab8c\" (UID: \"aa56df3b-fd52-46b8-a626-0dd7e827ab8c\") "
Nov 02 20:17:44.424590 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:44.424550    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa56df3b-fd52-46b8-a626-0dd7e827ab8c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa56df3b-fd52-46b8-a626-0dd7e827ab8c-metrics-certs\") pod \"aa56df3b-fd52-46b8-a626-0dd7e827ab8c\" (UID: \"aa56df3b-fd52-46b8-a626-0dd7e827ab8c\") "
Nov 02 20:17:44.424916 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:44.424883    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa56df3b-fd52-46b8-a626-0dd7e827ab8c-openshift-service-ca" pod "aa56df3b-fd52-46b8-a626-0dd7e827ab8c" (UID: "aa56df3b-fd52-46b8-a626-0dd7e827ab8c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:17:44.426648 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:44.426617    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa56df3b-fd52-46b8-a626-0dd7e827ab8c-kube-api-access-chq9t" pod "aa56df3b-fd52-46b8-a626-0dd7e827ab8c" (UID: "aa56df3b-fd52-46b8-a626-0dd7e827ab8c"). InnerVolumeSpecName "kube-api-access-chq9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:17:44.426763 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:44.426624    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa56df3b-fd52-46b8-a626-0dd7e827ab8c-maas-api-tls" pod "aa56df3b-fd52-46b8-a626-0dd7e827ab8c" (UID: "aa56df3b-fd52-46b8-a626-0dd7e827ab8c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:17:44.427045 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:44.426999    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa56df3b-fd52-46b8-a626-0dd7e827ab8c-metrics-certs" pod "aa56df3b-fd52-46b8-a626-0dd7e827ab8c" (UID: "aa56df3b-fd52-46b8-a626-0dd7e827ab8c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:17:44.525697 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:44.525663    4463 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa56df3b-fd52-46b8-a626-0dd7e827ab8c-openshift-service-ca\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:17:44.525697 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:44.525690    4463 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa56df3b-fd52-46b8-a626-0dd7e827ab8c-metrics-certs\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:17:44.525697 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:44.525700    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chq9t\" (UniqueName: \"kubernetes.io/projected/aa56df3b-fd52-46b8-a626-0dd7e827ab8c-kube-api-access-chq9t\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:17:44.525924 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:44.525710    4463 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa56df3b-fd52-46b8-a626-0dd7e827ab8c-maas-api-tls\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:17:44.546491 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:44.546459    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 20:17:44.814550 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:44.814470    4463 generic.go:356] "Generic (PLEG): container finished" podID="aa56df3b-fd52-46b8-a626-0dd7e827ab8c" containerID="34aa0a30d4ba98e7b72463224c419a7cad22a36effdbc70329f675d137d678d3" exitCode=0
Nov 02 20:17:44.814550 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:44.814519    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4848d2-565fb9d78c-pd59p"
Nov 02 20:17:44.814747 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:44.814545    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4848d2-565fb9d78c-pd59p" event={"ID":"aa56df3b-fd52-46b8-a626-0dd7e827ab8c","Type":"ContainerDied","Data":"34aa0a30d4ba98e7b72463224c419a7cad22a36effdbc70329f675d137d678d3"}
Nov 02 20:17:44.814747 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:44.814600    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4848d2-565fb9d78c-pd59p" event={"ID":"aa56df3b-fd52-46b8-a626-0dd7e827ab8c","Type":"ContainerDied","Data":"e45235736ec8ccffd5eac053f651e922702b3f70850baac7e082a61b204d3424"}
Nov 02 20:17:44.814747 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:44.814627    4463 scope.go:122] "RemoveContainer" containerID="34aa0a30d4ba98e7b72463224c419a7cad22a36effdbc70329f675d137d678d3"
Nov 02 20:17:44.823959 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:44.823942    4463 scope.go:122] "RemoveContainer" containerID="34aa0a30d4ba98e7b72463224c419a7cad22a36effdbc70329f675d137d678d3"
Nov 02 20:17:44.824296 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:17:44.824272    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34aa0a30d4ba98e7b72463224c419a7cad22a36effdbc70329f675d137d678d3\": container with ID starting with 34aa0a30d4ba98e7b72463224c419a7cad22a36effdbc70329f675d137d678d3 not found: ID does not exist" containerID="34aa0a30d4ba98e7b72463224c419a7cad22a36effdbc70329f675d137d678d3"
Nov 02 20:17:44.824413 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:44.824300    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34aa0a30d4ba98e7b72463224c419a7cad22a36effdbc70329f675d137d678d3"} err="failed to get container status \"34aa0a30d4ba98e7b72463224c419a7cad22a36effdbc70329f675d137d678d3\": rpc error: code = NotFound desc = could not find container \"34aa0a30d4ba98e7b72463224c419a7cad22a36effdbc70329f675d137d678d3\": container with ID starting with 34aa0a30d4ba98e7b72463224c419a7cad22a36effdbc70329f675d137d678d3 not found: ID does not exist"
Nov 02 20:17:44.967526 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:44.966898    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4848d2-565fb9d78c-pd59p"]
Nov 02 20:17:44.969156 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:44.969123    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4848d2-565fb9d78c-pd59p"]
Nov 02 20:17:45.122301 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:45.122204    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-24ef8b-7fff7cb988-vllfk"]
Nov 02 20:17:45.122749 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:45.122458    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-24ef8b-7fff7cb988-vllfk" podUID="a053056d-0fa4-4ea9-8091-85979931330f" containerName="maas-api" containerID="cri-o://42ca1f33848771f5acc256713cae4d411f5302377cb12b633e8ab34ba79e94f5" gracePeriod=30
Nov 02 20:17:45.369851 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:45.369823    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-24ef8b-7fff7cb988-vllfk"
Nov 02 20:17:45.535546 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:45.535516    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a053056d-0fa4-4ea9-8091-85979931330f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a053056d-0fa4-4ea9-8091-85979931330f-maas-api-tls\") pod \"a053056d-0fa4-4ea9-8091-85979931330f\" (UID: \"a053056d-0fa4-4ea9-8091-85979931330f\") "
Nov 02 20:17:45.535844 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:45.535563    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a053056d-0fa4-4ea9-8091-85979931330f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a053056d-0fa4-4ea9-8091-85979931330f-openshift-service-ca\") pod \"a053056d-0fa4-4ea9-8091-85979931330f\" (UID: \"a053056d-0fa4-4ea9-8091-85979931330f\") "
Nov 02 20:17:45.535844 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:45.535591    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a053056d-0fa4-4ea9-8091-85979931330f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a053056d-0fa4-4ea9-8091-85979931330f-metrics-certs\") pod \"a053056d-0fa4-4ea9-8091-85979931330f\" (UID: \"a053056d-0fa4-4ea9-8091-85979931330f\") "
Nov 02 20:17:45.535844 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:45.535697    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a053056d-0fa4-4ea9-8091-85979931330f-kube-api-access-s2ktt\" (UniqueName: \"kubernetes.io/projected/a053056d-0fa4-4ea9-8091-85979931330f-kube-api-access-s2ktt\") pod \"a053056d-0fa4-4ea9-8091-85979931330f\" (UID: \"a053056d-0fa4-4ea9-8091-85979931330f\") "
Nov 02 20:17:45.536082 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:45.535952    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a053056d-0fa4-4ea9-8091-85979931330f-openshift-service-ca" pod "a053056d-0fa4-4ea9-8091-85979931330f" (UID: "a053056d-0fa4-4ea9-8091-85979931330f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:17:45.537944 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:45.537919    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a053056d-0fa4-4ea9-8091-85979931330f-kube-api-access-s2ktt" pod "a053056d-0fa4-4ea9-8091-85979931330f" (UID: "a053056d-0fa4-4ea9-8091-85979931330f"). InnerVolumeSpecName "kube-api-access-s2ktt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:17:45.538297 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:45.538274    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a053056d-0fa4-4ea9-8091-85979931330f-metrics-certs" pod "a053056d-0fa4-4ea9-8091-85979931330f" (UID: "a053056d-0fa4-4ea9-8091-85979931330f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:17:45.538297 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:45.538291    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a053056d-0fa4-4ea9-8091-85979931330f-maas-api-tls" pod "a053056d-0fa4-4ea9-8091-85979931330f" (UID: "a053056d-0fa4-4ea9-8091-85979931330f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:17:45.582710 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:45.582684    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa56df3b-fd52-46b8-a626-0dd7e827ab8c" path="/var/lib/kubelet/pods/aa56df3b-fd52-46b8-a626-0dd7e827ab8c/volumes"
Nov 02 20:17:45.636973 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:45.636944    4463 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a053056d-0fa4-4ea9-8091-85979931330f-maas-api-tls\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:17:45.636973 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:45.636971    4463 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a053056d-0fa4-4ea9-8091-85979931330f-openshift-service-ca\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:17:45.637176 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:45.636984    4463 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a053056d-0fa4-4ea9-8091-85979931330f-metrics-certs\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:17:45.637176 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:45.636993    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2ktt\" (UniqueName: \"kubernetes.io/projected/a053056d-0fa4-4ea9-8091-85979931330f-kube-api-access-s2ktt\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:17:45.821617 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:45.821538    4463 generic.go:356] "Generic (PLEG): container finished" podID="a053056d-0fa4-4ea9-8091-85979931330f" containerID="42ca1f33848771f5acc256713cae4d411f5302377cb12b633e8ab34ba79e94f5" exitCode=0
Nov 02 20:17:45.821617 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:45.821583    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-24ef8b-7fff7cb988-vllfk"
Nov 02 20:17:45.821812 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:45.821614    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-24ef8b-7fff7cb988-vllfk" event={"ID":"a053056d-0fa4-4ea9-8091-85979931330f","Type":"ContainerDied","Data":"42ca1f33848771f5acc256713cae4d411f5302377cb12b633e8ab34ba79e94f5"}
Nov 02 20:17:45.821812 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:45.821681    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-24ef8b-7fff7cb988-vllfk" event={"ID":"a053056d-0fa4-4ea9-8091-85979931330f","Type":"ContainerDied","Data":"630a4f2e0ca646400c14f58a9f98e12976d2caaf2ee69dcec693248417a9a11e"}
Nov 02 20:17:45.821812 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:45.821704    4463 scope.go:122] "RemoveContainer" containerID="42ca1f33848771f5acc256713cae4d411f5302377cb12b633e8ab34ba79e94f5"
Nov 02 20:17:45.832742 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:45.832720    4463 scope.go:122] "RemoveContainer" containerID="42ca1f33848771f5acc256713cae4d411f5302377cb12b633e8ab34ba79e94f5"
Nov 02 20:17:45.833072 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:17:45.833053    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42ca1f33848771f5acc256713cae4d411f5302377cb12b633e8ab34ba79e94f5\": container with ID starting with 42ca1f33848771f5acc256713cae4d411f5302377cb12b633e8ab34ba79e94f5 not found: ID does not exist" containerID="42ca1f33848771f5acc256713cae4d411f5302377cb12b633e8ab34ba79e94f5"
Nov 02 20:17:45.833204 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:45.833080    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42ca1f33848771f5acc256713cae4d411f5302377cb12b633e8ab34ba79e94f5"} err="failed to get container status \"42ca1f33848771f5acc256713cae4d411f5302377cb12b633e8ab34ba79e94f5\": rpc error: code = NotFound desc = could not find container \"42ca1f33848771f5acc256713cae4d411f5302377cb12b633e8ab34ba79e94f5\": container with ID starting with 42ca1f33848771f5acc256713cae4d411f5302377cb12b633e8ab34ba79e94f5 not found: ID does not exist"
Nov 02 20:17:45.973205 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:45.973173    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-24ef8b-7fff7cb988-vllfk"]
Nov 02 20:17:45.976943 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:45.976919    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-24ef8b-7fff7cb988-vllfk"]
Nov 02 20:17:47.584499 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:47.584463    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a053056d-0fa4-4ea9-8091-85979931330f" path="/var/lib/kubelet/pods/a053056d-0fa4-4ea9-8091-85979931330f/volumes"
Nov 02 20:17:53.837065 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:53.837028    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-24ef8b-openshift-default-685d675f47-76xb9"]
Nov 02 20:17:53.837482 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:53.837244    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-24ef8b-openshift-default-685d675f47-76xb9" podUID="b2c0f86f-3e9e-4996-8710-664c445e9113" containerName="istio-proxy" containerID="cri-o://cf9f5f253bbb7e992b020a6d5848a9abe04be961c682b3b028fe46c434642e3c" gracePeriod=30
Nov 02 20:17:54.628115 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:54.628079    4463 patch_prober.go:29] interesting pod/e2e-shared-b-w6-24ef8b-openshift-default-685d675f47-76xb9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 20:17:54.628303 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:54.628142    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w6-24ef8b-openshift-default-685d675f47-76xb9" podUID="b2c0f86f-3e9e-4996-8710-664c445e9113" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 20:17:59.086190 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:59.086165    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-24ef8b-openshift-default-685d675f47-76xb9"
Nov 02 20:17:59.159276 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:59.159211    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2c0f86f-3e9e-4996-8710-664c445e9113-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2c0f86f-3e9e-4996-8710-664c445e9113-istio-data\") pod \"b2c0f86f-3e9e-4996-8710-664c445e9113\" (UID: \"b2c0f86f-3e9e-4996-8710-664c445e9113\") "
Nov 02 20:17:59.159276 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:59.159247    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2c0f86f-3e9e-4996-8710-664c445e9113-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2c0f86f-3e9e-4996-8710-664c445e9113-workload-certs\") pod \"b2c0f86f-3e9e-4996-8710-664c445e9113\" (UID: \"b2c0f86f-3e9e-4996-8710-664c445e9113\") "
Nov 02 20:17:59.159276 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:59.159272    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2c0f86f-3e9e-4996-8710-664c445e9113-istio-token\" (UniqueName: \"kubernetes.io/projected/b2c0f86f-3e9e-4996-8710-664c445e9113-istio-token\") pod \"b2c0f86f-3e9e-4996-8710-664c445e9113\" (UID: \"b2c0f86f-3e9e-4996-8710-664c445e9113\") "
Nov 02 20:17:59.159488 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:59.159338    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2c0f86f-3e9e-4996-8710-664c445e9113-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2c0f86f-3e9e-4996-8710-664c445e9113-workload-socket\") pod \"b2c0f86f-3e9e-4996-8710-664c445e9113\" (UID: \"b2c0f86f-3e9e-4996-8710-664c445e9113\") "
Nov 02 20:17:59.159488 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:59.159372    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2c0f86f-3e9e-4996-8710-664c445e9113-kube-api-access-pqw4l\" (UniqueName: \"kubernetes.io/projected/b2c0f86f-3e9e-4996-8710-664c445e9113-kube-api-access-pqw4l\") pod \"b2c0f86f-3e9e-4996-8710-664c445e9113\" (UID: \"b2c0f86f-3e9e-4996-8710-664c445e9113\") "
Nov 02 20:17:59.159488 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:59.159449    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2c0f86f-3e9e-4996-8710-664c445e9113-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2c0f86f-3e9e-4996-8710-664c445e9113-credential-socket\") pod \"b2c0f86f-3e9e-4996-8710-664c445e9113\" (UID: \"b2c0f86f-3e9e-4996-8710-664c445e9113\") "
Nov 02 20:17:59.159659 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:59.159520    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b2c0f86f-3e9e-4996-8710-664c445e9113-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2c0f86f-3e9e-4996-8710-664c445e9113-istio-podinfo\") pod \"b2c0f86f-3e9e-4996-8710-664c445e9113\" (UID: \"b2c0f86f-3e9e-4996-8710-664c445e9113\") "
Nov 02 20:17:59.159659 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:59.159525    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2c0f86f-3e9e-4996-8710-664c445e9113-istio-data" pod "b2c0f86f-3e9e-4996-8710-664c445e9113" (UID: "b2c0f86f-3e9e-4996-8710-664c445e9113"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:17:59.159659 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:59.159545    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2c0f86f-3e9e-4996-8710-664c445e9113-workload-certs" pod "b2c0f86f-3e9e-4996-8710-664c445e9113" (UID: "b2c0f86f-3e9e-4996-8710-664c445e9113"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:17:59.159659 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:59.159570    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2c0f86f-3e9e-4996-8710-664c445e9113-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2c0f86f-3e9e-4996-8710-664c445e9113-istio-envoy\") pod \"b2c0f86f-3e9e-4996-8710-664c445e9113\" (UID: \"b2c0f86f-3e9e-4996-8710-664c445e9113\") "
Nov 02 20:17:59.159659 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:59.159603    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2c0f86f-3e9e-4996-8710-664c445e9113-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2c0f86f-3e9e-4996-8710-664c445e9113-istiod-ca-cert\") pod \"b2c0f86f-3e9e-4996-8710-664c445e9113\" (UID: \"b2c0f86f-3e9e-4996-8710-664c445e9113\") "
Nov 02 20:17:59.159659 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:59.159650    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2c0f86f-3e9e-4996-8710-664c445e9113-workload-socket" pod "b2c0f86f-3e9e-4996-8710-664c445e9113" (UID: "b2c0f86f-3e9e-4996-8710-664c445e9113"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:17:59.159964 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:59.159848    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2c0f86f-3e9e-4996-8710-664c445e9113-credential-socket" pod "b2c0f86f-3e9e-4996-8710-664c445e9113" (UID: "b2c0f86f-3e9e-4996-8710-664c445e9113"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:17:59.160046 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:59.159980    4463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2c0f86f-3e9e-4996-8710-664c445e9113-istio-data\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:17:59.160046 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:59.160001    4463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2c0f86f-3e9e-4996-8710-664c445e9113-workload-certs\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:17:59.160046 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:59.160037    4463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2c0f86f-3e9e-4996-8710-664c445e9113-workload-socket\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:17:59.160046 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:59.160047    4463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2c0f86f-3e9e-4996-8710-664c445e9113-credential-socket\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:17:59.160273 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:59.160079    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2c0f86f-3e9e-4996-8710-664c445e9113-istiod-ca-cert" pod "b2c0f86f-3e9e-4996-8710-664c445e9113" (UID: "b2c0f86f-3e9e-4996-8710-664c445e9113"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:17:59.161851 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:59.161822    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2c0f86f-3e9e-4996-8710-664c445e9113-istio-envoy" pod "b2c0f86f-3e9e-4996-8710-664c445e9113" (UID: "b2c0f86f-3e9e-4996-8710-664c445e9113"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:17:59.161976 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:59.161885    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b2c0f86f-3e9e-4996-8710-664c445e9113-istio-podinfo" pod "b2c0f86f-3e9e-4996-8710-664c445e9113" (UID: "b2c0f86f-3e9e-4996-8710-664c445e9113"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:17:59.162108 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:59.161971    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2c0f86f-3e9e-4996-8710-664c445e9113-istio-token" pod "b2c0f86f-3e9e-4996-8710-664c445e9113" (UID: "b2c0f86f-3e9e-4996-8710-664c445e9113"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:17:59.162108 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:59.161987    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2c0f86f-3e9e-4996-8710-664c445e9113-kube-api-access-pqw4l" pod "b2c0f86f-3e9e-4996-8710-664c445e9113" (UID: "b2c0f86f-3e9e-4996-8710-664c445e9113"). InnerVolumeSpecName "kube-api-access-pqw4l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:17:59.261238 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:59.261207    4463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2c0f86f-3e9e-4996-8710-664c445e9113-istio-envoy\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:17:59.261238 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:59.261231    4463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2c0f86f-3e9e-4996-8710-664c445e9113-istiod-ca-cert\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:17:59.261238 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:59.261240    4463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2c0f86f-3e9e-4996-8710-664c445e9113-istio-token\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:17:59.261463 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:59.261249    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqw4l\" (UniqueName: \"kubernetes.io/projected/b2c0f86f-3e9e-4996-8710-664c445e9113-kube-api-access-pqw4l\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:17:59.261463 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:59.261258    4463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2c0f86f-3e9e-4996-8710-664c445e9113-istio-podinfo\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:17:59.288201 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:59.288169    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 20:17:59.888121 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:59.888008    4463 generic.go:356] "Generic (PLEG): container finished" podID="b2c0f86f-3e9e-4996-8710-664c445e9113" containerID="cf9f5f253bbb7e992b020a6d5848a9abe04be961c682b3b028fe46c434642e3c" exitCode=0
Nov 02 20:17:59.888121 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:59.888046    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-24ef8b-openshift-default-685d675f47-76xb9" event={"ID":"b2c0f86f-3e9e-4996-8710-664c445e9113","Type":"ContainerDied","Data":"cf9f5f253bbb7e992b020a6d5848a9abe04be961c682b3b028fe46c434642e3c"}
Nov 02 20:17:59.888121 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:59.888101    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-24ef8b-openshift-default-685d675f47-76xb9"
Nov 02 20:17:59.888386 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:59.888151    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-24ef8b-openshift-default-685d675f47-76xb9" event={"ID":"b2c0f86f-3e9e-4996-8710-664c445e9113","Type":"ContainerDied","Data":"02bd90f6b755cb4e6ccef328c61c999b80f551f74e182a625f178384a33515ba"}
Nov 02 20:17:59.888386 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:59.888184    4463 scope.go:122] "RemoveContainer" containerID="cf9f5f253bbb7e992b020a6d5848a9abe04be961c682b3b028fe46c434642e3c"
Nov 02 20:17:59.896999 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:59.896983    4463 scope.go:122] "RemoveContainer" containerID="36d39bfe6697d6d7172d4e0ead0d7f1b422687a8126f90f98969a12155c00d60"
Nov 02 20:17:59.904929 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:59.904909    4463 scope.go:122] "RemoveContainer" containerID="cf9f5f253bbb7e992b020a6d5848a9abe04be961c682b3b028fe46c434642e3c"
Nov 02 20:17:59.905192 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:17:59.905172    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf9f5f253bbb7e992b020a6d5848a9abe04be961c682b3b028fe46c434642e3c\": container with ID starting with cf9f5f253bbb7e992b020a6d5848a9abe04be961c682b3b028fe46c434642e3c not found: ID does not exist" containerID="cf9f5f253bbb7e992b020a6d5848a9abe04be961c682b3b028fe46c434642e3c"
Nov 02 20:17:59.905245 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:59.905201    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf9f5f253bbb7e992b020a6d5848a9abe04be961c682b3b028fe46c434642e3c"} err="failed to get container status \"cf9f5f253bbb7e992b020a6d5848a9abe04be961c682b3b028fe46c434642e3c\": rpc error: code = NotFound desc = could not find container \"cf9f5f253bbb7e992b020a6d5848a9abe04be961c682b3b028fe46c434642e3c\": container with ID starting with cf9f5f253bbb7e992b020a6d5848a9abe04be961c682b3b028fe46c434642e3c not found: ID does not exist"
Nov 02 20:17:59.905245 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:59.905218    4463 scope.go:122] "RemoveContainer" containerID="36d39bfe6697d6d7172d4e0ead0d7f1b422687a8126f90f98969a12155c00d60"
Nov 02 20:17:59.905447 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:17:59.905432    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36d39bfe6697d6d7172d4e0ead0d7f1b422687a8126f90f98969a12155c00d60\": container with ID starting with 36d39bfe6697d6d7172d4e0ead0d7f1b422687a8126f90f98969a12155c00d60 not found: ID does not exist" containerID="36d39bfe6697d6d7172d4e0ead0d7f1b422687a8126f90f98969a12155c00d60"
Nov 02 20:17:59.905487 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:17:59.905450    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36d39bfe6697d6d7172d4e0ead0d7f1b422687a8126f90f98969a12155c00d60"} err="failed to get container status \"36d39bfe6697d6d7172d4e0ead0d7f1b422687a8126f90f98969a12155c00d60\": rpc error: code = NotFound desc = could not find container \"36d39bfe6697d6d7172d4e0ead0d7f1b422687a8126f90f98969a12155c00d60\": container with ID starting with 36d39bfe6697d6d7172d4e0ead0d7f1b422687a8126f90f98969a12155c00d60 not found: ID does not exist"
Nov 02 20:18:00.042002 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:00.041965    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-24ef8b-openshift-default-685d675f47-76xb9"]
Nov 02 20:18:00.088309 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:00.088271    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-24ef8b-openshift-default-685d675f47-76xb9"]
Nov 02 20:18:01.582485 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:01.582447    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2c0f86f-3e9e-4996-8710-664c445e9113" path="/var/lib/kubelet/pods/b2c0f86f-3e9e-4996-8710-664c445e9113/volumes"
Nov 02 20:18:04.293340 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:18:04.293306    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 20:18:16.546122 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:16.546083    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 20:18:23.011404 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:23.011368    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-26cfbe/distinct-w1-26cfbe-kserve-685985bfb-tkgn2"]
Nov 02 20:18:23.058162 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:23.058125    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-26cfbe/simulator-w1-26cfbe-kserve-74c5578fbd-t79hq"]
Nov 02 20:18:30.157852 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:30.157777    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-26cfbe-674f8bdf75-kjvkz"]
Nov 02 20:18:34.293286 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:18:34.293248    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 20:18:35.158450 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:35.158402    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-26cfbe-674f8bdf75-kjvkz" podUID="03ec9733-7e8d-49fa-94d6-6000b30c0e16" containerName="payload-pre-processing" containerID="cri-o://5df7ddcae246819d46c049c0004141bab0c810f3b8573f22f2b5ac632fbf2565" gracePeriod=25
Nov 02 20:18:35.411153 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:35.411076    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-26cfbe-674f8bdf75-kjvkz"
Nov 02 20:18:35.493116 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:35.493082    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03ec9733-7e8d-49fa-94d6-6000b30c0e16-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03ec9733-7e8d-49fa-94d6-6000b30c0e16-plugins-config-volume\") pod \"03ec9733-7e8d-49fa-94d6-6000b30c0e16\" (UID: \"03ec9733-7e8d-49fa-94d6-6000b30c0e16\") "
Nov 02 20:18:35.493711 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:35.493588    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03ec9733-7e8d-49fa-94d6-6000b30c0e16-plugins-config-volume" pod "03ec9733-7e8d-49fa-94d6-6000b30c0e16" (UID: "03ec9733-7e8d-49fa-94d6-6000b30c0e16"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:18:35.493886 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:35.493767    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03ec9733-7e8d-49fa-94d6-6000b30c0e16-kube-api-access-6644h\" (UniqueName: \"kubernetes.io/projected/03ec9733-7e8d-49fa-94d6-6000b30c0e16-kube-api-access-6644h\") pod \"03ec9733-7e8d-49fa-94d6-6000b30c0e16\" (UID: \"03ec9733-7e8d-49fa-94d6-6000b30c0e16\") "
Nov 02 20:18:35.494286 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:35.494249    4463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03ec9733-7e8d-49fa-94d6-6000b30c0e16-plugins-config-volume\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:35.496969 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:35.496943    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03ec9733-7e8d-49fa-94d6-6000b30c0e16-kube-api-access-6644h" pod "03ec9733-7e8d-49fa-94d6-6000b30c0e16" (UID: "03ec9733-7e8d-49fa-94d6-6000b30c0e16"). InnerVolumeSpecName "kube-api-access-6644h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:18:35.596267 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:35.596235    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6644h\" (UniqueName: \"kubernetes.io/projected/03ec9733-7e8d-49fa-94d6-6000b30c0e16-kube-api-access-6644h\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:36.046917 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:36.046872    4463 generic.go:356] "Generic (PLEG): container finished" podID="03ec9733-7e8d-49fa-94d6-6000b30c0e16" containerID="5df7ddcae246819d46c049c0004141bab0c810f3b8573f22f2b5ac632fbf2565" exitCode=0
Nov 02 20:18:36.047116 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:36.046922    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-26cfbe-674f8bdf75-kjvkz" event={"ID":"03ec9733-7e8d-49fa-94d6-6000b30c0e16","Type":"ContainerDied","Data":"5df7ddcae246819d46c049c0004141bab0c810f3b8573f22f2b5ac632fbf2565"}
Nov 02 20:18:36.047116 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:36.046937    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-26cfbe-674f8bdf75-kjvkz"
Nov 02 20:18:36.047116 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:36.047008    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-26cfbe-674f8bdf75-kjvkz" event={"ID":"03ec9733-7e8d-49fa-94d6-6000b30c0e16","Type":"ContainerDied","Data":"acf2b941581620028a56e7e91b0c563df9185f5ff4d320e60cd0dabfe1daab9a"}
Nov 02 20:18:36.047116 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:36.047063    4463 scope.go:122] "RemoveContainer" containerID="5df7ddcae246819d46c049c0004141bab0c810f3b8573f22f2b5ac632fbf2565"
Nov 02 20:18:36.055353 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:36.055331    4463 scope.go:122] "RemoveContainer" containerID="5df7ddcae246819d46c049c0004141bab0c810f3b8573f22f2b5ac632fbf2565"
Nov 02 20:18:36.055624 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:18:36.055595    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5df7ddcae246819d46c049c0004141bab0c810f3b8573f22f2b5ac632fbf2565\": container with ID starting with 5df7ddcae246819d46c049c0004141bab0c810f3b8573f22f2b5ac632fbf2565 not found: ID does not exist" containerID="5df7ddcae246819d46c049c0004141bab0c810f3b8573f22f2b5ac632fbf2565"
Nov 02 20:18:36.055682 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:36.055632    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5df7ddcae246819d46c049c0004141bab0c810f3b8573f22f2b5ac632fbf2565"} err="failed to get container status \"5df7ddcae246819d46c049c0004141bab0c810f3b8573f22f2b5ac632fbf2565\": rpc error: code = NotFound desc = could not find container \"5df7ddcae246819d46c049c0004141bab0c810f3b8573f22f2b5ac632fbf2565\": container with ID starting with 5df7ddcae246819d46c049c0004141bab0c810f3b8573f22f2b5ac632fbf2565 not found: ID does not exist"
Nov 02 20:18:36.198124 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:36.198096    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-26cfbe-674f8bdf75-kjvkz"]
Nov 02 20:18:36.203561 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:36.203538    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-26cfbe-674f8bdf75-kjvkz"]
Nov 02 20:18:37.583823 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:37.583791    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03ec9733-7e8d-49fa-94d6-6000b30c0e16" path="/var/lib/kubelet/pods/03ec9733-7e8d-49fa-94d6-6000b30c0e16/volumes"
Nov 02 20:18:38.030939 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:38.030890    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-26cfbe/distinct-w1-26cfbe-kserve-685985bfb-tkgn2" podUID="7415aa62-467c-43af-94c8-d604f57ec1c3" containerName="main" containerID="cri-o://070cf61c57bcab497189c1d0adb97ea4d06c65d76aef34acbf1bbe8a56fd457f" gracePeriod=45
Nov 02 20:18:38.077620 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:38.077582    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-26cfbe/simulator-w1-26cfbe-kserve-74c5578fbd-t79hq" podUID="50dc76f4-aebd-41f4-9f23-799ddc8c70f4" containerName="main" containerID="cri-o://17ea621ec4532ca3d7f1a5ffd28fe026bc1d8b6f532e2139c3dd84cd5817f679" gracePeriod=45
Nov 02 20:18:38.323426 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:38.323401    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-26cfbe/distinct-w1-26cfbe-kserve-685985bfb-tkgn2"
Nov 02 20:18:38.347395 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:38.347375    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-26cfbe/simulator-w1-26cfbe-kserve-74c5578fbd-t79hq"
Nov 02 20:18:38.417888 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:38.417861    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7415aa62-467c-43af-94c8-d604f57ec1c3-tls-certs\" (UniqueName: \"kubernetes.io/secret/7415aa62-467c-43af-94c8-d604f57ec1c3-tls-certs\") pod \"7415aa62-467c-43af-94c8-d604f57ec1c3\" (UID: \"7415aa62-467c-43af-94c8-d604f57ec1c3\") "
Nov 02 20:18:38.418044 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:38.417896    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50dc76f4-aebd-41f4-9f23-799ddc8c70f4-tls-certs\" (UniqueName: \"kubernetes.io/secret/50dc76f4-aebd-41f4-9f23-799ddc8c70f4-tls-certs\") pod \"50dc76f4-aebd-41f4-9f23-799ddc8c70f4\" (UID: \"50dc76f4-aebd-41f4-9f23-799ddc8c70f4\") "
Nov 02 20:18:38.418044 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:38.417923    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7415aa62-467c-43af-94c8-d604f57ec1c3-kube-api-access-j9zgq\" (UniqueName: \"kubernetes.io/projected/7415aa62-467c-43af-94c8-d604f57ec1c3-kube-api-access-j9zgq\") pod \"7415aa62-467c-43af-94c8-d604f57ec1c3\" (UID: \"7415aa62-467c-43af-94c8-d604f57ec1c3\") "
Nov 02 20:18:38.418044 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:38.417954    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50dc76f4-aebd-41f4-9f23-799ddc8c70f4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50dc76f4-aebd-41f4-9f23-799ddc8c70f4-model-cache\") pod \"50dc76f4-aebd-41f4-9f23-799ddc8c70f4\" (UID: \"50dc76f4-aebd-41f4-9f23-799ddc8c70f4\") "
Nov 02 20:18:38.418044 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:38.418002    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7415aa62-467c-43af-94c8-d604f57ec1c3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7415aa62-467c-43af-94c8-d604f57ec1c3-model-cache\") pod \"7415aa62-467c-43af-94c8-d604f57ec1c3\" (UID: \"7415aa62-467c-43af-94c8-d604f57ec1c3\") "
Nov 02 20:18:38.418288 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:38.418044    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7415aa62-467c-43af-94c8-d604f57ec1c3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7415aa62-467c-43af-94c8-d604f57ec1c3-tmp-dir\") pod \"7415aa62-467c-43af-94c8-d604f57ec1c3\" (UID: \"7415aa62-467c-43af-94c8-d604f57ec1c3\") "
Nov 02 20:18:38.418288 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:38.418104    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50dc76f4-aebd-41f4-9f23-799ddc8c70f4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/50dc76f4-aebd-41f4-9f23-799ddc8c70f4-dshm\") pod \"50dc76f4-aebd-41f4-9f23-799ddc8c70f4\" (UID: \"50dc76f4-aebd-41f4-9f23-799ddc8c70f4\") "
Nov 02 20:18:38.418288 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:38.418131    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7415aa62-467c-43af-94c8-d604f57ec1c3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7415aa62-467c-43af-94c8-d604f57ec1c3-dshm\") pod \"7415aa62-467c-43af-94c8-d604f57ec1c3\" (UID: \"7415aa62-467c-43af-94c8-d604f57ec1c3\") "
Nov 02 20:18:38.418288 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:38.418184    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50dc76f4-aebd-41f4-9f23-799ddc8c70f4-home\" (UniqueName: \"kubernetes.io/empty-dir/50dc76f4-aebd-41f4-9f23-799ddc8c70f4-home\") pod \"50dc76f4-aebd-41f4-9f23-799ddc8c70f4\" (UID: \"50dc76f4-aebd-41f4-9f23-799ddc8c70f4\") "
Nov 02 20:18:38.418288 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:38.418220    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50dc76f4-aebd-41f4-9f23-799ddc8c70f4-kube-api-access-zk5g9\" (UniqueName: \"kubernetes.io/projected/50dc76f4-aebd-41f4-9f23-799ddc8c70f4-kube-api-access-zk5g9\") pod \"50dc76f4-aebd-41f4-9f23-799ddc8c70f4\" (UID: \"50dc76f4-aebd-41f4-9f23-799ddc8c70f4\") "
Nov 02 20:18:38.418288 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:38.418244    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7415aa62-467c-43af-94c8-d604f57ec1c3-home\" (UniqueName: \"kubernetes.io/empty-dir/7415aa62-467c-43af-94c8-d604f57ec1c3-home\") pod \"7415aa62-467c-43af-94c8-d604f57ec1c3\" (UID: \"7415aa62-467c-43af-94c8-d604f57ec1c3\") "
Nov 02 20:18:38.418607 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:38.418561    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7415aa62-467c-43af-94c8-d604f57ec1c3-model-cache" pod "7415aa62-467c-43af-94c8-d604f57ec1c3" (UID: "7415aa62-467c-43af-94c8-d604f57ec1c3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:18:38.419277 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:38.418754    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50dc76f4-aebd-41f4-9f23-799ddc8c70f4-model-cache" pod "50dc76f4-aebd-41f4-9f23-799ddc8c70f4" (UID: "50dc76f4-aebd-41f4-9f23-799ddc8c70f4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:18:38.419277 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:38.418978    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50dc76f4-aebd-41f4-9f23-799ddc8c70f4-home" pod "50dc76f4-aebd-41f4-9f23-799ddc8c70f4" (UID: "50dc76f4-aebd-41f4-9f23-799ddc8c70f4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:18:38.419541 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:38.419419    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7415aa62-467c-43af-94c8-d604f57ec1c3-tmp-dir" pod "7415aa62-467c-43af-94c8-d604f57ec1c3" (UID: "7415aa62-467c-43af-94c8-d604f57ec1c3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:18:38.419697 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:38.419669    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7415aa62-467c-43af-94c8-d604f57ec1c3-home" pod "7415aa62-467c-43af-94c8-d604f57ec1c3" (UID: "7415aa62-467c-43af-94c8-d604f57ec1c3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:18:38.421664 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:38.421628    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50dc76f4-aebd-41f4-9f23-799ddc8c70f4-kube-api-access-zk5g9" pod "50dc76f4-aebd-41f4-9f23-799ddc8c70f4" (UID: "50dc76f4-aebd-41f4-9f23-799ddc8c70f4"). InnerVolumeSpecName "kube-api-access-zk5g9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:18:38.421988 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:38.421964    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zk5g9\" (UniqueName: \"kubernetes.io/projected/50dc76f4-aebd-41f4-9f23-799ddc8c70f4-kube-api-access-zk5g9\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:38.422154 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:38.421996    4463 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7415aa62-467c-43af-94c8-d604f57ec1c3-home\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:38.422154 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:38.422028    4463 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50dc76f4-aebd-41f4-9f23-799ddc8c70f4-model-cache\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:38.422154 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:38.422044    4463 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7415aa62-467c-43af-94c8-d604f57ec1c3-model-cache\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:38.422154 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:38.422057    4463 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7415aa62-467c-43af-94c8-d604f57ec1c3-tmp-dir\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:38.422154 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:38.422072    4463 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50dc76f4-aebd-41f4-9f23-799ddc8c70f4-home\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:38.422426 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:38.422196    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7415aa62-467c-43af-94c8-d604f57ec1c3-kube-api-access-j9zgq" pod "7415aa62-467c-43af-94c8-d604f57ec1c3" (UID: "7415aa62-467c-43af-94c8-d604f57ec1c3"). InnerVolumeSpecName "kube-api-access-j9zgq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:18:38.422661 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:38.422643    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7415aa62-467c-43af-94c8-d604f57ec1c3-dshm" pod "7415aa62-467c-43af-94c8-d604f57ec1c3" (UID: "7415aa62-467c-43af-94c8-d604f57ec1c3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:18:38.422762 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:38.422737    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50dc76f4-aebd-41f4-9f23-799ddc8c70f4-tls-certs" pod "50dc76f4-aebd-41f4-9f23-799ddc8c70f4" (UID: "50dc76f4-aebd-41f4-9f23-799ddc8c70f4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:18:38.422846 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:38.422814    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7415aa62-467c-43af-94c8-d604f57ec1c3-tls-certs" pod "7415aa62-467c-43af-94c8-d604f57ec1c3" (UID: "7415aa62-467c-43af-94c8-d604f57ec1c3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:18:38.422889 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:38.422849    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50dc76f4-aebd-41f4-9f23-799ddc8c70f4-dshm" pod "50dc76f4-aebd-41f4-9f23-799ddc8c70f4" (UID: "50dc76f4-aebd-41f4-9f23-799ddc8c70f4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:18:38.523045 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:38.523001    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50dc76f4-aebd-41f4-9f23-799ddc8c70f4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50dc76f4-aebd-41f4-9f23-799ddc8c70f4-tmp-dir\") pod \"50dc76f4-aebd-41f4-9f23-799ddc8c70f4\" (UID: \"50dc76f4-aebd-41f4-9f23-799ddc8c70f4\") "
Nov 02 20:18:38.523557 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:38.523238    4463 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7415aa62-467c-43af-94c8-d604f57ec1c3-tls-certs\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:38.523557 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:38.523258    4463 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50dc76f4-aebd-41f4-9f23-799ddc8c70f4-tls-certs\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:38.523557 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:38.523272    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9zgq\" (UniqueName: \"kubernetes.io/projected/7415aa62-467c-43af-94c8-d604f57ec1c3-kube-api-access-j9zgq\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:38.523557 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:38.523271    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50dc76f4-aebd-41f4-9f23-799ddc8c70f4-tmp-dir" pod "50dc76f4-aebd-41f4-9f23-799ddc8c70f4" (UID: "50dc76f4-aebd-41f4-9f23-799ddc8c70f4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:18:38.523557 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:38.523284    4463 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50dc76f4-aebd-41f4-9f23-799ddc8c70f4-dshm\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:38.523557 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:38.523328    4463 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7415aa62-467c-43af-94c8-d604f57ec1c3-dshm\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:38.624396 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:38.624308    4463 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50dc76f4-aebd-41f4-9f23-799ddc8c70f4-tmp-dir\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:39.061119 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:39.061083    4463 generic.go:356] "Generic (PLEG): container finished" podID="7415aa62-467c-43af-94c8-d604f57ec1c3" containerID="070cf61c57bcab497189c1d0adb97ea4d06c65d76aef34acbf1bbe8a56fd457f" exitCode=0
Nov 02 20:18:39.061273 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:39.061134    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-26cfbe/distinct-w1-26cfbe-kserve-685985bfb-tkgn2"
Nov 02 20:18:39.061273 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:39.061163    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-26cfbe/distinct-w1-26cfbe-kserve-685985bfb-tkgn2" event={"ID":"7415aa62-467c-43af-94c8-d604f57ec1c3","Type":"ContainerDied","Data":"070cf61c57bcab497189c1d0adb97ea4d06c65d76aef34acbf1bbe8a56fd457f"}
Nov 02 20:18:39.061273 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:39.061219    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-26cfbe/distinct-w1-26cfbe-kserve-685985bfb-tkgn2" event={"ID":"7415aa62-467c-43af-94c8-d604f57ec1c3","Type":"ContainerDied","Data":"6c1c0ada948d819542f79b806cbec58476f63916e92721246c6a1cdc242cc775"}
Nov 02 20:18:39.061273 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:39.061239    4463 scope.go:122] "RemoveContainer" containerID="070cf61c57bcab497189c1d0adb97ea4d06c65d76aef34acbf1bbe8a56fd457f"
Nov 02 20:18:39.062534 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:39.062515    4463 generic.go:356] "Generic (PLEG): container finished" podID="50dc76f4-aebd-41f4-9f23-799ddc8c70f4" containerID="17ea621ec4532ca3d7f1a5ffd28fe026bc1d8b6f532e2139c3dd84cd5817f679" exitCode=0
Nov 02 20:18:39.062601 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:39.062554    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-26cfbe/simulator-w1-26cfbe-kserve-74c5578fbd-t79hq" event={"ID":"50dc76f4-aebd-41f4-9f23-799ddc8c70f4","Type":"ContainerDied","Data":"17ea621ec4532ca3d7f1a5ffd28fe026bc1d8b6f532e2139c3dd84cd5817f679"}
Nov 02 20:18:39.062601 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:39.062565    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-26cfbe/simulator-w1-26cfbe-kserve-74c5578fbd-t79hq"
Nov 02 20:18:39.062689 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:39.062575    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-26cfbe/simulator-w1-26cfbe-kserve-74c5578fbd-t79hq" event={"ID":"50dc76f4-aebd-41f4-9f23-799ddc8c70f4","Type":"ContainerDied","Data":"cf889241316dd46e6c7a9420fab5d6d958b2584551424f5fee3fd3fdc95dbd5e"}
Nov 02 20:18:39.070729 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:39.070714    4463 scope.go:122] "RemoveContainer" containerID="070cf61c57bcab497189c1d0adb97ea4d06c65d76aef34acbf1bbe8a56fd457f"
Nov 02 20:18:39.070947 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:18:39.070930    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"070cf61c57bcab497189c1d0adb97ea4d06c65d76aef34acbf1bbe8a56fd457f\": container with ID starting with 070cf61c57bcab497189c1d0adb97ea4d06c65d76aef34acbf1bbe8a56fd457f not found: ID does not exist" containerID="070cf61c57bcab497189c1d0adb97ea4d06c65d76aef34acbf1bbe8a56fd457f"
Nov 02 20:18:39.071001 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:39.070952    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"070cf61c57bcab497189c1d0adb97ea4d06c65d76aef34acbf1bbe8a56fd457f"} err="failed to get container status \"070cf61c57bcab497189c1d0adb97ea4d06c65d76aef34acbf1bbe8a56fd457f\": rpc error: code = NotFound desc = could not find container \"070cf61c57bcab497189c1d0adb97ea4d06c65d76aef34acbf1bbe8a56fd457f\": container with ID starting with 070cf61c57bcab497189c1d0adb97ea4d06c65d76aef34acbf1bbe8a56fd457f not found: ID does not exist"
Nov 02 20:18:39.071001 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:39.070969    4463 scope.go:122] "RemoveContainer" containerID="17ea621ec4532ca3d7f1a5ffd28fe026bc1d8b6f532e2139c3dd84cd5817f679"
Nov 02 20:18:39.079005 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:39.078983    4463 scope.go:122] "RemoveContainer" containerID="17ea621ec4532ca3d7f1a5ffd28fe026bc1d8b6f532e2139c3dd84cd5817f679"
Nov 02 20:18:39.082171 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:18:39.079603    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17ea621ec4532ca3d7f1a5ffd28fe026bc1d8b6f532e2139c3dd84cd5817f679\": container with ID starting with 17ea621ec4532ca3d7f1a5ffd28fe026bc1d8b6f532e2139c3dd84cd5817f679 not found: ID does not exist" containerID="17ea621ec4532ca3d7f1a5ffd28fe026bc1d8b6f532e2139c3dd84cd5817f679"
Nov 02 20:18:39.082171 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:39.079634    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17ea621ec4532ca3d7f1a5ffd28fe026bc1d8b6f532e2139c3dd84cd5817f679"} err="failed to get container status \"17ea621ec4532ca3d7f1a5ffd28fe026bc1d8b6f532e2139c3dd84cd5817f679\": rpc error: code = NotFound desc = could not find container \"17ea621ec4532ca3d7f1a5ffd28fe026bc1d8b6f532e2139c3dd84cd5817f679\": container with ID starting with 17ea621ec4532ca3d7f1a5ffd28fe026bc1d8b6f532e2139c3dd84cd5817f679 not found: ID does not exist"
Nov 02 20:18:39.225238 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:39.225204    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-26cfbe/distinct-w1-26cfbe-kserve-685985bfb-tkgn2"]
Nov 02 20:18:39.271317 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:39.271287    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-26cfbe/distinct-w1-26cfbe-kserve-685985bfb-tkgn2"]
Nov 02 20:18:39.329700 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:39.329623    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-26cfbe/simulator-w1-26cfbe-kserve-74c5578fbd-t79hq"]
Nov 02 20:18:39.333045 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:39.333002    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-26cfbe/simulator-w1-26cfbe-kserve-74c5578fbd-t79hq"]
Nov 02 20:18:39.583102 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:39.583006    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50dc76f4-aebd-41f4-9f23-799ddc8c70f4" path="/var/lib/kubelet/pods/50dc76f4-aebd-41f4-9f23-799ddc8c70f4/volumes"
Nov 02 20:18:39.583653 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:39.583633    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7415aa62-467c-43af-94c8-d604f57ec1c3" path="/var/lib/kubelet/pods/7415aa62-467c-43af-94c8-d604f57ec1c3/volumes"
Nov 02 20:18:42.554522 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:42.554486    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 20:18:53.521610 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:53.521575    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-26cfbe-76898cf9c6-rlrzb"]
Nov 02 20:18:53.522209 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:53.521764    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-26cfbe-76898cf9c6-rlrzb" podUID="91cbd659-c151-4a51-a9a9-236ddf601f8d" containerName="maas-api" containerID="cri-o://8dd82949eaf1ba814caed6432aa28aa0791d3ffa38521bb0f34fe1b89713b437" gracePeriod=30
Nov 02 20:18:53.760731 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:53.760704    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-26cfbe-76898cf9c6-rlrzb"
Nov 02 20:18:53.858271 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:53.858209    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91cbd659-c151-4a51-a9a9-236ddf601f8d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91cbd659-c151-4a51-a9a9-236ddf601f8d-maas-api-tls\") pod \"91cbd659-c151-4a51-a9a9-236ddf601f8d\" (UID: \"91cbd659-c151-4a51-a9a9-236ddf601f8d\") "
Nov 02 20:18:53.858271 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:53.858252    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91cbd659-c151-4a51-a9a9-236ddf601f8d-kube-api-access-dggw4\" (UniqueName: \"kubernetes.io/projected/91cbd659-c151-4a51-a9a9-236ddf601f8d-kube-api-access-dggw4\") pod \"91cbd659-c151-4a51-a9a9-236ddf601f8d\" (UID: \"91cbd659-c151-4a51-a9a9-236ddf601f8d\") "
Nov 02 20:18:53.858436 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:53.858330    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91cbd659-c151-4a51-a9a9-236ddf601f8d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/91cbd659-c151-4a51-a9a9-236ddf601f8d-metrics-certs\") pod \"91cbd659-c151-4a51-a9a9-236ddf601f8d\" (UID: \"91cbd659-c151-4a51-a9a9-236ddf601f8d\") "
Nov 02 20:18:53.858436 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:53.858365    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91cbd659-c151-4a51-a9a9-236ddf601f8d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91cbd659-c151-4a51-a9a9-236ddf601f8d-openshift-service-ca\") pod \"91cbd659-c151-4a51-a9a9-236ddf601f8d\" (UID: \"91cbd659-c151-4a51-a9a9-236ddf601f8d\") "
Nov 02 20:18:53.858754 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:53.858729    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91cbd659-c151-4a51-a9a9-236ddf601f8d-openshift-service-ca" pod "91cbd659-c151-4a51-a9a9-236ddf601f8d" (UID: "91cbd659-c151-4a51-a9a9-236ddf601f8d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:18:53.860440 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:53.860416    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91cbd659-c151-4a51-a9a9-236ddf601f8d-maas-api-tls" pod "91cbd659-c151-4a51-a9a9-236ddf601f8d" (UID: "91cbd659-c151-4a51-a9a9-236ddf601f8d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:18:53.860525 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:53.860436    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91cbd659-c151-4a51-a9a9-236ddf601f8d-metrics-certs" pod "91cbd659-c151-4a51-a9a9-236ddf601f8d" (UID: "91cbd659-c151-4a51-a9a9-236ddf601f8d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:18:53.860525 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:53.860439    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91cbd659-c151-4a51-a9a9-236ddf601f8d-kube-api-access-dggw4" pod "91cbd659-c151-4a51-a9a9-236ddf601f8d" (UID: "91cbd659-c151-4a51-a9a9-236ddf601f8d"). InnerVolumeSpecName "kube-api-access-dggw4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:18:53.959093 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:53.959062    4463 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91cbd659-c151-4a51-a9a9-236ddf601f8d-metrics-certs\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:53.959093 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:53.959089    4463 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91cbd659-c151-4a51-a9a9-236ddf601f8d-openshift-service-ca\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:53.959093 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:53.959100    4463 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91cbd659-c151-4a51-a9a9-236ddf601f8d-maas-api-tls\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:53.959308 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:53.959109    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dggw4\" (UniqueName: \"kubernetes.io/projected/91cbd659-c151-4a51-a9a9-236ddf601f8d-kube-api-access-dggw4\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:54.133066 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:54.132988    4463 generic.go:356] "Generic (PLEG): container finished" podID="91cbd659-c151-4a51-a9a9-236ddf601f8d" containerID="8dd82949eaf1ba814caed6432aa28aa0791d3ffa38521bb0f34fe1b89713b437" exitCode=0
Nov 02 20:18:54.133066 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:54.133056    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-26cfbe-76898cf9c6-rlrzb"
Nov 02 20:18:54.133236 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:54.133081    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-26cfbe-76898cf9c6-rlrzb" event={"ID":"91cbd659-c151-4a51-a9a9-236ddf601f8d","Type":"ContainerDied","Data":"8dd82949eaf1ba814caed6432aa28aa0791d3ffa38521bb0f34fe1b89713b437"}
Nov 02 20:18:54.133236 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:54.133167    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-26cfbe-76898cf9c6-rlrzb" event={"ID":"91cbd659-c151-4a51-a9a9-236ddf601f8d","Type":"ContainerDied","Data":"e4cff7acd3cb83eba1a3c2aa3c55237339d5467e59d2283b99ef361c26fe9501"}
Nov 02 20:18:54.133236 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:54.133198    4463 scope.go:122] "RemoveContainer" containerID="8dd82949eaf1ba814caed6432aa28aa0791d3ffa38521bb0f34fe1b89713b437"
Nov 02 20:18:54.141970 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:54.141950    4463 scope.go:122] "RemoveContainer" containerID="8dd82949eaf1ba814caed6432aa28aa0791d3ffa38521bb0f34fe1b89713b437"
Nov 02 20:18:54.142275 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:18:54.142251    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dd82949eaf1ba814caed6432aa28aa0791d3ffa38521bb0f34fe1b89713b437\": container with ID starting with 8dd82949eaf1ba814caed6432aa28aa0791d3ffa38521bb0f34fe1b89713b437 not found: ID does not exist" containerID="8dd82949eaf1ba814caed6432aa28aa0791d3ffa38521bb0f34fe1b89713b437"
Nov 02 20:18:54.142382 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:54.142284    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dd82949eaf1ba814caed6432aa28aa0791d3ffa38521bb0f34fe1b89713b437"} err="failed to get container status \"8dd82949eaf1ba814caed6432aa28aa0791d3ffa38521bb0f34fe1b89713b437\": rpc error: code = NotFound desc = could not find container \"8dd82949eaf1ba814caed6432aa28aa0791d3ffa38521bb0f34fe1b89713b437\": container with ID starting with 8dd82949eaf1ba814caed6432aa28aa0791d3ffa38521bb0f34fe1b89713b437 not found: ID does not exist"
Nov 02 20:18:54.283595 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:54.283567    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-26cfbe-76898cf9c6-rlrzb"]
Nov 02 20:18:54.289302 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:54.289280    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-26cfbe-76898cf9c6-rlrzb"]
Nov 02 20:18:55.584934 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:18:55.584899    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91cbd659-c151-4a51-a9a9-236ddf601f8d" path="/var/lib/kubelet/pods/91cbd659-c151-4a51-a9a9-236ddf601f8d/volumes"
Nov 02 20:19:01.029383 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.029350    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 20:19:01.418050 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.417958    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-p5rnp"]
Nov 02 20:19:01.418316 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418303    4463 cpu_manager.go:414] "containerMap: removing container" podUID="3e95fad5-268c-4d06-866e-ed10c87a78d0" containerName="main"
Nov 02 20:19:01.418368 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418318    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e95fad5-268c-4d06-866e-ed10c87a78d0" containerName="main"
Nov 02 20:19:01.418368 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418329    4463 cpu_manager.go:414] "containerMap: removing container" podUID="86b84714-1e20-47b0-9a7b-f289c2589ffc" containerName="payload-processing"
Nov 02 20:19:01.418368 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418334    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86b84714-1e20-47b0-9a7b-f289c2589ffc" containerName="payload-processing"
Nov 02 20:19:01.418368 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418348    4463 cpu_manager.go:414] "containerMap: removing container" podUID="db50c476-4694-4cd0-8fa0-cca68002faa8" containerName="payload-pre-processing"
Nov 02 20:19:01.418368 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418353    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db50c476-4694-4cd0-8fa0-cca68002faa8" containerName="payload-pre-processing"
Nov 02 20:19:01.418368 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418363    4463 cpu_manager.go:414] "containerMap: removing container" podUID="c4fc2355-c8a3-4c51-9302-f93ced5c343e" containerName="istio-proxy"
Nov 02 20:19:01.418368 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418368    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4fc2355-c8a3-4c51-9302-f93ced5c343e" containerName="istio-proxy"
Nov 02 20:19:01.418588 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418375    4463 cpu_manager.go:414] "containerMap: removing container" podUID="a053056d-0fa4-4ea9-8091-85979931330f" containerName="maas-api"
Nov 02 20:19:01.418588 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418380    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a053056d-0fa4-4ea9-8091-85979931330f" containerName="maas-api"
Nov 02 20:19:01.418588 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418390    4463 cpu_manager.go:414] "containerMap: removing container" podUID="ba35ca07-1e6a-4098-9d52-afa7a68c8805" containerName="payload-processing"
Nov 02 20:19:01.418588 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418394    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba35ca07-1e6a-4098-9d52-afa7a68c8805" containerName="payload-processing"
Nov 02 20:19:01.418588 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418400    4463 cpu_manager.go:414] "containerMap: removing container" podUID="33f548d7-2602-477f-a2b4-80196795d489" containerName="payload-processing"
Nov 02 20:19:01.418588 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418405    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33f548d7-2602-477f-a2b4-80196795d489" containerName="payload-processing"
Nov 02 20:19:01.418588 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418412    4463 cpu_manager.go:414] "containerMap: removing container" podUID="7f250661-9b39-4d06-af9c-6f067b82f8ca" containerName="payload-processing"
Nov 02 20:19:01.418588 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418416    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f250661-9b39-4d06-af9c-6f067b82f8ca" containerName="payload-processing"
Nov 02 20:19:01.418588 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418423    4463 cpu_manager.go:414] "containerMap: removing container" podUID="73818a40-7ba5-410c-aa9f-f6833bd95623" containerName="istio-proxy"
Nov 02 20:19:01.418588 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418427    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73818a40-7ba5-410c-aa9f-f6833bd95623" containerName="istio-proxy"
Nov 02 20:19:01.418588 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418433    4463 cpu_manager.go:414] "containerMap: removing container" podUID="d6bed081-d1c4-426f-9611-81894a7b3e57" containerName="payload-processing"
Nov 02 20:19:01.418588 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418438    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6bed081-d1c4-426f-9611-81894a7b3e57" containerName="payload-processing"
Nov 02 20:19:01.418588 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418445    4463 cpu_manager.go:414] "containerMap: removing container" podUID="b2c0f86f-3e9e-4996-8710-664c445e9113" containerName="istio-proxy"
Nov 02 20:19:01.418588 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418451    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2c0f86f-3e9e-4996-8710-664c445e9113" containerName="istio-proxy"
Nov 02 20:19:01.418588 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418457    4463 cpu_manager.go:414] "containerMap: removing container" podUID="20157f70-df1b-48c9-ab3b-48712bd899fc" containerName="maas-api"
Nov 02 20:19:01.418588 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418461    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20157f70-df1b-48c9-ab3b-48712bd899fc" containerName="maas-api"
Nov 02 20:19:01.418588 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418468    4463 cpu_manager.go:414] "containerMap: removing container" podUID="2a52fefb-fb79-4d25-9edf-719bd84cecdd" containerName="maas-api"
Nov 02 20:19:01.418588 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418472    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a52fefb-fb79-4d25-9edf-719bd84cecdd" containerName="maas-api"
Nov 02 20:19:01.418588 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418479    4463 cpu_manager.go:414] "containerMap: removing container" podUID="b2c0f86f-3e9e-4996-8710-664c445e9113" containerName="istio-proxy"
Nov 02 20:19:01.418588 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418483    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2c0f86f-3e9e-4996-8710-664c445e9113" containerName="istio-proxy"
Nov 02 20:19:01.418588 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418491    4463 cpu_manager.go:414] "containerMap: removing container" podUID="8709c9fb-f685-43fd-a3d5-9cb73cf8329d" containerName="istio-proxy"
Nov 02 20:19:01.418588 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418496    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8709c9fb-f685-43fd-a3d5-9cb73cf8329d" containerName="istio-proxy"
Nov 02 20:19:01.418588 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418502    4463 cpu_manager.go:414] "containerMap: removing container" podUID="b78a889a-2b5d-4ff0-a419-2eb7f7858579" containerName="maas-api"
Nov 02 20:19:01.418588 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418506    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b78a889a-2b5d-4ff0-a419-2eb7f7858579" containerName="maas-api"
Nov 02 20:19:01.418588 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418512    4463 cpu_manager.go:414] "containerMap: removing container" podUID="7ab3e969-44bd-4d38-a066-005a66561670" containerName="istio-proxy"
Nov 02 20:19:01.418588 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418517    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ab3e969-44bd-4d38-a066-005a66561670" containerName="istio-proxy"
Nov 02 20:19:01.418588 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418524    4463 cpu_manager.go:414] "containerMap: removing container" podUID="7ab3e969-44bd-4d38-a066-005a66561670" containerName="istio-proxy"
Nov 02 20:19:01.418588 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418528    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ab3e969-44bd-4d38-a066-005a66561670" containerName="istio-proxy"
Nov 02 20:19:01.418588 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418536    4463 cpu_manager.go:414] "containerMap: removing container" podUID="03ec9733-7e8d-49fa-94d6-6000b30c0e16" containerName="payload-pre-processing"
Nov 02 20:19:01.418588 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418541    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03ec9733-7e8d-49fa-94d6-6000b30c0e16" containerName="payload-pre-processing"
Nov 02 20:19:01.418588 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418549    4463 cpu_manager.go:414] "containerMap: removing container" podUID="b9f30411-f136-4711-9f5d-6f0ef7a24591" containerName="istio-proxy"
Nov 02 20:19:01.418588 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418553    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9f30411-f136-4711-9f5d-6f0ef7a24591" containerName="istio-proxy"
Nov 02 20:19:01.418588 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418558    4463 cpu_manager.go:414] "containerMap: removing container" podUID="9fbc861b-be1c-460c-833d-478346a9b840" containerName="payload-pre-processing"
Nov 02 20:19:01.418588 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418563    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fbc861b-be1c-460c-833d-478346a9b840" containerName="payload-pre-processing"
Nov 02 20:19:01.418588 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418569    4463 cpu_manager.go:414] "containerMap: removing container" podUID="e20408d3-f3a6-4ca9-8413-aafe209a1477" containerName="payload-processing"
Nov 02 20:19:01.418588 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418573    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e20408d3-f3a6-4ca9-8413-aafe209a1477" containerName="payload-processing"
Nov 02 20:19:01.418588 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418581    4463 cpu_manager.go:414] "containerMap: removing container" podUID="c2091abf-b755-49b3-86ad-b4bac6e2e4c5" containerName="payload-processing"
Nov 02 20:19:01.418588 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418586    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2091abf-b755-49b3-86ad-b4bac6e2e4c5" containerName="payload-processing"
Nov 02 20:19:01.418588 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418593    4463 cpu_manager.go:414] "containerMap: removing container" podUID="b4de3a3a-cbd8-4488-9d04-4ec03e4d53ce" containerName="maas-api"
Nov 02 20:19:01.418588 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418597    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4de3a3a-cbd8-4488-9d04-4ec03e4d53ce" containerName="maas-api"
Nov 02 20:19:01.418588 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418605    4463 cpu_manager.go:414] "containerMap: removing container" podUID="8709c9fb-f685-43fd-a3d5-9cb73cf8329d" containerName="istio-proxy"
Nov 02 20:19:01.419864 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418609    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8709c9fb-f685-43fd-a3d5-9cb73cf8329d" containerName="istio-proxy"
Nov 02 20:19:01.419864 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418616    4463 cpu_manager.go:414] "containerMap: removing container" podUID="7415aa62-467c-43af-94c8-d604f57ec1c3" containerName="main"
Nov 02 20:19:01.419864 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418621    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7415aa62-467c-43af-94c8-d604f57ec1c3" containerName="main"
Nov 02 20:19:01.419864 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418631    4463 cpu_manager.go:414] "containerMap: removing container" podUID="50dc76f4-aebd-41f4-9f23-799ddc8c70f4" containerName="main"
Nov 02 20:19:01.419864 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418636    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50dc76f4-aebd-41f4-9f23-799ddc8c70f4" containerName="main"
Nov 02 20:19:01.419864 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418642    4463 cpu_manager.go:414] "containerMap: removing container" podUID="058aa7d8-1404-494f-b25b-70e8b702455b" containerName="payload-pre-processing"
Nov 02 20:19:01.419864 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418647    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="058aa7d8-1404-494f-b25b-70e8b702455b" containerName="payload-pre-processing"
Nov 02 20:19:01.419864 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418653    4463 cpu_manager.go:414] "containerMap: removing container" podUID="8dd6362e-7dbd-489b-b6d8-0d9caab0fa69" containerName="main"
Nov 02 20:19:01.419864 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418657    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8dd6362e-7dbd-489b-b6d8-0d9caab0fa69" containerName="main"
Nov 02 20:19:01.419864 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418663    4463 cpu_manager.go:414] "containerMap: removing container" podUID="93875a90-855a-489f-8ea9-971e26ad15eb" containerName="payload-pre-processing"
Nov 02 20:19:01.419864 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418668    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93875a90-855a-489f-8ea9-971e26ad15eb" containerName="payload-pre-processing"
Nov 02 20:19:01.419864 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418675    4463 cpu_manager.go:414] "containerMap: removing container" podUID="7be32df1-5049-4f79-be23-a827f3f0a85f" containerName="main"
Nov 02 20:19:01.419864 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418679    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7be32df1-5049-4f79-be23-a827f3f0a85f" containerName="main"
Nov 02 20:19:01.419864 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418684    4463 cpu_manager.go:414] "containerMap: removing container" podUID="2ec406bc-f697-42e1-9a76-857c3a28d0d6" containerName="maas-api"
Nov 02 20:19:01.419864 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418688    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ec406bc-f697-42e1-9a76-857c3a28d0d6" containerName="maas-api"
Nov 02 20:19:01.419864 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418696    4463 cpu_manager.go:414] "containerMap: removing container" podUID="aa56df3b-fd52-46b8-a626-0dd7e827ab8c" containerName="maas-api"
Nov 02 20:19:01.419864 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418700    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa56df3b-fd52-46b8-a626-0dd7e827ab8c" containerName="maas-api"
Nov 02 20:19:01.419864 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418705    4463 cpu_manager.go:414] "containerMap: removing container" podUID="a33f6a91-4468-4ea7-a107-c569cc82a306" containerName="maas-api"
Nov 02 20:19:01.419864 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418710    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a33f6a91-4468-4ea7-a107-c569cc82a306" containerName="maas-api"
Nov 02 20:19:01.419864 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418715    4463 cpu_manager.go:414] "containerMap: removing container" podUID="dc539282-391f-4350-8c34-95084fe87e65" containerName="maas-api"
Nov 02 20:19:01.419864 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418720    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc539282-391f-4350-8c34-95084fe87e65" containerName="maas-api"
Nov 02 20:19:01.419864 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418726    4463 cpu_manager.go:414] "containerMap: removing container" podUID="30d2bf45-98f1-4394-957b-18c5b768f902" containerName="payload-pre-processing"
Nov 02 20:19:01.419864 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418730    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30d2bf45-98f1-4394-957b-18c5b768f902" containerName="payload-pre-processing"
Nov 02 20:19:01.419864 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418737    4463 cpu_manager.go:414] "containerMap: removing container" podUID="9f6ba8f5-7ed9-4807-8435-6794197fee7c" containerName="istio-proxy"
Nov 02 20:19:01.419864 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418741    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f6ba8f5-7ed9-4807-8435-6794197fee7c" containerName="istio-proxy"
Nov 02 20:19:01.419864 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418746    4463 cpu_manager.go:414] "containerMap: removing container" podUID="91cbd659-c151-4a51-a9a9-236ddf601f8d" containerName="maas-api"
Nov 02 20:19:01.419864 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418750    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91cbd659-c151-4a51-a9a9-236ddf601f8d" containerName="maas-api"
Nov 02 20:19:01.419864 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418757    4463 cpu_manager.go:414] "containerMap: removing container" podUID="f40668dc-8c36-4c34-ae69-c87d8c71e338" containerName="istio-proxy"
Nov 02 20:19:01.419864 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418761    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f40668dc-8c36-4c34-ae69-c87d8c71e338" containerName="istio-proxy"
Nov 02 20:19:01.419864 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418812    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f250661-9b39-4d06-af9c-6f067b82f8ca" containerName="payload-processing"
Nov 02 20:19:01.419864 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418820    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="73818a40-7ba5-410c-aa9f-f6833bd95623" containerName="istio-proxy"
Nov 02 20:19:01.419864 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418827    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6bed081-d1c4-426f-9611-81894a7b3e57" containerName="payload-processing"
Nov 02 20:19:01.419864 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418833    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2c0f86f-3e9e-4996-8710-664c445e9113" containerName="istio-proxy"
Nov 02 20:19:01.419864 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418839    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="20157f70-df1b-48c9-ab3b-48712bd899fc" containerName="maas-api"
Nov 02 20:19:01.419864 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418844    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a52fefb-fb79-4d25-9edf-719bd84cecdd" containerName="maas-api"
Nov 02 20:19:01.419864 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418851    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2c0f86f-3e9e-4996-8710-664c445e9113" containerName="istio-proxy"
Nov 02 20:19:01.419864 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418856    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="8709c9fb-f685-43fd-a3d5-9cb73cf8329d" containerName="istio-proxy"
Nov 02 20:19:01.419864 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418862    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="b78a889a-2b5d-4ff0-a419-2eb7f7858579" containerName="maas-api"
Nov 02 20:19:01.419864 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418867    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ab3e969-44bd-4d38-a066-005a66561670" containerName="istio-proxy"
Nov 02 20:19:01.419864 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418872    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ab3e969-44bd-4d38-a066-005a66561670" containerName="istio-proxy"
Nov 02 20:19:01.419864 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418879    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="03ec9733-7e8d-49fa-94d6-6000b30c0e16" containerName="payload-pre-processing"
Nov 02 20:19:01.419864 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418884    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9f30411-f136-4711-9f5d-6f0ef7a24591" containerName="istio-proxy"
Nov 02 20:19:01.419864 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418889    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fbc861b-be1c-460c-833d-478346a9b840" containerName="payload-pre-processing"
Nov 02 20:19:01.419864 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418894    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="e20408d3-f3a6-4ca9-8413-aafe209a1477" containerName="payload-processing"
Nov 02 20:19:01.419864 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418900    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2091abf-b755-49b3-86ad-b4bac6e2e4c5" containerName="payload-processing"
Nov 02 20:19:01.419864 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418906    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4de3a3a-cbd8-4488-9d04-4ec03e4d53ce" containerName="maas-api"
Nov 02 20:19:01.419864 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418911    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4fc2355-c8a3-4c51-9302-f93ced5c343e" containerName="istio-proxy"
Nov 02 20:19:01.419864 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418916    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="7415aa62-467c-43af-94c8-d604f57ec1c3" containerName="main"
Nov 02 20:19:01.419864 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418923    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="50dc76f4-aebd-41f4-9f23-799ddc8c70f4" containerName="main"
Nov 02 20:19:01.419864 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418928    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="058aa7d8-1404-494f-b25b-70e8b702455b" containerName="payload-pre-processing"
Nov 02 20:19:01.419864 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418933    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="8dd6362e-7dbd-489b-b6d8-0d9caab0fa69" containerName="main"
Nov 02 20:19:01.419864 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418937    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="93875a90-855a-489f-8ea9-971e26ad15eb" containerName="payload-pre-processing"
Nov 02 20:19:01.419864 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418942    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="7be32df1-5049-4f79-be23-a827f3f0a85f" containerName="main"
Nov 02 20:19:01.419864 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418947    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ec406bc-f697-42e1-9a76-857c3a28d0d6" containerName="maas-api"
Nov 02 20:19:01.419864 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418953    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa56df3b-fd52-46b8-a626-0dd7e827ab8c" containerName="maas-api"
Nov 02 20:19:01.419864 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418957    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="a33f6a91-4468-4ea7-a107-c569cc82a306" containerName="maas-api"
Nov 02 20:19:01.419864 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418961    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc539282-391f-4350-8c34-95084fe87e65" containerName="maas-api"
Nov 02 20:19:01.419864 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418966    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="30d2bf45-98f1-4394-957b-18c5b768f902" containerName="payload-pre-processing"
Nov 02 20:19:01.419864 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418971    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f6ba8f5-7ed9-4807-8435-6794197fee7c" containerName="istio-proxy"
Nov 02 20:19:01.419864 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418976    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="91cbd659-c151-4a51-a9a9-236ddf601f8d" containerName="maas-api"
Nov 02 20:19:01.419864 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418980    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="f40668dc-8c36-4c34-ae69-c87d8c71e338" containerName="istio-proxy"
Nov 02 20:19:01.419864 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418985    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e95fad5-268c-4d06-866e-ed10c87a78d0" containerName="main"
Nov 02 20:19:01.419864 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418989    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="86b84714-1e20-47b0-9a7b-f289c2589ffc" containerName="payload-processing"
Nov 02 20:19:01.419864 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418994    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="db50c476-4694-4cd0-8fa0-cca68002faa8" containerName="payload-pre-processing"
Nov 02 20:19:01.419864 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.418999    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4fc2355-c8a3-4c51-9302-f93ced5c343e" containerName="istio-proxy"
Nov 02 20:19:01.419864 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.419004    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="a053056d-0fa4-4ea9-8091-85979931330f" containerName="maas-api"
Nov 02 20:19:01.419864 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.419022    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba35ca07-1e6a-4098-9d52-afa7a68c8805" containerName="payload-processing"
Nov 02 20:19:01.419864 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.419027    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="33f548d7-2602-477f-a2b4-80196795d489" containerName="payload-processing"
Nov 02 20:19:01.423648 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.423632    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-p5rnp"
Nov 02 20:19:01.472171 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.472146    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 02 20:19:01.472171 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.472146    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-26cfbe-openshift-default-dockercfg-msctx\""
Nov 02 20:19:01.520035 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.519985    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-p5rnp"]
Nov 02 20:19:01.529174 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.529126    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dd9bc70-07e1-4955-9a45-eadff2469364-workload-socket\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-p5rnp\" (UID: \"4dd9bc70-07e1-4955-9a45-eadff2469364\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-p5rnp"
Nov 02 20:19:01.529304 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.529187    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dd9bc70-07e1-4955-9a45-eadff2469364-istio-podinfo\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-p5rnp\" (UID: \"4dd9bc70-07e1-4955-9a45-eadff2469364\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-p5rnp"
Nov 02 20:19:01.529304 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.529286    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dd9bc70-07e1-4955-9a45-eadff2469364-credential-socket\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-p5rnp\" (UID: \"4dd9bc70-07e1-4955-9a45-eadff2469364\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-p5rnp"
Nov 02 20:19:01.529504 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.529374    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dd9bc70-07e1-4955-9a45-eadff2469364-istio-token\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-p5rnp\" (UID: \"4dd9bc70-07e1-4955-9a45-eadff2469364\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-p5rnp"
Nov 02 20:19:01.529504 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.529443    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dd9bc70-07e1-4955-9a45-eadff2469364-istiod-ca-cert\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-p5rnp\" (UID: \"4dd9bc70-07e1-4955-9a45-eadff2469364\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-p5rnp"
Nov 02 20:19:01.529632 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.529537    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dd9bc70-07e1-4955-9a45-eadff2469364-istio-data\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-p5rnp\" (UID: \"4dd9bc70-07e1-4955-9a45-eadff2469364\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-p5rnp"
Nov 02 20:19:01.529632 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.529570    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87vg7\" (UniqueName: \"kubernetes.io/projected/4dd9bc70-07e1-4955-9a45-eadff2469364-kube-api-access-87vg7\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-p5rnp\" (UID: \"4dd9bc70-07e1-4955-9a45-eadff2469364\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-p5rnp"
Nov 02 20:19:01.529730 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.529661    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dd9bc70-07e1-4955-9a45-eadff2469364-istio-envoy\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-p5rnp\" (UID: \"4dd9bc70-07e1-4955-9a45-eadff2469364\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-p5rnp"
Nov 02 20:19:01.529797 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.529716    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dd9bc70-07e1-4955-9a45-eadff2469364-workload-certs\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-p5rnp\" (UID: \"4dd9bc70-07e1-4955-9a45-eadff2469364\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-p5rnp"
Nov 02 20:19:01.630472 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.630441    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dd9bc70-07e1-4955-9a45-eadff2469364-istio-token\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-p5rnp\" (UID: \"4dd9bc70-07e1-4955-9a45-eadff2469364\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-p5rnp"
Nov 02 20:19:01.630618 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.630493    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dd9bc70-07e1-4955-9a45-eadff2469364-istiod-ca-cert\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-p5rnp\" (UID: \"4dd9bc70-07e1-4955-9a45-eadff2469364\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-p5rnp"
Nov 02 20:19:01.630618 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.630526    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dd9bc70-07e1-4955-9a45-eadff2469364-istio-data\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-p5rnp\" (UID: \"4dd9bc70-07e1-4955-9a45-eadff2469364\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-p5rnp"
Nov 02 20:19:01.630618 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.630543    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87vg7\" (UniqueName: \"kubernetes.io/projected/4dd9bc70-07e1-4955-9a45-eadff2469364-kube-api-access-87vg7\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-p5rnp\" (UID: \"4dd9bc70-07e1-4955-9a45-eadff2469364\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-p5rnp"
Nov 02 20:19:01.630618 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.630558    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dd9bc70-07e1-4955-9a45-eadff2469364-istio-envoy\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-p5rnp\" (UID: \"4dd9bc70-07e1-4955-9a45-eadff2469364\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-p5rnp"
Nov 02 20:19:01.630618 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.630587    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dd9bc70-07e1-4955-9a45-eadff2469364-workload-certs\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-p5rnp\" (UID: \"4dd9bc70-07e1-4955-9a45-eadff2469364\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-p5rnp"
Nov 02 20:19:01.630882 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.630619    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dd9bc70-07e1-4955-9a45-eadff2469364-workload-socket\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-p5rnp\" (UID: \"4dd9bc70-07e1-4955-9a45-eadff2469364\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-p5rnp"
Nov 02 20:19:01.630882 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.630643    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dd9bc70-07e1-4955-9a45-eadff2469364-istio-podinfo\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-p5rnp\" (UID: \"4dd9bc70-07e1-4955-9a45-eadff2469364\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-p5rnp"
Nov 02 20:19:01.630882 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.630672    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dd9bc70-07e1-4955-9a45-eadff2469364-credential-socket\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-p5rnp\" (UID: \"4dd9bc70-07e1-4955-9a45-eadff2469364\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-p5rnp"
Nov 02 20:19:01.631066 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.630968    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dd9bc70-07e1-4955-9a45-eadff2469364-istio-data\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-p5rnp\" (UID: \"4dd9bc70-07e1-4955-9a45-eadff2469364\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-p5rnp"
Nov 02 20:19:01.631126 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.631059    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dd9bc70-07e1-4955-9a45-eadff2469364-workload-certs\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-p5rnp\" (UID: \"4dd9bc70-07e1-4955-9a45-eadff2469364\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-p5rnp"
Nov 02 20:19:01.631126 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.631094    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dd9bc70-07e1-4955-9a45-eadff2469364-credential-socket\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-p5rnp\" (UID: \"4dd9bc70-07e1-4955-9a45-eadff2469364\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-p5rnp"
Nov 02 20:19:01.631228 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.631156    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dd9bc70-07e1-4955-9a45-eadff2469364-workload-socket\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-p5rnp\" (UID: \"4dd9bc70-07e1-4955-9a45-eadff2469364\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-p5rnp"
Nov 02 20:19:01.631279 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.631260    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dd9bc70-07e1-4955-9a45-eadff2469364-istiod-ca-cert\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-p5rnp\" (UID: \"4dd9bc70-07e1-4955-9a45-eadff2469364\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-p5rnp"
Nov 02 20:19:01.633060 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.633039    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dd9bc70-07e1-4955-9a45-eadff2469364-istio-envoy\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-p5rnp\" (UID: \"4dd9bc70-07e1-4955-9a45-eadff2469364\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-p5rnp"
Nov 02 20:19:01.633168 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.633132    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dd9bc70-07e1-4955-9a45-eadff2469364-istio-podinfo\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-p5rnp\" (UID: \"4dd9bc70-07e1-4955-9a45-eadff2469364\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-p5rnp"
Nov 02 20:19:01.683437 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.683409    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dd9bc70-07e1-4955-9a45-eadff2469364-istio-token\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-p5rnp\" (UID: \"4dd9bc70-07e1-4955-9a45-eadff2469364\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-p5rnp"
Nov 02 20:19:01.683832 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.683813    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87vg7\" (UniqueName: \"kubernetes.io/projected/4dd9bc70-07e1-4955-9a45-eadff2469364-kube-api-access-87vg7\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-p5rnp\" (UID: \"4dd9bc70-07e1-4955-9a45-eadff2469364\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-p5rnp"
Nov 02 20:19:01.736004 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:01.735981    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-p5rnp"
Nov 02 20:19:02.200248 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:02.200165    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-p5rnp"]
Nov 02 20:19:02.202520 ip-10-0-133-9 kubenswrapper[4463]: W1102 20:19:02.202490    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dd9bc70_07e1_4955_9a45_eadff2469364.slice/crio-d8d1ec9ddebcf4216002205c1c0a12fbbc3011e6d175abeb2973a0c60195617f WatchSource:0}: Error finding container d8d1ec9ddebcf4216002205c1c0a12fbbc3011e6d175abeb2973a0c60195617f: Status 404 returned error can't find the container with id d8d1ec9ddebcf4216002205c1c0a12fbbc3011e6d175abeb2973a0c60195617f
Nov 02 20:19:02.204550 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:02.204517    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:19:02.204651 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:02.204592    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:19:02.204651 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:02.204642    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:19:02.494866 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:02.494774    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-p5rnp"]
Nov 02 20:19:03.173240 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:03.173198    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-p5rnp" event={"ID":"4dd9bc70-07e1-4955-9a45-eadff2469364","Type":"ContainerStarted","Data":"0b846195aa10d4349bc90f13723b04c711735a73cda2cf5f0f84f003d9b2ac5a"}
Nov 02 20:19:03.173422 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:03.173265    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-p5rnp" event={"ID":"4dd9bc70-07e1-4955-9a45-eadff2469364","Type":"ContainerStarted","Data":"d8d1ec9ddebcf4216002205c1c0a12fbbc3011e6d175abeb2973a0c60195617f"}
Nov 02 20:19:03.173422 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:03.173386    4463 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-p5rnp" secret="" err="secret \"e2e-worker-w1-26cfbe-openshift-default-dockercfg-msctx\" not found"
Nov 02 20:19:03.282559 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:03.282509    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-p5rnp" podStartSLOduration=2.2824955239999998 podStartE2EDuration="2.282495524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:19:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:19:03.280755519 +0000 UTC m=+3049.887789306" watchObservedRunningTime="2026-11-02 20:19:03.282495524 +0000 UTC m=+3049.889529292"
Nov 02 20:19:03.736738 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:03.736705    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-p5rnp"
Nov 02 20:19:03.738259 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:03.738225    4463 patch_prober.go:29] interesting pod/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-p5rnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body=
Nov 02 20:19:03.738385 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:03.738273    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-p5rnp" podUID="4dd9bc70-07e1-4955-9a45-eadff2469364" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused"
Nov 02 20:19:04.177489 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:04.177446    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-p5rnp" podUID="4dd9bc70-07e1-4955-9a45-eadff2469364" containerName="istio-proxy" containerID="cri-o://0b846195aa10d4349bc90f13723b04c711735a73cda2cf5f0f84f003d9b2ac5a" gracePeriod=30
Nov 02 20:19:04.294266 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:19:04.294232    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 20:19:09.418803 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:09.418778    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-p5rnp"
Nov 02 20:19:09.493872 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:09.493817    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4dd9bc70-07e1-4955-9a45-eadff2469364-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dd9bc70-07e1-4955-9a45-eadff2469364-credential-socket\") pod \"4dd9bc70-07e1-4955-9a45-eadff2469364\" (UID: \"4dd9bc70-07e1-4955-9a45-eadff2469364\") "
Nov 02 20:19:09.494001 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:09.493892    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4dd9bc70-07e1-4955-9a45-eadff2469364-kube-api-access-87vg7\" (UniqueName: \"kubernetes.io/projected/4dd9bc70-07e1-4955-9a45-eadff2469364-kube-api-access-87vg7\") pod \"4dd9bc70-07e1-4955-9a45-eadff2469364\" (UID: \"4dd9bc70-07e1-4955-9a45-eadff2469364\") "
Nov 02 20:19:09.494001 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:09.493936    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4dd9bc70-07e1-4955-9a45-eadff2469364-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dd9bc70-07e1-4955-9a45-eadff2469364-istio-envoy\") pod \"4dd9bc70-07e1-4955-9a45-eadff2469364\" (UID: \"4dd9bc70-07e1-4955-9a45-eadff2469364\") "
Nov 02 20:19:09.494001 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:09.493971    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4dd9bc70-07e1-4955-9a45-eadff2469364-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dd9bc70-07e1-4955-9a45-eadff2469364-workload-certs\") pod \"4dd9bc70-07e1-4955-9a45-eadff2469364\" (UID: \"4dd9bc70-07e1-4955-9a45-eadff2469364\") "
Nov 02 20:19:09.494193 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:09.494002    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4dd9bc70-07e1-4955-9a45-eadff2469364-istio-token\" (UniqueName: \"kubernetes.io/projected/4dd9bc70-07e1-4955-9a45-eadff2469364-istio-token\") pod \"4dd9bc70-07e1-4955-9a45-eadff2469364\" (UID: \"4dd9bc70-07e1-4955-9a45-eadff2469364\") "
Nov 02 20:19:09.494193 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:09.494045    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4dd9bc70-07e1-4955-9a45-eadff2469364-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dd9bc70-07e1-4955-9a45-eadff2469364-workload-socket\") pod \"4dd9bc70-07e1-4955-9a45-eadff2469364\" (UID: \"4dd9bc70-07e1-4955-9a45-eadff2469364\") "
Nov 02 20:19:09.494193 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:09.494076    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4dd9bc70-07e1-4955-9a45-eadff2469364-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dd9bc70-07e1-4955-9a45-eadff2469364-istio-data\") pod \"4dd9bc70-07e1-4955-9a45-eadff2469364\" (UID: \"4dd9bc70-07e1-4955-9a45-eadff2469364\") "
Nov 02 20:19:09.494193 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:09.494143    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4dd9bc70-07e1-4955-9a45-eadff2469364-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dd9bc70-07e1-4955-9a45-eadff2469364-istio-podinfo\") pod \"4dd9bc70-07e1-4955-9a45-eadff2469364\" (UID: \"4dd9bc70-07e1-4955-9a45-eadff2469364\") "
Nov 02 20:19:09.494193 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:09.494182    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4dd9bc70-07e1-4955-9a45-eadff2469364-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dd9bc70-07e1-4955-9a45-eadff2469364-istiod-ca-cert\") pod \"4dd9bc70-07e1-4955-9a45-eadff2469364\" (UID: \"4dd9bc70-07e1-4955-9a45-eadff2469364\") "
Nov 02 20:19:09.494457 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:09.494184    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dd9bc70-07e1-4955-9a45-eadff2469364-credential-socket" pod "4dd9bc70-07e1-4955-9a45-eadff2469364" (UID: "4dd9bc70-07e1-4955-9a45-eadff2469364"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:19:09.494457 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:09.494377    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dd9bc70-07e1-4955-9a45-eadff2469364-workload-certs" pod "4dd9bc70-07e1-4955-9a45-eadff2469364" (UID: "4dd9bc70-07e1-4955-9a45-eadff2469364"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:19:09.494457 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:09.494409    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dd9bc70-07e1-4955-9a45-eadff2469364-workload-socket" pod "4dd9bc70-07e1-4955-9a45-eadff2469364" (UID: "4dd9bc70-07e1-4955-9a45-eadff2469364"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:19:09.494609 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:09.494537    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dd9bc70-07e1-4955-9a45-eadff2469364-istiod-ca-cert" pod "4dd9bc70-07e1-4955-9a45-eadff2469364" (UID: "4dd9bc70-07e1-4955-9a45-eadff2469364"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:19:09.494609 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:09.494599    4463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dd9bc70-07e1-4955-9a45-eadff2469364-workload-certs\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:19:09.494716 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:09.494620    4463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dd9bc70-07e1-4955-9a45-eadff2469364-workload-socket\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:19:09.494716 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:09.494636    4463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dd9bc70-07e1-4955-9a45-eadff2469364-credential-socket\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:19:09.494858 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:09.494831    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dd9bc70-07e1-4955-9a45-eadff2469364-istio-data" pod "4dd9bc70-07e1-4955-9a45-eadff2469364" (UID: "4dd9bc70-07e1-4955-9a45-eadff2469364"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:19:09.496161 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:09.496138    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dd9bc70-07e1-4955-9a45-eadff2469364-kube-api-access-87vg7" pod "4dd9bc70-07e1-4955-9a45-eadff2469364" (UID: "4dd9bc70-07e1-4955-9a45-eadff2469364"). InnerVolumeSpecName "kube-api-access-87vg7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:19:09.496578 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:09.496557    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dd9bc70-07e1-4955-9a45-eadff2469364-istio-envoy" pod "4dd9bc70-07e1-4955-9a45-eadff2469364" (UID: "4dd9bc70-07e1-4955-9a45-eadff2469364"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:19:09.496718 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:09.496695    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dd9bc70-07e1-4955-9a45-eadff2469364-istio-token" pod "4dd9bc70-07e1-4955-9a45-eadff2469364" (UID: "4dd9bc70-07e1-4955-9a45-eadff2469364"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:19:09.496800 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:09.496747    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4dd9bc70-07e1-4955-9a45-eadff2469364-istio-podinfo" pod "4dd9bc70-07e1-4955-9a45-eadff2469364" (UID: "4dd9bc70-07e1-4955-9a45-eadff2469364"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:19:09.595582 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:09.595559    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87vg7\" (UniqueName: \"kubernetes.io/projected/4dd9bc70-07e1-4955-9a45-eadff2469364-kube-api-access-87vg7\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:19:09.595582 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:09.595580    4463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dd9bc70-07e1-4955-9a45-eadff2469364-istio-envoy\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:19:09.595754 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:09.595589    4463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dd9bc70-07e1-4955-9a45-eadff2469364-istio-token\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:19:09.595754 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:09.595599    4463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dd9bc70-07e1-4955-9a45-eadff2469364-istio-data\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:19:09.595754 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:09.595607    4463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dd9bc70-07e1-4955-9a45-eadff2469364-istio-podinfo\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:19:09.595754 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:09.595615    4463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dd9bc70-07e1-4955-9a45-eadff2469364-istiod-ca-cert\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:19:10.203860 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:10.203819    4463 generic.go:356] "Generic (PLEG): container finished" podID="4dd9bc70-07e1-4955-9a45-eadff2469364" containerID="0b846195aa10d4349bc90f13723b04c711735a73cda2cf5f0f84f003d9b2ac5a" exitCode=0
Nov 02 20:19:10.204066 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:10.203870    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-p5rnp"
Nov 02 20:19:10.204066 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:10.203904    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-p5rnp" event={"ID":"4dd9bc70-07e1-4955-9a45-eadff2469364","Type":"ContainerDied","Data":"0b846195aa10d4349bc90f13723b04c711735a73cda2cf5f0f84f003d9b2ac5a"}
Nov 02 20:19:10.204066 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:10.203958    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-p5rnp" event={"ID":"4dd9bc70-07e1-4955-9a45-eadff2469364","Type":"ContainerDied","Data":"d8d1ec9ddebcf4216002205c1c0a12fbbc3011e6d175abeb2973a0c60195617f"}
Nov 02 20:19:10.204066 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:10.203982    4463 scope.go:122] "RemoveContainer" containerID="0b846195aa10d4349bc90f13723b04c711735a73cda2cf5f0f84f003d9b2ac5a"
Nov 02 20:19:10.214546 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:10.214528    4463 scope.go:122] "RemoveContainer" containerID="0b846195aa10d4349bc90f13723b04c711735a73cda2cf5f0f84f003d9b2ac5a"
Nov 02 20:19:10.214813 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:19:10.214789    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b846195aa10d4349bc90f13723b04c711735a73cda2cf5f0f84f003d9b2ac5a\": container with ID starting with 0b846195aa10d4349bc90f13723b04c711735a73cda2cf5f0f84f003d9b2ac5a not found: ID does not exist" containerID="0b846195aa10d4349bc90f13723b04c711735a73cda2cf5f0f84f003d9b2ac5a"
Nov 02 20:19:10.214905 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:10.214817    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b846195aa10d4349bc90f13723b04c711735a73cda2cf5f0f84f003d9b2ac5a"} err="failed to get container status \"0b846195aa10d4349bc90f13723b04c711735a73cda2cf5f0f84f003d9b2ac5a\": rpc error: code = NotFound desc = could not find container \"0b846195aa10d4349bc90f13723b04c711735a73cda2cf5f0f84f003d9b2ac5a\": container with ID starting with 0b846195aa10d4349bc90f13723b04c711735a73cda2cf5f0f84f003d9b2ac5a not found: ID does not exist"
Nov 02 20:19:10.402302 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:10.402276    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-p5rnp"]
Nov 02 20:19:10.448836 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:10.448809    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-p5rnp"]
Nov 02 20:19:11.583540 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:11.583510    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4dd9bc70-07e1-4955-9a45-eadff2469364" path="/var/lib/kubelet/pods/4dd9bc70-07e1-4955-9a45-eadff2469364/volumes"
Nov 02 20:19:31.614072 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:31.614032    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fm6qk"]
Nov 02 20:19:31.614590 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:31.614395    4463 cpu_manager.go:414] "containerMap: removing container" podUID="c4fc2355-c8a3-4c51-9302-f93ced5c343e" containerName="istio-proxy"
Nov 02 20:19:31.614590 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:31.614407    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4fc2355-c8a3-4c51-9302-f93ced5c343e" containerName="istio-proxy"
Nov 02 20:19:31.614590 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:31.614430    4463 cpu_manager.go:414] "containerMap: removing container" podUID="4dd9bc70-07e1-4955-9a45-eadff2469364" containerName="istio-proxy"
Nov 02 20:19:31.614590 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:31.614436    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4dd9bc70-07e1-4955-9a45-eadff2469364" containerName="istio-proxy"
Nov 02 20:19:31.614590 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:31.614498    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="4dd9bc70-07e1-4955-9a45-eadff2469364" containerName="istio-proxy"
Nov 02 20:19:31.614590 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:31.614511    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="8709c9fb-f685-43fd-a3d5-9cb73cf8329d" containerName="istio-proxy"
Nov 02 20:19:31.617414 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:31.617397    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fm6qk"
Nov 02 20:19:31.665972 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:31.665943    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 02 20:19:31.694292 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:31.694265    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae15f3c4-bd17-4252-837c-4483162c3542-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-fm6qk\" (UID: \"ae15f3c4-bd17-4252-837c-4483162c3542\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fm6qk"
Nov 02 20:19:31.694420 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:31.694297    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae15f3c4-bd17-4252-837c-4483162c3542-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-fm6qk\" (UID: \"ae15f3c4-bd17-4252-837c-4483162c3542\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fm6qk"
Nov 02 20:19:31.694420 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:31.694322    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae15f3c4-bd17-4252-837c-4483162c3542-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-fm6qk\" (UID: \"ae15f3c4-bd17-4252-837c-4483162c3542\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fm6qk"
Nov 02 20:19:31.694497 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:31.694429    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcjqg\" (UniqueName: \"kubernetes.io/projected/ae15f3c4-bd17-4252-837c-4483162c3542-kube-api-access-tcjqg\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-fm6qk\" (UID: \"ae15f3c4-bd17-4252-837c-4483162c3542\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fm6qk"
Nov 02 20:19:31.694497 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:31.694461    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae15f3c4-bd17-4252-837c-4483162c3542-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-fm6qk\" (UID: \"ae15f3c4-bd17-4252-837c-4483162c3542\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fm6qk"
Nov 02 20:19:31.694497 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:31.694490    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae15f3c4-bd17-4252-837c-4483162c3542-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-fm6qk\" (UID: \"ae15f3c4-bd17-4252-837c-4483162c3542\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fm6qk"
Nov 02 20:19:31.694597 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:31.694579    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae15f3c4-bd17-4252-837c-4483162c3542-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-fm6qk\" (UID: \"ae15f3c4-bd17-4252-837c-4483162c3542\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fm6qk"
Nov 02 20:19:31.694631 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:31.694618    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae15f3c4-bd17-4252-837c-4483162c3542-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-fm6qk\" (UID: \"ae15f3c4-bd17-4252-837c-4483162c3542\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fm6qk"
Nov 02 20:19:31.694665 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:31.694656    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae15f3c4-bd17-4252-837c-4483162c3542-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-fm6qk\" (UID: \"ae15f3c4-bd17-4252-837c-4483162c3542\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fm6qk"
Nov 02 20:19:31.713708 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:31.713677    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fm6qk"]
Nov 02 20:19:31.721119 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:31.721093    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-qmwl4\""
Nov 02 20:19:31.795775 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:31.795737    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae15f3c4-bd17-4252-837c-4483162c3542-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-fm6qk\" (UID: \"ae15f3c4-bd17-4252-837c-4483162c3542\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fm6qk"
Nov 02 20:19:31.795775 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:31.795790    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcjqg\" (UniqueName: \"kubernetes.io/projected/ae15f3c4-bd17-4252-837c-4483162c3542-kube-api-access-tcjqg\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-fm6qk\" (UID: \"ae15f3c4-bd17-4252-837c-4483162c3542\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fm6qk"
Nov 02 20:19:31.796083 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:31.795810    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae15f3c4-bd17-4252-837c-4483162c3542-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-fm6qk\" (UID: \"ae15f3c4-bd17-4252-837c-4483162c3542\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fm6qk"
Nov 02 20:19:31.796083 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:31.795844    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae15f3c4-bd17-4252-837c-4483162c3542-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-fm6qk\" (UID: \"ae15f3c4-bd17-4252-837c-4483162c3542\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fm6qk"
Nov 02 20:19:31.796083 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:31.795861    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae15f3c4-bd17-4252-837c-4483162c3542-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-fm6qk\" (UID: \"ae15f3c4-bd17-4252-837c-4483162c3542\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fm6qk"
Nov 02 20:19:31.796083 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:31.795883    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae15f3c4-bd17-4252-837c-4483162c3542-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-fm6qk\" (UID: \"ae15f3c4-bd17-4252-837c-4483162c3542\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fm6qk"
Nov 02 20:19:31.796083 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:31.795940    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae15f3c4-bd17-4252-837c-4483162c3542-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-fm6qk\" (UID: \"ae15f3c4-bd17-4252-837c-4483162c3542\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fm6qk"
Nov 02 20:19:31.796083 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:31.795973    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae15f3c4-bd17-4252-837c-4483162c3542-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-fm6qk\" (UID: \"ae15f3c4-bd17-4252-837c-4483162c3542\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fm6qk"
Nov 02 20:19:31.796083 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:31.796007    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae15f3c4-bd17-4252-837c-4483162c3542-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-fm6qk\" (UID: \"ae15f3c4-bd17-4252-837c-4483162c3542\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fm6qk"
Nov 02 20:19:31.796479 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:31.796358    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae15f3c4-bd17-4252-837c-4483162c3542-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-fm6qk\" (UID: \"ae15f3c4-bd17-4252-837c-4483162c3542\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fm6qk"
Nov 02 20:19:31.796541 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:31.796520    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae15f3c4-bd17-4252-837c-4483162c3542-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-fm6qk\" (UID: \"ae15f3c4-bd17-4252-837c-4483162c3542\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fm6qk"
Nov 02 20:19:31.796603 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:31.796559    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae15f3c4-bd17-4252-837c-4483162c3542-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-fm6qk\" (UID: \"ae15f3c4-bd17-4252-837c-4483162c3542\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fm6qk"
Nov 02 20:19:31.796644 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:31.796621    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae15f3c4-bd17-4252-837c-4483162c3542-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-fm6qk\" (UID: \"ae15f3c4-bd17-4252-837c-4483162c3542\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fm6qk"
Nov 02 20:19:31.796920 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:31.796891    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae15f3c4-bd17-4252-837c-4483162c3542-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-fm6qk\" (UID: \"ae15f3c4-bd17-4252-837c-4483162c3542\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fm6qk"
Nov 02 20:19:31.798505 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:31.798485    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae15f3c4-bd17-4252-837c-4483162c3542-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-fm6qk\" (UID: \"ae15f3c4-bd17-4252-837c-4483162c3542\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fm6qk"
Nov 02 20:19:31.798748 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:31.798730    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae15f3c4-bd17-4252-837c-4483162c3542-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-fm6qk\" (UID: \"ae15f3c4-bd17-4252-837c-4483162c3542\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fm6qk"
Nov 02 20:19:31.848318 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:31.848290    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae15f3c4-bd17-4252-837c-4483162c3542-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-fm6qk\" (UID: \"ae15f3c4-bd17-4252-837c-4483162c3542\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fm6qk"
Nov 02 20:19:31.848594 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:31.848570    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcjqg\" (UniqueName: \"kubernetes.io/projected/ae15f3c4-bd17-4252-837c-4483162c3542-kube-api-access-tcjqg\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-fm6qk\" (UID: \"ae15f3c4-bd17-4252-837c-4483162c3542\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fm6qk"
Nov 02 20:19:31.928487 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:31.928462    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fm6qk"
Nov 02 20:19:32.338959 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:32.338931    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fm6qk"]
Nov 02 20:19:32.341443 ip-10-0-133-9 kubenswrapper[4463]: W1102 20:19:32.341413    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae15f3c4_bd17_4252_837c_4483162c3542.slice/crio-badc2e216c7252ec83117c7ab6eed4079a21dbeea602a02cac68a4358a035670 WatchSource:0}: Error finding container badc2e216c7252ec83117c7ab6eed4079a21dbeea602a02cac68a4358a035670: Status 404 returned error can't find the container with id badc2e216c7252ec83117c7ab6eed4079a21dbeea602a02cac68a4358a035670
Nov 02 20:19:32.343462 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:32.343431    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:19:32.343523 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:32.343495    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:19:32.343559 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:32.343522    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:19:33.303671 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:33.303629    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fm6qk" event={"ID":"ae15f3c4-bd17-4252-837c-4483162c3542","Type":"ContainerStarted","Data":"fb73542cf74948b2d7b31688a9706ae910a15f841e0736b03901ad4d80eb7db0"}
Nov 02 20:19:33.304105 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:33.303698    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fm6qk" event={"ID":"ae15f3c4-bd17-4252-837c-4483162c3542","Type":"ContainerStarted","Data":"badc2e216c7252ec83117c7ab6eed4079a21dbeea602a02cac68a4358a035670"}
Nov 02 20:19:33.929494 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:33.929463    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fm6qk"
Nov 02 20:19:33.930921 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:33.930892    4463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-fm6qk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body=
Nov 02 20:19:33.931080 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:33.930946    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fm6qk" podUID="ae15f3c4-bd17-4252-837c-4483162c3542" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused"
Nov 02 20:19:34.298733 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:19:34.298650    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 20:19:34.929353 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:34.929320    4463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-fm6qk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body=
Nov 02 20:19:34.929805 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:34.929375    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fm6qk" podUID="ae15f3c4-bd17-4252-837c-4483162c3542" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused"
Nov 02 20:19:35.929428 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:35.929396    4463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-fm6qk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body=
Nov 02 20:19:35.929822 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:35.929458    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fm6qk" podUID="ae15f3c4-bd17-4252-837c-4483162c3542" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused"
Nov 02 20:19:36.928952 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:36.928911    4463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-fm6qk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body=
Nov 02 20:19:36.929153 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:36.928981    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fm6qk" podUID="ae15f3c4-bd17-4252-837c-4483162c3542" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused"
Nov 02 20:19:37.929068 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:37.929003    4463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-fm6qk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body=
Nov 02 20:19:37.929489 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:37.929092    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fm6qk" podUID="ae15f3c4-bd17-4252-837c-4483162c3542" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused"
Nov 02 20:19:38.928840 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:38.928806    4463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-fm6qk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body=
Nov 02 20:19:38.930883 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:38.928861    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fm6qk" podUID="ae15f3c4-bd17-4252-837c-4483162c3542" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused"
Nov 02 20:19:40.929662 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:40.929608    4463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-fm6qk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 20:19:40.930143 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:40.929668    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fm6qk" podUID="ae15f3c4-bd17-4252-837c-4483162c3542" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:19:41.971178 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:41.971131    4463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-fm6qk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 20:19:41.971699 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:41.971186    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fm6qk" podUID="ae15f3c4-bd17-4252-837c-4483162c3542" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:19:42.659462 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:42.659431    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fm6qk"
Nov 02 20:19:42.660165 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:42.660137    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fm6qk"
Nov 02 20:19:42.660798 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:42.660777    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fm6qk"
Nov 02 20:19:42.861410 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:19:42.861351    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fm6qk" podStartSLOduration=11.861334513 podStartE2EDuration="11.861334513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:19:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:19:33.518994819 +0000 UTC m=+3080.126028587" watchObservedRunningTime="2026-11-02 20:19:42.861334513 +0000 UTC m=+3089.468368282"
Nov 02 20:20:04.293413 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:20:04.293372    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 20:20:34.292445 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:20:34.292405    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 20:20:41.212553 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:20:41.212513    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 20:21:02.256067 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:02.256032    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-dw7ld"]
Nov 02 20:21:02.264809 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:02.264763    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-dw7ld"
Nov 02 20:21:02.311136 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:02.311092    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a85b7c93-b1b1-4bc3-b840-0de225a8ec58-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-dw7ld\" (UID: \"a85b7c93-b1b1-4bc3-b840-0de225a8ec58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-dw7ld"
Nov 02 20:21:02.311319 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:02.311149    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a85b7c93-b1b1-4bc3-b840-0de225a8ec58-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-dw7ld\" (UID: \"a85b7c93-b1b1-4bc3-b840-0de225a8ec58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-dw7ld"
Nov 02 20:21:02.311319 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:02.311241    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a85b7c93-b1b1-4bc3-b840-0de225a8ec58-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-dw7ld\" (UID: \"a85b7c93-b1b1-4bc3-b840-0de225a8ec58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-dw7ld"
Nov 02 20:21:02.311319 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:02.311275    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a85b7c93-b1b1-4bc3-b840-0de225a8ec58-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-dw7ld\" (UID: \"a85b7c93-b1b1-4bc3-b840-0de225a8ec58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-dw7ld"
Nov 02 20:21:02.311461 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:02.311368    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a85b7c93-b1b1-4bc3-b840-0de225a8ec58-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-dw7ld\" (UID: \"a85b7c93-b1b1-4bc3-b840-0de225a8ec58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-dw7ld"
Nov 02 20:21:02.311461 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:02.311419    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a85b7c93-b1b1-4bc3-b840-0de225a8ec58-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-dw7ld\" (UID: \"a85b7c93-b1b1-4bc3-b840-0de225a8ec58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-dw7ld"
Nov 02 20:21:02.311461 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:02.311440    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a85b7c93-b1b1-4bc3-b840-0de225a8ec58-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-dw7ld\" (UID: \"a85b7c93-b1b1-4bc3-b840-0de225a8ec58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-dw7ld"
Nov 02 20:21:02.311580 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:02.311485    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a85b7c93-b1b1-4bc3-b840-0de225a8ec58-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-dw7ld\" (UID: \"a85b7c93-b1b1-4bc3-b840-0de225a8ec58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-dw7ld"
Nov 02 20:21:02.311580 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:02.311548    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vwxn\" (UniqueName: \"kubernetes.io/projected/a85b7c93-b1b1-4bc3-b840-0de225a8ec58-kube-api-access-8vwxn\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-dw7ld\" (UID: \"a85b7c93-b1b1-4bc3-b840-0de225a8ec58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-dw7ld"
Nov 02 20:21:02.412682 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:02.412649    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a85b7c93-b1b1-4bc3-b840-0de225a8ec58-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-dw7ld\" (UID: \"a85b7c93-b1b1-4bc3-b840-0de225a8ec58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-dw7ld"
Nov 02 20:21:02.412854 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:02.412689    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a85b7c93-b1b1-4bc3-b840-0de225a8ec58-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-dw7ld\" (UID: \"a85b7c93-b1b1-4bc3-b840-0de225a8ec58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-dw7ld"
Nov 02 20:21:02.412854 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:02.412718    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a85b7c93-b1b1-4bc3-b840-0de225a8ec58-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-dw7ld\" (UID: \"a85b7c93-b1b1-4bc3-b840-0de225a8ec58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-dw7ld"
Nov 02 20:21:02.412854 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:02.412788    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a85b7c93-b1b1-4bc3-b840-0de225a8ec58-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-dw7ld\" (UID: \"a85b7c93-b1b1-4bc3-b840-0de225a8ec58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-dw7ld"
Nov 02 20:21:02.412854 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:02.412843    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a85b7c93-b1b1-4bc3-b840-0de225a8ec58-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-dw7ld\" (UID: \"a85b7c93-b1b1-4bc3-b840-0de225a8ec58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-dw7ld"
Nov 02 20:21:02.413112 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:02.412867    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a85b7c93-b1b1-4bc3-b840-0de225a8ec58-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-dw7ld\" (UID: \"a85b7c93-b1b1-4bc3-b840-0de225a8ec58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-dw7ld"
Nov 02 20:21:02.413112 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:02.412892    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a85b7c93-b1b1-4bc3-b840-0de225a8ec58-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-dw7ld\" (UID: \"a85b7c93-b1b1-4bc3-b840-0de225a8ec58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-dw7ld"
Nov 02 20:21:02.413112 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:02.412919    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vwxn\" (UniqueName: \"kubernetes.io/projected/a85b7c93-b1b1-4bc3-b840-0de225a8ec58-kube-api-access-8vwxn\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-dw7ld\" (UID: \"a85b7c93-b1b1-4bc3-b840-0de225a8ec58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-dw7ld"
Nov 02 20:21:02.413112 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:02.412952    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a85b7c93-b1b1-4bc3-b840-0de225a8ec58-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-dw7ld\" (UID: \"a85b7c93-b1b1-4bc3-b840-0de225a8ec58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-dw7ld"
Nov 02 20:21:02.413335 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:02.413117    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a85b7c93-b1b1-4bc3-b840-0de225a8ec58-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-dw7ld\" (UID: \"a85b7c93-b1b1-4bc3-b840-0de225a8ec58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-dw7ld"
Nov 02 20:21:02.413335 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:02.413288    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a85b7c93-b1b1-4bc3-b840-0de225a8ec58-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-dw7ld\" (UID: \"a85b7c93-b1b1-4bc3-b840-0de225a8ec58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-dw7ld"
Nov 02 20:21:02.413335 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:02.413321    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a85b7c93-b1b1-4bc3-b840-0de225a8ec58-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-dw7ld\" (UID: \"a85b7c93-b1b1-4bc3-b840-0de225a8ec58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-dw7ld"
Nov 02 20:21:02.413553 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:02.413364    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a85b7c93-b1b1-4bc3-b840-0de225a8ec58-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-dw7ld\" (UID: \"a85b7c93-b1b1-4bc3-b840-0de225a8ec58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-dw7ld"
Nov 02 20:21:02.413732 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:02.413706    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a85b7c93-b1b1-4bc3-b840-0de225a8ec58-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-dw7ld\" (UID: \"a85b7c93-b1b1-4bc3-b840-0de225a8ec58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-dw7ld"
Nov 02 20:21:02.415137 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:02.415116    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a85b7c93-b1b1-4bc3-b840-0de225a8ec58-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-dw7ld\" (UID: \"a85b7c93-b1b1-4bc3-b840-0de225a8ec58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-dw7ld"
Nov 02 20:21:02.415258 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:02.415242    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a85b7c93-b1b1-4bc3-b840-0de225a8ec58-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-dw7ld\" (UID: \"a85b7c93-b1b1-4bc3-b840-0de225a8ec58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-dw7ld"
Nov 02 20:21:02.449234 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:02.449207    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-dw7ld"]
Nov 02 20:21:02.465358 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:02.465336    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a85b7c93-b1b1-4bc3-b840-0de225a8ec58-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-dw7ld\" (UID: \"a85b7c93-b1b1-4bc3-b840-0de225a8ec58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-dw7ld"
Nov 02 20:21:02.465358 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:02.465347    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vwxn\" (UniqueName: \"kubernetes.io/projected/a85b7c93-b1b1-4bc3-b840-0de225a8ec58-kube-api-access-8vwxn\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-dw7ld\" (UID: \"a85b7c93-b1b1-4bc3-b840-0de225a8ec58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-dw7ld"
Nov 02 20:21:02.533033 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:02.532933    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 20:21:02.577098 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:02.577071    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-dw7ld"
Nov 02 20:21:03.032099 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:03.032048    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-dw7ld"]
Nov 02 20:21:03.034312 ip-10-0-133-9 kubenswrapper[4463]: W1102 20:21:03.034287    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda85b7c93_b1b1_4bc3_b840_0de225a8ec58.slice/crio-b1fecd09028ca87571ef0efe5c76da401240a795380241577928f55fedc7877c WatchSource:0}: Error finding container b1fecd09028ca87571ef0efe5c76da401240a795380241577928f55fedc7877c: Status 404 returned error can't find the container with id b1fecd09028ca87571ef0efe5c76da401240a795380241577928f55fedc7877c
Nov 02 20:21:03.036486 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:03.036456    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:21:03.036555 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:03.036517    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:21:03.036555 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:03.036544    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:21:03.699246 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:03.699206    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-dw7ld" event={"ID":"a85b7c93-b1b1-4bc3-b840-0de225a8ec58","Type":"ContainerStarted","Data":"60146ffb72afc16683284278d1ac77dca6a95d2add91bfd692faec6205ab4798"}
Nov 02 20:21:03.699634 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:03.699271    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-dw7ld" event={"ID":"a85b7c93-b1b1-4bc3-b840-0de225a8ec58","Type":"ContainerStarted","Data":"b1fecd09028ca87571ef0efe5c76da401240a795380241577928f55fedc7877c"}
Nov 02 20:21:03.861036 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:03.859538    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-dw7ld" podStartSLOduration=1.859516813 podStartE2EDuration="1.859516813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:21:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:21:03.806329994 +0000 UTC m=+3170.413363763" watchObservedRunningTime="2026-11-02 20:21:03.859516813 +0000 UTC m=+3170.466550583"
Nov 02 20:21:04.299981 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:21:04.299936    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 20:21:04.578235 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:04.578134    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-dw7ld"
Nov 02 20:21:04.579692 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:04.579662    4463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-dw7ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Nov 02 20:21:04.579822 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:04.579722    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-dw7ld" podUID="a85b7c93-b1b1-4bc3-b840-0de225a8ec58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Nov 02 20:21:05.577578 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:05.577539    4463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-dw7ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Nov 02 20:21:05.578088 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:05.577606    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-dw7ld" podUID="a85b7c93-b1b1-4bc3-b840-0de225a8ec58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Nov 02 20:21:06.577470 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:06.577429    4463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-dw7ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Nov 02 20:21:06.577683 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:06.577480    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-dw7ld" podUID="a85b7c93-b1b1-4bc3-b840-0de225a8ec58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Nov 02 20:21:07.580514 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:07.580477    4463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-dw7ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Nov 02 20:21:07.581004 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:07.580561    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-dw7ld" podUID="a85b7c93-b1b1-4bc3-b840-0de225a8ec58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Nov 02 20:21:08.579662 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:08.579627    4463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-dw7ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Nov 02 20:21:08.579969 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:08.579932    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-dw7ld" podUID="a85b7c93-b1b1-4bc3-b840-0de225a8ec58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Nov 02 20:21:09.577569 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:09.577530    4463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-dw7ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Nov 02 20:21:09.578194 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:09.578158    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-dw7ld" podUID="a85b7c93-b1b1-4bc3-b840-0de225a8ec58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Nov 02 20:21:11.578952 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:11.578864    4463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-dw7ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 20:21:11.578952 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:11.578945    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-dw7ld" podUID="a85b7c93-b1b1-4bc3-b840-0de225a8ec58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:21:12.621158 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:12.621124    4463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-dw7ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 20:21:12.621651 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:12.621180    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-dw7ld" podUID="a85b7c93-b1b1-4bc3-b840-0de225a8ec58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:21:12.718605 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:12.718576    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 20:21:13.259609 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:13.259576    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-dw7ld"
Nov 02 20:21:13.259951 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:13.259929    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-dw7ld"
Nov 02 20:21:13.260624 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:13.260604    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-dw7ld"
Nov 02 20:21:17.366625 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:17.366590    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-khwqc"]
Nov 02 20:21:17.369750 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:17.369734    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-khwqc"
Nov 02 20:21:17.459028 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:17.458988    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-khwqc"]
Nov 02 20:21:17.480304 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:17.480273    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bec6162-6f9a-4871-8185-cc3592f4075c-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-khwqc\" (UID: \"2bec6162-6f9a-4871-8185-cc3592f4075c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-khwqc"
Nov 02 20:21:17.480304 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:17.480308    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bec6162-6f9a-4871-8185-cc3592f4075c-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-khwqc\" (UID: \"2bec6162-6f9a-4871-8185-cc3592f4075c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-khwqc"
Nov 02 20:21:17.480571 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:17.480348    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bec6162-6f9a-4871-8185-cc3592f4075c-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-khwqc\" (UID: \"2bec6162-6f9a-4871-8185-cc3592f4075c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-khwqc"
Nov 02 20:21:17.480571 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:17.480457    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bec6162-6f9a-4871-8185-cc3592f4075c-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-khwqc\" (UID: \"2bec6162-6f9a-4871-8185-cc3592f4075c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-khwqc"
Nov 02 20:21:17.480571 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:17.480478    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6n74\" (UniqueName: \"kubernetes.io/projected/2bec6162-6f9a-4871-8185-cc3592f4075c-kube-api-access-w6n74\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-khwqc\" (UID: \"2bec6162-6f9a-4871-8185-cc3592f4075c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-khwqc"
Nov 02 20:21:17.480571 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:17.480503    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bec6162-6f9a-4871-8185-cc3592f4075c-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-khwqc\" (UID: \"2bec6162-6f9a-4871-8185-cc3592f4075c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-khwqc"
Nov 02 20:21:17.480571 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:17.480520    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bec6162-6f9a-4871-8185-cc3592f4075c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-khwqc\" (UID: \"2bec6162-6f9a-4871-8185-cc3592f4075c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-khwqc"
Nov 02 20:21:17.480878 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:17.480649    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bec6162-6f9a-4871-8185-cc3592f4075c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-khwqc\" (UID: \"2bec6162-6f9a-4871-8185-cc3592f4075c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-khwqc"
Nov 02 20:21:17.480878 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:17.480687    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bec6162-6f9a-4871-8185-cc3592f4075c-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-khwqc\" (UID: \"2bec6162-6f9a-4871-8185-cc3592f4075c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-khwqc"
Nov 02 20:21:17.581206 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:17.581171    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bec6162-6f9a-4871-8185-cc3592f4075c-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-khwqc\" (UID: \"2bec6162-6f9a-4871-8185-cc3592f4075c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-khwqc"
Nov 02 20:21:17.581206 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:17.581209    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6n74\" (UniqueName: \"kubernetes.io/projected/2bec6162-6f9a-4871-8185-cc3592f4075c-kube-api-access-w6n74\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-khwqc\" (UID: \"2bec6162-6f9a-4871-8185-cc3592f4075c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-khwqc"
Nov 02 20:21:17.581467 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:17.581240    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bec6162-6f9a-4871-8185-cc3592f4075c-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-khwqc\" (UID: \"2bec6162-6f9a-4871-8185-cc3592f4075c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-khwqc"
Nov 02 20:21:17.581467 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:17.581266    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bec6162-6f9a-4871-8185-cc3592f4075c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-khwqc\" (UID: \"2bec6162-6f9a-4871-8185-cc3592f4075c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-khwqc"
Nov 02 20:21:17.581467 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:17.581442    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bec6162-6f9a-4871-8185-cc3592f4075c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-khwqc\" (UID: \"2bec6162-6f9a-4871-8185-cc3592f4075c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-khwqc"
Nov 02 20:21:17.581633 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:17.581501    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bec6162-6f9a-4871-8185-cc3592f4075c-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-khwqc\" (UID: \"2bec6162-6f9a-4871-8185-cc3592f4075c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-khwqc"
Nov 02 20:21:17.581633 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:17.581538    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bec6162-6f9a-4871-8185-cc3592f4075c-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-khwqc\" (UID: \"2bec6162-6f9a-4871-8185-cc3592f4075c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-khwqc"
Nov 02 20:21:17.581633 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:17.581579    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bec6162-6f9a-4871-8185-cc3592f4075c-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-khwqc\" (UID: \"2bec6162-6f9a-4871-8185-cc3592f4075c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-khwqc"
Nov 02 20:21:17.581633 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:17.581620    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bec6162-6f9a-4871-8185-cc3592f4075c-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-khwqc\" (UID: \"2bec6162-6f9a-4871-8185-cc3592f4075c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-khwqc"
Nov 02 20:21:17.581839 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:17.581690    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bec6162-6f9a-4871-8185-cc3592f4075c-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-khwqc\" (UID: \"2bec6162-6f9a-4871-8185-cc3592f4075c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-khwqc"
Nov 02 20:21:17.581977 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:17.581952    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bec6162-6f9a-4871-8185-cc3592f4075c-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-khwqc\" (UID: \"2bec6162-6f9a-4871-8185-cc3592f4075c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-khwqc"
Nov 02 20:21:17.582067 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:17.582047    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bec6162-6f9a-4871-8185-cc3592f4075c-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-khwqc\" (UID: \"2bec6162-6f9a-4871-8185-cc3592f4075c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-khwqc"
Nov 02 20:21:17.582194 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:17.582174    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bec6162-6f9a-4871-8185-cc3592f4075c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-khwqc\" (UID: \"2bec6162-6f9a-4871-8185-cc3592f4075c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-khwqc"
Nov 02 20:21:17.582322 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:17.582302    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bec6162-6f9a-4871-8185-cc3592f4075c-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-khwqc\" (UID: \"2bec6162-6f9a-4871-8185-cc3592f4075c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-khwqc"
Nov 02 20:21:17.584529 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:17.584499    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bec6162-6f9a-4871-8185-cc3592f4075c-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-khwqc\" (UID: \"2bec6162-6f9a-4871-8185-cc3592f4075c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-khwqc"
Nov 02 20:21:17.584656 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:17.584553    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bec6162-6f9a-4871-8185-cc3592f4075c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-khwqc\" (UID: \"2bec6162-6f9a-4871-8185-cc3592f4075c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-khwqc"
Nov 02 20:21:17.633926 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:17.633853    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bec6162-6f9a-4871-8185-cc3592f4075c-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-khwqc\" (UID: \"2bec6162-6f9a-4871-8185-cc3592f4075c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-khwqc"
Nov 02 20:21:17.634094 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:17.633950    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6n74\" (UniqueName: \"kubernetes.io/projected/2bec6162-6f9a-4871-8185-cc3592f4075c-kube-api-access-w6n74\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-khwqc\" (UID: \"2bec6162-6f9a-4871-8185-cc3592f4075c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-khwqc"
Nov 02 20:21:17.682739 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:17.682710    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-khwqc"
Nov 02 20:21:18.098796 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:18.098768    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-khwqc"]
Nov 02 20:21:18.101063 ip-10-0-133-9 kubenswrapper[4463]: W1102 20:21:18.101036    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bec6162_6f9a_4871_8185_cc3592f4075c.slice/crio-d75995eefd882d257979f600950aedc89a5984eb7e072c1abe7c203c20ee8923 WatchSource:0}: Error finding container d75995eefd882d257979f600950aedc89a5984eb7e072c1abe7c203c20ee8923: Status 404 returned error can't find the container with id d75995eefd882d257979f600950aedc89a5984eb7e072c1abe7c203c20ee8923
Nov 02 20:21:18.103228 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:18.103199    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:21:18.103298 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:18.103266    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:21:18.103340 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:18.103296    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:21:18.779645 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:18.779606    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-khwqc" event={"ID":"2bec6162-6f9a-4871-8185-cc3592f4075c","Type":"ContainerStarted","Data":"1a623d19c94466b55e806cc76503cfa4709563e244a9d22f3ae4ecb5198cd1d4"}
Nov 02 20:21:18.780062 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:18.779662    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-khwqc" event={"ID":"2bec6162-6f9a-4871-8185-cc3592f4075c","Type":"ContainerStarted","Data":"d75995eefd882d257979f600950aedc89a5984eb7e072c1abe7c203c20ee8923"}
Nov 02 20:21:18.889769 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:18.889705    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-khwqc" podStartSLOduration=1.889692334 podStartE2EDuration="1.889692334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:21:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:21:18.887813625 +0000 UTC m=+3185.494847415" watchObservedRunningTime="2026-11-02 20:21:18.889692334 +0000 UTC m=+3185.496726102"
Nov 02 20:21:19.683661 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:19.683627    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-khwqc"
Nov 02 20:21:19.685023 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:19.684990    4463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-khwqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Nov 02 20:21:19.685105 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:19.685059    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-khwqc" podUID="2bec6162-6f9a-4871-8185-cc3592f4075c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Nov 02 20:21:20.682993 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:20.682957    4463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-khwqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Nov 02 20:21:20.683408 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:20.683043    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-khwqc" podUID="2bec6162-6f9a-4871-8185-cc3592f4075c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Nov 02 20:21:21.683917 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:21.683878    4463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-khwqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Nov 02 20:21:21.684336 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:21.683943    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-khwqc" podUID="2bec6162-6f9a-4871-8185-cc3592f4075c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Nov 02 20:21:22.683534 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:22.683496    4463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-khwqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Nov 02 20:21:22.683719 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:22.683554    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-khwqc" podUID="2bec6162-6f9a-4871-8185-cc3592f4075c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Nov 02 20:21:23.683231 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:23.683195    4463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-khwqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Nov 02 20:21:23.683643 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:23.683246    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-khwqc" podUID="2bec6162-6f9a-4871-8185-cc3592f4075c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Nov 02 20:21:24.683850 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:24.683810    4463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-khwqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Nov 02 20:21:24.684262 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:24.683866    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-khwqc" podUID="2bec6162-6f9a-4871-8185-cc3592f4075c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Nov 02 20:21:26.270952 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:26.270872    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-khwqc"
Nov 02 20:21:26.271427 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:26.271333    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-khwqc"
Nov 02 20:21:26.272273 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:26.272246    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-khwqc"
Nov 02 20:21:34.294162 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:21:34.294122    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 20:21:37.358504 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:37.358467    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 20:21:52.210346 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:21:52.210306    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 20:22:04.293612 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:22:04.293575    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 20:22:24.958311 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:22:24.958273    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 20:22:27.314148 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:22:27.314108    4463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-khwqc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 20:22:27.314763 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:22:27.314730    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-khwqc" podUID="2bec6162-6f9a-4871-8185-cc3592f4075c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:22:34.296720 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:22:34.296672    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 20:22:35.636681 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:22:35.636647    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 20:22:44.313161 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:22:44.313119    4463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-dw7ld container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 20:22:44.313761 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:22:44.313725    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-dw7ld" podUID="a85b7c93-b1b1-4bc3-b840-0de225a8ec58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:22:47.344109 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:22:47.344075    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 20:23:04.292943 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:23:04.292901    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 20:23:10.134992 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:23:10.134953    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 20:23:19.118777 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:23:19.118740    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 20:23:34.294270 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:23:34.294236    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 20:23:38.229996 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:23:38.229961    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 20:23:50.142777 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:23:50.142739    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 20:24:04.293784 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:24:04.293749    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 20:24:10.363595 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:24:10.363556    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 20:24:29.156247 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:24:29.156162    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 20:24:34.293597 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:24:34.293560    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 20:24:42.220357 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:24:42.220305    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 20:25:04.293762 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:25:04.293715    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 20:25:14.810079 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:25:14.810045    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 20:25:34.293238 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:25:34.293197    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 20:25:36.551989 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:25:36.551956    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 20:25:47.118547 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:25:47.118504    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 20:26:04.293112 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:26:04.293072    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 20:26:07.880566 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:07.880523    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-9ff720-openshift-default-58fd5fdb7cc6sn5"]
Nov 02 20:26:07.884009 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:07.883992    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-9ff720-openshift-default-58fd5fdb7cc6sn5"
Nov 02 20:26:07.934588 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:07.934549    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-9ff720-openshift-default-dockercfg-tv96t\""
Nov 02 20:26:07.966467 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:07.966423    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2da1ff8e-3060-42a6-896e-75e07469e4b5-istio-data\") pod \"e2e-worker-single-main-9ff720-openshift-default-58fd5fdb7cc6sn5\" (UID: \"2da1ff8e-3060-42a6-896e-75e07469e4b5\") " pod="openshift-ingress/e2e-worker-single-main-9ff720-openshift-default-58fd5fdb7cc6sn5"
Nov 02 20:26:07.966664 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:07.966480    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2da1ff8e-3060-42a6-896e-75e07469e4b5-istiod-ca-cert\") pod \"e2e-worker-single-main-9ff720-openshift-default-58fd5fdb7cc6sn5\" (UID: \"2da1ff8e-3060-42a6-896e-75e07469e4b5\") " pod="openshift-ingress/e2e-worker-single-main-9ff720-openshift-default-58fd5fdb7cc6sn5"
Nov 02 20:26:07.966664 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:07.966543    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2da1ff8e-3060-42a6-896e-75e07469e4b5-istio-podinfo\") pod \"e2e-worker-single-main-9ff720-openshift-default-58fd5fdb7cc6sn5\" (UID: \"2da1ff8e-3060-42a6-896e-75e07469e4b5\") " pod="openshift-ingress/e2e-worker-single-main-9ff720-openshift-default-58fd5fdb7cc6sn5"
Nov 02 20:26:07.966664 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:07.966575    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2da1ff8e-3060-42a6-896e-75e07469e4b5-workload-socket\") pod \"e2e-worker-single-main-9ff720-openshift-default-58fd5fdb7cc6sn5\" (UID: \"2da1ff8e-3060-42a6-896e-75e07469e4b5\") " pod="openshift-ingress/e2e-worker-single-main-9ff720-openshift-default-58fd5fdb7cc6sn5"
Nov 02 20:26:07.966823 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:07.966667    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2da1ff8e-3060-42a6-896e-75e07469e4b5-istio-token\") pod \"e2e-worker-single-main-9ff720-openshift-default-58fd5fdb7cc6sn5\" (UID: \"2da1ff8e-3060-42a6-896e-75e07469e4b5\") " pod="openshift-ingress/e2e-worker-single-main-9ff720-openshift-default-58fd5fdb7cc6sn5"
Nov 02 20:26:07.966823 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:07.966712    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2da1ff8e-3060-42a6-896e-75e07469e4b5-credential-socket\") pod \"e2e-worker-single-main-9ff720-openshift-default-58fd5fdb7cc6sn5\" (UID: \"2da1ff8e-3060-42a6-896e-75e07469e4b5\") " pod="openshift-ingress/e2e-worker-single-main-9ff720-openshift-default-58fd5fdb7cc6sn5"
Nov 02 20:26:07.966911 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:07.966831    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2da1ff8e-3060-42a6-896e-75e07469e4b5-istio-envoy\") pod \"e2e-worker-single-main-9ff720-openshift-default-58fd5fdb7cc6sn5\" (UID: \"2da1ff8e-3060-42a6-896e-75e07469e4b5\") " pod="openshift-ingress/e2e-worker-single-main-9ff720-openshift-default-58fd5fdb7cc6sn5"
Nov 02 20:26:07.966911 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:07.966863    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrsm4\" (UniqueName: \"kubernetes.io/projected/2da1ff8e-3060-42a6-896e-75e07469e4b5-kube-api-access-qrsm4\") pod \"e2e-worker-single-main-9ff720-openshift-default-58fd5fdb7cc6sn5\" (UID: \"2da1ff8e-3060-42a6-896e-75e07469e4b5\") " pod="openshift-ingress/e2e-worker-single-main-9ff720-openshift-default-58fd5fdb7cc6sn5"
Nov 02 20:26:07.966911 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:07.966892    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2da1ff8e-3060-42a6-896e-75e07469e4b5-workload-certs\") pod \"e2e-worker-single-main-9ff720-openshift-default-58fd5fdb7cc6sn5\" (UID: \"2da1ff8e-3060-42a6-896e-75e07469e4b5\") " pod="openshift-ingress/e2e-worker-single-main-9ff720-openshift-default-58fd5fdb7cc6sn5"
Nov 02 20:26:07.982870 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:07.982838    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-9ff720-openshift-default-58fd5fdb7cc6sn5"]
Nov 02 20:26:08.068524 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:08.068484    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2da1ff8e-3060-42a6-896e-75e07469e4b5-istio-token\") pod \"e2e-worker-single-main-9ff720-openshift-default-58fd5fdb7cc6sn5\" (UID: \"2da1ff8e-3060-42a6-896e-75e07469e4b5\") " pod="openshift-ingress/e2e-worker-single-main-9ff720-openshift-default-58fd5fdb7cc6sn5"
Nov 02 20:26:08.068695 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:08.068541    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2da1ff8e-3060-42a6-896e-75e07469e4b5-credential-socket\") pod \"e2e-worker-single-main-9ff720-openshift-default-58fd5fdb7cc6sn5\" (UID: \"2da1ff8e-3060-42a6-896e-75e07469e4b5\") " pod="openshift-ingress/e2e-worker-single-main-9ff720-openshift-default-58fd5fdb7cc6sn5"
Nov 02 20:26:08.068695 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:08.068686    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2da1ff8e-3060-42a6-896e-75e07469e4b5-istio-envoy\") pod \"e2e-worker-single-main-9ff720-openshift-default-58fd5fdb7cc6sn5\" (UID: \"2da1ff8e-3060-42a6-896e-75e07469e4b5\") " pod="openshift-ingress/e2e-worker-single-main-9ff720-openshift-default-58fd5fdb7cc6sn5"
Nov 02 20:26:08.068790 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:08.068715    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrsm4\" (UniqueName: \"kubernetes.io/projected/2da1ff8e-3060-42a6-896e-75e07469e4b5-kube-api-access-qrsm4\") pod \"e2e-worker-single-main-9ff720-openshift-default-58fd5fdb7cc6sn5\" (UID: \"2da1ff8e-3060-42a6-896e-75e07469e4b5\") " pod="openshift-ingress/e2e-worker-single-main-9ff720-openshift-default-58fd5fdb7cc6sn5"
Nov 02 20:26:08.068790 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:08.068746    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2da1ff8e-3060-42a6-896e-75e07469e4b5-workload-certs\") pod \"e2e-worker-single-main-9ff720-openshift-default-58fd5fdb7cc6sn5\" (UID: \"2da1ff8e-3060-42a6-896e-75e07469e4b5\") " pod="openshift-ingress/e2e-worker-single-main-9ff720-openshift-default-58fd5fdb7cc6sn5"
Nov 02 20:26:08.068880 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:08.068862    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2da1ff8e-3060-42a6-896e-75e07469e4b5-istio-data\") pod \"e2e-worker-single-main-9ff720-openshift-default-58fd5fdb7cc6sn5\" (UID: \"2da1ff8e-3060-42a6-896e-75e07469e4b5\") " pod="openshift-ingress/e2e-worker-single-main-9ff720-openshift-default-58fd5fdb7cc6sn5"
Nov 02 20:26:08.068939 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:08.068894    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2da1ff8e-3060-42a6-896e-75e07469e4b5-istiod-ca-cert\") pod \"e2e-worker-single-main-9ff720-openshift-default-58fd5fdb7cc6sn5\" (UID: \"2da1ff8e-3060-42a6-896e-75e07469e4b5\") " pod="openshift-ingress/e2e-worker-single-main-9ff720-openshift-default-58fd5fdb7cc6sn5"
Nov 02 20:26:08.068939 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:08.068920    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2da1ff8e-3060-42a6-896e-75e07469e4b5-istio-podinfo\") pod \"e2e-worker-single-main-9ff720-openshift-default-58fd5fdb7cc6sn5\" (UID: \"2da1ff8e-3060-42a6-896e-75e07469e4b5\") " pod="openshift-ingress/e2e-worker-single-main-9ff720-openshift-default-58fd5fdb7cc6sn5"
Nov 02 20:26:08.069084 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:08.068945    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2da1ff8e-3060-42a6-896e-75e07469e4b5-workload-socket\") pod \"e2e-worker-single-main-9ff720-openshift-default-58fd5fdb7cc6sn5\" (UID: \"2da1ff8e-3060-42a6-896e-75e07469e4b5\") " pod="openshift-ingress/e2e-worker-single-main-9ff720-openshift-default-58fd5fdb7cc6sn5"
Nov 02 20:26:08.069084 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:08.069062    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2da1ff8e-3060-42a6-896e-75e07469e4b5-credential-socket\") pod \"e2e-worker-single-main-9ff720-openshift-default-58fd5fdb7cc6sn5\" (UID: \"2da1ff8e-3060-42a6-896e-75e07469e4b5\") " pod="openshift-ingress/e2e-worker-single-main-9ff720-openshift-default-58fd5fdb7cc6sn5"
Nov 02 20:26:08.069212 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:08.069194    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2da1ff8e-3060-42a6-896e-75e07469e4b5-workload-certs\") pod \"e2e-worker-single-main-9ff720-openshift-default-58fd5fdb7cc6sn5\" (UID: \"2da1ff8e-3060-42a6-896e-75e07469e4b5\") " pod="openshift-ingress/e2e-worker-single-main-9ff720-openshift-default-58fd5fdb7cc6sn5"
Nov 02 20:26:08.069559 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:08.069533    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2da1ff8e-3060-42a6-896e-75e07469e4b5-workload-socket\") pod \"e2e-worker-single-main-9ff720-openshift-default-58fd5fdb7cc6sn5\" (UID: \"2da1ff8e-3060-42a6-896e-75e07469e4b5\") " pod="openshift-ingress/e2e-worker-single-main-9ff720-openshift-default-58fd5fdb7cc6sn5"
Nov 02 20:26:08.069693 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:08.069584    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2da1ff8e-3060-42a6-896e-75e07469e4b5-istio-data\") pod \"e2e-worker-single-main-9ff720-openshift-default-58fd5fdb7cc6sn5\" (UID: \"2da1ff8e-3060-42a6-896e-75e07469e4b5\") " pod="openshift-ingress/e2e-worker-single-main-9ff720-openshift-default-58fd5fdb7cc6sn5"
Nov 02 20:26:08.069693 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:08.069669    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2da1ff8e-3060-42a6-896e-75e07469e4b5-istiod-ca-cert\") pod \"e2e-worker-single-main-9ff720-openshift-default-58fd5fdb7cc6sn5\" (UID: \"2da1ff8e-3060-42a6-896e-75e07469e4b5\") " pod="openshift-ingress/e2e-worker-single-main-9ff720-openshift-default-58fd5fdb7cc6sn5"
Nov 02 20:26:08.070995 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:08.070972    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2da1ff8e-3060-42a6-896e-75e07469e4b5-istio-envoy\") pod \"e2e-worker-single-main-9ff720-openshift-default-58fd5fdb7cc6sn5\" (UID: \"2da1ff8e-3060-42a6-896e-75e07469e4b5\") " pod="openshift-ingress/e2e-worker-single-main-9ff720-openshift-default-58fd5fdb7cc6sn5"
Nov 02 20:26:08.071318 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:08.071299    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2da1ff8e-3060-42a6-896e-75e07469e4b5-istio-podinfo\") pod \"e2e-worker-single-main-9ff720-openshift-default-58fd5fdb7cc6sn5\" (UID: \"2da1ff8e-3060-42a6-896e-75e07469e4b5\") " pod="openshift-ingress/e2e-worker-single-main-9ff720-openshift-default-58fd5fdb7cc6sn5"
Nov 02 20:26:08.121857 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:08.121823    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2da1ff8e-3060-42a6-896e-75e07469e4b5-istio-token\") pod \"e2e-worker-single-main-9ff720-openshift-default-58fd5fdb7cc6sn5\" (UID: \"2da1ff8e-3060-42a6-896e-75e07469e4b5\") " pod="openshift-ingress/e2e-worker-single-main-9ff720-openshift-default-58fd5fdb7cc6sn5"
Nov 02 20:26:08.122063 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:08.122039    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrsm4\" (UniqueName: \"kubernetes.io/projected/2da1ff8e-3060-42a6-896e-75e07469e4b5-kube-api-access-qrsm4\") pod \"e2e-worker-single-main-9ff720-openshift-default-58fd5fdb7cc6sn5\" (UID: \"2da1ff8e-3060-42a6-896e-75e07469e4b5\") " pod="openshift-ingress/e2e-worker-single-main-9ff720-openshift-default-58fd5fdb7cc6sn5"
Nov 02 20:26:08.196567 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:08.196537    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-9ff720-openshift-default-58fd5fdb7cc6sn5"
Nov 02 20:26:08.613042 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:08.612920    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-9ff720-openshift-default-58fd5fdb7cc6sn5"]
Nov 02 20:26:08.615796 ip-10-0-133-9 kubenswrapper[4463]: W1102 20:26:08.615767    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2da1ff8e_3060_42a6_896e_75e07469e4b5.slice/crio-39a88e9989e45fd067f3f619bba92a0ccfac57750320773d1741f3e117c1ed0c WatchSource:0}: Error finding container 39a88e9989e45fd067f3f619bba92a0ccfac57750320773d1741f3e117c1ed0c: Status 404 returned error can't find the container with id 39a88e9989e45fd067f3f619bba92a0ccfac57750320773d1741f3e117c1ed0c
Nov 02 20:26:08.617962 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:08.617928    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:26:08.618047 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:08.617993    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:26:08.618047 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:08.618035    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:26:09.180275 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:09.180242    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-9ff720-openshift-default-58fd5fdb7cc6sn5" event={"ID":"2da1ff8e-3060-42a6-896e-75e07469e4b5","Type":"ContainerStarted","Data":"27dcbc5c37d6a56e748897372a1c4805e32469e377fe098df37c8f1b6c382746"}
Nov 02 20:26:09.180663 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:09.180294    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-9ff720-openshift-default-58fd5fdb7cc6sn5" event={"ID":"2da1ff8e-3060-42a6-896e-75e07469e4b5","Type":"ContainerStarted","Data":"39a88e9989e45fd067f3f619bba92a0ccfac57750320773d1741f3e117c1ed0c"}
Nov 02 20:26:09.197683 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:09.197660    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-9ff720-openshift-default-58fd5fdb7cc6sn5"
Nov 02 20:26:09.202568 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:09.202549    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-9ff720-openshift-default-58fd5fdb7cc6sn5"
Nov 02 20:26:09.289594 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:09.289548    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-9ff720-openshift-default-58fd5fdb7cc6sn5" podStartSLOduration=2.289536 podStartE2EDuration="2.289536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:26:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:26:09.288377239 +0000 UTC m=+3475.895411019" watchObservedRunningTime="2026-11-02 20:26:09.289536 +0000 UTC m=+3475.896569768"
Nov 02 20:26:10.184708 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:10.184679    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-9ff720-openshift-default-58fd5fdb7cc6sn5"
Nov 02 20:26:10.185683 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:10.185663    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-9ff720-openshift-default-58fd5fdb7cc6sn5"
Nov 02 20:26:15.806807 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:15.806766    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-9ff720-77546dfpjc"]
Nov 02 20:26:15.810932 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:15.810905    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9ff720-77546dfpjc"
Nov 02 20:26:15.908931 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:15.908897    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-9ff720\""
Nov 02 20:26:15.908931 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:15.908914    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-ef8d045f-dockercfg-f8nm7\""
Nov 02 20:26:15.911722 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:15.911693    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-9ff720-77546dfpjc"]
Nov 02 20:26:15.942209 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:15.942176    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8de16bdf-930e-4435-bbe6-87a0275c5c83-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-9ff720-77546dfpjc\" (UID: \"8de16bdf-930e-4435-bbe6-87a0275c5c83\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9ff720-77546dfpjc"
Nov 02 20:26:15.942411 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:15.942351    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7npc\" (UniqueName: \"kubernetes.io/projected/8de16bdf-930e-4435-bbe6-87a0275c5c83-kube-api-access-s7npc\") pod \"payload-pre-processing-e2e-worker-single-main-9ff720-77546dfpjc\" (UID: \"8de16bdf-930e-4435-bbe6-87a0275c5c83\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9ff720-77546dfpjc"
Nov 02 20:26:16.043247 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:16.043213    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8de16bdf-930e-4435-bbe6-87a0275c5c83-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-9ff720-77546dfpjc\" (UID: \"8de16bdf-930e-4435-bbe6-87a0275c5c83\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9ff720-77546dfpjc"
Nov 02 20:26:16.043424 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:16.043309    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7npc\" (UniqueName: \"kubernetes.io/projected/8de16bdf-930e-4435-bbe6-87a0275c5c83-kube-api-access-s7npc\") pod \"payload-pre-processing-e2e-worker-single-main-9ff720-77546dfpjc\" (UID: \"8de16bdf-930e-4435-bbe6-87a0275c5c83\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9ff720-77546dfpjc"
Nov 02 20:26:16.043962 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:16.043942    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8de16bdf-930e-4435-bbe6-87a0275c5c83-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-9ff720-77546dfpjc\" (UID: \"8de16bdf-930e-4435-bbe6-87a0275c5c83\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9ff720-77546dfpjc"
Nov 02 20:26:16.096390 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:16.096322    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7npc\" (UniqueName: \"kubernetes.io/projected/8de16bdf-930e-4435-bbe6-87a0275c5c83-kube-api-access-s7npc\") pod \"payload-pre-processing-e2e-worker-single-main-9ff720-77546dfpjc\" (UID: \"8de16bdf-930e-4435-bbe6-87a0275c5c83\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9ff720-77546dfpjc"
Nov 02 20:26:16.124170 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:16.124145    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9ff720-77546dfpjc"
Nov 02 20:26:16.488446 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:16.488350    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-9ff720-77546dfpjc"]
Nov 02 20:26:16.490886 ip-10-0-133-9 kubenswrapper[4463]: W1102 20:26:16.490856    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8de16bdf_930e_4435_bbe6_87a0275c5c83.slice/crio-220382a8cb8289e2a06c9ded58b84653d4732254619d99132455c8994a0ec387 WatchSource:0}: Error finding container 220382a8cb8289e2a06c9ded58b84653d4732254619d99132455c8994a0ec387: Status 404 returned error can't find the container with id 220382a8cb8289e2a06c9ded58b84653d4732254619d99132455c8994a0ec387
Nov 02 20:26:17.218462 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:17.218426    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9ff720-77546dfpjc" event={"ID":"8de16bdf-930e-4435-bbe6-87a0275c5c83","Type":"ContainerStarted","Data":"e8d4c8cc0cbb62ce9266751dac286a46005e01c1bee03785777907d3ea6c4e98"}
Nov 02 20:26:17.218897 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:17.218479    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9ff720-77546dfpjc" event={"ID":"8de16bdf-930e-4435-bbe6-87a0275c5c83","Type":"ContainerStarted","Data":"220382a8cb8289e2a06c9ded58b84653d4732254619d99132455c8994a0ec387"}
Nov 02 20:26:17.218897 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:17.218535    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9ff720-77546dfpjc"
Nov 02 20:26:17.326383 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:17.326329    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9ff720-77546dfpjc" podStartSLOduration=2.32631419 podStartE2EDuration="2.32631419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:26:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:26:17.324355666 +0000 UTC m=+3483.931389463" watchObservedRunningTime="2026-11-02 20:26:17.32631419 +0000 UTC m=+3483.933347958"
Nov 02 20:26:17.774547 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:17.774511    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9ff720-8547684996-29x56"]
Nov 02 20:26:17.778066 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:17.778048    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9ff720-8547684996-29x56"
Nov 02 20:26:17.826494 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:17.826456    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-9ff720-serving-cert\""
Nov 02 20:26:17.832362 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:17.832281    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9ff720-8547684996-29x56"]
Nov 02 20:26:17.863204 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:17.863166    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/151a28da-8cc2-4b0e-8263-bb11e40e6470-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-9ff720-8547684996-29x56\" (UID: \"151a28da-8cc2-4b0e-8263-bb11e40e6470\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9ff720-8547684996-29x56"
Nov 02 20:26:17.863390 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:17.863235    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/151a28da-8cc2-4b0e-8263-bb11e40e6470-metrics-certs\") pod \"maas-api-e2e-worker-single-main-9ff720-8547684996-29x56\" (UID: \"151a28da-8cc2-4b0e-8263-bb11e40e6470\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9ff720-8547684996-29x56"
Nov 02 20:26:17.863390 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:17.863338    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/151a28da-8cc2-4b0e-8263-bb11e40e6470-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-9ff720-8547684996-29x56\" (UID: \"151a28da-8cc2-4b0e-8263-bb11e40e6470\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9ff720-8547684996-29x56"
Nov 02 20:26:17.863390 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:17.863374    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpknw\" (UniqueName: \"kubernetes.io/projected/151a28da-8cc2-4b0e-8263-bb11e40e6470-kube-api-access-vpknw\") pod \"maas-api-e2e-worker-single-main-9ff720-8547684996-29x56\" (UID: \"151a28da-8cc2-4b0e-8263-bb11e40e6470\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9ff720-8547684996-29x56"
Nov 02 20:26:17.964463 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:17.964418    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/151a28da-8cc2-4b0e-8263-bb11e40e6470-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-9ff720-8547684996-29x56\" (UID: \"151a28da-8cc2-4b0e-8263-bb11e40e6470\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9ff720-8547684996-29x56"
Nov 02 20:26:17.964463 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:17.964481    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/151a28da-8cc2-4b0e-8263-bb11e40e6470-metrics-certs\") pod \"maas-api-e2e-worker-single-main-9ff720-8547684996-29x56\" (UID: \"151a28da-8cc2-4b0e-8263-bb11e40e6470\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9ff720-8547684996-29x56"
Nov 02 20:26:17.964754 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:17.964508    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/151a28da-8cc2-4b0e-8263-bb11e40e6470-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-9ff720-8547684996-29x56\" (UID: \"151a28da-8cc2-4b0e-8263-bb11e40e6470\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9ff720-8547684996-29x56"
Nov 02 20:26:17.964754 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:17.964528    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpknw\" (UniqueName: \"kubernetes.io/projected/151a28da-8cc2-4b0e-8263-bb11e40e6470-kube-api-access-vpknw\") pod \"maas-api-e2e-worker-single-main-9ff720-8547684996-29x56\" (UID: \"151a28da-8cc2-4b0e-8263-bb11e40e6470\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9ff720-8547684996-29x56"
Nov 02 20:26:17.964754 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:26:17.964577    4463 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9ff720-serving-cert: secret "maas-api-e2e-worker-single-main-9ff720-serving-cert" not found
Nov 02 20:26:17.964754 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:26:17.964642    4463 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/151a28da-8cc2-4b0e-8263-bb11e40e6470-maas-api-tls podName:151a28da-8cc2-4b0e-8263-bb11e40e6470 nodeName:}" failed. No retries permitted until 2026-11-02 20:26:18.464624065 +0000 UTC m=+3485.071657825 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/151a28da-8cc2-4b0e-8263-bb11e40e6470-maas-api-tls") pod "maas-api-e2e-worker-single-main-9ff720-8547684996-29x56" (UID: "151a28da-8cc2-4b0e-8263-bb11e40e6470") : secret "maas-api-e2e-worker-single-main-9ff720-serving-cert" not found
Nov 02 20:26:17.965273 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:17.965255    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/151a28da-8cc2-4b0e-8263-bb11e40e6470-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-9ff720-8547684996-29x56\" (UID: \"151a28da-8cc2-4b0e-8263-bb11e40e6470\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9ff720-8547684996-29x56"
Nov 02 20:26:17.967266 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:17.967231    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/151a28da-8cc2-4b0e-8263-bb11e40e6470-metrics-certs\") pod \"maas-api-e2e-worker-single-main-9ff720-8547684996-29x56\" (UID: \"151a28da-8cc2-4b0e-8263-bb11e40e6470\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9ff720-8547684996-29x56"
Nov 02 20:26:18.029852 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:18.029777    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpknw\" (UniqueName: \"kubernetes.io/projected/151a28da-8cc2-4b0e-8263-bb11e40e6470-kube-api-access-vpknw\") pod \"maas-api-e2e-worker-single-main-9ff720-8547684996-29x56\" (UID: \"151a28da-8cc2-4b0e-8263-bb11e40e6470\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9ff720-8547684996-29x56"
Nov 02 20:26:18.469544 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:18.469504    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/151a28da-8cc2-4b0e-8263-bb11e40e6470-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-9ff720-8547684996-29x56\" (UID: \"151a28da-8cc2-4b0e-8263-bb11e40e6470\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9ff720-8547684996-29x56"
Nov 02 20:26:18.471899 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:18.471869    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/151a28da-8cc2-4b0e-8263-bb11e40e6470-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-9ff720-8547684996-29x56\" (UID: \"151a28da-8cc2-4b0e-8263-bb11e40e6470\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9ff720-8547684996-29x56"
Nov 02 20:26:18.690255 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:18.690226    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9ff720-8547684996-29x56"
Nov 02 20:26:19.053528 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:19.053446    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9ff720-8547684996-29x56"]
Nov 02 20:26:19.059653 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:19.059627    4463 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 20:26:19.228874 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:19.228838    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9ff720-8547684996-29x56" event={"ID":"151a28da-8cc2-4b0e-8263-bb11e40e6470","Type":"ContainerStarted","Data":"9d2256535988c5f86e1614325aaede3a946ad2507f17238b15cc5a638ea5cba4"}
Nov 02 20:26:20.235216 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:20.235119    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9ff720-8547684996-29x56" event={"ID":"151a28da-8cc2-4b0e-8263-bb11e40e6470","Type":"ContainerStarted","Data":"3da9a261015b903aa18ea4e1daa63478a987de3625d0d6740f2850466e0f48f7"}
Nov 02 20:26:20.235677 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:20.235248    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9ff720-8547684996-29x56"
Nov 02 20:26:20.342457 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:20.342397    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9ff720-8547684996-29x56" podStartSLOduration=2.441407785 podStartE2EDuration="3.342378658s" totalImagesPullingTime="900.970873ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:26:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:26:20.341550461 +0000 UTC m=+3486.948584230" watchObservedRunningTime="2026-11-02 20:26:20.342378658 +0000 UTC m=+3486.949412428"
Nov 02 20:26:25.538259 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:25.538225    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 20:26:26.247004 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:26.246975    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9ff720-8547684996-29x56"
Nov 02 20:26:28.226657 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:28.226628    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9ff720-77546dfpjc"
Nov 02 20:26:34.293795 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:26:34.293715    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 20:26:34.318282 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:34.318253    4463 generic.go:356] "Generic (PLEG): container finished" podID="2da1ff8e-3060-42a6-896e-75e07469e4b5" containerID="27dcbc5c37d6a56e748897372a1c4805e32469e377fe098df37c8f1b6c382746" exitCode=0
Nov 02 20:26:34.318409 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:34.318314    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-9ff720-openshift-default-58fd5fdb7cc6sn5" event={"ID":"2da1ff8e-3060-42a6-896e-75e07469e4b5","Type":"ContainerDied","Data":"27dcbc5c37d6a56e748897372a1c4805e32469e377fe098df37c8f1b6c382746"}
Nov 02 20:26:34.318744 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:34.318726    4463 scope.go:122] "RemoveContainer" containerID="27dcbc5c37d6a56e748897372a1c4805e32469e377fe098df37c8f1b6c382746"
Nov 02 20:26:34.319855 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:34.319829    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:26:34.319918 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:34.319893    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:26:34.319953 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:34.319923    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:26:35.325139 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:35.325099    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-9ff720-openshift-default-58fd5fdb7cc6sn5" event={"ID":"2da1ff8e-3060-42a6-896e-75e07469e4b5","Type":"ContainerStarted","Data":"301c12fc3dfab2000211a6f7ad8efa83fe058dde666ffafdbad094d27efd1a19"}
Nov 02 20:26:36.197287 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:36.197236    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-9ff720-openshift-default-58fd5fdb7cc6sn5"
Nov 02 20:26:36.198636 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:36.198613    4463 patch_prober.go:29] interesting pod/e2e-worker-single-main-9ff720-openshift-default-58fd5fdb7cc6sn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Nov 02 20:26:36.198753 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:36.198666    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-9ff720-openshift-default-58fd5fdb7cc6sn5" podUID="2da1ff8e-3060-42a6-896e-75e07469e4b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Nov 02 20:26:37.197466 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:37.197426    4463 patch_prober.go:29] interesting pod/e2e-worker-single-main-9ff720-openshift-default-58fd5fdb7cc6sn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Nov 02 20:26:37.197946 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:37.197482    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-9ff720-openshift-default-58fd5fdb7cc6sn5" podUID="2da1ff8e-3060-42a6-896e-75e07469e4b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Nov 02 20:26:38.197375 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:38.197330    4463 patch_prober.go:29] interesting pod/e2e-worker-single-main-9ff720-openshift-default-58fd5fdb7cc6sn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Nov 02 20:26:38.197589 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:38.197391    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-9ff720-openshift-default-58fd5fdb7cc6sn5" podUID="2da1ff8e-3060-42a6-896e-75e07469e4b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Nov 02 20:26:39.197193 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:39.197158    4463 patch_prober.go:29] interesting pod/e2e-worker-single-main-9ff720-openshift-default-58fd5fdb7cc6sn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Nov 02 20:26:39.197384 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:39.197212    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-9ff720-openshift-default-58fd5fdb7cc6sn5" podUID="2da1ff8e-3060-42a6-896e-75e07469e4b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Nov 02 20:26:40.184632 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:40.184593    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-9ff720-openshift-default-58fd5fdb7cc6sn5"
Nov 02 20:26:40.197536 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:40.197505    4463 patch_prober.go:29] interesting pod/e2e-worker-single-main-9ff720-openshift-default-58fd5fdb7cc6sn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Nov 02 20:26:40.197680 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:40.197583    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-9ff720-openshift-default-58fd5fdb7cc6sn5" podUID="2da1ff8e-3060-42a6-896e-75e07469e4b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Nov 02 20:26:41.201311 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:41.201279    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-9ff720-openshift-default-58fd5fdb7cc6sn5"
Nov 02 20:26:41.202386 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:41.202359    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-9ff720-openshift-default-58fd5fdb7cc6sn5"
Nov 02 20:26:45.753806 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:26:45.753766    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 20:27:04.294334 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:27:04.294289    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 20:27:10.858167 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:27:10.858132    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 20:27:34.293229 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:27:34.293191    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 20:27:40.058549 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:27:40.058513    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 20:27:49.204059 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:27:49.204004    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-9ff720-77546dfpjc"]
Nov 02 20:27:54.204858 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:27:54.204801    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9ff720-77546dfpjc" podUID="8de16bdf-930e-4435-bbe6-87a0275c5c83" containerName="payload-pre-processing" containerID="cri-o://e8d4c8cc0cbb62ce9266751dac286a46005e01c1bee03785777907d3ea6c4e98" gracePeriod=25
Nov 02 20:27:55.039698 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:27:55.039663    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9ff720-8547684996-29x56"]
Nov 02 20:27:55.039879 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:27:55.039836    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9ff720-8547684996-29x56" podUID="151a28da-8cc2-4b0e-8263-bb11e40e6470" containerName="maas-api" containerID="cri-o://3da9a261015b903aa18ea4e1daa63478a987de3625d0d6740f2850466e0f48f7" gracePeriod=30
Nov 02 20:27:56.287330 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:27:56.287305    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9ff720-8547684996-29x56"
Nov 02 20:27:56.350610 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:27:56.350576    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/151a28da-8cc2-4b0e-8263-bb11e40e6470-kube-api-access-vpknw\" (UniqueName: \"kubernetes.io/projected/151a28da-8cc2-4b0e-8263-bb11e40e6470-kube-api-access-vpknw\") pod \"151a28da-8cc2-4b0e-8263-bb11e40e6470\" (UID: \"151a28da-8cc2-4b0e-8263-bb11e40e6470\") "
Nov 02 20:27:56.350815 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:27:56.350663    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/151a28da-8cc2-4b0e-8263-bb11e40e6470-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/151a28da-8cc2-4b0e-8263-bb11e40e6470-maas-api-tls\") pod \"151a28da-8cc2-4b0e-8263-bb11e40e6470\" (UID: \"151a28da-8cc2-4b0e-8263-bb11e40e6470\") "
Nov 02 20:27:56.350815 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:27:56.350737    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/151a28da-8cc2-4b0e-8263-bb11e40e6470-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/151a28da-8cc2-4b0e-8263-bb11e40e6470-openshift-service-ca\") pod \"151a28da-8cc2-4b0e-8263-bb11e40e6470\" (UID: \"151a28da-8cc2-4b0e-8263-bb11e40e6470\") "
Nov 02 20:27:56.350815 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:27:56.350761    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/151a28da-8cc2-4b0e-8263-bb11e40e6470-metrics-certs\" (UniqueName: \"kubernetes.io/secret/151a28da-8cc2-4b0e-8263-bb11e40e6470-metrics-certs\") pod \"151a28da-8cc2-4b0e-8263-bb11e40e6470\" (UID: \"151a28da-8cc2-4b0e-8263-bb11e40e6470\") "
Nov 02 20:27:56.351097 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:27:56.351069    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/151a28da-8cc2-4b0e-8263-bb11e40e6470-openshift-service-ca" pod "151a28da-8cc2-4b0e-8263-bb11e40e6470" (UID: "151a28da-8cc2-4b0e-8263-bb11e40e6470"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:27:56.352854 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:27:56.352831    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/151a28da-8cc2-4b0e-8263-bb11e40e6470-maas-api-tls" pod "151a28da-8cc2-4b0e-8263-bb11e40e6470" (UID: "151a28da-8cc2-4b0e-8263-bb11e40e6470"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:27:56.352927 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:27:56.352880    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/151a28da-8cc2-4b0e-8263-bb11e40e6470-kube-api-access-vpknw" pod "151a28da-8cc2-4b0e-8263-bb11e40e6470" (UID: "151a28da-8cc2-4b0e-8263-bb11e40e6470"). InnerVolumeSpecName "kube-api-access-vpknw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:27:56.352927 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:27:56.352914    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/151a28da-8cc2-4b0e-8263-bb11e40e6470-metrics-certs" pod "151a28da-8cc2-4b0e-8263-bb11e40e6470" (UID: "151a28da-8cc2-4b0e-8263-bb11e40e6470"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:27:56.451425 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:27:56.451397    4463 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/151a28da-8cc2-4b0e-8263-bb11e40e6470-metrics-certs\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:27:56.451425 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:27:56.451423    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpknw\" (UniqueName: \"kubernetes.io/projected/151a28da-8cc2-4b0e-8263-bb11e40e6470-kube-api-access-vpknw\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:27:56.451425 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:27:56.451433    4463 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/151a28da-8cc2-4b0e-8263-bb11e40e6470-maas-api-tls\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:27:56.451652 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:27:56.451444    4463 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/151a28da-8cc2-4b0e-8263-bb11e40e6470-openshift-service-ca\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:27:56.717992 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:27:56.717905    4463 generic.go:356] "Generic (PLEG): container finished" podID="151a28da-8cc2-4b0e-8263-bb11e40e6470" containerID="3da9a261015b903aa18ea4e1daa63478a987de3625d0d6740f2850466e0f48f7" exitCode=0
Nov 02 20:27:56.717992 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:27:56.717949    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9ff720-8547684996-29x56"
Nov 02 20:27:56.718231 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:27:56.717987    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9ff720-8547684996-29x56" event={"ID":"151a28da-8cc2-4b0e-8263-bb11e40e6470","Type":"ContainerDied","Data":"3da9a261015b903aa18ea4e1daa63478a987de3625d0d6740f2850466e0f48f7"}
Nov 02 20:27:56.718231 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:27:56.718069    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9ff720-8547684996-29x56" event={"ID":"151a28da-8cc2-4b0e-8263-bb11e40e6470","Type":"ContainerDied","Data":"9d2256535988c5f86e1614325aaede3a946ad2507f17238b15cc5a638ea5cba4"}
Nov 02 20:27:56.718231 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:27:56.718095    4463 scope.go:122] "RemoveContainer" containerID="3da9a261015b903aa18ea4e1daa63478a987de3625d0d6740f2850466e0f48f7"
Nov 02 20:27:56.727087 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:27:56.727071    4463 scope.go:122] "RemoveContainer" containerID="3da9a261015b903aa18ea4e1daa63478a987de3625d0d6740f2850466e0f48f7"
Nov 02 20:27:56.727339 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:27:56.727322    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3da9a261015b903aa18ea4e1daa63478a987de3625d0d6740f2850466e0f48f7\": container with ID starting with 3da9a261015b903aa18ea4e1daa63478a987de3625d0d6740f2850466e0f48f7 not found: ID does not exist" containerID="3da9a261015b903aa18ea4e1daa63478a987de3625d0d6740f2850466e0f48f7"
Nov 02 20:27:56.727391 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:27:56.727346    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3da9a261015b903aa18ea4e1daa63478a987de3625d0d6740f2850466e0f48f7"} err="failed to get container status \"3da9a261015b903aa18ea4e1daa63478a987de3625d0d6740f2850466e0f48f7\": rpc error: code = NotFound desc = could not find container \"3da9a261015b903aa18ea4e1daa63478a987de3625d0d6740f2850466e0f48f7\": container with ID starting with 3da9a261015b903aa18ea4e1daa63478a987de3625d0d6740f2850466e0f48f7 not found: ID does not exist"
Nov 02 20:27:56.869274 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:27:56.869243    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9ff720-8547684996-29x56"]
Nov 02 20:27:56.915522 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:27:56.915491    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9ff720-8547684996-29x56"]
Nov 02 20:27:57.583384 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:27:57.583349    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="151a28da-8cc2-4b0e-8263-bb11e40e6470" path="/var/lib/kubelet/pods/151a28da-8cc2-4b0e-8263-bb11e40e6470/volumes"
Nov 02 20:27:58.224810 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:27:58.224769    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9ff720-77546dfpjc" podUID="8de16bdf-930e-4435-bbe6-87a0275c5c83" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 20:27:59.241757 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:27:59.241715    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9ff720-8547684996-29x56" podUID="151a28da-8cc2-4b0e-8263-bb11e40e6470" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.138:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:27:59.805851 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:27:59.805816    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 20:28:03.587968 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:28:03.587937    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-9ff720-openshift-default-58fd5fdb7cc6sn5"]
Nov 02 20:28:03.588532 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:28:03.588173    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-9ff720-openshift-default-58fd5fdb7cc6sn5" podUID="2da1ff8e-3060-42a6-896e-75e07469e4b5" containerName="istio-proxy" containerID="cri-o://301c12fc3dfab2000211a6f7ad8efa83fe058dde666ffafdbad094d27efd1a19" gracePeriod=30
Nov 02 20:28:04.293773 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:28:04.293727    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 20:28:08.224956 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:28:08.224911    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9ff720-77546dfpjc" podUID="8de16bdf-930e-4435-bbe6-87a0275c5c83" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 20:28:08.773659 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:28:08.773630    4463 generic.go:356] "Generic (PLEG): container finished" podID="2da1ff8e-3060-42a6-896e-75e07469e4b5" containerID="301c12fc3dfab2000211a6f7ad8efa83fe058dde666ffafdbad094d27efd1a19" exitCode=0
Nov 02 20:28:08.773807 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:28:08.773705    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-9ff720-openshift-default-58fd5fdb7cc6sn5" event={"ID":"2da1ff8e-3060-42a6-896e-75e07469e4b5","Type":"ContainerDied","Data":"301c12fc3dfab2000211a6f7ad8efa83fe058dde666ffafdbad094d27efd1a19"}
Nov 02 20:28:08.773807 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:28:08.773769    4463 scope.go:122] "RemoveContainer" containerID="27dcbc5c37d6a56e748897372a1c4805e32469e377fe098df37c8f1b6c382746"
Nov 02 20:28:08.836736 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:28:08.836711    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-9ff720-openshift-default-58fd5fdb7cc6sn5"
Nov 02 20:28:08.973698 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:28:08.973669    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2da1ff8e-3060-42a6-896e-75e07469e4b5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2da1ff8e-3060-42a6-896e-75e07469e4b5-istio-data\") pod \"2da1ff8e-3060-42a6-896e-75e07469e4b5\" (UID: \"2da1ff8e-3060-42a6-896e-75e07469e4b5\") "
Nov 02 20:28:08.973841 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:28:08.973708    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2da1ff8e-3060-42a6-896e-75e07469e4b5-istio-token\" (UniqueName: \"kubernetes.io/projected/2da1ff8e-3060-42a6-896e-75e07469e4b5-istio-token\") pod \"2da1ff8e-3060-42a6-896e-75e07469e4b5\" (UID: \"2da1ff8e-3060-42a6-896e-75e07469e4b5\") "
Nov 02 20:28:08.973841 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:28:08.973737    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2da1ff8e-3060-42a6-896e-75e07469e4b5-kube-api-access-qrsm4\" (UniqueName: \"kubernetes.io/projected/2da1ff8e-3060-42a6-896e-75e07469e4b5-kube-api-access-qrsm4\") pod \"2da1ff8e-3060-42a6-896e-75e07469e4b5\" (UID: \"2da1ff8e-3060-42a6-896e-75e07469e4b5\") "
Nov 02 20:28:08.973934 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:28:08.973854    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2da1ff8e-3060-42a6-896e-75e07469e4b5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2da1ff8e-3060-42a6-896e-75e07469e4b5-istio-envoy\") pod \"2da1ff8e-3060-42a6-896e-75e07469e4b5\" (UID: \"2da1ff8e-3060-42a6-896e-75e07469e4b5\") "
Nov 02 20:28:08.973934 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:28:08.973910    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2da1ff8e-3060-42a6-896e-75e07469e4b5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2da1ff8e-3060-42a6-896e-75e07469e4b5-workload-socket\") pod \"2da1ff8e-3060-42a6-896e-75e07469e4b5\" (UID: \"2da1ff8e-3060-42a6-896e-75e07469e4b5\") "
Nov 02 20:28:08.974055 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:28:08.973961    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2da1ff8e-3060-42a6-896e-75e07469e4b5-istio-data" pod "2da1ff8e-3060-42a6-896e-75e07469e4b5" (UID: "2da1ff8e-3060-42a6-896e-75e07469e4b5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:28:08.974055 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:28:08.973965    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2da1ff8e-3060-42a6-896e-75e07469e4b5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2da1ff8e-3060-42a6-896e-75e07469e4b5-workload-certs\") pod \"2da1ff8e-3060-42a6-896e-75e07469e4b5\" (UID: \"2da1ff8e-3060-42a6-896e-75e07469e4b5\") "
Nov 02 20:28:08.974168 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:28:08.974065    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2da1ff8e-3060-42a6-896e-75e07469e4b5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2da1ff8e-3060-42a6-896e-75e07469e4b5-istio-podinfo\") pod \"2da1ff8e-3060-42a6-896e-75e07469e4b5\" (UID: \"2da1ff8e-3060-42a6-896e-75e07469e4b5\") "
Nov 02 20:28:08.974168 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:28:08.974098    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2da1ff8e-3060-42a6-896e-75e07469e4b5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2da1ff8e-3060-42a6-896e-75e07469e4b5-credential-socket\") pod \"2da1ff8e-3060-42a6-896e-75e07469e4b5\" (UID: \"2da1ff8e-3060-42a6-896e-75e07469e4b5\") "
Nov 02 20:28:08.974285 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:28:08.974170    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2da1ff8e-3060-42a6-896e-75e07469e4b5-workload-certs" pod "2da1ff8e-3060-42a6-896e-75e07469e4b5" (UID: "2da1ff8e-3060-42a6-896e-75e07469e4b5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:28:08.974285 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:28:08.974185    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2da1ff8e-3060-42a6-896e-75e07469e4b5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2da1ff8e-3060-42a6-896e-75e07469e4b5-istiod-ca-cert\") pod \"2da1ff8e-3060-42a6-896e-75e07469e4b5\" (UID: \"2da1ff8e-3060-42a6-896e-75e07469e4b5\") "
Nov 02 20:28:08.974285 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:28:08.974228    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2da1ff8e-3060-42a6-896e-75e07469e4b5-workload-socket" pod "2da1ff8e-3060-42a6-896e-75e07469e4b5" (UID: "2da1ff8e-3060-42a6-896e-75e07469e4b5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:28:08.974446 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:28:08.974392    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2da1ff8e-3060-42a6-896e-75e07469e4b5-credential-socket" pod "2da1ff8e-3060-42a6-896e-75e07469e4b5" (UID: "2da1ff8e-3060-42a6-896e-75e07469e4b5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:28:08.974621 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:28:08.974604    4463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2da1ff8e-3060-42a6-896e-75e07469e4b5-workload-socket\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:28:08.974702 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:28:08.974626    4463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2da1ff8e-3060-42a6-896e-75e07469e4b5-workload-certs\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:28:08.974702 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:28:08.974634    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2da1ff8e-3060-42a6-896e-75e07469e4b5-istiod-ca-cert" pod "2da1ff8e-3060-42a6-896e-75e07469e4b5" (UID: "2da1ff8e-3060-42a6-896e-75e07469e4b5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:28:08.974702 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:28:08.974641    4463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2da1ff8e-3060-42a6-896e-75e07469e4b5-credential-socket\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:28:08.974702 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:28:08.974678    4463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2da1ff8e-3060-42a6-896e-75e07469e4b5-istio-data\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:28:08.976106 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:28:08.976081    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2da1ff8e-3060-42a6-896e-75e07469e4b5-kube-api-access-qrsm4" pod "2da1ff8e-3060-42a6-896e-75e07469e4b5" (UID: "2da1ff8e-3060-42a6-896e-75e07469e4b5"). InnerVolumeSpecName "kube-api-access-qrsm4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:28:08.976218 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:28:08.976195    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2da1ff8e-3060-42a6-896e-75e07469e4b5-istio-podinfo" pod "2da1ff8e-3060-42a6-896e-75e07469e4b5" (UID: "2da1ff8e-3060-42a6-896e-75e07469e4b5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:28:08.976381 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:28:08.976360    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2da1ff8e-3060-42a6-896e-75e07469e4b5-istio-envoy" pod "2da1ff8e-3060-42a6-896e-75e07469e4b5" (UID: "2da1ff8e-3060-42a6-896e-75e07469e4b5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:28:08.976441 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:28:08.976377    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2da1ff8e-3060-42a6-896e-75e07469e4b5-istio-token" pod "2da1ff8e-3060-42a6-896e-75e07469e4b5" (UID: "2da1ff8e-3060-42a6-896e-75e07469e4b5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:28:09.075668 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:28:09.075638    4463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2da1ff8e-3060-42a6-896e-75e07469e4b5-istio-podinfo\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:28:09.075668 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:28:09.075664    4463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2da1ff8e-3060-42a6-896e-75e07469e4b5-istiod-ca-cert\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:28:09.075668 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:28:09.075675    4463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2da1ff8e-3060-42a6-896e-75e07469e4b5-istio-token\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:28:09.075895 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:28:09.075685    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrsm4\" (UniqueName: \"kubernetes.io/projected/2da1ff8e-3060-42a6-896e-75e07469e4b5-kube-api-access-qrsm4\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:28:09.075895 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:28:09.075696    4463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2da1ff8e-3060-42a6-896e-75e07469e4b5-istio-envoy\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:28:09.779337 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:28:09.779307    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-9ff720-openshift-default-58fd5fdb7cc6sn5"
Nov 02 20:28:09.779337 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:28:09.779331    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-9ff720-openshift-default-58fd5fdb7cc6sn5" event={"ID":"2da1ff8e-3060-42a6-896e-75e07469e4b5","Type":"ContainerDied","Data":"39a88e9989e45fd067f3f619bba92a0ccfac57750320773d1741f3e117c1ed0c"}
Nov 02 20:28:09.779776 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:28:09.779386    4463 scope.go:122] "RemoveContainer" containerID="301c12fc3dfab2000211a6f7ad8efa83fe058dde666ffafdbad094d27efd1a19"
Nov 02 20:28:09.978495 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:28:09.978457    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-9ff720-openshift-default-58fd5fdb7cc6sn5"]
Nov 02 20:28:10.024723 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:28:10.024676    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-9ff720-openshift-default-58fd5fdb7cc6sn5"]
Nov 02 20:28:11.583647 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:28:11.583610    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2da1ff8e-3060-42a6-896e-75e07469e4b5" path="/var/lib/kubelet/pods/2da1ff8e-3060-42a6-896e-75e07469e4b5/volumes"
Nov 02 20:28:14.455502 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:28:14.455477    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9ff720-77546dfpjc"
Nov 02 20:28:14.630641 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:28:14.630542    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8de16bdf-930e-4435-bbe6-87a0275c5c83-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8de16bdf-930e-4435-bbe6-87a0275c5c83-plugins-config-volume\") pod \"8de16bdf-930e-4435-bbe6-87a0275c5c83\" (UID: \"8de16bdf-930e-4435-bbe6-87a0275c5c83\") "
Nov 02 20:28:14.630641 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:28:14.630607    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8de16bdf-930e-4435-bbe6-87a0275c5c83-kube-api-access-s7npc\" (UniqueName: \"kubernetes.io/projected/8de16bdf-930e-4435-bbe6-87a0275c5c83-kube-api-access-s7npc\") pod \"8de16bdf-930e-4435-bbe6-87a0275c5c83\" (UID: \"8de16bdf-930e-4435-bbe6-87a0275c5c83\") "
Nov 02 20:28:14.630981 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:28:14.630956    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8de16bdf-930e-4435-bbe6-87a0275c5c83-plugins-config-volume" pod "8de16bdf-930e-4435-bbe6-87a0275c5c83" (UID: "8de16bdf-930e-4435-bbe6-87a0275c5c83"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:28:14.632650 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:28:14.632630    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8de16bdf-930e-4435-bbe6-87a0275c5c83-kube-api-access-s7npc" pod "8de16bdf-930e-4435-bbe6-87a0275c5c83" (UID: "8de16bdf-930e-4435-bbe6-87a0275c5c83"). InnerVolumeSpecName "kube-api-access-s7npc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:28:14.731737 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:28:14.731710    4463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8de16bdf-930e-4435-bbe6-87a0275c5c83-plugins-config-volume\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:28:14.731737 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:28:14.731735    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7npc\" (UniqueName: \"kubernetes.io/projected/8de16bdf-930e-4435-bbe6-87a0275c5c83-kube-api-access-s7npc\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:28:14.806131 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:28:14.806099    4463 generic.go:356] "Generic (PLEG): container finished" podID="8de16bdf-930e-4435-bbe6-87a0275c5c83" containerID="e8d4c8cc0cbb62ce9266751dac286a46005e01c1bee03785777907d3ea6c4e98" exitCode=0
Nov 02 20:28:14.806230 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:28:14.806148    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9ff720-77546dfpjc"
Nov 02 20:28:14.806230 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:28:14.806183    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9ff720-77546dfpjc" event={"ID":"8de16bdf-930e-4435-bbe6-87a0275c5c83","Type":"ContainerDied","Data":"e8d4c8cc0cbb62ce9266751dac286a46005e01c1bee03785777907d3ea6c4e98"}
Nov 02 20:28:14.806309 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:28:14.806235    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9ff720-77546dfpjc" event={"ID":"8de16bdf-930e-4435-bbe6-87a0275c5c83","Type":"ContainerDied","Data":"220382a8cb8289e2a06c9ded58b84653d4732254619d99132455c8994a0ec387"}
Nov 02 20:28:14.806309 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:28:14.806261    4463 scope.go:122] "RemoveContainer" containerID="e8d4c8cc0cbb62ce9266751dac286a46005e01c1bee03785777907d3ea6c4e98"
Nov 02 20:28:14.814102 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:28:14.814075    4463 scope.go:122] "RemoveContainer" containerID="e8d4c8cc0cbb62ce9266751dac286a46005e01c1bee03785777907d3ea6c4e98"
Nov 02 20:28:14.814357 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:28:14.814336    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8d4c8cc0cbb62ce9266751dac286a46005e01c1bee03785777907d3ea6c4e98\": container with ID starting with e8d4c8cc0cbb62ce9266751dac286a46005e01c1bee03785777907d3ea6c4e98 not found: ID does not exist" containerID="e8d4c8cc0cbb62ce9266751dac286a46005e01c1bee03785777907d3ea6c4e98"
Nov 02 20:28:14.814416 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:28:14.814363    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8d4c8cc0cbb62ce9266751dac286a46005e01c1bee03785777907d3ea6c4e98"} err="failed to get container status \"e8d4c8cc0cbb62ce9266751dac286a46005e01c1bee03785777907d3ea6c4e98\": rpc error: code = NotFound desc = could not find container \"e8d4c8cc0cbb62ce9266751dac286a46005e01c1bee03785777907d3ea6c4e98\": container with ID starting with e8d4c8cc0cbb62ce9266751dac286a46005e01c1bee03785777907d3ea6c4e98 not found: ID does not exist"
Nov 02 20:28:14.957830 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:28:14.957801    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-9ff720-77546dfpjc"]
Nov 02 20:28:15.003805 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:28:15.003776    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-9ff720-77546dfpjc"]
Nov 02 20:28:15.586186 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:28:15.584313    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8de16bdf-930e-4435-bbe6-87a0275c5c83" path="/var/lib/kubelet/pods/8de16bdf-930e-4435-bbe6-87a0275c5c83/volumes"
Nov 02 20:28:34.294050 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:28:34.293996    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 20:29:04.294390 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:29:04.294345    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 20:29:34.293225 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:29:34.293179    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 20:29:52.427113 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:29:52.427074    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 20:30:04.294450 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:30:04.294400    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 20:30:09.530906 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:30:09.530868    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 20:30:18.346876 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:30:18.346834    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 20:30:34.294032 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:30:34.293925    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 20:30:38.435395 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:30:38.435360    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 20:30:38.714692 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:30:38.714603    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-a52dfa-openshift-default-84c659644f4r87k"]
Nov 02 20:30:38.715104 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:30:38.715085    4463 cpu_manager.go:414] "containerMap: removing container" podUID="151a28da-8cc2-4b0e-8263-bb11e40e6470" containerName="maas-api"
Nov 02 20:30:38.715212 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:30:38.715105    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="151a28da-8cc2-4b0e-8263-bb11e40e6470" containerName="maas-api"
Nov 02 20:30:38.715212 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:30:38.715127    4463 cpu_manager.go:414] "containerMap: removing container" podUID="2da1ff8e-3060-42a6-896e-75e07469e4b5" containerName="istio-proxy"
Nov 02 20:30:38.715212 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:30:38.715134    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2da1ff8e-3060-42a6-896e-75e07469e4b5" containerName="istio-proxy"
Nov 02 20:30:38.715212 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:30:38.715179    4463 cpu_manager.go:414] "containerMap: removing container" podUID="8de16bdf-930e-4435-bbe6-87a0275c5c83" containerName="payload-pre-processing"
Nov 02 20:30:38.715212 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:30:38.715188    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8de16bdf-930e-4435-bbe6-87a0275c5c83" containerName="payload-pre-processing"
Nov 02 20:30:38.715488 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:30:38.715293    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="8de16bdf-930e-4435-bbe6-87a0275c5c83" containerName="payload-pre-processing"
Nov 02 20:30:38.715488 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:30:38.715308    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="2da1ff8e-3060-42a6-896e-75e07469e4b5" containerName="istio-proxy"
Nov 02 20:30:38.715488 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:30:38.715318    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="151a28da-8cc2-4b0e-8263-bb11e40e6470" containerName="maas-api"
Nov 02 20:30:38.715488 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:30:38.715328    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="2da1ff8e-3060-42a6-896e-75e07469e4b5" containerName="istio-proxy"
Nov 02 20:30:38.718647 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:30:38.718627    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-a52dfa-openshift-default-84c659644f4r87k"
Nov 02 20:30:38.770356 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:30:38.770326    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-a52dfa-openshift-default-dockercfg-q9k5q\""
Nov 02 20:30:38.826593 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:30:38.826556    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-a52dfa-openshift-default-84c659644f4r87k"]
Nov 02 20:30:38.871360 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:30:38.871311    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c46e1a4-84cc-483e-8b1d-7786349ef4d5-istio-token\") pod \"e2e-worker-single-main-a52dfa-openshift-default-84c659644f4r87k\" (UID: \"2c46e1a4-84cc-483e-8b1d-7786349ef4d5\") " pod="openshift-ingress/e2e-worker-single-main-a52dfa-openshift-default-84c659644f4r87k"
Nov 02 20:30:38.871553 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:30:38.871379    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c46e1a4-84cc-483e-8b1d-7786349ef4d5-workload-socket\") pod \"e2e-worker-single-main-a52dfa-openshift-default-84c659644f4r87k\" (UID: \"2c46e1a4-84cc-483e-8b1d-7786349ef4d5\") " pod="openshift-ingress/e2e-worker-single-main-a52dfa-openshift-default-84c659644f4r87k"
Nov 02 20:30:38.871553 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:30:38.871488    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c46e1a4-84cc-483e-8b1d-7786349ef4d5-istio-envoy\") pod \"e2e-worker-single-main-a52dfa-openshift-default-84c659644f4r87k\" (UID: \"2c46e1a4-84cc-483e-8b1d-7786349ef4d5\") " pod="openshift-ingress/e2e-worker-single-main-a52dfa-openshift-default-84c659644f4r87k"
Nov 02 20:30:38.871553 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:30:38.871537    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skwm7\" (UniqueName: \"kubernetes.io/projected/2c46e1a4-84cc-483e-8b1d-7786349ef4d5-kube-api-access-skwm7\") pod \"e2e-worker-single-main-a52dfa-openshift-default-84c659644f4r87k\" (UID: \"2c46e1a4-84cc-483e-8b1d-7786349ef4d5\") " pod="openshift-ingress/e2e-worker-single-main-a52dfa-openshift-default-84c659644f4r87k"
Nov 02 20:30:38.871729 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:30:38.871616    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c46e1a4-84cc-483e-8b1d-7786349ef4d5-istio-podinfo\") pod \"e2e-worker-single-main-a52dfa-openshift-default-84c659644f4r87k\" (UID: \"2c46e1a4-84cc-483e-8b1d-7786349ef4d5\") " pod="openshift-ingress/e2e-worker-single-main-a52dfa-openshift-default-84c659644f4r87k"
Nov 02 20:30:38.871729 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:30:38.871671    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c46e1a4-84cc-483e-8b1d-7786349ef4d5-istiod-ca-cert\") pod \"e2e-worker-single-main-a52dfa-openshift-default-84c659644f4r87k\" (UID: \"2c46e1a4-84cc-483e-8b1d-7786349ef4d5\") " pod="openshift-ingress/e2e-worker-single-main-a52dfa-openshift-default-84c659644f4r87k"
Nov 02 20:30:38.871729 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:30:38.871707    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c46e1a4-84cc-483e-8b1d-7786349ef4d5-credential-socket\") pod \"e2e-worker-single-main-a52dfa-openshift-default-84c659644f4r87k\" (UID: \"2c46e1a4-84cc-483e-8b1d-7786349ef4d5\") " pod="openshift-ingress/e2e-worker-single-main-a52dfa-openshift-default-84c659644f4r87k"
Nov 02 20:30:38.871921 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:30:38.871819    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c46e1a4-84cc-483e-8b1d-7786349ef4d5-workload-certs\") pod \"e2e-worker-single-main-a52dfa-openshift-default-84c659644f4r87k\" (UID: \"2c46e1a4-84cc-483e-8b1d-7786349ef4d5\") " pod="openshift-ingress/e2e-worker-single-main-a52dfa-openshift-default-84c659644f4r87k"
Nov 02 20:30:38.871995 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:30:38.871934    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c46e1a4-84cc-483e-8b1d-7786349ef4d5-istio-data\") pod \"e2e-worker-single-main-a52dfa-openshift-default-84c659644f4r87k\" (UID: \"2c46e1a4-84cc-483e-8b1d-7786349ef4d5\") " pod="openshift-ingress/e2e-worker-single-main-a52dfa-openshift-default-84c659644f4r87k"
Nov 02 20:30:38.972954 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:30:38.972879    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c46e1a4-84cc-483e-8b1d-7786349ef4d5-istio-token\") pod \"e2e-worker-single-main-a52dfa-openshift-default-84c659644f4r87k\" (UID: \"2c46e1a4-84cc-483e-8b1d-7786349ef4d5\") " pod="openshift-ingress/e2e-worker-single-main-a52dfa-openshift-default-84c659644f4r87k"
Nov 02 20:30:38.972954 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:30:38.972924    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c46e1a4-84cc-483e-8b1d-7786349ef4d5-workload-socket\") pod \"e2e-worker-single-main-a52dfa-openshift-default-84c659644f4r87k\" (UID: \"2c46e1a4-84cc-483e-8b1d-7786349ef4d5\") " pod="openshift-ingress/e2e-worker-single-main-a52dfa-openshift-default-84c659644f4r87k"
Nov 02 20:30:38.972954 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:30:38.972943    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c46e1a4-84cc-483e-8b1d-7786349ef4d5-istio-envoy\") pod \"e2e-worker-single-main-a52dfa-openshift-default-84c659644f4r87k\" (UID: \"2c46e1a4-84cc-483e-8b1d-7786349ef4d5\") " pod="openshift-ingress/e2e-worker-single-main-a52dfa-openshift-default-84c659644f4r87k"
Nov 02 20:30:38.973255 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:30:38.972965    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skwm7\" (UniqueName: \"kubernetes.io/projected/2c46e1a4-84cc-483e-8b1d-7786349ef4d5-kube-api-access-skwm7\") pod \"e2e-worker-single-main-a52dfa-openshift-default-84c659644f4r87k\" (UID: \"2c46e1a4-84cc-483e-8b1d-7786349ef4d5\") " pod="openshift-ingress/e2e-worker-single-main-a52dfa-openshift-default-84c659644f4r87k"
Nov 02 20:30:38.973255 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:30:38.973090    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c46e1a4-84cc-483e-8b1d-7786349ef4d5-istio-podinfo\") pod \"e2e-worker-single-main-a52dfa-openshift-default-84c659644f4r87k\" (UID: \"2c46e1a4-84cc-483e-8b1d-7786349ef4d5\") " pod="openshift-ingress/e2e-worker-single-main-a52dfa-openshift-default-84c659644f4r87k"
Nov 02 20:30:38.973255 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:30:38.973161    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c46e1a4-84cc-483e-8b1d-7786349ef4d5-istiod-ca-cert\") pod \"e2e-worker-single-main-a52dfa-openshift-default-84c659644f4r87k\" (UID: \"2c46e1a4-84cc-483e-8b1d-7786349ef4d5\") " pod="openshift-ingress/e2e-worker-single-main-a52dfa-openshift-default-84c659644f4r87k"
Nov 02 20:30:38.973255 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:30:38.973208    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c46e1a4-84cc-483e-8b1d-7786349ef4d5-credential-socket\") pod \"e2e-worker-single-main-a52dfa-openshift-default-84c659644f4r87k\" (UID: \"2c46e1a4-84cc-483e-8b1d-7786349ef4d5\") " pod="openshift-ingress/e2e-worker-single-main-a52dfa-openshift-default-84c659644f4r87k"
Nov 02 20:30:38.973255 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:30:38.973253    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c46e1a4-84cc-483e-8b1d-7786349ef4d5-workload-certs\") pod \"e2e-worker-single-main-a52dfa-openshift-default-84c659644f4r87k\" (UID: \"2c46e1a4-84cc-483e-8b1d-7786349ef4d5\") " pod="openshift-ingress/e2e-worker-single-main-a52dfa-openshift-default-84c659644f4r87k"
Nov 02 20:30:38.973506 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:30:38.973333    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c46e1a4-84cc-483e-8b1d-7786349ef4d5-istio-data\") pod \"e2e-worker-single-main-a52dfa-openshift-default-84c659644f4r87k\" (UID: \"2c46e1a4-84cc-483e-8b1d-7786349ef4d5\") " pod="openshift-ingress/e2e-worker-single-main-a52dfa-openshift-default-84c659644f4r87k"
Nov 02 20:30:38.973506 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:30:38.973360    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c46e1a4-84cc-483e-8b1d-7786349ef4d5-workload-socket\") pod \"e2e-worker-single-main-a52dfa-openshift-default-84c659644f4r87k\" (UID: \"2c46e1a4-84cc-483e-8b1d-7786349ef4d5\") " pod="openshift-ingress/e2e-worker-single-main-a52dfa-openshift-default-84c659644f4r87k"
Nov 02 20:30:38.973621 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:30:38.973559    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c46e1a4-84cc-483e-8b1d-7786349ef4d5-credential-socket\") pod \"e2e-worker-single-main-a52dfa-openshift-default-84c659644f4r87k\" (UID: \"2c46e1a4-84cc-483e-8b1d-7786349ef4d5\") " pod="openshift-ingress/e2e-worker-single-main-a52dfa-openshift-default-84c659644f4r87k"
Nov 02 20:30:38.973682 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:30:38.973616    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c46e1a4-84cc-483e-8b1d-7786349ef4d5-workload-certs\") pod \"e2e-worker-single-main-a52dfa-openshift-default-84c659644f4r87k\" (UID: \"2c46e1a4-84cc-483e-8b1d-7786349ef4d5\") " pod="openshift-ingress/e2e-worker-single-main-a52dfa-openshift-default-84c659644f4r87k"
Nov 02 20:30:38.973765 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:30:38.973746    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c46e1a4-84cc-483e-8b1d-7786349ef4d5-istio-data\") pod \"e2e-worker-single-main-a52dfa-openshift-default-84c659644f4r87k\" (UID: \"2c46e1a4-84cc-483e-8b1d-7786349ef4d5\") " pod="openshift-ingress/e2e-worker-single-main-a52dfa-openshift-default-84c659644f4r87k"
Nov 02 20:30:38.974354 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:30:38.974333    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c46e1a4-84cc-483e-8b1d-7786349ef4d5-istiod-ca-cert\") pod \"e2e-worker-single-main-a52dfa-openshift-default-84c659644f4r87k\" (UID: \"2c46e1a4-84cc-483e-8b1d-7786349ef4d5\") " pod="openshift-ingress/e2e-worker-single-main-a52dfa-openshift-default-84c659644f4r87k"
Nov 02 20:30:38.975482 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:30:38.975455    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c46e1a4-84cc-483e-8b1d-7786349ef4d5-istio-envoy\") pod \"e2e-worker-single-main-a52dfa-openshift-default-84c659644f4r87k\" (UID: \"2c46e1a4-84cc-483e-8b1d-7786349ef4d5\") " pod="openshift-ingress/e2e-worker-single-main-a52dfa-openshift-default-84c659644f4r87k"
Nov 02 20:30:38.975809 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:30:38.975790    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c46e1a4-84cc-483e-8b1d-7786349ef4d5-istio-podinfo\") pod \"e2e-worker-single-main-a52dfa-openshift-default-84c659644f4r87k\" (UID: \"2c46e1a4-84cc-483e-8b1d-7786349ef4d5\") " pod="openshift-ingress/e2e-worker-single-main-a52dfa-openshift-default-84c659644f4r87k"
Nov 02 20:30:39.025227 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:30:39.025204    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c46e1a4-84cc-483e-8b1d-7786349ef4d5-istio-token\") pod \"e2e-worker-single-main-a52dfa-openshift-default-84c659644f4r87k\" (UID: \"2c46e1a4-84cc-483e-8b1d-7786349ef4d5\") " pod="openshift-ingress/e2e-worker-single-main-a52dfa-openshift-default-84c659644f4r87k"
Nov 02 20:30:39.025439 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:30:39.025419    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skwm7\" (UniqueName: \"kubernetes.io/projected/2c46e1a4-84cc-483e-8b1d-7786349ef4d5-kube-api-access-skwm7\") pod \"e2e-worker-single-main-a52dfa-openshift-default-84c659644f4r87k\" (UID: \"2c46e1a4-84cc-483e-8b1d-7786349ef4d5\") " pod="openshift-ingress/e2e-worker-single-main-a52dfa-openshift-default-84c659644f4r87k"
Nov 02 20:30:39.031087 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:30:39.031065    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-a52dfa-openshift-default-84c659644f4r87k"
Nov 02 20:30:39.442654 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:30:39.442457    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-a52dfa-openshift-default-84c659644f4r87k"]
Nov 02 20:30:39.445333 ip-10-0-133-9 kubenswrapper[4463]: W1102 20:30:39.445308    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c46e1a4_84cc_483e_8b1d_7786349ef4d5.slice/crio-a1dc1d34b2316eea8ae7f7ba38d49c7f479779d13c6ec74cf86d6237bfab2314 WatchSource:0}: Error finding container a1dc1d34b2316eea8ae7f7ba38d49c7f479779d13c6ec74cf86d6237bfab2314: Status 404 returned error can't find the container with id a1dc1d34b2316eea8ae7f7ba38d49c7f479779d13c6ec74cf86d6237bfab2314
Nov 02 20:30:39.447319 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:30:39.447289    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:30:39.447396 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:30:39.447349    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:30:39.447396 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:30:39.447377    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:30:40.437992 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:30:40.437948    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-a52dfa-openshift-default-84c659644f4r87k" event={"ID":"2c46e1a4-84cc-483e-8b1d-7786349ef4d5","Type":"ContainerStarted","Data":"90019f02c4a5485b58444d4564769fec4b9635d6f43d2202828bc001eec00f27"}
Nov 02 20:30:40.438309 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:30:40.438038    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-a52dfa-openshift-default-84c659644f4r87k" event={"ID":"2c46e1a4-84cc-483e-8b1d-7786349ef4d5","Type":"ContainerStarted","Data":"a1dc1d34b2316eea8ae7f7ba38d49c7f479779d13c6ec74cf86d6237bfab2314"}
Nov 02 20:30:40.545951 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:30:40.545890    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-a52dfa-openshift-default-84c659644f4r87k" podStartSLOduration=2.545873905 podStartE2EDuration="2.545873905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:30:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:30:40.545067352 +0000 UTC m=+3747.152101121" watchObservedRunningTime="2026-11-02 20:30:40.545873905 +0000 UTC m=+3747.152907675"
Nov 02 20:30:41.031245 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:30:41.031210    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-a52dfa-openshift-default-84c659644f4r87k"
Nov 02 20:30:41.036672 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:30:41.036643    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-a52dfa-openshift-default-84c659644f4r87k"
Nov 02 20:30:41.442909 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:30:41.442883    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-a52dfa-openshift-default-84c659644f4r87k"
Nov 02 20:30:41.443961 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:30:41.443938    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-a52dfa-openshift-default-84c659644f4r87k"
Nov 02 20:30:47.039057 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:30:47.039007    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-a52dfa-77fd7lwnps"]
Nov 02 20:30:47.039492 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:30:47.039475    4463 cpu_manager.go:414] "containerMap: removing container" podUID="2da1ff8e-3060-42a6-896e-75e07469e4b5" containerName="istio-proxy"
Nov 02 20:30:47.039541 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:30:47.039494    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2da1ff8e-3060-42a6-896e-75e07469e4b5" containerName="istio-proxy"
Nov 02 20:30:47.042682 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:30:47.042667    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a52dfa-77fd7lwnps"
Nov 02 20:30:47.096719 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:30:47.096684    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-a52dfa-77fd7lwnps"]
Nov 02 20:30:47.140744 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:30:47.140712    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-6d135020-dockercfg-hj45f\""
Nov 02 20:30:47.140997 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:30:47.140773    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-a52dfa\""
Nov 02 20:30:47.146390 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:30:47.146361    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/570c4f8a-38a7-4415-bbb9-b089a6b2af00-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-a52dfa-77fd7lwnps\" (UID: \"570c4f8a-38a7-4415-bbb9-b089a6b2af00\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a52dfa-77fd7lwnps"
Nov 02 20:30:47.146522 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:30:47.146442    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbbgp\" (UniqueName: \"kubernetes.io/projected/570c4f8a-38a7-4415-bbb9-b089a6b2af00-kube-api-access-rbbgp\") pod \"payload-pre-processing-e2e-worker-single-main-a52dfa-77fd7lwnps\" (UID: \"570c4f8a-38a7-4415-bbb9-b089a6b2af00\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a52dfa-77fd7lwnps"
Nov 02 20:30:47.247370 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:30:47.247328    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/570c4f8a-38a7-4415-bbb9-b089a6b2af00-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-a52dfa-77fd7lwnps\" (UID: \"570c4f8a-38a7-4415-bbb9-b089a6b2af00\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a52dfa-77fd7lwnps"
Nov 02 20:30:47.247608 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:30:47.247432    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbbgp\" (UniqueName: \"kubernetes.io/projected/570c4f8a-38a7-4415-bbb9-b089a6b2af00-kube-api-access-rbbgp\") pod \"payload-pre-processing-e2e-worker-single-main-a52dfa-77fd7lwnps\" (UID: \"570c4f8a-38a7-4415-bbb9-b089a6b2af00\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a52dfa-77fd7lwnps"
Nov 02 20:30:47.248143 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:30:47.248113    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/570c4f8a-38a7-4415-bbb9-b089a6b2af00-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-a52dfa-77fd7lwnps\" (UID: \"570c4f8a-38a7-4415-bbb9-b089a6b2af00\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a52dfa-77fd7lwnps"
Nov 02 20:30:47.299889 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:30:47.299829    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbbgp\" (UniqueName: \"kubernetes.io/projected/570c4f8a-38a7-4415-bbb9-b089a6b2af00-kube-api-access-rbbgp\") pod \"payload-pre-processing-e2e-worker-single-main-a52dfa-77fd7lwnps\" (UID: \"570c4f8a-38a7-4415-bbb9-b089a6b2af00\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a52dfa-77fd7lwnps"
Nov 02 20:30:47.354963 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:30:47.354932    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a52dfa-77fd7lwnps"
Nov 02 20:30:47.716968 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:30:47.716926    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-a52dfa-77fd7lwnps"]
Nov 02 20:30:47.719967 ip-10-0-133-9 kubenswrapper[4463]: W1102 20:30:47.719930    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod570c4f8a_38a7_4415_bbb9_b089a6b2af00.slice/crio-23525edfb252ea121a0aa2f5d526218d4a81a2191123159e406af60a7f64d154 WatchSource:0}: Error finding container 23525edfb252ea121a0aa2f5d526218d4a81a2191123159e406af60a7f64d154: Status 404 returned error can't find the container with id 23525edfb252ea121a0aa2f5d526218d4a81a2191123159e406af60a7f64d154
Nov 02 20:30:48.476460 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:30:48.476422    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a52dfa-77fd7lwnps" event={"ID":"570c4f8a-38a7-4415-bbb9-b089a6b2af00","Type":"ContainerStarted","Data":"3ba2aab3fce3aa9d38e337a2f8e3996a568dfd645b4b76f98cf41ea7c570d993"}
Nov 02 20:30:48.476861 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:30:48.476478    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a52dfa-77fd7lwnps" event={"ID":"570c4f8a-38a7-4415-bbb9-b089a6b2af00","Type":"ContainerStarted","Data":"23525edfb252ea121a0aa2f5d526218d4a81a2191123159e406af60a7f64d154"}
Nov 02 20:30:48.476861 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:30:48.476532    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a52dfa-77fd7lwnps"
Nov 02 20:30:48.583558 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:30:48.583496    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a52dfa-77fd7lwnps" podStartSLOduration=1.583477695 podStartE2EDuration="1.583477695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:30:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:30:48.581357975 +0000 UTC m=+3755.188391744" watchObservedRunningTime="2026-11-02 20:30:48.583477695 +0000 UTC m=+3755.190511462"
Nov 02 20:30:48.906295 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:30:48.906219    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a52dfa-86f5cdf6f5-vbqdr"]
Nov 02 20:30:48.909695 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:30:48.909677    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a52dfa-86f5cdf6f5-vbqdr"
Nov 02 20:30:48.960077 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:30:48.960049    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-a52dfa-serving-cert\""
Nov 02 20:30:48.962858 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:30:48.962824    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4x9q\" (UniqueName: \"kubernetes.io/projected/e31394c0-4327-47e7-9bcc-ca2b3808bb61-kube-api-access-t4x9q\") pod \"maas-api-e2e-worker-single-main-a52dfa-86f5cdf6f5-vbqdr\" (UID: \"e31394c0-4327-47e7-9bcc-ca2b3808bb61\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a52dfa-86f5cdf6f5-vbqdr"
Nov 02 20:30:48.963021 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:30:48.962882    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e31394c0-4327-47e7-9bcc-ca2b3808bb61-metrics-certs\") pod \"maas-api-e2e-worker-single-main-a52dfa-86f5cdf6f5-vbqdr\" (UID: \"e31394c0-4327-47e7-9bcc-ca2b3808bb61\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a52dfa-86f5cdf6f5-vbqdr"
Nov 02 20:30:48.963021 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:30:48.962956    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e31394c0-4327-47e7-9bcc-ca2b3808bb61-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-a52dfa-86f5cdf6f5-vbqdr\" (UID: \"e31394c0-4327-47e7-9bcc-ca2b3808bb61\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a52dfa-86f5cdf6f5-vbqdr"
Nov 02 20:30:48.963021 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:30:48.962997    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e31394c0-4327-47e7-9bcc-ca2b3808bb61-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-a52dfa-86f5cdf6f5-vbqdr\" (UID: \"e31394c0-4327-47e7-9bcc-ca2b3808bb61\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a52dfa-86f5cdf6f5-vbqdr"
Nov 02 20:30:49.007987 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:30:49.007951    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a52dfa-86f5cdf6f5-vbqdr"]
Nov 02 20:30:49.063800 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:30:49.063758    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4x9q\" (UniqueName: \"kubernetes.io/projected/e31394c0-4327-47e7-9bcc-ca2b3808bb61-kube-api-access-t4x9q\") pod \"maas-api-e2e-worker-single-main-a52dfa-86f5cdf6f5-vbqdr\" (UID: \"e31394c0-4327-47e7-9bcc-ca2b3808bb61\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a52dfa-86f5cdf6f5-vbqdr"
Nov 02 20:30:49.063800 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:30:49.063803    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e31394c0-4327-47e7-9bcc-ca2b3808bb61-metrics-certs\") pod \"maas-api-e2e-worker-single-main-a52dfa-86f5cdf6f5-vbqdr\" (UID: \"e31394c0-4327-47e7-9bcc-ca2b3808bb61\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a52dfa-86f5cdf6f5-vbqdr"
Nov 02 20:30:49.064084 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:30:49.063829    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e31394c0-4327-47e7-9bcc-ca2b3808bb61-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-a52dfa-86f5cdf6f5-vbqdr\" (UID: \"e31394c0-4327-47e7-9bcc-ca2b3808bb61\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a52dfa-86f5cdf6f5-vbqdr"
Nov 02 20:30:49.064084 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:30:49.063980    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e31394c0-4327-47e7-9bcc-ca2b3808bb61-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-a52dfa-86f5cdf6f5-vbqdr\" (UID: \"e31394c0-4327-47e7-9bcc-ca2b3808bb61\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a52dfa-86f5cdf6f5-vbqdr"
Nov 02 20:30:49.064195 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:30:49.064164    4463 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a52dfa-serving-cert: secret "maas-api-e2e-worker-single-main-a52dfa-serving-cert" not found
Nov 02 20:30:49.064263 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:30:49.064246    4463 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e31394c0-4327-47e7-9bcc-ca2b3808bb61-maas-api-tls podName:e31394c0-4327-47e7-9bcc-ca2b3808bb61 nodeName:}" failed. No retries permitted until 2026-11-02 20:30:49.564222627 +0000 UTC m=+3756.171256379 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e31394c0-4327-47e7-9bcc-ca2b3808bb61-maas-api-tls") pod "maas-api-e2e-worker-single-main-a52dfa-86f5cdf6f5-vbqdr" (UID: "e31394c0-4327-47e7-9bcc-ca2b3808bb61") : secret "maas-api-e2e-worker-single-main-a52dfa-serving-cert" not found
Nov 02 20:30:49.064593 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:30:49.064571    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e31394c0-4327-47e7-9bcc-ca2b3808bb61-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-a52dfa-86f5cdf6f5-vbqdr\" (UID: \"e31394c0-4327-47e7-9bcc-ca2b3808bb61\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a52dfa-86f5cdf6f5-vbqdr"
Nov 02 20:30:49.066425 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:30:49.066399    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e31394c0-4327-47e7-9bcc-ca2b3808bb61-metrics-certs\") pod \"maas-api-e2e-worker-single-main-a52dfa-86f5cdf6f5-vbqdr\" (UID: \"e31394c0-4327-47e7-9bcc-ca2b3808bb61\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a52dfa-86f5cdf6f5-vbqdr"
Nov 02 20:30:49.118961 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:30:49.118926    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4x9q\" (UniqueName: \"kubernetes.io/projected/e31394c0-4327-47e7-9bcc-ca2b3808bb61-kube-api-access-t4x9q\") pod \"maas-api-e2e-worker-single-main-a52dfa-86f5cdf6f5-vbqdr\" (UID: \"e31394c0-4327-47e7-9bcc-ca2b3808bb61\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a52dfa-86f5cdf6f5-vbqdr"
Nov 02 20:30:49.569989 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:30:49.569941    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e31394c0-4327-47e7-9bcc-ca2b3808bb61-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-a52dfa-86f5cdf6f5-vbqdr\" (UID: \"e31394c0-4327-47e7-9bcc-ca2b3808bb61\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a52dfa-86f5cdf6f5-vbqdr"
Nov 02 20:30:49.572410 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:30:49.572378    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e31394c0-4327-47e7-9bcc-ca2b3808bb61-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-a52dfa-86f5cdf6f5-vbqdr\" (UID: \"e31394c0-4327-47e7-9bcc-ca2b3808bb61\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a52dfa-86f5cdf6f5-vbqdr"
Nov 02 20:30:49.821839 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:30:49.821741    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a52dfa-86f5cdf6f5-vbqdr"
Nov 02 20:30:50.186129 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:30:50.186099    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a52dfa-86f5cdf6f5-vbqdr"]
Nov 02 20:30:50.188773 ip-10-0-133-9 kubenswrapper[4463]: W1102 20:30:50.188745    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode31394c0_4327_47e7_9bcc_ca2b3808bb61.slice/crio-2440691e11aa207859057143b32d4a93359b502e44a0e87ef02b6c6f00b56b1a WatchSource:0}: Error finding container 2440691e11aa207859057143b32d4a93359b502e44a0e87ef02b6c6f00b56b1a: Status 404 returned error can't find the container with id 2440691e11aa207859057143b32d4a93359b502e44a0e87ef02b6c6f00b56b1a
Nov 02 20:30:50.486493 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:30:50.486412    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a52dfa-86f5cdf6f5-vbqdr" event={"ID":"e31394c0-4327-47e7-9bcc-ca2b3808bb61","Type":"ContainerStarted","Data":"2440691e11aa207859057143b32d4a93359b502e44a0e87ef02b6c6f00b56b1a"}
Nov 02 20:30:51.500112 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:30:51.500052    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a52dfa-86f5cdf6f5-vbqdr" event={"ID":"e31394c0-4327-47e7-9bcc-ca2b3808bb61","Type":"ContainerStarted","Data":"0994d80c56c5b8133200c5e79efd8d67acff00c2f8df07e8d6eba27771573f1f"}
Nov 02 20:30:51.500630 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:30:51.500247    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a52dfa-86f5cdf6f5-vbqdr"
Nov 02 20:30:51.607692 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:30:51.607632    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a52dfa-86f5cdf6f5-vbqdr" podStartSLOduration=2.805113826 podStartE2EDuration="3.607616038s" totalImagesPullingTime="802.502212ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:30:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:30:51.605929373 +0000 UTC m=+3758.212963141" watchObservedRunningTime="2026-11-02 20:30:51.607616038 +0000 UTC m=+3758.214649806"
Nov 02 20:30:56.366188 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:30:56.366147    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 20:30:57.509999 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:30:57.509969    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a52dfa-86f5cdf6f5-vbqdr"
Nov 02 20:30:59.484307 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:30:59.484272    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a52dfa-77fd7lwnps"
Nov 02 20:31:04.294431 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:31:04.294380    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 20:31:05.568141 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:31:05.568052    4463 generic.go:356] "Generic (PLEG): container finished" podID="2c46e1a4-84cc-483e-8b1d-7786349ef4d5" containerID="90019f02c4a5485b58444d4564769fec4b9635d6f43d2202828bc001eec00f27" exitCode=0
Nov 02 20:31:05.568693 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:31:05.568130    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-a52dfa-openshift-default-84c659644f4r87k" event={"ID":"2c46e1a4-84cc-483e-8b1d-7786349ef4d5","Type":"ContainerDied","Data":"90019f02c4a5485b58444d4564769fec4b9635d6f43d2202828bc001eec00f27"}
Nov 02 20:31:05.568693 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:31:05.568611    4463 scope.go:122] "RemoveContainer" containerID="90019f02c4a5485b58444d4564769fec4b9635d6f43d2202828bc001eec00f27"
Nov 02 20:31:05.569827 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:31:05.569797    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:31:05.569892 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:31:05.569872    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:31:05.569929 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:31:05.569904    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:31:06.579878 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:31:06.579845    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-a52dfa-openshift-default-84c659644f4r87k" event={"ID":"2c46e1a4-84cc-483e-8b1d-7786349ef4d5","Type":"ContainerStarted","Data":"9582d0e23a16760762884d59efafd70b64c1d43d58e0c7bb845fb2c8b7a68d66"}
Nov 02 20:31:07.031929 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:31:07.031892    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-a52dfa-openshift-default-84c659644f4r87k"
Nov 02 20:31:07.033202 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:31:07.033177    4463 patch_prober.go:29] interesting pod/e2e-worker-single-main-a52dfa-openshift-default-84c659644f4r87k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Nov 02 20:31:07.033264 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:31:07.033237    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-a52dfa-openshift-default-84c659644f4r87k" podUID="2c46e1a4-84cc-483e-8b1d-7786349ef4d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Nov 02 20:31:08.031833 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:31:08.031797    4463 patch_prober.go:29] interesting pod/e2e-worker-single-main-a52dfa-openshift-default-84c659644f4r87k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Nov 02 20:31:08.032352 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:31:08.031870    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-a52dfa-openshift-default-84c659644f4r87k" podUID="2c46e1a4-84cc-483e-8b1d-7786349ef4d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Nov 02 20:31:09.032033 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:31:09.031976    4463 patch_prober.go:29] interesting pod/e2e-worker-single-main-a52dfa-openshift-default-84c659644f4r87k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Nov 02 20:31:09.032425 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:31:09.032059    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-a52dfa-openshift-default-84c659644f4r87k" podUID="2c46e1a4-84cc-483e-8b1d-7786349ef4d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Nov 02 20:31:10.032543 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:31:10.032502    4463 patch_prober.go:29] interesting pod/e2e-worker-single-main-a52dfa-openshift-default-84c659644f4r87k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Nov 02 20:31:10.032963 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:31:10.032560    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-a52dfa-openshift-default-84c659644f4r87k" podUID="2c46e1a4-84cc-483e-8b1d-7786349ef4d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Nov 02 20:31:10.640206 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:31:10.640175    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 20:31:11.031879 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:31:11.031840    4463 patch_prober.go:29] interesting pod/e2e-worker-single-main-a52dfa-openshift-default-84c659644f4r87k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Nov 02 20:31:11.032093 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:31:11.031898    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-a52dfa-openshift-default-84c659644f4r87k" podUID="2c46e1a4-84cc-483e-8b1d-7786349ef4d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Nov 02 20:31:11.443645 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:31:11.443611    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-a52dfa-openshift-default-84c659644f4r87k"
Nov 02 20:31:12.093539 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:31:12.093503    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-a52dfa-openshift-default-84c659644f4r87k"
Nov 02 20:31:12.095526 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:31:12.095503    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-a52dfa-openshift-default-84c659644f4r87k"
Nov 02 20:31:34.293704 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:31:34.293659    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 20:31:36.238983 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:31:36.238954    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 20:32:04.251997 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:32:04.251961    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 20:32:04.293433 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:32:04.293401    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 20:32:24.575149 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:32:24.575107    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-a52dfa-77fd7lwnps"]
Nov 02 20:32:26.343196 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:32:26.343155    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a52dfa-86f5cdf6f5-vbqdr"]
Nov 02 20:32:26.343668 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:32:26.343365    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a52dfa-86f5cdf6f5-vbqdr" podUID="e31394c0-4327-47e7-9bcc-ca2b3808bb61" containerName="maas-api" containerID="cri-o://0994d80c56c5b8133200c5e79efd8d67acff00c2f8df07e8d6eba27771573f1f" gracePeriod=30
Nov 02 20:32:27.504987 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:32:27.504947    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a52dfa-86f5cdf6f5-vbqdr" podUID="e31394c0-4327-47e7-9bcc-ca2b3808bb61" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.141:8443/health\": dial tcp 10.133.0.141:8443: connect: connection refused"
Nov 02 20:32:27.598278 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:32:27.598246    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a52dfa-86f5cdf6f5-vbqdr"
Nov 02 20:32:27.644440 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:32:27.644408    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e31394c0-4327-47e7-9bcc-ca2b3808bb61-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e31394c0-4327-47e7-9bcc-ca2b3808bb61-maas-api-tls\") pod \"e31394c0-4327-47e7-9bcc-ca2b3808bb61\" (UID: \"e31394c0-4327-47e7-9bcc-ca2b3808bb61\") "
Nov 02 20:32:27.646518 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:32:27.646488    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e31394c0-4327-47e7-9bcc-ca2b3808bb61-maas-api-tls" pod "e31394c0-4327-47e7-9bcc-ca2b3808bb61" (UID: "e31394c0-4327-47e7-9bcc-ca2b3808bb61"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:32:27.745615 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:32:27.745585    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e31394c0-4327-47e7-9bcc-ca2b3808bb61-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e31394c0-4327-47e7-9bcc-ca2b3808bb61-metrics-certs\") pod \"e31394c0-4327-47e7-9bcc-ca2b3808bb61\" (UID: \"e31394c0-4327-47e7-9bcc-ca2b3808bb61\") "
Nov 02 20:32:27.745807 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:32:27.745626    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e31394c0-4327-47e7-9bcc-ca2b3808bb61-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e31394c0-4327-47e7-9bcc-ca2b3808bb61-openshift-service-ca\") pod \"e31394c0-4327-47e7-9bcc-ca2b3808bb61\" (UID: \"e31394c0-4327-47e7-9bcc-ca2b3808bb61\") "
Nov 02 20:32:27.745807 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:32:27.745663    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e31394c0-4327-47e7-9bcc-ca2b3808bb61-kube-api-access-t4x9q\" (UniqueName: \"kubernetes.io/projected/e31394c0-4327-47e7-9bcc-ca2b3808bb61-kube-api-access-t4x9q\") pod \"e31394c0-4327-47e7-9bcc-ca2b3808bb61\" (UID: \"e31394c0-4327-47e7-9bcc-ca2b3808bb61\") "
Nov 02 20:32:27.746082 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:32:27.746058    4463 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e31394c0-4327-47e7-9bcc-ca2b3808bb61-maas-api-tls\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:32:27.746172 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:32:27.746098    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e31394c0-4327-47e7-9bcc-ca2b3808bb61-openshift-service-ca" pod "e31394c0-4327-47e7-9bcc-ca2b3808bb61" (UID: "e31394c0-4327-47e7-9bcc-ca2b3808bb61"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:32:27.747886 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:32:27.747864    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e31394c0-4327-47e7-9bcc-ca2b3808bb61-kube-api-access-t4x9q" pod "e31394c0-4327-47e7-9bcc-ca2b3808bb61" (UID: "e31394c0-4327-47e7-9bcc-ca2b3808bb61"). InnerVolumeSpecName "kube-api-access-t4x9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:32:27.747976 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:32:27.747926    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e31394c0-4327-47e7-9bcc-ca2b3808bb61-metrics-certs" pod "e31394c0-4327-47e7-9bcc-ca2b3808bb61" (UID: "e31394c0-4327-47e7-9bcc-ca2b3808bb61"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:32:27.846708 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:32:27.846677    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4x9q\" (UniqueName: \"kubernetes.io/projected/e31394c0-4327-47e7-9bcc-ca2b3808bb61-kube-api-access-t4x9q\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:32:27.846708 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:32:27.846706    4463 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e31394c0-4327-47e7-9bcc-ca2b3808bb61-metrics-certs\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:32:27.846904 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:32:27.846716    4463 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e31394c0-4327-47e7-9bcc-ca2b3808bb61-openshift-service-ca\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:32:27.958739 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:32:27.958707    4463 generic.go:356] "Generic (PLEG): container finished" podID="e31394c0-4327-47e7-9bcc-ca2b3808bb61" containerID="0994d80c56c5b8133200c5e79efd8d67acff00c2f8df07e8d6eba27771573f1f" exitCode=0
Nov 02 20:32:27.958915 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:32:27.958754    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a52dfa-86f5cdf6f5-vbqdr"
Nov 02 20:32:27.958915 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:32:27.958780    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a52dfa-86f5cdf6f5-vbqdr" event={"ID":"e31394c0-4327-47e7-9bcc-ca2b3808bb61","Type":"ContainerDied","Data":"0994d80c56c5b8133200c5e79efd8d67acff00c2f8df07e8d6eba27771573f1f"}
Nov 02 20:32:27.958915 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:32:27.958821    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a52dfa-86f5cdf6f5-vbqdr" event={"ID":"e31394c0-4327-47e7-9bcc-ca2b3808bb61","Type":"ContainerDied","Data":"2440691e11aa207859057143b32d4a93359b502e44a0e87ef02b6c6f00b56b1a"}
Nov 02 20:32:27.958915 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:32:27.958842    4463 scope.go:122] "RemoveContainer" containerID="0994d80c56c5b8133200c5e79efd8d67acff00c2f8df07e8d6eba27771573f1f"
Nov 02 20:32:27.967720 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:32:27.967701    4463 scope.go:122] "RemoveContainer" containerID="0994d80c56c5b8133200c5e79efd8d67acff00c2f8df07e8d6eba27771573f1f"
Nov 02 20:32:27.967969 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:32:27.967953    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0994d80c56c5b8133200c5e79efd8d67acff00c2f8df07e8d6eba27771573f1f\": container with ID starting with 0994d80c56c5b8133200c5e79efd8d67acff00c2f8df07e8d6eba27771573f1f not found: ID does not exist" containerID="0994d80c56c5b8133200c5e79efd8d67acff00c2f8df07e8d6eba27771573f1f"
Nov 02 20:32:27.968022 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:32:27.967976    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0994d80c56c5b8133200c5e79efd8d67acff00c2f8df07e8d6eba27771573f1f"} err="failed to get container status \"0994d80c56c5b8133200c5e79efd8d67acff00c2f8df07e8d6eba27771573f1f\": rpc error: code = NotFound desc = could not find container \"0994d80c56c5b8133200c5e79efd8d67acff00c2f8df07e8d6eba27771573f1f\": container with ID starting with 0994d80c56c5b8133200c5e79efd8d67acff00c2f8df07e8d6eba27771573f1f not found: ID does not exist"
Nov 02 20:32:28.110415 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:32:28.110343    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a52dfa-86f5cdf6f5-vbqdr"]
Nov 02 20:32:28.156545 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:32:28.156509    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a52dfa-86f5cdf6f5-vbqdr"]
Nov 02 20:32:29.575944 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:32:29.575901    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a52dfa-77fd7lwnps" podUID="570c4f8a-38a7-4415-bbb9-b089a6b2af00" containerName="payload-pre-processing" containerID="cri-o://3ba2aab3fce3aa9d38e337a2f8e3996a568dfd645b4b76f98cf41ea7c570d993" gracePeriod=25
Nov 02 20:32:29.584819 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:32:29.584786    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e31394c0-4327-47e7-9bcc-ca2b3808bb61" path="/var/lib/kubelet/pods/e31394c0-4327-47e7-9bcc-ca2b3808bb61/volumes"
Nov 02 20:32:31.022449 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:32:31.022409    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 20:32:34.293396 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:32:34.293351    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 20:32:39.482565 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:32:39.482528    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a52dfa-77fd7lwnps" podUID="570c4f8a-38a7-4415-bbb9-b089a6b2af00" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 20:32:40.246610 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:32:40.246573    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-a52dfa-openshift-default-84c659644f4r87k"]
Nov 02 20:32:40.246835 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:32:40.246812    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-a52dfa-openshift-default-84c659644f4r87k" podUID="2c46e1a4-84cc-483e-8b1d-7786349ef4d5" containerName="istio-proxy" containerID="cri-o://9582d0e23a16760762884d59efafd70b64c1d43d58e0c7bb845fb2c8b7a68d66" gracePeriod=30
Nov 02 20:32:42.096322 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:32:42.096278    4463 patch_prober.go:29] interesting pod/e2e-worker-single-main-a52dfa-openshift-default-84c659644f4r87k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 20:32:42.096718 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:32:42.096349    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-a52dfa-openshift-default-84c659644f4r87k" podUID="2c46e1a4-84cc-483e-8b1d-7786349ef4d5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 20:32:45.497439 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:32:45.497408    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-a52dfa-openshift-default-84c659644f4r87k"
Nov 02 20:32:45.612211 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:32:45.612138    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c46e1a4-84cc-483e-8b1d-7786349ef4d5-istio-token\" (UniqueName: \"kubernetes.io/projected/2c46e1a4-84cc-483e-8b1d-7786349ef4d5-istio-token\") pod \"2c46e1a4-84cc-483e-8b1d-7786349ef4d5\" (UID: \"2c46e1a4-84cc-483e-8b1d-7786349ef4d5\") "
Nov 02 20:32:45.612211 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:32:45.612174    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2c46e1a4-84cc-483e-8b1d-7786349ef4d5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c46e1a4-84cc-483e-8b1d-7786349ef4d5-istio-podinfo\") pod \"2c46e1a4-84cc-483e-8b1d-7786349ef4d5\" (UID: \"2c46e1a4-84cc-483e-8b1d-7786349ef4d5\") "
Nov 02 20:32:45.612454 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:32:45.612246    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c46e1a4-84cc-483e-8b1d-7786349ef4d5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c46e1a4-84cc-483e-8b1d-7786349ef4d5-istiod-ca-cert\") pod \"2c46e1a4-84cc-483e-8b1d-7786349ef4d5\" (UID: \"2c46e1a4-84cc-483e-8b1d-7786349ef4d5\") "
Nov 02 20:32:45.612454 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:32:45.612305    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c46e1a4-84cc-483e-8b1d-7786349ef4d5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c46e1a4-84cc-483e-8b1d-7786349ef4d5-istio-envoy\") pod \"2c46e1a4-84cc-483e-8b1d-7786349ef4d5\" (UID: \"2c46e1a4-84cc-483e-8b1d-7786349ef4d5\") "
Nov 02 20:32:45.612454 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:32:45.612327    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c46e1a4-84cc-483e-8b1d-7786349ef4d5-kube-api-access-skwm7\" (UniqueName: \"kubernetes.io/projected/2c46e1a4-84cc-483e-8b1d-7786349ef4d5-kube-api-access-skwm7\") pod \"2c46e1a4-84cc-483e-8b1d-7786349ef4d5\" (UID: \"2c46e1a4-84cc-483e-8b1d-7786349ef4d5\") "
Nov 02 20:32:45.612454 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:32:45.612364    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c46e1a4-84cc-483e-8b1d-7786349ef4d5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c46e1a4-84cc-483e-8b1d-7786349ef4d5-credential-socket\") pod \"2c46e1a4-84cc-483e-8b1d-7786349ef4d5\" (UID: \"2c46e1a4-84cc-483e-8b1d-7786349ef4d5\") "
Nov 02 20:32:45.612454 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:32:45.612402    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c46e1a4-84cc-483e-8b1d-7786349ef4d5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c46e1a4-84cc-483e-8b1d-7786349ef4d5-istio-data\") pod \"2c46e1a4-84cc-483e-8b1d-7786349ef4d5\" (UID: \"2c46e1a4-84cc-483e-8b1d-7786349ef4d5\") "
Nov 02 20:32:45.612454 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:32:45.612434    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c46e1a4-84cc-483e-8b1d-7786349ef4d5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c46e1a4-84cc-483e-8b1d-7786349ef4d5-workload-certs\") pod \"2c46e1a4-84cc-483e-8b1d-7786349ef4d5\" (UID: \"2c46e1a4-84cc-483e-8b1d-7786349ef4d5\") "
Nov 02 20:32:45.612791 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:32:45.612462    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c46e1a4-84cc-483e-8b1d-7786349ef4d5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c46e1a4-84cc-483e-8b1d-7786349ef4d5-workload-socket\") pod \"2c46e1a4-84cc-483e-8b1d-7786349ef4d5\" (UID: \"2c46e1a4-84cc-483e-8b1d-7786349ef4d5\") "
Nov 02 20:32:45.612859 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:32:45.612808    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c46e1a4-84cc-483e-8b1d-7786349ef4d5-istio-data" pod "2c46e1a4-84cc-483e-8b1d-7786349ef4d5" (UID: "2c46e1a4-84cc-483e-8b1d-7786349ef4d5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:32:45.612918 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:32:45.612905    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c46e1a4-84cc-483e-8b1d-7786349ef4d5-credential-socket" pod "2c46e1a4-84cc-483e-8b1d-7786349ef4d5" (UID: "2c46e1a4-84cc-483e-8b1d-7786349ef4d5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:32:45.612976 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:32:45.612916    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c46e1a4-84cc-483e-8b1d-7786349ef4d5-istiod-ca-cert" pod "2c46e1a4-84cc-483e-8b1d-7786349ef4d5" (UID: "2c46e1a4-84cc-483e-8b1d-7786349ef4d5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:32:45.612976 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:32:45.612941    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c46e1a4-84cc-483e-8b1d-7786349ef4d5-workload-certs" pod "2c46e1a4-84cc-483e-8b1d-7786349ef4d5" (UID: "2c46e1a4-84cc-483e-8b1d-7786349ef4d5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:32:45.613115 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:32:45.613068    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c46e1a4-84cc-483e-8b1d-7786349ef4d5-workload-socket" pod "2c46e1a4-84cc-483e-8b1d-7786349ef4d5" (UID: "2c46e1a4-84cc-483e-8b1d-7786349ef4d5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:32:45.613115 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:32:45.613083    4463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c46e1a4-84cc-483e-8b1d-7786349ef4d5-istiod-ca-cert\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:32:45.613234 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:32:45.613118    4463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c46e1a4-84cc-483e-8b1d-7786349ef4d5-credential-socket\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:32:45.613234 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:32:45.613133    4463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c46e1a4-84cc-483e-8b1d-7786349ef4d5-istio-data\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:32:45.613234 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:32:45.613147    4463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c46e1a4-84cc-483e-8b1d-7786349ef4d5-workload-certs\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:32:45.614526 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:32:45.614498    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c46e1a4-84cc-483e-8b1d-7786349ef4d5-istio-token" pod "2c46e1a4-84cc-483e-8b1d-7786349ef4d5" (UID: "2c46e1a4-84cc-483e-8b1d-7786349ef4d5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:32:45.614635 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:32:45.614524    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c46e1a4-84cc-483e-8b1d-7786349ef4d5-kube-api-access-skwm7" pod "2c46e1a4-84cc-483e-8b1d-7786349ef4d5" (UID: "2c46e1a4-84cc-483e-8b1d-7786349ef4d5"). InnerVolumeSpecName "kube-api-access-skwm7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:32:45.614635 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:32:45.614532    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c46e1a4-84cc-483e-8b1d-7786349ef4d5-istio-envoy" pod "2c46e1a4-84cc-483e-8b1d-7786349ef4d5" (UID: "2c46e1a4-84cc-483e-8b1d-7786349ef4d5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:32:45.614711 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:32:45.614652    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2c46e1a4-84cc-483e-8b1d-7786349ef4d5-istio-podinfo" pod "2c46e1a4-84cc-483e-8b1d-7786349ef4d5" (UID: "2c46e1a4-84cc-483e-8b1d-7786349ef4d5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:32:45.713854 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:32:45.713815    4463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c46e1a4-84cc-483e-8b1d-7786349ef4d5-istio-envoy\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:32:45.713854 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:32:45.713847    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skwm7\" (UniqueName: \"kubernetes.io/projected/2c46e1a4-84cc-483e-8b1d-7786349ef4d5-kube-api-access-skwm7\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:32:45.714099 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:32:45.713865    4463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c46e1a4-84cc-483e-8b1d-7786349ef4d5-workload-socket\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:32:45.714099 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:32:45.713876    4463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c46e1a4-84cc-483e-8b1d-7786349ef4d5-istio-token\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:32:45.714099 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:32:45.713889    4463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c46e1a4-84cc-483e-8b1d-7786349ef4d5-istio-podinfo\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:32:46.043988 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:32:46.043952    4463 generic.go:356] "Generic (PLEG): container finished" podID="2c46e1a4-84cc-483e-8b1d-7786349ef4d5" containerID="9582d0e23a16760762884d59efafd70b64c1d43d58e0c7bb845fb2c8b7a68d66" exitCode=0
Nov 02 20:32:46.044223 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:32:46.044001    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-a52dfa-openshift-default-84c659644f4r87k"
Nov 02 20:32:46.044223 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:32:46.044042    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-a52dfa-openshift-default-84c659644f4r87k" event={"ID":"2c46e1a4-84cc-483e-8b1d-7786349ef4d5","Type":"ContainerDied","Data":"9582d0e23a16760762884d59efafd70b64c1d43d58e0c7bb845fb2c8b7a68d66"}
Nov 02 20:32:46.044223 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:32:46.044110    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-a52dfa-openshift-default-84c659644f4r87k" event={"ID":"2c46e1a4-84cc-483e-8b1d-7786349ef4d5","Type":"ContainerDied","Data":"a1dc1d34b2316eea8ae7f7ba38d49c7f479779d13c6ec74cf86d6237bfab2314"}
Nov 02 20:32:46.044223 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:32:46.044144    4463 scope.go:122] "RemoveContainer" containerID="9582d0e23a16760762884d59efafd70b64c1d43d58e0c7bb845fb2c8b7a68d66"
Nov 02 20:32:46.053643 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:32:46.053624    4463 scope.go:122] "RemoveContainer" containerID="90019f02c4a5485b58444d4564769fec4b9635d6f43d2202828bc001eec00f27"
Nov 02 20:32:46.062049 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:32:46.062004    4463 scope.go:122] "RemoveContainer" containerID="9582d0e23a16760762884d59efafd70b64c1d43d58e0c7bb845fb2c8b7a68d66"
Nov 02 20:32:46.062301 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:32:46.062284    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9582d0e23a16760762884d59efafd70b64c1d43d58e0c7bb845fb2c8b7a68d66\": container with ID starting with 9582d0e23a16760762884d59efafd70b64c1d43d58e0c7bb845fb2c8b7a68d66 not found: ID does not exist" containerID="9582d0e23a16760762884d59efafd70b64c1d43d58e0c7bb845fb2c8b7a68d66"
Nov 02 20:32:46.062348 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:32:46.062311    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9582d0e23a16760762884d59efafd70b64c1d43d58e0c7bb845fb2c8b7a68d66"} err="failed to get container status \"9582d0e23a16760762884d59efafd70b64c1d43d58e0c7bb845fb2c8b7a68d66\": rpc error: code = NotFound desc = could not find container \"9582d0e23a16760762884d59efafd70b64c1d43d58e0c7bb845fb2c8b7a68d66\": container with ID starting with 9582d0e23a16760762884d59efafd70b64c1d43d58e0c7bb845fb2c8b7a68d66 not found: ID does not exist"
Nov 02 20:32:46.062348 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:32:46.062330    4463 scope.go:122] "RemoveContainer" containerID="90019f02c4a5485b58444d4564769fec4b9635d6f43d2202828bc001eec00f27"
Nov 02 20:32:46.062571 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:32:46.062549    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90019f02c4a5485b58444d4564769fec4b9635d6f43d2202828bc001eec00f27\": container with ID starting with 90019f02c4a5485b58444d4564769fec4b9635d6f43d2202828bc001eec00f27 not found: ID does not exist" containerID="90019f02c4a5485b58444d4564769fec4b9635d6f43d2202828bc001eec00f27"
Nov 02 20:32:46.062617 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:32:46.062577    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90019f02c4a5485b58444d4564769fec4b9635d6f43d2202828bc001eec00f27"} err="failed to get container status \"90019f02c4a5485b58444d4564769fec4b9635d6f43d2202828bc001eec00f27\": rpc error: code = NotFound desc = could not find container \"90019f02c4a5485b58444d4564769fec4b9635d6f43d2202828bc001eec00f27\": container with ID starting with 90019f02c4a5485b58444d4564769fec4b9635d6f43d2202828bc001eec00f27 not found: ID does not exist"
Nov 02 20:32:46.195789 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:32:46.195754    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-a52dfa-openshift-default-84c659644f4r87k"]
Nov 02 20:32:46.242156 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:32:46.242114    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-a52dfa-openshift-default-84c659644f4r87k"]
Nov 02 20:32:47.584454 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:32:47.584415    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c46e1a4-84cc-483e-8b1d-7786349ef4d5" path="/var/lib/kubelet/pods/2c46e1a4-84cc-483e-8b1d-7786349ef4d5/volumes"
Nov 02 20:32:49.482725 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:32:49.482681    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a52dfa-77fd7lwnps" podUID="570c4f8a-38a7-4415-bbb9-b089a6b2af00" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 20:32:49.826505 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:32:49.826479    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a52dfa-77fd7lwnps"
Nov 02 20:32:49.954706 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:32:49.954677    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/570c4f8a-38a7-4415-bbb9-b089a6b2af00-kube-api-access-rbbgp\" (UniqueName: \"kubernetes.io/projected/570c4f8a-38a7-4415-bbb9-b089a6b2af00-kube-api-access-rbbgp\") pod \"570c4f8a-38a7-4415-bbb9-b089a6b2af00\" (UID: \"570c4f8a-38a7-4415-bbb9-b089a6b2af00\") "
Nov 02 20:32:49.954883 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:32:49.954783    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/570c4f8a-38a7-4415-bbb9-b089a6b2af00-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/570c4f8a-38a7-4415-bbb9-b089a6b2af00-plugins-config-volume\") pod \"570c4f8a-38a7-4415-bbb9-b089a6b2af00\" (UID: \"570c4f8a-38a7-4415-bbb9-b089a6b2af00\") "
Nov 02 20:32:49.955222 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:32:49.955199    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/570c4f8a-38a7-4415-bbb9-b089a6b2af00-plugins-config-volume" pod "570c4f8a-38a7-4415-bbb9-b089a6b2af00" (UID: "570c4f8a-38a7-4415-bbb9-b089a6b2af00"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:32:49.956876 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:32:49.956845    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/570c4f8a-38a7-4415-bbb9-b089a6b2af00-kube-api-access-rbbgp" pod "570c4f8a-38a7-4415-bbb9-b089a6b2af00" (UID: "570c4f8a-38a7-4415-bbb9-b089a6b2af00"). InnerVolumeSpecName "kube-api-access-rbbgp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:32:50.055826 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:32:50.055737    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbbgp\" (UniqueName: \"kubernetes.io/projected/570c4f8a-38a7-4415-bbb9-b089a6b2af00-kube-api-access-rbbgp\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:32:50.055826 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:32:50.055771    4463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/570c4f8a-38a7-4415-bbb9-b089a6b2af00-plugins-config-volume\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:32:50.065199 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:32:50.065168    4463 generic.go:356] "Generic (PLEG): container finished" podID="570c4f8a-38a7-4415-bbb9-b089a6b2af00" containerID="3ba2aab3fce3aa9d38e337a2f8e3996a568dfd645b4b76f98cf41ea7c570d993" exitCode=0
Nov 02 20:32:50.065339 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:32:50.065215    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a52dfa-77fd7lwnps"
Nov 02 20:32:50.065339 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:32:50.065244    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a52dfa-77fd7lwnps" event={"ID":"570c4f8a-38a7-4415-bbb9-b089a6b2af00","Type":"ContainerDied","Data":"3ba2aab3fce3aa9d38e337a2f8e3996a568dfd645b4b76f98cf41ea7c570d993"}
Nov 02 20:32:50.065339 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:32:50.065283    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a52dfa-77fd7lwnps" event={"ID":"570c4f8a-38a7-4415-bbb9-b089a6b2af00","Type":"ContainerDied","Data":"23525edfb252ea121a0aa2f5d526218d4a81a2191123159e406af60a7f64d154"}
Nov 02 20:32:50.065339 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:32:50.065305    4463 scope.go:122] "RemoveContainer" containerID="3ba2aab3fce3aa9d38e337a2f8e3996a568dfd645b4b76f98cf41ea7c570d993"
Nov 02 20:32:50.073779 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:32:50.073759    4463 scope.go:122] "RemoveContainer" containerID="3ba2aab3fce3aa9d38e337a2f8e3996a568dfd645b4b76f98cf41ea7c570d993"
Nov 02 20:32:50.074049 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:32:50.074029    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ba2aab3fce3aa9d38e337a2f8e3996a568dfd645b4b76f98cf41ea7c570d993\": container with ID starting with 3ba2aab3fce3aa9d38e337a2f8e3996a568dfd645b4b76f98cf41ea7c570d993 not found: ID does not exist" containerID="3ba2aab3fce3aa9d38e337a2f8e3996a568dfd645b4b76f98cf41ea7c570d993"
Nov 02 20:32:50.074102 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:32:50.074056    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ba2aab3fce3aa9d38e337a2f8e3996a568dfd645b4b76f98cf41ea7c570d993"} err="failed to get container status \"3ba2aab3fce3aa9d38e337a2f8e3996a568dfd645b4b76f98cf41ea7c570d993\": rpc error: code = NotFound desc = could not find container \"3ba2aab3fce3aa9d38e337a2f8e3996a568dfd645b4b76f98cf41ea7c570d993\": container with ID starting with 3ba2aab3fce3aa9d38e337a2f8e3996a568dfd645b4b76f98cf41ea7c570d993 not found: ID does not exist"
Nov 02 20:32:50.216234 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:32:50.216200    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-a52dfa-77fd7lwnps"]
Nov 02 20:32:50.221561 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:32:50.221537    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-a52dfa-77fd7lwnps"]
Nov 02 20:32:51.583275 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:32:51.583242    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="570c4f8a-38a7-4415-bbb9-b089a6b2af00" path="/var/lib/kubelet/pods/570c4f8a-38a7-4415-bbb9-b089a6b2af00/volumes"
Nov 02 20:33:04.294196 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:33:04.294159    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 20:33:21.446298 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:33:21.446263    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 20:33:34.293450 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:33:34.293374    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 20:33:43.895357 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:33:43.895317    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-7265e7a4-kserve-7449956697-wrftb"]
Nov 02 20:33:43.895835 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:33:43.895706    4463 cpu_manager.go:414] "containerMap: removing container" podUID="2c46e1a4-84cc-483e-8b1d-7786349ef4d5" containerName="istio-proxy"
Nov 02 20:33:43.895835 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:33:43.895718    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c46e1a4-84cc-483e-8b1d-7786349ef4d5" containerName="istio-proxy"
Nov 02 20:33:43.895835 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:33:43.895735    4463 cpu_manager.go:414] "containerMap: removing container" podUID="2c46e1a4-84cc-483e-8b1d-7786349ef4d5" containerName="istio-proxy"
Nov 02 20:33:43.895835 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:33:43.895744    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c46e1a4-84cc-483e-8b1d-7786349ef4d5" containerName="istio-proxy"
Nov 02 20:33:43.895835 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:33:43.895770    4463 cpu_manager.go:414] "containerMap: removing container" podUID="570c4f8a-38a7-4415-bbb9-b089a6b2af00" containerName="payload-pre-processing"
Nov 02 20:33:43.895835 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:33:43.895778    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="570c4f8a-38a7-4415-bbb9-b089a6b2af00" containerName="payload-pre-processing"
Nov 02 20:33:43.895835 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:33:43.895807    4463 cpu_manager.go:414] "containerMap: removing container" podUID="e31394c0-4327-47e7-9bcc-ca2b3808bb61" containerName="maas-api"
Nov 02 20:33:43.895835 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:33:43.895814    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e31394c0-4327-47e7-9bcc-ca2b3808bb61" containerName="maas-api"
Nov 02 20:33:43.896168 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:33:43.895876    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="570c4f8a-38a7-4415-bbb9-b089a6b2af00" containerName="payload-pre-processing"
Nov 02 20:33:43.896168 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:33:43.895888    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="e31394c0-4327-47e7-9bcc-ca2b3808bb61" containerName="maas-api"
Nov 02 20:33:43.896168 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:33:43.895894    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c46e1a4-84cc-483e-8b1d-7786349ef4d5" containerName="istio-proxy"
Nov 02 20:33:43.896168 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:33:43.895902    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c46e1a4-84cc-483e-8b1d-7786349ef4d5" containerName="istio-proxy"
Nov 02 20:33:43.899159 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:33:43.899141    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-7265e7a4-kserve-7449956697-wrftb"
Nov 02 20:33:43.995792 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:33:43.995759    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-7265e7a4-kserve-self-signed-certs\""
Nov 02 20:33:43.997343 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:33:43.997322    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-7265e7a4-kserve-7449956697-wrftb"]
Nov 02 20:33:44.005155 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:33:44.005127    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca8df6d8-68ce-43ba-b76c-a23f9d792a18-dshm\") pod \"e2e-same-id-b-7265e7a4-kserve-7449956697-wrftb\" (UID: \"ca8df6d8-68ce-43ba-b76c-a23f9d792a18\") " pod="llm/e2e-same-id-b-7265e7a4-kserve-7449956697-wrftb"
Nov 02 20:33:44.005288 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:33:44.005157    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bx4m\" (UniqueName: \"kubernetes.io/projected/ca8df6d8-68ce-43ba-b76c-a23f9d792a18-kube-api-access-4bx4m\") pod \"e2e-same-id-b-7265e7a4-kserve-7449956697-wrftb\" (UID: \"ca8df6d8-68ce-43ba-b76c-a23f9d792a18\") " pod="llm/e2e-same-id-b-7265e7a4-kserve-7449956697-wrftb"
Nov 02 20:33:44.005288 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:33:44.005198    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca8df6d8-68ce-43ba-b76c-a23f9d792a18-home\") pod \"e2e-same-id-b-7265e7a4-kserve-7449956697-wrftb\" (UID: \"ca8df6d8-68ce-43ba-b76c-a23f9d792a18\") " pod="llm/e2e-same-id-b-7265e7a4-kserve-7449956697-wrftb"
Nov 02 20:33:44.005412 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:33:44.005304    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca8df6d8-68ce-43ba-b76c-a23f9d792a18-model-cache\") pod \"e2e-same-id-b-7265e7a4-kserve-7449956697-wrftb\" (UID: \"ca8df6d8-68ce-43ba-b76c-a23f9d792a18\") " pod="llm/e2e-same-id-b-7265e7a4-kserve-7449956697-wrftb"
Nov 02 20:33:44.005412 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:33:44.005356    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca8df6d8-68ce-43ba-b76c-a23f9d792a18-tmp-dir\") pod \"e2e-same-id-b-7265e7a4-kserve-7449956697-wrftb\" (UID: \"ca8df6d8-68ce-43ba-b76c-a23f9d792a18\") " pod="llm/e2e-same-id-b-7265e7a4-kserve-7449956697-wrftb"
Nov 02 20:33:44.005524 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:33:44.005420    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca8df6d8-68ce-43ba-b76c-a23f9d792a18-tls-certs\") pod \"e2e-same-id-b-7265e7a4-kserve-7449956697-wrftb\" (UID: \"ca8df6d8-68ce-43ba-b76c-a23f9d792a18\") " pod="llm/e2e-same-id-b-7265e7a4-kserve-7449956697-wrftb"
Nov 02 20:33:44.106405 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:33:44.106372    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca8df6d8-68ce-43ba-b76c-a23f9d792a18-tls-certs\") pod \"e2e-same-id-b-7265e7a4-kserve-7449956697-wrftb\" (UID: \"ca8df6d8-68ce-43ba-b76c-a23f9d792a18\") " pod="llm/e2e-same-id-b-7265e7a4-kserve-7449956697-wrftb"
Nov 02 20:33:44.106605 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:33:44.106427    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca8df6d8-68ce-43ba-b76c-a23f9d792a18-dshm\") pod \"e2e-same-id-b-7265e7a4-kserve-7449956697-wrftb\" (UID: \"ca8df6d8-68ce-43ba-b76c-a23f9d792a18\") " pod="llm/e2e-same-id-b-7265e7a4-kserve-7449956697-wrftb"
Nov 02 20:33:44.106605 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:33:44.106507    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bx4m\" (UniqueName: \"kubernetes.io/projected/ca8df6d8-68ce-43ba-b76c-a23f9d792a18-kube-api-access-4bx4m\") pod \"e2e-same-id-b-7265e7a4-kserve-7449956697-wrftb\" (UID: \"ca8df6d8-68ce-43ba-b76c-a23f9d792a18\") " pod="llm/e2e-same-id-b-7265e7a4-kserve-7449956697-wrftb"
Nov 02 20:33:44.106605 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:33:44.106585    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca8df6d8-68ce-43ba-b76c-a23f9d792a18-home\") pod \"e2e-same-id-b-7265e7a4-kserve-7449956697-wrftb\" (UID: \"ca8df6d8-68ce-43ba-b76c-a23f9d792a18\") " pod="llm/e2e-same-id-b-7265e7a4-kserve-7449956697-wrftb"
Nov 02 20:33:44.106782 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:33:44.106628    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca8df6d8-68ce-43ba-b76c-a23f9d792a18-model-cache\") pod \"e2e-same-id-b-7265e7a4-kserve-7449956697-wrftb\" (UID: \"ca8df6d8-68ce-43ba-b76c-a23f9d792a18\") " pod="llm/e2e-same-id-b-7265e7a4-kserve-7449956697-wrftb"
Nov 02 20:33:44.106782 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:33:44.106658    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca8df6d8-68ce-43ba-b76c-a23f9d792a18-tmp-dir\") pod \"e2e-same-id-b-7265e7a4-kserve-7449956697-wrftb\" (UID: \"ca8df6d8-68ce-43ba-b76c-a23f9d792a18\") " pod="llm/e2e-same-id-b-7265e7a4-kserve-7449956697-wrftb"
Nov 02 20:33:44.106997 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:33:44.106971    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca8df6d8-68ce-43ba-b76c-a23f9d792a18-home\") pod \"e2e-same-id-b-7265e7a4-kserve-7449956697-wrftb\" (UID: \"ca8df6d8-68ce-43ba-b76c-a23f9d792a18\") " pod="llm/e2e-same-id-b-7265e7a4-kserve-7449956697-wrftb"
Nov 02 20:33:44.107138 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:33:44.107052    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca8df6d8-68ce-43ba-b76c-a23f9d792a18-tmp-dir\") pod \"e2e-same-id-b-7265e7a4-kserve-7449956697-wrftb\" (UID: \"ca8df6d8-68ce-43ba-b76c-a23f9d792a18\") " pod="llm/e2e-same-id-b-7265e7a4-kserve-7449956697-wrftb"
Nov 02 20:33:44.107138 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:33:44.107089    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca8df6d8-68ce-43ba-b76c-a23f9d792a18-model-cache\") pod \"e2e-same-id-b-7265e7a4-kserve-7449956697-wrftb\" (UID: \"ca8df6d8-68ce-43ba-b76c-a23f9d792a18\") " pod="llm/e2e-same-id-b-7265e7a4-kserve-7449956697-wrftb"
Nov 02 20:33:44.108717 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:33:44.108692    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca8df6d8-68ce-43ba-b76c-a23f9d792a18-dshm\") pod \"e2e-same-id-b-7265e7a4-kserve-7449956697-wrftb\" (UID: \"ca8df6d8-68ce-43ba-b76c-a23f9d792a18\") " pod="llm/e2e-same-id-b-7265e7a4-kserve-7449956697-wrftb"
Nov 02 20:33:44.109050 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:33:44.109004    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca8df6d8-68ce-43ba-b76c-a23f9d792a18-tls-certs\") pod \"e2e-same-id-b-7265e7a4-kserve-7449956697-wrftb\" (UID: \"ca8df6d8-68ce-43ba-b76c-a23f9d792a18\") " pod="llm/e2e-same-id-b-7265e7a4-kserve-7449956697-wrftb"
Nov 02 20:33:44.159112 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:33:44.159058    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bx4m\" (UniqueName: \"kubernetes.io/projected/ca8df6d8-68ce-43ba-b76c-a23f9d792a18-kube-api-access-4bx4m\") pod \"e2e-same-id-b-7265e7a4-kserve-7449956697-wrftb\" (UID: \"ca8df6d8-68ce-43ba-b76c-a23f9d792a18\") " pod="llm/e2e-same-id-b-7265e7a4-kserve-7449956697-wrftb"
Nov 02 20:33:44.210662 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:33:44.210636    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-7265e7a4-kserve-7449956697-wrftb"
Nov 02 20:33:44.575151 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:33:44.575115    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-7265e7a4-kserve-7449956697-wrftb"]
Nov 02 20:33:44.578695 ip-10-0-133-9 kubenswrapper[4463]: W1102 20:33:44.578666    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podca8df6d8_68ce_43ba_b76c_a23f9d792a18.slice/crio-ffb58a0494360abd6fce95afc400634638573ba6dd72aab7d51a2df128e25f87 WatchSource:0}: Error finding container ffb58a0494360abd6fce95afc400634638573ba6dd72aab7d51a2df128e25f87: Status 404 returned error can't find the container with id ffb58a0494360abd6fce95afc400634638573ba6dd72aab7d51a2df128e25f87
Nov 02 20:33:45.329535 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:33:45.329502    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-7265e7a4-kserve-7449956697-wrftb" event={"ID":"ca8df6d8-68ce-43ba-b76c-a23f9d792a18","Type":"ContainerStarted","Data":"e46b67735a5b478ce523d1a5be61cafd6bd2edc8fbc08dee8333e98dfb8efc9a"}
Nov 02 20:33:45.329535 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:33:45.329542    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-7265e7a4-kserve-7449956697-wrftb" event={"ID":"ca8df6d8-68ce-43ba-b76c-a23f9d792a18","Type":"ContainerStarted","Data":"ffb58a0494360abd6fce95afc400634638573ba6dd72aab7d51a2df128e25f87"}
Nov 02 20:33:45.434955 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:33:45.434893    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-7265e7a4-kserve-7449956697-wrftb" podStartSLOduration=2.434876674 podStartE2EDuration="2.434876674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:33:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:33:45.433972148 +0000 UTC m=+3932.041005917" watchObservedRunningTime="2026-11-02 20:33:45.434876674 +0000 UTC m=+3932.041910444"
Nov 02 20:33:46.210984 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:33:46.210955    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-7265e7a4-kserve-7449956697-wrftb"
Nov 02 20:33:48.328301 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:33:48.328270    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 20:33:54.210871 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:33:54.210830    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-7265e7a4-kserve-7449956697-wrftb"
Nov 02 20:33:54.223293 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:33:54.223268    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-7265e7a4-kserve-7449956697-wrftb"
Nov 02 20:33:54.403237 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:33:54.403206    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-7265e7a4-kserve-7449956697-wrftb"
Nov 02 20:34:04.293909 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:34:04.293865    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 20:34:13.819621 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:34:13.819581    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 20:34:27.832258 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:34:27.832221    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-7265e7a4-kserve-7449956697-wrftb"]
Nov 02 20:34:29.131873 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:34:29.131837    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 20:34:34.293858 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:34:34.293817    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 20:34:40.450107 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:34:40.450065    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 20:34:42.851637 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:34:42.851565    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-7265e7a4-kserve-7449956697-wrftb" podUID="ca8df6d8-68ce-43ba-b76c-a23f9d792a18" containerName="main" containerID="cri-o://e46b67735a5b478ce523d1a5be61cafd6bd2edc8fbc08dee8333e98dfb8efc9a" gracePeriod=45
Nov 02 20:34:43.093835 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:34:43.093812    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-7265e7a4-kserve-7449956697-wrftb"
Nov 02 20:34:43.164926 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:34:43.164861    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca8df6d8-68ce-43ba-b76c-a23f9d792a18-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca8df6d8-68ce-43ba-b76c-a23f9d792a18-tmp-dir\") pod \"ca8df6d8-68ce-43ba-b76c-a23f9d792a18\" (UID: \"ca8df6d8-68ce-43ba-b76c-a23f9d792a18\") "
Nov 02 20:34:43.164926 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:34:43.164904    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca8df6d8-68ce-43ba-b76c-a23f9d792a18-kube-api-access-4bx4m\" (UniqueName: \"kubernetes.io/projected/ca8df6d8-68ce-43ba-b76c-a23f9d792a18-kube-api-access-4bx4m\") pod \"ca8df6d8-68ce-43ba-b76c-a23f9d792a18\" (UID: \"ca8df6d8-68ce-43ba-b76c-a23f9d792a18\") "
Nov 02 20:34:43.164926 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:34:43.164921    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca8df6d8-68ce-43ba-b76c-a23f9d792a18-home\" (UniqueName: \"kubernetes.io/empty-dir/ca8df6d8-68ce-43ba-b76c-a23f9d792a18-home\") pod \"ca8df6d8-68ce-43ba-b76c-a23f9d792a18\" (UID: \"ca8df6d8-68ce-43ba-b76c-a23f9d792a18\") "
Nov 02 20:34:43.165223 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:34:43.164989    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca8df6d8-68ce-43ba-b76c-a23f9d792a18-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca8df6d8-68ce-43ba-b76c-a23f9d792a18-dshm\") pod \"ca8df6d8-68ce-43ba-b76c-a23f9d792a18\" (UID: \"ca8df6d8-68ce-43ba-b76c-a23f9d792a18\") "
Nov 02 20:34:43.165223 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:34:43.165095    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca8df6d8-68ce-43ba-b76c-a23f9d792a18-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca8df6d8-68ce-43ba-b76c-a23f9d792a18-model-cache\") pod \"ca8df6d8-68ce-43ba-b76c-a23f9d792a18\" (UID: \"ca8df6d8-68ce-43ba-b76c-a23f9d792a18\") "
Nov 02 20:34:43.165223 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:34:43.165125    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca8df6d8-68ce-43ba-b76c-a23f9d792a18-tls-certs\" (UniqueName: \"kubernetes.io/secret/ca8df6d8-68ce-43ba-b76c-a23f9d792a18-tls-certs\") pod \"ca8df6d8-68ce-43ba-b76c-a23f9d792a18\" (UID: \"ca8df6d8-68ce-43ba-b76c-a23f9d792a18\") "
Nov 02 20:34:43.165223 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:34:43.165188    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca8df6d8-68ce-43ba-b76c-a23f9d792a18-tmp-dir" pod "ca8df6d8-68ce-43ba-b76c-a23f9d792a18" (UID: "ca8df6d8-68ce-43ba-b76c-a23f9d792a18"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:34:43.165223 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:34:43.165203    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca8df6d8-68ce-43ba-b76c-a23f9d792a18-home" pod "ca8df6d8-68ce-43ba-b76c-a23f9d792a18" (UID: "ca8df6d8-68ce-43ba-b76c-a23f9d792a18"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:34:43.165431 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:34:43.165411    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca8df6d8-68ce-43ba-b76c-a23f9d792a18-model-cache" pod "ca8df6d8-68ce-43ba-b76c-a23f9d792a18" (UID: "ca8df6d8-68ce-43ba-b76c-a23f9d792a18"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:34:43.165565 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:34:43.165545    4463 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca8df6d8-68ce-43ba-b76c-a23f9d792a18-home\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:34:43.165565 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:34:43.165565    4463 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca8df6d8-68ce-43ba-b76c-a23f9d792a18-model-cache\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:34:43.165666 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:34:43.165574    4463 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca8df6d8-68ce-43ba-b76c-a23f9d792a18-tmp-dir\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:34:43.167322 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:34:43.167299    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca8df6d8-68ce-43ba-b76c-a23f9d792a18-kube-api-access-4bx4m" pod "ca8df6d8-68ce-43ba-b76c-a23f9d792a18" (UID: "ca8df6d8-68ce-43ba-b76c-a23f9d792a18"). InnerVolumeSpecName "kube-api-access-4bx4m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:34:43.167322 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:34:43.167304    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca8df6d8-68ce-43ba-b76c-a23f9d792a18-tls-certs" pod "ca8df6d8-68ce-43ba-b76c-a23f9d792a18" (UID: "ca8df6d8-68ce-43ba-b76c-a23f9d792a18"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:34:43.167461 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:34:43.167301    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca8df6d8-68ce-43ba-b76c-a23f9d792a18-dshm" pod "ca8df6d8-68ce-43ba-b76c-a23f9d792a18" (UID: "ca8df6d8-68ce-43ba-b76c-a23f9d792a18"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:34:43.266843 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:34:43.266810    4463 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca8df6d8-68ce-43ba-b76c-a23f9d792a18-dshm\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:34:43.266843 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:34:43.266836    4463 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca8df6d8-68ce-43ba-b76c-a23f9d792a18-tls-certs\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:34:43.266843 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:34:43.266847    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bx4m\" (UniqueName: \"kubernetes.io/projected/ca8df6d8-68ce-43ba-b76c-a23f9d792a18-kube-api-access-4bx4m\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:34:43.708043 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:34:43.707999    4463 generic.go:356] "Generic (PLEG): container finished" podID="ca8df6d8-68ce-43ba-b76c-a23f9d792a18" containerID="e46b67735a5b478ce523d1a5be61cafd6bd2edc8fbc08dee8333e98dfb8efc9a" exitCode=0
Nov 02 20:34:43.708222 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:34:43.708064    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-7265e7a4-kserve-7449956697-wrftb"
Nov 02 20:34:43.708222 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:34:43.708080    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-7265e7a4-kserve-7449956697-wrftb" event={"ID":"ca8df6d8-68ce-43ba-b76c-a23f9d792a18","Type":"ContainerDied","Data":"e46b67735a5b478ce523d1a5be61cafd6bd2edc8fbc08dee8333e98dfb8efc9a"}
Nov 02 20:34:43.708222 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:34:43.708122    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-7265e7a4-kserve-7449956697-wrftb" event={"ID":"ca8df6d8-68ce-43ba-b76c-a23f9d792a18","Type":"ContainerDied","Data":"ffb58a0494360abd6fce95afc400634638573ba6dd72aab7d51a2df128e25f87"}
Nov 02 20:34:43.708222 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:34:43.708139    4463 scope.go:122] "RemoveContainer" containerID="e46b67735a5b478ce523d1a5be61cafd6bd2edc8fbc08dee8333e98dfb8efc9a"
Nov 02 20:34:43.716723 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:34:43.716706    4463 scope.go:122] "RemoveContainer" containerID="e46b67735a5b478ce523d1a5be61cafd6bd2edc8fbc08dee8333e98dfb8efc9a"
Nov 02 20:34:43.716982 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:34:43.716959    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e46b67735a5b478ce523d1a5be61cafd6bd2edc8fbc08dee8333e98dfb8efc9a\": container with ID starting with e46b67735a5b478ce523d1a5be61cafd6bd2edc8fbc08dee8333e98dfb8efc9a not found: ID does not exist" containerID="e46b67735a5b478ce523d1a5be61cafd6bd2edc8fbc08dee8333e98dfb8efc9a"
Nov 02 20:34:43.717077 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:34:43.716993    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e46b67735a5b478ce523d1a5be61cafd6bd2edc8fbc08dee8333e98dfb8efc9a"} err="failed to get container status \"e46b67735a5b478ce523d1a5be61cafd6bd2edc8fbc08dee8333e98dfb8efc9a\": rpc error: code = NotFound desc = could not find container \"e46b67735a5b478ce523d1a5be61cafd6bd2edc8fbc08dee8333e98dfb8efc9a\": container with ID starting with e46b67735a5b478ce523d1a5be61cafd6bd2edc8fbc08dee8333e98dfb8efc9a not found: ID does not exist"
Nov 02 20:34:43.859260 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:34:43.859185    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-7265e7a4-kserve-7449956697-wrftb"]
Nov 02 20:34:43.905175 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:34:43.905147    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-7265e7a4-kserve-7449956697-wrftb"]
Nov 02 20:34:45.583665 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:34:45.583634    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca8df6d8-68ce-43ba-b76c-a23f9d792a18" path="/var/lib/kubelet/pods/ca8df6d8-68ce-43ba-b76c-a23f9d792a18/volumes"
Nov 02 20:34:49.824047 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:34:49.823992    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 20:35:04.293267 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:35:04.293222    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 20:35:12.937594 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:12.937554    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 20:35:13.348543 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:13.348465    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-f6584b448-n45m7"]
Nov 02 20:35:13.348845 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:13.348833    4463 cpu_manager.go:414] "containerMap: removing container" podUID="ca8df6d8-68ce-43ba-b76c-a23f9d792a18" containerName="main"
Nov 02 20:35:13.348893 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:13.348847    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca8df6d8-68ce-43ba-b76c-a23f9d792a18" containerName="main"
Nov 02 20:35:13.348931 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:13.348918    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca8df6d8-68ce-43ba-b76c-a23f9d792a18" containerName="main"
Nov 02 20:35:13.352173 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:13.352151    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-f6584b448-n45m7"
Nov 02 20:35:13.427656 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:13.427626    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-f6584b448-n45m7"]
Nov 02 20:35:13.494755 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:13.494722    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32e8cbc9-818f-44ce-a617-1551fbc35930-metrics-certs\") pod \"maas-api-f6584b448-n45m7\" (UID: \"32e8cbc9-818f-44ce-a617-1551fbc35930\") " pod="odh-ai-gateway-infra/maas-api-f6584b448-n45m7"
Nov 02 20:35:13.494930 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:13.494861    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32e8cbc9-818f-44ce-a617-1551fbc35930-openshift-service-ca\") pod \"maas-api-f6584b448-n45m7\" (UID: \"32e8cbc9-818f-44ce-a617-1551fbc35930\") " pod="odh-ai-gateway-infra/maas-api-f6584b448-n45m7"
Nov 02 20:35:13.494930 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:13.494905    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32e8cbc9-818f-44ce-a617-1551fbc35930-maas-api-tls\") pod \"maas-api-f6584b448-n45m7\" (UID: \"32e8cbc9-818f-44ce-a617-1551fbc35930\") " pod="odh-ai-gateway-infra/maas-api-f6584b448-n45m7"
Nov 02 20:35:13.495040 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:13.494931    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6n5p\" (UniqueName: \"kubernetes.io/projected/32e8cbc9-818f-44ce-a617-1551fbc35930-kube-api-access-w6n5p\") pod \"maas-api-f6584b448-n45m7\" (UID: \"32e8cbc9-818f-44ce-a617-1551fbc35930\") " pod="odh-ai-gateway-infra/maas-api-f6584b448-n45m7"
Nov 02 20:35:13.596456 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:13.596427    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32e8cbc9-818f-44ce-a617-1551fbc35930-maas-api-tls\") pod \"maas-api-f6584b448-n45m7\" (UID: \"32e8cbc9-818f-44ce-a617-1551fbc35930\") " pod="odh-ai-gateway-infra/maas-api-f6584b448-n45m7"
Nov 02 20:35:13.596621 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:13.596462    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6n5p\" (UniqueName: \"kubernetes.io/projected/32e8cbc9-818f-44ce-a617-1551fbc35930-kube-api-access-w6n5p\") pod \"maas-api-f6584b448-n45m7\" (UID: \"32e8cbc9-818f-44ce-a617-1551fbc35930\") " pod="odh-ai-gateway-infra/maas-api-f6584b448-n45m7"
Nov 02 20:35:13.596621 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:13.596494    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32e8cbc9-818f-44ce-a617-1551fbc35930-metrics-certs\") pod \"maas-api-f6584b448-n45m7\" (UID: \"32e8cbc9-818f-44ce-a617-1551fbc35930\") " pod="odh-ai-gateway-infra/maas-api-f6584b448-n45m7"
Nov 02 20:35:13.596621 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:13.596564    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32e8cbc9-818f-44ce-a617-1551fbc35930-openshift-service-ca\") pod \"maas-api-f6584b448-n45m7\" (UID: \"32e8cbc9-818f-44ce-a617-1551fbc35930\") " pod="odh-ai-gateway-infra/maas-api-f6584b448-n45m7"
Nov 02 20:35:13.597276 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:13.597250    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32e8cbc9-818f-44ce-a617-1551fbc35930-openshift-service-ca\") pod \"maas-api-f6584b448-n45m7\" (UID: \"32e8cbc9-818f-44ce-a617-1551fbc35930\") " pod="odh-ai-gateway-infra/maas-api-f6584b448-n45m7"
Nov 02 20:35:13.598717 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:13.598664    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32e8cbc9-818f-44ce-a617-1551fbc35930-maas-api-tls\") pod \"maas-api-f6584b448-n45m7\" (UID: \"32e8cbc9-818f-44ce-a617-1551fbc35930\") " pod="odh-ai-gateway-infra/maas-api-f6584b448-n45m7"
Nov 02 20:35:13.598837 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:13.598771    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32e8cbc9-818f-44ce-a617-1551fbc35930-metrics-certs\") pod \"maas-api-f6584b448-n45m7\" (UID: \"32e8cbc9-818f-44ce-a617-1551fbc35930\") " pod="odh-ai-gateway-infra/maas-api-f6584b448-n45m7"
Nov 02 20:35:13.648969 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:13.648939    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6n5p\" (UniqueName: \"kubernetes.io/projected/32e8cbc9-818f-44ce-a617-1551fbc35930-kube-api-access-w6n5p\") pod \"maas-api-f6584b448-n45m7\" (UID: \"32e8cbc9-818f-44ce-a617-1551fbc35930\") " pod="odh-ai-gateway-infra/maas-api-f6584b448-n45m7"
Nov 02 20:35:13.664289 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:13.664269    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-f6584b448-n45m7"
Nov 02 20:35:14.026461 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:14.026383    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-f6584b448-n45m7"]
Nov 02 20:35:14.028901 ip-10-0-133-9 kubenswrapper[4463]: W1102 20:35:14.028869    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32e8cbc9_818f_44ce_a617_1551fbc35930.slice/crio-0b9673230e0c214399ddb411f949de4e3baae4b0b49f1c76f5898e9501cbd711 WatchSource:0}: Error finding container 0b9673230e0c214399ddb411f949de4e3baae4b0b49f1c76f5898e9501cbd711: Status 404 returned error can't find the container with id 0b9673230e0c214399ddb411f949de4e3baae4b0b49f1c76f5898e9501cbd711
Nov 02 20:35:14.030201 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:14.030184    4463 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 20:35:14.868887 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:14.868849    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-f6584b448-n45m7" event={"ID":"32e8cbc9-818f-44ce-a617-1551fbc35930","Type":"ContainerStarted","Data":"0b9673230e0c214399ddb411f949de4e3baae4b0b49f1c76f5898e9501cbd711"}
Nov 02 20:35:15.874369 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:15.874324    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-f6584b448-n45m7" event={"ID":"32e8cbc9-818f-44ce-a617-1551fbc35930","Type":"ContainerStarted","Data":"ee3ccb4d5ef81412346b36bb368b896dea125de5f66dd98e8eda1b5282e148f8"}
Nov 02 20:35:15.874811 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:15.874487    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-f6584b448-n45m7"
Nov 02 20:35:15.982245 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:15.982195    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-f6584b448-n45m7" podStartSLOduration=2.165483436 podStartE2EDuration="2.982182793s" totalImagesPullingTime="816.699357ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:35:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:35:15.980653346 +0000 UTC m=+4022.587687115" watchObservedRunningTime="2026-11-02 20:35:15.982182793 +0000 UTC m=+4022.589216561"
Nov 02 20:35:21.884136 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:21.884104    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-f6584b448-n45m7"
Nov 02 20:35:22.038453 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:22.038419    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-769845dd8d-nthbb"]
Nov 02 20:35:22.038667 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:22.038637    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-769845dd8d-nthbb" podUID="48058634-fd1d-4c53-a367-d43833c583b4" containerName="maas-api" containerID="cri-o://03093b0ae89cc9037e21046ef214c1f4d422e1c3c738a2cca50cc63ddccc6083" gracePeriod=30
Nov 02 20:35:23.277426 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:23.277394    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-769845dd8d-nthbb"
Nov 02 20:35:23.282641 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:23.282619    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48058634-fd1d-4c53-a367-d43833c583b4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48058634-fd1d-4c53-a367-d43833c583b4-openshift-service-ca\") pod \"48058634-fd1d-4c53-a367-d43833c583b4\" (UID: \"48058634-fd1d-4c53-a367-d43833c583b4\") "
Nov 02 20:35:23.282756 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:23.282728    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48058634-fd1d-4c53-a367-d43833c583b4-kube-api-access-4fvsw\" (UniqueName: \"kubernetes.io/projected/48058634-fd1d-4c53-a367-d43833c583b4-kube-api-access-4fvsw\") pod \"48058634-fd1d-4c53-a367-d43833c583b4\" (UID: \"48058634-fd1d-4c53-a367-d43833c583b4\") "
Nov 02 20:35:23.282823 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:23.282786    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48058634-fd1d-4c53-a367-d43833c583b4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48058634-fd1d-4c53-a367-d43833c583b4-maas-api-tls\") pod \"48058634-fd1d-4c53-a367-d43833c583b4\" (UID: \"48058634-fd1d-4c53-a367-d43833c583b4\") "
Nov 02 20:35:23.282879 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:23.282833    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48058634-fd1d-4c53-a367-d43833c583b4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/48058634-fd1d-4c53-a367-d43833c583b4-metrics-certs\") pod \"48058634-fd1d-4c53-a367-d43833c583b4\" (UID: \"48058634-fd1d-4c53-a367-d43833c583b4\") "
Nov 02 20:35:23.282933 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:23.282915    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48058634-fd1d-4c53-a367-d43833c583b4-openshift-service-ca" pod "48058634-fd1d-4c53-a367-d43833c583b4" (UID: "48058634-fd1d-4c53-a367-d43833c583b4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:35:23.283121 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:23.283105    4463 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48058634-fd1d-4c53-a367-d43833c583b4-openshift-service-ca\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:35:23.284803 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:23.284779    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48058634-fd1d-4c53-a367-d43833c583b4-kube-api-access-4fvsw" pod "48058634-fd1d-4c53-a367-d43833c583b4" (UID: "48058634-fd1d-4c53-a367-d43833c583b4"). InnerVolumeSpecName "kube-api-access-4fvsw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:35:23.284894 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:23.284827    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48058634-fd1d-4c53-a367-d43833c583b4-maas-api-tls" pod "48058634-fd1d-4c53-a367-d43833c583b4" (UID: "48058634-fd1d-4c53-a367-d43833c583b4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:35:23.284894 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:23.284856    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48058634-fd1d-4c53-a367-d43833c583b4-metrics-certs" pod "48058634-fd1d-4c53-a367-d43833c583b4" (UID: "48058634-fd1d-4c53-a367-d43833c583b4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:35:23.383791 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:23.383730    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fvsw\" (UniqueName: \"kubernetes.io/projected/48058634-fd1d-4c53-a367-d43833c583b4-kube-api-access-4fvsw\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:35:23.383791 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:23.383755    4463 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48058634-fd1d-4c53-a367-d43833c583b4-maas-api-tls\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:35:23.383791 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:23.383770    4463 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48058634-fd1d-4c53-a367-d43833c583b4-metrics-certs\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:35:23.910991 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:23.910953    4463 generic.go:356] "Generic (PLEG): container finished" podID="48058634-fd1d-4c53-a367-d43833c583b4" containerID="03093b0ae89cc9037e21046ef214c1f4d422e1c3c738a2cca50cc63ddccc6083" exitCode=0
Nov 02 20:35:23.911181 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:23.911003    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-769845dd8d-nthbb"
Nov 02 20:35:23.911181 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:23.911038    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-769845dd8d-nthbb" event={"ID":"48058634-fd1d-4c53-a367-d43833c583b4","Type":"ContainerDied","Data":"03093b0ae89cc9037e21046ef214c1f4d422e1c3c738a2cca50cc63ddccc6083"}
Nov 02 20:35:23.911181 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:23.911098    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-769845dd8d-nthbb" event={"ID":"48058634-fd1d-4c53-a367-d43833c583b4","Type":"ContainerDied","Data":"0ec8f029adf057f3d66a992a33035a7958f1245520e4aa34a9c36f24aa3db3e9"}
Nov 02 20:35:23.911181 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:23.911120    4463 scope.go:122] "RemoveContainer" containerID="03093b0ae89cc9037e21046ef214c1f4d422e1c3c738a2cca50cc63ddccc6083"
Nov 02 20:35:23.919494 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:23.919474    4463 scope.go:122] "RemoveContainer" containerID="03093b0ae89cc9037e21046ef214c1f4d422e1c3c738a2cca50cc63ddccc6083"
Nov 02 20:35:23.919791 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:35:23.919764    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03093b0ae89cc9037e21046ef214c1f4d422e1c3c738a2cca50cc63ddccc6083\": container with ID starting with 03093b0ae89cc9037e21046ef214c1f4d422e1c3c738a2cca50cc63ddccc6083 not found: ID does not exist" containerID="03093b0ae89cc9037e21046ef214c1f4d422e1c3c738a2cca50cc63ddccc6083"
Nov 02 20:35:23.919899 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:23.919792    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03093b0ae89cc9037e21046ef214c1f4d422e1c3c738a2cca50cc63ddccc6083"} err="failed to get container status \"03093b0ae89cc9037e21046ef214c1f4d422e1c3c738a2cca50cc63ddccc6083\": rpc error: code = NotFound desc = could not find container \"03093b0ae89cc9037e21046ef214c1f4d422e1c3c738a2cca50cc63ddccc6083\": container with ID starting with 03093b0ae89cc9037e21046ef214c1f4d422e1c3c738a2cca50cc63ddccc6083 not found: ID does not exist"
Nov 02 20:35:24.063520 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:24.063476    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-769845dd8d-nthbb"]
Nov 02 20:35:24.109455 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:24.109427    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-769845dd8d-nthbb"]
Nov 02 20:35:25.585166 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:25.585130    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48058634-fd1d-4c53-a367-d43833c583b4" path="/var/lib/kubelet/pods/48058634-fd1d-4c53-a367-d43833c583b4/volumes"
Nov 02 20:35:26.978512 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:26.978477    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5b04055d-5f9db46f6d-sfcpf"]
Nov 02 20:35:26.978902 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:26.978888    4463 cpu_manager.go:414] "containerMap: removing container" podUID="48058634-fd1d-4c53-a367-d43833c583b4" containerName="maas-api"
Nov 02 20:35:26.978902 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:26.978900    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48058634-fd1d-4c53-a367-d43833c583b4" containerName="maas-api"
Nov 02 20:35:26.978992 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:26.978979    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="48058634-fd1d-4c53-a367-d43833c583b4" containerName="maas-api"
Nov 02 20:35:26.983497 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:26.983473    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b04055d-5f9db46f6d-sfcpf"
Nov 02 20:35:27.015559 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:27.015529    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/352d3c8c-a544-424e-9a29-f36c353e706d-metrics-certs\") pod \"maas-api-e2e-ait-5b04055d-5f9db46f6d-sfcpf\" (UID: \"352d3c8c-a544-424e-9a29-f36c353e706d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b04055d-5f9db46f6d-sfcpf"
Nov 02 20:35:27.015687 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:27.015646    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vz2r\" (UniqueName: \"kubernetes.io/projected/352d3c8c-a544-424e-9a29-f36c353e706d-kube-api-access-6vz2r\") pod \"maas-api-e2e-ait-5b04055d-5f9db46f6d-sfcpf\" (UID: \"352d3c8c-a544-424e-9a29-f36c353e706d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b04055d-5f9db46f6d-sfcpf"
Nov 02 20:35:27.015687 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:27.015681    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/352d3c8c-a544-424e-9a29-f36c353e706d-maas-api-tls\") pod \"maas-api-e2e-ait-5b04055d-5f9db46f6d-sfcpf\" (UID: \"352d3c8c-a544-424e-9a29-f36c353e706d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b04055d-5f9db46f6d-sfcpf"
Nov 02 20:35:27.015787 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:27.015770    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/352d3c8c-a544-424e-9a29-f36c353e706d-openshift-service-ca\") pod \"maas-api-e2e-ait-5b04055d-5f9db46f6d-sfcpf\" (UID: \"352d3c8c-a544-424e-9a29-f36c353e706d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b04055d-5f9db46f6d-sfcpf"
Nov 02 20:35:27.031583 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:27.031558    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-5b04055d-serving-cert\""
Nov 02 20:35:27.079613 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:27.079577    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5b04055d-5f9db46f6d-sfcpf"]
Nov 02 20:35:27.116553 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:27.116516    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vz2r\" (UniqueName: \"kubernetes.io/projected/352d3c8c-a544-424e-9a29-f36c353e706d-kube-api-access-6vz2r\") pod \"maas-api-e2e-ait-5b04055d-5f9db46f6d-sfcpf\" (UID: \"352d3c8c-a544-424e-9a29-f36c353e706d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b04055d-5f9db46f6d-sfcpf"
Nov 02 20:35:27.116778 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:27.116559    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/352d3c8c-a544-424e-9a29-f36c353e706d-maas-api-tls\") pod \"maas-api-e2e-ait-5b04055d-5f9db46f6d-sfcpf\" (UID: \"352d3c8c-a544-424e-9a29-f36c353e706d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b04055d-5f9db46f6d-sfcpf"
Nov 02 20:35:27.116778 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:27.116620    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/352d3c8c-a544-424e-9a29-f36c353e706d-openshift-service-ca\") pod \"maas-api-e2e-ait-5b04055d-5f9db46f6d-sfcpf\" (UID: \"352d3c8c-a544-424e-9a29-f36c353e706d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b04055d-5f9db46f6d-sfcpf"
Nov 02 20:35:27.116778 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:27.116666    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/352d3c8c-a544-424e-9a29-f36c353e706d-metrics-certs\") pod \"maas-api-e2e-ait-5b04055d-5f9db46f6d-sfcpf\" (UID: \"352d3c8c-a544-424e-9a29-f36c353e706d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b04055d-5f9db46f6d-sfcpf"
Nov 02 20:35:27.116778 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:35:27.116765    4463 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-5b04055d-serving-cert: secret "maas-api-e2e-ait-5b04055d-serving-cert" not found
Nov 02 20:35:27.117043 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:35:27.116850    4463 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/352d3c8c-a544-424e-9a29-f36c353e706d-maas-api-tls podName:352d3c8c-a544-424e-9a29-f36c353e706d nodeName:}" failed. No retries permitted until 2026-11-02 20:35:27.616828582 +0000 UTC m=+4034.223862350 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/352d3c8c-a544-424e-9a29-f36c353e706d-maas-api-tls") pod "maas-api-e2e-ait-5b04055d-5f9db46f6d-sfcpf" (UID: "352d3c8c-a544-424e-9a29-f36c353e706d") : secret "maas-api-e2e-ait-5b04055d-serving-cert" not found
Nov 02 20:35:27.117492 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:27.117467    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/352d3c8c-a544-424e-9a29-f36c353e706d-openshift-service-ca\") pod \"maas-api-e2e-ait-5b04055d-5f9db46f6d-sfcpf\" (UID: \"352d3c8c-a544-424e-9a29-f36c353e706d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b04055d-5f9db46f6d-sfcpf"
Nov 02 20:35:27.119283 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:27.119265    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/352d3c8c-a544-424e-9a29-f36c353e706d-metrics-certs\") pod \"maas-api-e2e-ait-5b04055d-5f9db46f6d-sfcpf\" (UID: \"352d3c8c-a544-424e-9a29-f36c353e706d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b04055d-5f9db46f6d-sfcpf"
Nov 02 20:35:27.169999 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:27.169962    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vz2r\" (UniqueName: \"kubernetes.io/projected/352d3c8c-a544-424e-9a29-f36c353e706d-kube-api-access-6vz2r\") pod \"maas-api-e2e-ait-5b04055d-5f9db46f6d-sfcpf\" (UID: \"352d3c8c-a544-424e-9a29-f36c353e706d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b04055d-5f9db46f6d-sfcpf"
Nov 02 20:35:27.427230 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:27.427194    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5b04055d-597945b9c5-5rmbf"]
Nov 02 20:35:27.432841 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:27.432813    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5b04055d-597945b9c5-5rmbf"
Nov 02 20:35:27.484614 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:27.484549    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-5b04055d\""
Nov 02 20:35:27.484805 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:27.484612    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-5b04055d-dockercfg-zpt6k\""
Nov 02 20:35:27.486601 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:27.486576    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5b04055d-597945b9c5-5rmbf"]
Nov 02 20:35:27.520357 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:27.520329    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7efe97f-30f2-4d56-a8fc-7e29f23a70f9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5b04055d-597945b9c5-5rmbf\" (UID: \"d7efe97f-30f2-4d56-a8fc-7e29f23a70f9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5b04055d-597945b9c5-5rmbf"
Nov 02 20:35:27.520545 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:27.520461    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rlkp\" (UniqueName: \"kubernetes.io/projected/d7efe97f-30f2-4d56-a8fc-7e29f23a70f9-kube-api-access-2rlkp\") pod \"payload-pre-processing-e2e-ait-5b04055d-597945b9c5-5rmbf\" (UID: \"d7efe97f-30f2-4d56-a8fc-7e29f23a70f9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5b04055d-597945b9c5-5rmbf"
Nov 02 20:35:27.584521 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:27.584491    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5b04055d-7d9574f7d5-8pnkc"]
Nov 02 20:35:27.588026 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:27.587999    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5b04055d-7d9574f7d5-8pnkc"
Nov 02 20:35:27.621111 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:27.621085    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rlkp\" (UniqueName: \"kubernetes.io/projected/d7efe97f-30f2-4d56-a8fc-7e29f23a70f9-kube-api-access-2rlkp\") pod \"payload-pre-processing-e2e-ait-5b04055d-597945b9c5-5rmbf\" (UID: \"d7efe97f-30f2-4d56-a8fc-7e29f23a70f9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5b04055d-597945b9c5-5rmbf"
Nov 02 20:35:27.621202 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:27.621122    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wk7v2\" (UniqueName: \"kubernetes.io/projected/68891943-f1fa-42f4-8ebb-50f1d768b23e-kube-api-access-wk7v2\") pod \"payload-processing-e2e-ait-5b04055d-7d9574f7d5-8pnkc\" (UID: \"68891943-f1fa-42f4-8ebb-50f1d768b23e\") " pod="openshift-ingress/payload-processing-e2e-ait-5b04055d-7d9574f7d5-8pnkc"
Nov 02 20:35:27.621202 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:27.621162    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/352d3c8c-a544-424e-9a29-f36c353e706d-maas-api-tls\") pod \"maas-api-e2e-ait-5b04055d-5f9db46f6d-sfcpf\" (UID: \"352d3c8c-a544-424e-9a29-f36c353e706d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b04055d-5f9db46f6d-sfcpf"
Nov 02 20:35:27.621297 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:27.621199    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7efe97f-30f2-4d56-a8fc-7e29f23a70f9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5b04055d-597945b9c5-5rmbf\" (UID: \"d7efe97f-30f2-4d56-a8fc-7e29f23a70f9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5b04055d-597945b9c5-5rmbf"
Nov 02 20:35:27.621297 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:27.621224    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68891943-f1fa-42f4-8ebb-50f1d768b23e-plugins-config-volume\") pod \"payload-processing-e2e-ait-5b04055d-7d9574f7d5-8pnkc\" (UID: \"68891943-f1fa-42f4-8ebb-50f1d768b23e\") " pod="openshift-ingress/payload-processing-e2e-ait-5b04055d-7d9574f7d5-8pnkc"
Nov 02 20:35:27.621889 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:27.621866    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7efe97f-30f2-4d56-a8fc-7e29f23a70f9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5b04055d-597945b9c5-5rmbf\" (UID: \"d7efe97f-30f2-4d56-a8fc-7e29f23a70f9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5b04055d-597945b9c5-5rmbf"
Nov 02 20:35:27.623523 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:27.623502    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/352d3c8c-a544-424e-9a29-f36c353e706d-maas-api-tls\") pod \"maas-api-e2e-ait-5b04055d-5f9db46f6d-sfcpf\" (UID: \"352d3c8c-a544-424e-9a29-f36c353e706d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b04055d-5f9db46f6d-sfcpf"
Nov 02 20:35:27.640575 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:27.640541    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5b04055d-7d9574f7d5-8pnkc"]
Nov 02 20:35:27.674504 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:27.674472    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rlkp\" (UniqueName: \"kubernetes.io/projected/d7efe97f-30f2-4d56-a8fc-7e29f23a70f9-kube-api-access-2rlkp\") pod \"payload-pre-processing-e2e-ait-5b04055d-597945b9c5-5rmbf\" (UID: \"d7efe97f-30f2-4d56-a8fc-7e29f23a70f9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5b04055d-597945b9c5-5rmbf"
Nov 02 20:35:27.721880 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:27.721822    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wk7v2\" (UniqueName: \"kubernetes.io/projected/68891943-f1fa-42f4-8ebb-50f1d768b23e-kube-api-access-wk7v2\") pod \"payload-processing-e2e-ait-5b04055d-7d9574f7d5-8pnkc\" (UID: \"68891943-f1fa-42f4-8ebb-50f1d768b23e\") " pod="openshift-ingress/payload-processing-e2e-ait-5b04055d-7d9574f7d5-8pnkc"
Nov 02 20:35:27.721880 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:27.721869    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68891943-f1fa-42f4-8ebb-50f1d768b23e-plugins-config-volume\") pod \"payload-processing-e2e-ait-5b04055d-7d9574f7d5-8pnkc\" (UID: \"68891943-f1fa-42f4-8ebb-50f1d768b23e\") " pod="openshift-ingress/payload-processing-e2e-ait-5b04055d-7d9574f7d5-8pnkc"
Nov 02 20:35:27.722533 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:27.722514    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68891943-f1fa-42f4-8ebb-50f1d768b23e-plugins-config-volume\") pod \"payload-processing-e2e-ait-5b04055d-7d9574f7d5-8pnkc\" (UID: \"68891943-f1fa-42f4-8ebb-50f1d768b23e\") " pod="openshift-ingress/payload-processing-e2e-ait-5b04055d-7d9574f7d5-8pnkc"
Nov 02 20:35:27.745775 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:27.745749    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5b04055d-597945b9c5-5rmbf"
Nov 02 20:35:27.775386 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:27.775354    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wk7v2\" (UniqueName: \"kubernetes.io/projected/68891943-f1fa-42f4-8ebb-50f1d768b23e-kube-api-access-wk7v2\") pod \"payload-processing-e2e-ait-5b04055d-7d9574f7d5-8pnkc\" (UID: \"68891943-f1fa-42f4-8ebb-50f1d768b23e\") " pod="openshift-ingress/payload-processing-e2e-ait-5b04055d-7d9574f7d5-8pnkc"
Nov 02 20:35:27.896629 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:27.896594    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b04055d-5f9db46f6d-sfcpf"
Nov 02 20:35:27.900375 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:27.900347    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5b04055d-7d9574f7d5-8pnkc"
Nov 02 20:35:28.108331 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:28.108210    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5b04055d-597945b9c5-5rmbf"]
Nov 02 20:35:28.110764 ip-10-0-133-9 kubenswrapper[4463]: W1102 20:35:28.110740    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7efe97f_30f2_4d56_a8fc_7e29f23a70f9.slice/crio-2f630f61d0be615032ce773a4d01f4bffe25a14d29a86b6b98dc43eb0cdc3b2a WatchSource:0}: Error finding container 2f630f61d0be615032ce773a4d01f4bffe25a14d29a86b6b98dc43eb0cdc3b2a: Status 404 returned error can't find the container with id 2f630f61d0be615032ce773a4d01f4bffe25a14d29a86b6b98dc43eb0cdc3b2a
Nov 02 20:35:28.264358 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:28.264332    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5b04055d-5f9db46f6d-sfcpf"]
Nov 02 20:35:28.266536 ip-10-0-133-9 kubenswrapper[4463]: W1102 20:35:28.266510    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod352d3c8c_a544_424e_9a29_f36c353e706d.slice/crio-7af5cc7e25d3c59b215db614d640d1249c4d59fc1c9cf2d19f0b4ce32b8a805a WatchSource:0}: Error finding container 7af5cc7e25d3c59b215db614d640d1249c4d59fc1c9cf2d19f0b4ce32b8a805a: Status 404 returned error can't find the container with id 7af5cc7e25d3c59b215db614d640d1249c4d59fc1c9cf2d19f0b4ce32b8a805a
Nov 02 20:35:28.309770 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:28.309744    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5b04055d-7d9574f7d5-8pnkc"]
Nov 02 20:35:28.312548 ip-10-0-133-9 kubenswrapper[4463]: W1102 20:35:28.312523    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68891943_f1fa_42f4_8ebb_50f1d768b23e.slice/crio-07debdd5332f9dc7f2ced303297304a29522bd0c9dad3c0c485cf1edc7aa3d47 WatchSource:0}: Error finding container 07debdd5332f9dc7f2ced303297304a29522bd0c9dad3c0c485cf1edc7aa3d47: Status 404 returned error can't find the container with id 07debdd5332f9dc7f2ced303297304a29522bd0c9dad3c0c485cf1edc7aa3d47
Nov 02 20:35:28.936787 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:28.936746    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b04055d-5f9db46f6d-sfcpf" event={"ID":"352d3c8c-a544-424e-9a29-f36c353e706d","Type":"ContainerStarted","Data":"7af5cc7e25d3c59b215db614d640d1249c4d59fc1c9cf2d19f0b4ce32b8a805a"}
Nov 02 20:35:28.938261 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:28.938235    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5b04055d-597945b9c5-5rmbf" event={"ID":"d7efe97f-30f2-4d56-a8fc-7e29f23a70f9","Type":"ContainerStarted","Data":"91a5679e946a702e33cf2f5a2c880489c679d88e25abbabf282602cd1dfd7d1d"}
Nov 02 20:35:28.938422 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:28.938273    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5b04055d-597945b9c5-5rmbf" event={"ID":"d7efe97f-30f2-4d56-a8fc-7e29f23a70f9","Type":"ContainerStarted","Data":"2f630f61d0be615032ce773a4d01f4bffe25a14d29a86b6b98dc43eb0cdc3b2a"}
Nov 02 20:35:28.938422 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:28.938333    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-5b04055d-597945b9c5-5rmbf"
Nov 02 20:35:28.939750 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:28.939729    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5b04055d-7d9574f7d5-8pnkc" event={"ID":"68891943-f1fa-42f4-8ebb-50f1d768b23e","Type":"ContainerStarted","Data":"84e695031ead88522d469e11ce9ce61cd2e4fc6f4050a3071cbb5d347bca25fa"}
Nov 02 20:35:28.939832 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:28.939769    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5b04055d-7d9574f7d5-8pnkc" event={"ID":"68891943-f1fa-42f4-8ebb-50f1d768b23e","Type":"ContainerStarted","Data":"07debdd5332f9dc7f2ced303297304a29522bd0c9dad3c0c485cf1edc7aa3d47"}
Nov 02 20:35:28.939832 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:28.939797    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-5b04055d-7d9574f7d5-8pnkc"
Nov 02 20:35:29.056752 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:29.056708    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-5b04055d-597945b9c5-5rmbf" podStartSLOduration=2.056694024 podStartE2EDuration="2.056694024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:35:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:35:29.054505925 +0000 UTC m=+4035.661539694" watchObservedRunningTime="2026-11-02 20:35:29.056694024 +0000 UTC m=+4035.663727851"
Nov 02 20:35:29.171302 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:29.171250    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-5b04055d-7d9574f7d5-8pnkc" podStartSLOduration=2.171235189 podStartE2EDuration="2.171235189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:35:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:35:29.169598012 +0000 UTC m=+4035.776631791" watchObservedRunningTime="2026-11-02 20:35:29.171235189 +0000 UTC m=+4035.778268955"
Nov 02 20:35:29.948239 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:29.948200    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b04055d-5f9db46f6d-sfcpf" event={"ID":"352d3c8c-a544-424e-9a29-f36c353e706d","Type":"ContainerStarted","Data":"097dd23a58aa4e5c0b6ee3311df799da146ffd1f906228cfe8f0d7bbd9112fa3"}
Nov 02 20:35:29.948584 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:29.948559    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b04055d-5f9db46f6d-sfcpf"
Nov 02 20:35:30.055236 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:30.055186    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b04055d-5f9db46f6d-sfcpf" podStartSLOduration=3.273071368 podStartE2EDuration="4.0551673s" totalImagesPullingTime="782.095932ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:35:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:35:30.053726096 +0000 UTC m=+4036.660759884" watchObservedRunningTime="2026-11-02 20:35:30.0551673 +0000 UTC m=+4036.662201063"
Nov 02 20:35:34.293573 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:35:34.293529    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 20:35:34.718473 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:34.718442    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 20:35:35.959228 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:35.959196    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b04055d-5f9db46f6d-sfcpf"
Nov 02 20:35:39.951332 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:39.951302    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-5b04055d-597945b9c5-5rmbf"
Nov 02 20:35:39.951833 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:35:39.951378    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-5b04055d-7d9574f7d5-8pnkc"
Nov 02 20:36:04.292935 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:36:04.292899    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 20:36:06.586394 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:06.586351    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5b04055d-7d9574f7d5-8pnkc"]
Nov 02 20:36:06.700763 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:06.700728    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5b04055d-597945b9c5-5rmbf"]
Nov 02 20:36:11.360748 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:11.360707    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5b04055d-5f9db46f6d-sfcpf"]
Nov 02 20:36:11.361285 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:11.360954    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b04055d-5f9db46f6d-sfcpf" podUID="352d3c8c-a544-424e-9a29-f36c353e706d" containerName="maas-api" containerID="cri-o://097dd23a58aa4e5c0b6ee3311df799da146ffd1f906228cfe8f0d7bbd9112fa3" gracePeriod=30
Nov 02 20:36:11.586817 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:11.586761    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-5b04055d-7d9574f7d5-8pnkc" podUID="68891943-f1fa-42f4-8ebb-50f1d768b23e" containerName="payload-processing" containerID="cri-o://84e695031ead88522d469e11ce9ce61cd2e4fc6f4050a3071cbb5d347bca25fa" gracePeriod=25
Nov 02 20:36:11.701845 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:11.701787    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-5b04055d-597945b9c5-5rmbf" podUID="d7efe97f-30f2-4d56-a8fc-7e29f23a70f9" containerName="payload-pre-processing" containerID="cri-o://91a5679e946a702e33cf2f5a2c880489c679d88e25abbabf282602cd1dfd7d1d" gracePeriod=25
Nov 02 20:36:11.711604 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:11.711583    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b04055d-5f9db46f6d-sfcpf"
Nov 02 20:36:11.751159 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:11.751128    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/352d3c8c-a544-424e-9a29-f36c353e706d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/352d3c8c-a544-424e-9a29-f36c353e706d-openshift-service-ca\") pod \"352d3c8c-a544-424e-9a29-f36c353e706d\" (UID: \"352d3c8c-a544-424e-9a29-f36c353e706d\") "
Nov 02 20:36:11.751345 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:11.751182    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/352d3c8c-a544-424e-9a29-f36c353e706d-kube-api-access-6vz2r\" (UniqueName: \"kubernetes.io/projected/352d3c8c-a544-424e-9a29-f36c353e706d-kube-api-access-6vz2r\") pod \"352d3c8c-a544-424e-9a29-f36c353e706d\" (UID: \"352d3c8c-a544-424e-9a29-f36c353e706d\") "
Nov 02 20:36:11.751345 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:11.751262    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/352d3c8c-a544-424e-9a29-f36c353e706d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/352d3c8c-a544-424e-9a29-f36c353e706d-maas-api-tls\") pod \"352d3c8c-a544-424e-9a29-f36c353e706d\" (UID: \"352d3c8c-a544-424e-9a29-f36c353e706d\") "
Nov 02 20:36:11.751786 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:11.751394    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/352d3c8c-a544-424e-9a29-f36c353e706d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/352d3c8c-a544-424e-9a29-f36c353e706d-metrics-certs\") pod \"352d3c8c-a544-424e-9a29-f36c353e706d\" (UID: \"352d3c8c-a544-424e-9a29-f36c353e706d\") "
Nov 02 20:36:11.751786 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:11.751547    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/352d3c8c-a544-424e-9a29-f36c353e706d-openshift-service-ca" pod "352d3c8c-a544-424e-9a29-f36c353e706d" (UID: "352d3c8c-a544-424e-9a29-f36c353e706d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:36:11.751786 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:11.751728    4463 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/352d3c8c-a544-424e-9a29-f36c353e706d-openshift-service-ca\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:36:11.754061 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:11.754028    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/352d3c8c-a544-424e-9a29-f36c353e706d-kube-api-access-6vz2r" pod "352d3c8c-a544-424e-9a29-f36c353e706d" (UID: "352d3c8c-a544-424e-9a29-f36c353e706d"). InnerVolumeSpecName "kube-api-access-6vz2r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:36:11.754645 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:11.754608    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/352d3c8c-a544-424e-9a29-f36c353e706d-maas-api-tls" pod "352d3c8c-a544-424e-9a29-f36c353e706d" (UID: "352d3c8c-a544-424e-9a29-f36c353e706d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:36:11.755282 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:11.755247    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/352d3c8c-a544-424e-9a29-f36c353e706d-metrics-certs" pod "352d3c8c-a544-424e-9a29-f36c353e706d" (UID: "352d3c8c-a544-424e-9a29-f36c353e706d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:36:11.833370 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:11.833328    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5b04055d-7d9574f7d5-8pnkc"
Nov 02 20:36:11.852648 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:11.852568    4463 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/352d3c8c-a544-424e-9a29-f36c353e706d-metrics-certs\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:36:11.852648 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:11.852606    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vz2r\" (UniqueName: \"kubernetes.io/projected/352d3c8c-a544-424e-9a29-f36c353e706d-kube-api-access-6vz2r\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:36:11.852648 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:11.852621    4463 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/352d3c8c-a544-424e-9a29-f36c353e706d-maas-api-tls\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:36:11.941320 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:11.941008    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5b04055d-597945b9c5-5rmbf"
Nov 02 20:36:11.953801 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:11.953774    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68891943-f1fa-42f4-8ebb-50f1d768b23e-kube-api-access-wk7v2\" (UniqueName: \"kubernetes.io/projected/68891943-f1fa-42f4-8ebb-50f1d768b23e-kube-api-access-wk7v2\") pod \"68891943-f1fa-42f4-8ebb-50f1d768b23e\" (UID: \"68891943-f1fa-42f4-8ebb-50f1d768b23e\") "
Nov 02 20:36:11.953950 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:11.953838    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68891943-f1fa-42f4-8ebb-50f1d768b23e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68891943-f1fa-42f4-8ebb-50f1d768b23e-plugins-config-volume\") pod \"68891943-f1fa-42f4-8ebb-50f1d768b23e\" (UID: \"68891943-f1fa-42f4-8ebb-50f1d768b23e\") "
Nov 02 20:36:11.954233 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:11.954209    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68891943-f1fa-42f4-8ebb-50f1d768b23e-plugins-config-volume" pod "68891943-f1fa-42f4-8ebb-50f1d768b23e" (UID: "68891943-f1fa-42f4-8ebb-50f1d768b23e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:36:11.955794 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:11.955769    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68891943-f1fa-42f4-8ebb-50f1d768b23e-kube-api-access-wk7v2" pod "68891943-f1fa-42f4-8ebb-50f1d768b23e" (UID: "68891943-f1fa-42f4-8ebb-50f1d768b23e"). InnerVolumeSpecName "kube-api-access-wk7v2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:36:12.054783 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:12.054758    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7efe97f-30f2-4d56-a8fc-7e29f23a70f9-kube-api-access-2rlkp\" (UniqueName: \"kubernetes.io/projected/d7efe97f-30f2-4d56-a8fc-7e29f23a70f9-kube-api-access-2rlkp\") pod \"d7efe97f-30f2-4d56-a8fc-7e29f23a70f9\" (UID: \"d7efe97f-30f2-4d56-a8fc-7e29f23a70f9\") "
Nov 02 20:36:12.054951 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:12.054860    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7efe97f-30f2-4d56-a8fc-7e29f23a70f9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7efe97f-30f2-4d56-a8fc-7e29f23a70f9-plugins-config-volume\") pod \"d7efe97f-30f2-4d56-a8fc-7e29f23a70f9\" (UID: \"d7efe97f-30f2-4d56-a8fc-7e29f23a70f9\") "
Nov 02 20:36:12.055181 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:12.055164    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wk7v2\" (UniqueName: \"kubernetes.io/projected/68891943-f1fa-42f4-8ebb-50f1d768b23e-kube-api-access-wk7v2\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:36:12.055265 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:12.055188    4463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68891943-f1fa-42f4-8ebb-50f1d768b23e-plugins-config-volume\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:36:12.055265 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:12.055236    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7efe97f-30f2-4d56-a8fc-7e29f23a70f9-plugins-config-volume" pod "d7efe97f-30f2-4d56-a8fc-7e29f23a70f9" (UID: "d7efe97f-30f2-4d56-a8fc-7e29f23a70f9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:36:12.056835 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:12.056819    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7efe97f-30f2-4d56-a8fc-7e29f23a70f9-kube-api-access-2rlkp" pod "d7efe97f-30f2-4d56-a8fc-7e29f23a70f9" (UID: "d7efe97f-30f2-4d56-a8fc-7e29f23a70f9"). InnerVolumeSpecName "kube-api-access-2rlkp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:36:12.136710 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:12.136677    4463 generic.go:356] "Generic (PLEG): container finished" podID="d7efe97f-30f2-4d56-a8fc-7e29f23a70f9" containerID="91a5679e946a702e33cf2f5a2c880489c679d88e25abbabf282602cd1dfd7d1d" exitCode=0
Nov 02 20:36:12.136838 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:12.136725    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5b04055d-597945b9c5-5rmbf"
Nov 02 20:36:12.136838 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:12.136765    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5b04055d-597945b9c5-5rmbf" event={"ID":"d7efe97f-30f2-4d56-a8fc-7e29f23a70f9","Type":"ContainerDied","Data":"91a5679e946a702e33cf2f5a2c880489c679d88e25abbabf282602cd1dfd7d1d"}
Nov 02 20:36:12.136838 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:12.136817    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5b04055d-597945b9c5-5rmbf" event={"ID":"d7efe97f-30f2-4d56-a8fc-7e29f23a70f9","Type":"ContainerDied","Data":"2f630f61d0be615032ce773a4d01f4bffe25a14d29a86b6b98dc43eb0cdc3b2a"}
Nov 02 20:36:12.136985 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:12.136846    4463 scope.go:122] "RemoveContainer" containerID="91a5679e946a702e33cf2f5a2c880489c679d88e25abbabf282602cd1dfd7d1d"
Nov 02 20:36:12.138073 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:12.138045    4463 generic.go:356] "Generic (PLEG): container finished" podID="68891943-f1fa-42f4-8ebb-50f1d768b23e" containerID="84e695031ead88522d469e11ce9ce61cd2e4fc6f4050a3071cbb5d347bca25fa" exitCode=0
Nov 02 20:36:12.138188 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:12.138129    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5b04055d-7d9574f7d5-8pnkc" event={"ID":"68891943-f1fa-42f4-8ebb-50f1d768b23e","Type":"ContainerDied","Data":"84e695031ead88522d469e11ce9ce61cd2e4fc6f4050a3071cbb5d347bca25fa"}
Nov 02 20:36:12.138188 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:12.138152    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5b04055d-7d9574f7d5-8pnkc"
Nov 02 20:36:12.138311 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:12.138191    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5b04055d-7d9574f7d5-8pnkc" event={"ID":"68891943-f1fa-42f4-8ebb-50f1d768b23e","Type":"ContainerDied","Data":"07debdd5332f9dc7f2ced303297304a29522bd0c9dad3c0c485cf1edc7aa3d47"}
Nov 02 20:36:12.139427 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:12.139406    4463 generic.go:356] "Generic (PLEG): container finished" podID="352d3c8c-a544-424e-9a29-f36c353e706d" containerID="097dd23a58aa4e5c0b6ee3311df799da146ffd1f906228cfe8f0d7bbd9112fa3" exitCode=0
Nov 02 20:36:12.139549 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:12.139439    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b04055d-5f9db46f6d-sfcpf" event={"ID":"352d3c8c-a544-424e-9a29-f36c353e706d","Type":"ContainerDied","Data":"097dd23a58aa4e5c0b6ee3311df799da146ffd1f906228cfe8f0d7bbd9112fa3"}
Nov 02 20:36:12.139549 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:12.139462    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b04055d-5f9db46f6d-sfcpf"
Nov 02 20:36:12.139549 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:12.139487    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b04055d-5f9db46f6d-sfcpf" event={"ID":"352d3c8c-a544-424e-9a29-f36c353e706d","Type":"ContainerDied","Data":"7af5cc7e25d3c59b215db614d640d1249c4d59fc1c9cf2d19f0b4ce32b8a805a"}
Nov 02 20:36:12.149848 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:12.149827    4463 scope.go:122] "RemoveContainer" containerID="91a5679e946a702e33cf2f5a2c880489c679d88e25abbabf282602cd1dfd7d1d"
Nov 02 20:36:12.150267 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:36:12.150237    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91a5679e946a702e33cf2f5a2c880489c679d88e25abbabf282602cd1dfd7d1d\": container with ID starting with 91a5679e946a702e33cf2f5a2c880489c679d88e25abbabf282602cd1dfd7d1d not found: ID does not exist" containerID="91a5679e946a702e33cf2f5a2c880489c679d88e25abbabf282602cd1dfd7d1d"
Nov 02 20:36:12.150386 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:12.150274    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91a5679e946a702e33cf2f5a2c880489c679d88e25abbabf282602cd1dfd7d1d"} err="failed to get container status \"91a5679e946a702e33cf2f5a2c880489c679d88e25abbabf282602cd1dfd7d1d\": rpc error: code = NotFound desc = could not find container \"91a5679e946a702e33cf2f5a2c880489c679d88e25abbabf282602cd1dfd7d1d\": container with ID starting with 91a5679e946a702e33cf2f5a2c880489c679d88e25abbabf282602cd1dfd7d1d not found: ID does not exist"
Nov 02 20:36:12.150386 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:12.150295    4463 scope.go:122] "RemoveContainer" containerID="84e695031ead88522d469e11ce9ce61cd2e4fc6f4050a3071cbb5d347bca25fa"
Nov 02 20:36:12.155742 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:12.155722    4463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7efe97f-30f2-4d56-a8fc-7e29f23a70f9-plugins-config-volume\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:36:12.155828 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:12.155746    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rlkp\" (UniqueName: \"kubernetes.io/projected/d7efe97f-30f2-4d56-a8fc-7e29f23a70f9-kube-api-access-2rlkp\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:36:12.158461 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:12.158443    4463 scope.go:122] "RemoveContainer" containerID="84e695031ead88522d469e11ce9ce61cd2e4fc6f4050a3071cbb5d347bca25fa"
Nov 02 20:36:12.158756 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:36:12.158736    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84e695031ead88522d469e11ce9ce61cd2e4fc6f4050a3071cbb5d347bca25fa\": container with ID starting with 84e695031ead88522d469e11ce9ce61cd2e4fc6f4050a3071cbb5d347bca25fa not found: ID does not exist" containerID="84e695031ead88522d469e11ce9ce61cd2e4fc6f4050a3071cbb5d347bca25fa"
Nov 02 20:36:12.158811 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:12.158763    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84e695031ead88522d469e11ce9ce61cd2e4fc6f4050a3071cbb5d347bca25fa"} err="failed to get container status \"84e695031ead88522d469e11ce9ce61cd2e4fc6f4050a3071cbb5d347bca25fa\": rpc error: code = NotFound desc = could not find container \"84e695031ead88522d469e11ce9ce61cd2e4fc6f4050a3071cbb5d347bca25fa\": container with ID starting with 84e695031ead88522d469e11ce9ce61cd2e4fc6f4050a3071cbb5d347bca25fa not found: ID does not exist"
Nov 02 20:36:12.158811 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:12.158780    4463 scope.go:122] "RemoveContainer" containerID="097dd23a58aa4e5c0b6ee3311df799da146ffd1f906228cfe8f0d7bbd9112fa3"
Nov 02 20:36:12.166007 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:12.165992    4463 scope.go:122] "RemoveContainer" containerID="097dd23a58aa4e5c0b6ee3311df799da146ffd1f906228cfe8f0d7bbd9112fa3"
Nov 02 20:36:12.166284 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:36:12.166266    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"097dd23a58aa4e5c0b6ee3311df799da146ffd1f906228cfe8f0d7bbd9112fa3\": container with ID starting with 097dd23a58aa4e5c0b6ee3311df799da146ffd1f906228cfe8f0d7bbd9112fa3 not found: ID does not exist" containerID="097dd23a58aa4e5c0b6ee3311df799da146ffd1f906228cfe8f0d7bbd9112fa3"
Nov 02 20:36:12.166327 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:12.166289    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"097dd23a58aa4e5c0b6ee3311df799da146ffd1f906228cfe8f0d7bbd9112fa3"} err="failed to get container status \"097dd23a58aa4e5c0b6ee3311df799da146ffd1f906228cfe8f0d7bbd9112fa3\": rpc error: code = NotFound desc = could not find container \"097dd23a58aa4e5c0b6ee3311df799da146ffd1f906228cfe8f0d7bbd9112fa3\": container with ID starting with 097dd23a58aa4e5c0b6ee3311df799da146ffd1f906228cfe8f0d7bbd9112fa3 not found: ID does not exist"
Nov 02 20:36:12.290350 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:12.290263    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5b04055d-597945b9c5-5rmbf"]
Nov 02 20:36:12.292613 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:12.292584    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5b04055d-597945b9c5-5rmbf"]
Nov 02 20:36:12.394354 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:12.394320    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5b04055d-7d9574f7d5-8pnkc"]
Nov 02 20:36:12.398187 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:12.398160    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5b04055d-7d9574f7d5-8pnkc"]
Nov 02 20:36:12.499052 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:12.499023    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5b04055d-5f9db46f6d-sfcpf"]
Nov 02 20:36:12.503119 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:12.503095    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5b04055d-5f9db46f6d-sfcpf"]
Nov 02 20:36:13.582457 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:13.582422    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="352d3c8c-a544-424e-9a29-f36c353e706d" path="/var/lib/kubelet/pods/352d3c8c-a544-424e-9a29-f36c353e706d/volumes"
Nov 02 20:36:13.582827 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:13.582800    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68891943-f1fa-42f4-8ebb-50f1d768b23e" path="/var/lib/kubelet/pods/68891943-f1fa-42f4-8ebb-50f1d768b23e/volumes"
Nov 02 20:36:13.583138 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:13.583125    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7efe97f-30f2-4d56-a8fc-7e29f23a70f9" path="/var/lib/kubelet/pods/d7efe97f-30f2-4d56-a8fc-7e29f23a70f9/volumes"
Nov 02 20:36:15.657007 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:15.656969    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 20:36:17.849766 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:17.849730    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-authcfg-http-cfa6c5-openshift-default-679d974589-kjk92"]
Nov 02 20:36:17.850200 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:17.850115    4463 cpu_manager.go:414] "containerMap: removing container" podUID="352d3c8c-a544-424e-9a29-f36c353e706d" containerName="maas-api"
Nov 02 20:36:17.850200 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:17.850127    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="352d3c8c-a544-424e-9a29-f36c353e706d" containerName="maas-api"
Nov 02 20:36:17.850200 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:17.850146    4463 cpu_manager.go:414] "containerMap: removing container" podUID="d7efe97f-30f2-4d56-a8fc-7e29f23a70f9" containerName="payload-pre-processing"
Nov 02 20:36:17.850200 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:17.850152    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7efe97f-30f2-4d56-a8fc-7e29f23a70f9" containerName="payload-pre-processing"
Nov 02 20:36:17.850200 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:17.850173    4463 cpu_manager.go:414] "containerMap: removing container" podUID="68891943-f1fa-42f4-8ebb-50f1d768b23e" containerName="payload-processing"
Nov 02 20:36:17.850200 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:17.850178    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68891943-f1fa-42f4-8ebb-50f1d768b23e" containerName="payload-processing"
Nov 02 20:36:17.850414 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:17.850232    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7efe97f-30f2-4d56-a8fc-7e29f23a70f9" containerName="payload-pre-processing"
Nov 02 20:36:17.850414 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:17.850243    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="68891943-f1fa-42f4-8ebb-50f1d768b23e" containerName="payload-processing"
Nov 02 20:36:17.850414 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:17.850251    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="352d3c8c-a544-424e-9a29-f36c353e706d" containerName="maas-api"
Nov 02 20:36:17.855151 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:17.855132    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-cfa6c5-openshift-default-679d974589-kjk92"
Nov 02 20:36:17.911654 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:17.911618    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/877de52e-ed20-431b-a88e-85bca29f9a4a-istio-data\") pod \"e2e-authcfg-http-cfa6c5-openshift-default-679d974589-kjk92\" (UID: \"877de52e-ed20-431b-a88e-85bca29f9a4a\") " pod="openshift-ingress/e2e-authcfg-http-cfa6c5-openshift-default-679d974589-kjk92"
Nov 02 20:36:17.911654 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:17.911659    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/877de52e-ed20-431b-a88e-85bca29f9a4a-istiod-ca-cert\") pod \"e2e-authcfg-http-cfa6c5-openshift-default-679d974589-kjk92\" (UID: \"877de52e-ed20-431b-a88e-85bca29f9a4a\") " pod="openshift-ingress/e2e-authcfg-http-cfa6c5-openshift-default-679d974589-kjk92"
Nov 02 20:36:17.911930 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:17.911691    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/877de52e-ed20-431b-a88e-85bca29f9a4a-credential-socket\") pod \"e2e-authcfg-http-cfa6c5-openshift-default-679d974589-kjk92\" (UID: \"877de52e-ed20-431b-a88e-85bca29f9a4a\") " pod="openshift-ingress/e2e-authcfg-http-cfa6c5-openshift-default-679d974589-kjk92"
Nov 02 20:36:17.911930 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:17.911711    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/877de52e-ed20-431b-a88e-85bca29f9a4a-workload-socket\") pod \"e2e-authcfg-http-cfa6c5-openshift-default-679d974589-kjk92\" (UID: \"877de52e-ed20-431b-a88e-85bca29f9a4a\") " pod="openshift-ingress/e2e-authcfg-http-cfa6c5-openshift-default-679d974589-kjk92"
Nov 02 20:36:17.911930 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:17.911871    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/877de52e-ed20-431b-a88e-85bca29f9a4a-istio-token\") pod \"e2e-authcfg-http-cfa6c5-openshift-default-679d974589-kjk92\" (UID: \"877de52e-ed20-431b-a88e-85bca29f9a4a\") " pod="openshift-ingress/e2e-authcfg-http-cfa6c5-openshift-default-679d974589-kjk92"
Nov 02 20:36:17.912106 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:17.911961    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/877de52e-ed20-431b-a88e-85bca29f9a4a-istio-podinfo\") pod \"e2e-authcfg-http-cfa6c5-openshift-default-679d974589-kjk92\" (UID: \"877de52e-ed20-431b-a88e-85bca29f9a4a\") " pod="openshift-ingress/e2e-authcfg-http-cfa6c5-openshift-default-679d974589-kjk92"
Nov 02 20:36:17.912106 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:17.911995    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx4mb\" (UniqueName: \"kubernetes.io/projected/877de52e-ed20-431b-a88e-85bca29f9a4a-kube-api-access-lx4mb\") pod \"e2e-authcfg-http-cfa6c5-openshift-default-679d974589-kjk92\" (UID: \"877de52e-ed20-431b-a88e-85bca29f9a4a\") " pod="openshift-ingress/e2e-authcfg-http-cfa6c5-openshift-default-679d974589-kjk92"
Nov 02 20:36:17.912106 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:17.912047    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/877de52e-ed20-431b-a88e-85bca29f9a4a-workload-certs\") pod \"e2e-authcfg-http-cfa6c5-openshift-default-679d974589-kjk92\" (UID: \"877de52e-ed20-431b-a88e-85bca29f9a4a\") " pod="openshift-ingress/e2e-authcfg-http-cfa6c5-openshift-default-679d974589-kjk92"
Nov 02 20:36:17.912274 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:17.912130    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/877de52e-ed20-431b-a88e-85bca29f9a4a-istio-envoy\") pod \"e2e-authcfg-http-cfa6c5-openshift-default-679d974589-kjk92\" (UID: \"877de52e-ed20-431b-a88e-85bca29f9a4a\") " pod="openshift-ingress/e2e-authcfg-http-cfa6c5-openshift-default-679d974589-kjk92"
Nov 02 20:36:17.945041 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:17.944986    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-authcfg-http-cfa6c5-openshift-default-dockercfg-qhnsc\""
Nov 02 20:36:17.952453 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:17.952419    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-cfa6c5-openshift-default-679d974589-kjk92"]
Nov 02 20:36:18.012984 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:18.012948    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/877de52e-ed20-431b-a88e-85bca29f9a4a-istio-data\") pod \"e2e-authcfg-http-cfa6c5-openshift-default-679d974589-kjk92\" (UID: \"877de52e-ed20-431b-a88e-85bca29f9a4a\") " pod="openshift-ingress/e2e-authcfg-http-cfa6c5-openshift-default-679d974589-kjk92"
Nov 02 20:36:18.013179 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:18.012993    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/877de52e-ed20-431b-a88e-85bca29f9a4a-istiod-ca-cert\") pod \"e2e-authcfg-http-cfa6c5-openshift-default-679d974589-kjk92\" (UID: \"877de52e-ed20-431b-a88e-85bca29f9a4a\") " pod="openshift-ingress/e2e-authcfg-http-cfa6c5-openshift-default-679d974589-kjk92"
Nov 02 20:36:18.013179 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:18.013056    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/877de52e-ed20-431b-a88e-85bca29f9a4a-credential-socket\") pod \"e2e-authcfg-http-cfa6c5-openshift-default-679d974589-kjk92\" (UID: \"877de52e-ed20-431b-a88e-85bca29f9a4a\") " pod="openshift-ingress/e2e-authcfg-http-cfa6c5-openshift-default-679d974589-kjk92"
Nov 02 20:36:18.013179 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:18.013082    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/877de52e-ed20-431b-a88e-85bca29f9a4a-workload-socket\") pod \"e2e-authcfg-http-cfa6c5-openshift-default-679d974589-kjk92\" (UID: \"877de52e-ed20-431b-a88e-85bca29f9a4a\") " pod="openshift-ingress/e2e-authcfg-http-cfa6c5-openshift-default-679d974589-kjk92"
Nov 02 20:36:18.013179 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:18.013117    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/877de52e-ed20-431b-a88e-85bca29f9a4a-istio-token\") pod \"e2e-authcfg-http-cfa6c5-openshift-default-679d974589-kjk92\" (UID: \"877de52e-ed20-431b-a88e-85bca29f9a4a\") " pod="openshift-ingress/e2e-authcfg-http-cfa6c5-openshift-default-679d974589-kjk92"
Nov 02 20:36:18.013179 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:18.013168    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/877de52e-ed20-431b-a88e-85bca29f9a4a-istio-podinfo\") pod \"e2e-authcfg-http-cfa6c5-openshift-default-679d974589-kjk92\" (UID: \"877de52e-ed20-431b-a88e-85bca29f9a4a\") " pod="openshift-ingress/e2e-authcfg-http-cfa6c5-openshift-default-679d974589-kjk92"
Nov 02 20:36:18.014559 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:18.013208    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx4mb\" (UniqueName: \"kubernetes.io/projected/877de52e-ed20-431b-a88e-85bca29f9a4a-kube-api-access-lx4mb\") pod \"e2e-authcfg-http-cfa6c5-openshift-default-679d974589-kjk92\" (UID: \"877de52e-ed20-431b-a88e-85bca29f9a4a\") " pod="openshift-ingress/e2e-authcfg-http-cfa6c5-openshift-default-679d974589-kjk92"
Nov 02 20:36:18.014559 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:18.013239    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/877de52e-ed20-431b-a88e-85bca29f9a4a-workload-certs\") pod \"e2e-authcfg-http-cfa6c5-openshift-default-679d974589-kjk92\" (UID: \"877de52e-ed20-431b-a88e-85bca29f9a4a\") " pod="openshift-ingress/e2e-authcfg-http-cfa6c5-openshift-default-679d974589-kjk92"
Nov 02 20:36:18.014559 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:18.013270    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/877de52e-ed20-431b-a88e-85bca29f9a4a-istio-envoy\") pod \"e2e-authcfg-http-cfa6c5-openshift-default-679d974589-kjk92\" (UID: \"877de52e-ed20-431b-a88e-85bca29f9a4a\") " pod="openshift-ingress/e2e-authcfg-http-cfa6c5-openshift-default-679d974589-kjk92"
Nov 02 20:36:18.014559 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:18.013678    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/877de52e-ed20-431b-a88e-85bca29f9a4a-istio-data\") pod \"e2e-authcfg-http-cfa6c5-openshift-default-679d974589-kjk92\" (UID: \"877de52e-ed20-431b-a88e-85bca29f9a4a\") " pod="openshift-ingress/e2e-authcfg-http-cfa6c5-openshift-default-679d974589-kjk92"
Nov 02 20:36:18.014559 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:18.014061    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/877de52e-ed20-431b-a88e-85bca29f9a4a-credential-socket\") pod \"e2e-authcfg-http-cfa6c5-openshift-default-679d974589-kjk92\" (UID: \"877de52e-ed20-431b-a88e-85bca29f9a4a\") " pod="openshift-ingress/e2e-authcfg-http-cfa6c5-openshift-default-679d974589-kjk92"
Nov 02 20:36:18.014559 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:18.014227    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/877de52e-ed20-431b-a88e-85bca29f9a4a-workload-socket\") pod \"e2e-authcfg-http-cfa6c5-openshift-default-679d974589-kjk92\" (UID: \"877de52e-ed20-431b-a88e-85bca29f9a4a\") " pod="openshift-ingress/e2e-authcfg-http-cfa6c5-openshift-default-679d974589-kjk92"
Nov 02 20:36:18.014559 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:18.014239    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/877de52e-ed20-431b-a88e-85bca29f9a4a-istiod-ca-cert\") pod \"e2e-authcfg-http-cfa6c5-openshift-default-679d974589-kjk92\" (UID: \"877de52e-ed20-431b-a88e-85bca29f9a4a\") " pod="openshift-ingress/e2e-authcfg-http-cfa6c5-openshift-default-679d974589-kjk92"
Nov 02 20:36:18.014559 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:18.014484    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/877de52e-ed20-431b-a88e-85bca29f9a4a-workload-certs\") pod \"e2e-authcfg-http-cfa6c5-openshift-default-679d974589-kjk92\" (UID: \"877de52e-ed20-431b-a88e-85bca29f9a4a\") " pod="openshift-ingress/e2e-authcfg-http-cfa6c5-openshift-default-679d974589-kjk92"
Nov 02 20:36:18.016750 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:18.016726    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/877de52e-ed20-431b-a88e-85bca29f9a4a-istio-podinfo\") pod \"e2e-authcfg-http-cfa6c5-openshift-default-679d974589-kjk92\" (UID: \"877de52e-ed20-431b-a88e-85bca29f9a4a\") " pod="openshift-ingress/e2e-authcfg-http-cfa6c5-openshift-default-679d974589-kjk92"
Nov 02 20:36:18.016979 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:18.016960    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/877de52e-ed20-431b-a88e-85bca29f9a4a-istio-envoy\") pod \"e2e-authcfg-http-cfa6c5-openshift-default-679d974589-kjk92\" (UID: \"877de52e-ed20-431b-a88e-85bca29f9a4a\") " pod="openshift-ingress/e2e-authcfg-http-cfa6c5-openshift-default-679d974589-kjk92"
Nov 02 20:36:18.066869 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:18.066833    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/877de52e-ed20-431b-a88e-85bca29f9a4a-istio-token\") pod \"e2e-authcfg-http-cfa6c5-openshift-default-679d974589-kjk92\" (UID: \"877de52e-ed20-431b-a88e-85bca29f9a4a\") " pod="openshift-ingress/e2e-authcfg-http-cfa6c5-openshift-default-679d974589-kjk92"
Nov 02 20:36:18.067142 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:18.067119    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx4mb\" (UniqueName: \"kubernetes.io/projected/877de52e-ed20-431b-a88e-85bca29f9a4a-kube-api-access-lx4mb\") pod \"e2e-authcfg-http-cfa6c5-openshift-default-679d974589-kjk92\" (UID: \"877de52e-ed20-431b-a88e-85bca29f9a4a\") " pod="openshift-ingress/e2e-authcfg-http-cfa6c5-openshift-default-679d974589-kjk92"
Nov 02 20:36:18.167000 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:18.166916    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-cfa6c5-openshift-default-679d974589-kjk92"
Nov 02 20:36:18.579572 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:18.579546    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-cfa6c5-openshift-default-679d974589-kjk92"]
Nov 02 20:36:18.581366 ip-10-0-133-9 kubenswrapper[4463]: W1102 20:36:18.581335    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod877de52e_ed20_431b_a88e_85bca29f9a4a.slice/crio-9902492635dc39b5824448cfee5d81c1a373d3640a82131470a7133612b58763 WatchSource:0}: Error finding container 9902492635dc39b5824448cfee5d81c1a373d3640a82131470a7133612b58763: Status 404 returned error can't find the container with id 9902492635dc39b5824448cfee5d81c1a373d3640a82131470a7133612b58763
Nov 02 20:36:18.583584 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:18.583552    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:36:18.583748 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:18.583614    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:36:18.583748 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:18.583639    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:36:19.173095 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:19.173050    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-cfa6c5-openshift-default-679d974589-kjk92" event={"ID":"877de52e-ed20-431b-a88e-85bca29f9a4a","Type":"ContainerStarted","Data":"73e23dbc75b058c5fae76e5915aefae48eff00b322048fb60286d4365be8f6af"}
Nov 02 20:36:19.173507 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:19.173148    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-cfa6c5-openshift-default-679d974589-kjk92" event={"ID":"877de52e-ed20-431b-a88e-85bca29f9a4a","Type":"ContainerStarted","Data":"9902492635dc39b5824448cfee5d81c1a373d3640a82131470a7133612b58763"}
Nov 02 20:36:19.281783 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:19.281719    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-authcfg-http-cfa6c5-openshift-default-679d974589-kjk92" podStartSLOduration=2.281701413 podStartE2EDuration="2.281701413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:36:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:36:19.281269102 +0000 UTC m=+4085.888302882" watchObservedRunningTime="2026-11-02 20:36:19.281701413 +0000 UTC m=+4085.888735181"
Nov 02 20:36:20.167451 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:20.167416    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-cfa6c5-openshift-default-679d974589-kjk92"
Nov 02 20:36:20.172689 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:20.172665    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-cfa6c5-openshift-default-679d974589-kjk92"
Nov 02 20:36:20.178261 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:20.178238    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-cfa6c5-openshift-default-679d974589-kjk92"
Nov 02 20:36:20.179091 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:20.179069    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-cfa6c5-openshift-default-679d974589-kjk92"
Nov 02 20:36:25.803591 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:25.803550    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-cfa6c5-54b7955cc5-wjfgd"]
Nov 02 20:36:25.808397 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:25.808367    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-cfa6c5-54b7955cc5-wjfgd"
Nov 02 20:36:25.859712 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:25.859681    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-authcfg-http-cfa6c5\""
Nov 02 20:36:25.888231 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:25.888195    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q87ww\" (UniqueName: \"kubernetes.io/projected/b42b3241-59a2-4493-a057-8b0656d60625-kube-api-access-q87ww\") pod \"payload-processing-e2e-authcfg-http-cfa6c5-54b7955cc5-wjfgd\" (UID: \"b42b3241-59a2-4493-a057-8b0656d60625\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-cfa6c5-54b7955cc5-wjfgd"
Nov 02 20:36:25.888410 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:25.888303    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b42b3241-59a2-4493-a057-8b0656d60625-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-cfa6c5-54b7955cc5-wjfgd\" (UID: \"b42b3241-59a2-4493-a057-8b0656d60625\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-cfa6c5-54b7955cc5-wjfgd"
Nov 02 20:36:25.905925 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:25.905897    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-authcfg-http-cfa6c5-dockercfg-lpnvq\""
Nov 02 20:36:25.907573 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:25.907552    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-cfa6c5-54b7955cc5-wjfgd"]
Nov 02 20:36:25.988808 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:25.988778    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q87ww\" (UniqueName: \"kubernetes.io/projected/b42b3241-59a2-4493-a057-8b0656d60625-kube-api-access-q87ww\") pod \"payload-processing-e2e-authcfg-http-cfa6c5-54b7955cc5-wjfgd\" (UID: \"b42b3241-59a2-4493-a057-8b0656d60625\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-cfa6c5-54b7955cc5-wjfgd"
Nov 02 20:36:25.989001 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:25.988837    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b42b3241-59a2-4493-a057-8b0656d60625-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-cfa6c5-54b7955cc5-wjfgd\" (UID: \"b42b3241-59a2-4493-a057-8b0656d60625\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-cfa6c5-54b7955cc5-wjfgd"
Nov 02 20:36:25.989547 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:25.989525    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b42b3241-59a2-4493-a057-8b0656d60625-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-cfa6c5-54b7955cc5-wjfgd\" (UID: \"b42b3241-59a2-4493-a057-8b0656d60625\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-cfa6c5-54b7955cc5-wjfgd"
Nov 02 20:36:26.042537 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:26.042500    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q87ww\" (UniqueName: \"kubernetes.io/projected/b42b3241-59a2-4493-a057-8b0656d60625-kube-api-access-q87ww\") pod \"payload-processing-e2e-authcfg-http-cfa6c5-54b7955cc5-wjfgd\" (UID: \"b42b3241-59a2-4493-a057-8b0656d60625\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-cfa6c5-54b7955cc5-wjfgd"
Nov 02 20:36:26.130034 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:26.129917    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-cfa6c5-54b7955cc5-wjfgd"
Nov 02 20:36:26.493156 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:26.493126    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-cfa6c5-54b7955cc5-wjfgd"]
Nov 02 20:36:26.495711 ip-10-0-133-9 kubenswrapper[4463]: W1102 20:36:26.495687    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb42b3241_59a2_4493_a057_8b0656d60625.slice/crio-a9fddf2213452ce46393d119239f8fb52ccc43f8b396ae1c587fd88bb59b3fa6 WatchSource:0}: Error finding container a9fddf2213452ce46393d119239f8fb52ccc43f8b396ae1c587fd88bb59b3fa6: Status 404 returned error can't find the container with id a9fddf2213452ce46393d119239f8fb52ccc43f8b396ae1c587fd88bb59b3fa6
Nov 02 20:36:27.218912 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:27.218870    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-cfa6c5-54b7955cc5-wjfgd" event={"ID":"b42b3241-59a2-4493-a057-8b0656d60625","Type":"ContainerStarted","Data":"c15dcfe906098fb8d24e2123f0653eaa959ad6f2e5a36a531253f775ae925faa"}
Nov 02 20:36:27.219440 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:27.218941    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-cfa6c5-54b7955cc5-wjfgd" event={"ID":"b42b3241-59a2-4493-a057-8b0656d60625","Type":"ContainerStarted","Data":"a9fddf2213452ce46393d119239f8fb52ccc43f8b396ae1c587fd88bb59b3fa6"}
Nov 02 20:36:27.219440 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:27.218973    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-cfa6c5-54b7955cc5-wjfgd"
Nov 02 20:36:27.324802 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:27.324747    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-authcfg-http-cfa6c5-54b7955cc5-wjfgd" podStartSLOduration=2.3247341280000002 podStartE2EDuration="2.324734128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:36:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:36:27.323890942 +0000 UTC m=+4093.930924712" watchObservedRunningTime="2026-11-02 20:36:27.324734128 +0000 UTC m=+4093.931767895"
Nov 02 20:36:27.583660 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:27.583589    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-cfa6c5-558d8df998-xnjhm"]
Nov 02 20:36:27.586998 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:27.586976    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-cfa6c5-558d8df998-xnjhm"
Nov 02 20:36:27.637150 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:27.637125    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-authcfg-http-cfa6c5-serving-cert\""
Nov 02 20:36:27.639888 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:27.639867    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-cfa6c5-558d8df998-xnjhm"]
Nov 02 20:36:27.706068 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:27.706042    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f0c2f2b-202a-4391-83e0-8c84bb587119-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-cfa6c5-558d8df998-xnjhm\" (UID: \"9f0c2f2b-202a-4391-83e0-8c84bb587119\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-cfa6c5-558d8df998-xnjhm"
Nov 02 20:36:27.706234 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:27.706212    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f0c2f2b-202a-4391-83e0-8c84bb587119-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-cfa6c5-558d8df998-xnjhm\" (UID: \"9f0c2f2b-202a-4391-83e0-8c84bb587119\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-cfa6c5-558d8df998-xnjhm"
Nov 02 20:36:27.706302 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:27.706288    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lflv\" (UniqueName: \"kubernetes.io/projected/9f0c2f2b-202a-4391-83e0-8c84bb587119-kube-api-access-4lflv\") pod \"maas-api-e2e-authcfg-http-cfa6c5-558d8df998-xnjhm\" (UID: \"9f0c2f2b-202a-4391-83e0-8c84bb587119\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-cfa6c5-558d8df998-xnjhm"
Nov 02 20:36:27.706503 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:27.706482    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f0c2f2b-202a-4391-83e0-8c84bb587119-metrics-certs\") pod \"maas-api-e2e-authcfg-http-cfa6c5-558d8df998-xnjhm\" (UID: \"9f0c2f2b-202a-4391-83e0-8c84bb587119\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-cfa6c5-558d8df998-xnjhm"
Nov 02 20:36:27.807925 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:27.807889    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f0c2f2b-202a-4391-83e0-8c84bb587119-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-cfa6c5-558d8df998-xnjhm\" (UID: \"9f0c2f2b-202a-4391-83e0-8c84bb587119\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-cfa6c5-558d8df998-xnjhm"
Nov 02 20:36:27.808137 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:27.807960    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f0c2f2b-202a-4391-83e0-8c84bb587119-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-cfa6c5-558d8df998-xnjhm\" (UID: \"9f0c2f2b-202a-4391-83e0-8c84bb587119\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-cfa6c5-558d8df998-xnjhm"
Nov 02 20:36:27.808137 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:27.807997    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lflv\" (UniqueName: \"kubernetes.io/projected/9f0c2f2b-202a-4391-83e0-8c84bb587119-kube-api-access-4lflv\") pod \"maas-api-e2e-authcfg-http-cfa6c5-558d8df998-xnjhm\" (UID: \"9f0c2f2b-202a-4391-83e0-8c84bb587119\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-cfa6c5-558d8df998-xnjhm"
Nov 02 20:36:27.808137 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:36:27.808053    4463 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-authcfg-http-cfa6c5-serving-cert: secret "maas-api-e2e-authcfg-http-cfa6c5-serving-cert" not found
Nov 02 20:36:27.808137 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:27.808102    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f0c2f2b-202a-4391-83e0-8c84bb587119-metrics-certs\") pod \"maas-api-e2e-authcfg-http-cfa6c5-558d8df998-xnjhm\" (UID: \"9f0c2f2b-202a-4391-83e0-8c84bb587119\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-cfa6c5-558d8df998-xnjhm"
Nov 02 20:36:27.808137 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:36:27.808118    4463 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f0c2f2b-202a-4391-83e0-8c84bb587119-maas-api-tls podName:9f0c2f2b-202a-4391-83e0-8c84bb587119 nodeName:}" failed. No retries permitted until 2026-11-02 20:36:28.308097558 +0000 UTC m=+4094.915131321 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9f0c2f2b-202a-4391-83e0-8c84bb587119-maas-api-tls") pod "maas-api-e2e-authcfg-http-cfa6c5-558d8df998-xnjhm" (UID: "9f0c2f2b-202a-4391-83e0-8c84bb587119") : secret "maas-api-e2e-authcfg-http-cfa6c5-serving-cert" not found
Nov 02 20:36:27.808640 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:27.808618    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f0c2f2b-202a-4391-83e0-8c84bb587119-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-cfa6c5-558d8df998-xnjhm\" (UID: \"9f0c2f2b-202a-4391-83e0-8c84bb587119\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-cfa6c5-558d8df998-xnjhm"
Nov 02 20:36:27.810688 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:27.810660    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f0c2f2b-202a-4391-83e0-8c84bb587119-metrics-certs\") pod \"maas-api-e2e-authcfg-http-cfa6c5-558d8df998-xnjhm\" (UID: \"9f0c2f2b-202a-4391-83e0-8c84bb587119\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-cfa6c5-558d8df998-xnjhm"
Nov 02 20:36:27.863216 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:27.863150    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lflv\" (UniqueName: \"kubernetes.io/projected/9f0c2f2b-202a-4391-83e0-8c84bb587119-kube-api-access-4lflv\") pod \"maas-api-e2e-authcfg-http-cfa6c5-558d8df998-xnjhm\" (UID: \"9f0c2f2b-202a-4391-83e0-8c84bb587119\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-cfa6c5-558d8df998-xnjhm"
Nov 02 20:36:28.312669 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:28.312636    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f0c2f2b-202a-4391-83e0-8c84bb587119-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-cfa6c5-558d8df998-xnjhm\" (UID: \"9f0c2f2b-202a-4391-83e0-8c84bb587119\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-cfa6c5-558d8df998-xnjhm"
Nov 02 20:36:28.315192 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:28.315157    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f0c2f2b-202a-4391-83e0-8c84bb587119-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-cfa6c5-558d8df998-xnjhm\" (UID: \"9f0c2f2b-202a-4391-83e0-8c84bb587119\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-cfa6c5-558d8df998-xnjhm"
Nov 02 20:36:28.502170 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:28.502141    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-cfa6c5-558d8df998-xnjhm"
Nov 02 20:36:28.866611 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:28.866585    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-cfa6c5-558d8df998-xnjhm"]
Nov 02 20:36:28.869040 ip-10-0-133-9 kubenswrapper[4463]: W1102 20:36:28.869000    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f0c2f2b_202a_4391_83e0_8c84bb587119.slice/crio-3f0fa9fde5d51890fbf543c20128b78197741baccdd86138aaf5abf8b30eebb6 WatchSource:0}: Error finding container 3f0fa9fde5d51890fbf543c20128b78197741baccdd86138aaf5abf8b30eebb6: Status 404 returned error can't find the container with id 3f0fa9fde5d51890fbf543c20128b78197741baccdd86138aaf5abf8b30eebb6
Nov 02 20:36:29.229452 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:29.229417    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-cfa6c5-558d8df998-xnjhm" event={"ID":"9f0c2f2b-202a-4391-83e0-8c84bb587119","Type":"ContainerStarted","Data":"3f0fa9fde5d51890fbf543c20128b78197741baccdd86138aaf5abf8b30eebb6"}
Nov 02 20:36:30.235627 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:30.235586    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-cfa6c5-558d8df998-xnjhm" event={"ID":"9f0c2f2b-202a-4391-83e0-8c84bb587119","Type":"ContainerStarted","Data":"b5e80842a07be0234177e25c0d2818ff7d0a344628d2c0e58778f4f6471f32c1"}
Nov 02 20:36:30.236153 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:30.235851    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-cfa6c5-558d8df998-xnjhm"
Nov 02 20:36:30.344991 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:30.344926    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-cfa6c5-558d8df998-xnjhm" podStartSLOduration=2.445530254 podStartE2EDuration="3.34491177s" totalImagesPullingTime="899.381516ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:36:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:36:30.342324055 +0000 UTC m=+4096.949357843" watchObservedRunningTime="2026-11-02 20:36:30.34491177 +0000 UTC m=+4096.951945539"
Nov 02 20:36:34.294292 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:36:34.294255    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 20:36:35.219259 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:35.219218    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 20:36:36.246064 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:36.246035    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-cfa6c5-558d8df998-xnjhm"
Nov 02 20:36:38.226375 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:38.226345    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-cfa6c5-54b7955cc5-wjfgd"
Nov 02 20:36:44.319312 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:44.319219    4463 generic.go:356] "Generic (PLEG): container finished" podID="877de52e-ed20-431b-a88e-85bca29f9a4a" containerID="73e23dbc75b058c5fae76e5915aefae48eff00b322048fb60286d4365be8f6af" exitCode=0
Nov 02 20:36:44.319312 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:44.319290    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-cfa6c5-openshift-default-679d974589-kjk92" event={"ID":"877de52e-ed20-431b-a88e-85bca29f9a4a","Type":"ContainerDied","Data":"73e23dbc75b058c5fae76e5915aefae48eff00b322048fb60286d4365be8f6af"}
Nov 02 20:36:44.319754 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:44.319658    4463 scope.go:122] "RemoveContainer" containerID="73e23dbc75b058c5fae76e5915aefae48eff00b322048fb60286d4365be8f6af"
Nov 02 20:36:44.320825 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:44.320797    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:36:44.320888 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:44.320874    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:36:44.320925 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:44.320911    4463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:36:45.325519 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:45.325487    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-cfa6c5-openshift-default-679d974589-kjk92" event={"ID":"877de52e-ed20-431b-a88e-85bca29f9a4a","Type":"ContainerStarted","Data":"7f73cd71842c61ac6071ad3341b3c83045235b396df50addfdc2f38d6d9b7eaf"}
Nov 02 20:36:46.174058 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:46.173380    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-cfa6c5-openshift-default-679d974589-kjk92"
Nov 02 20:36:46.174586 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:46.174564    4463 patch_prober.go:29] interesting pod/e2e-authcfg-http-cfa6c5-openshift-default-679d974589-kjk92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Nov 02 20:36:46.174637 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:46.174610    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-cfa6c5-openshift-default-679d974589-kjk92" podUID="877de52e-ed20-431b-a88e-85bca29f9a4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Nov 02 20:36:47.170518 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:47.170475    4463 patch_prober.go:29] interesting pod/e2e-authcfg-http-cfa6c5-openshift-default-679d974589-kjk92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Nov 02 20:36:47.171171 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:47.171141    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-cfa6c5-openshift-default-679d974589-kjk92" podUID="877de52e-ed20-431b-a88e-85bca29f9a4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Nov 02 20:36:48.170035 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:48.169488    4463 patch_prober.go:29] interesting pod/e2e-authcfg-http-cfa6c5-openshift-default-679d974589-kjk92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Nov 02 20:36:48.170035 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:48.169595    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-cfa6c5-openshift-default-679d974589-kjk92" podUID="877de52e-ed20-431b-a88e-85bca29f9a4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Nov 02 20:36:49.171043 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:49.169426    4463 patch_prober.go:29] interesting pod/e2e-authcfg-http-cfa6c5-openshift-default-679d974589-kjk92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Nov 02 20:36:49.171043 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:49.169504    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-cfa6c5-openshift-default-679d974589-kjk92" podUID="877de52e-ed20-431b-a88e-85bca29f9a4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Nov 02 20:36:50.168190 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:50.168141    4463 patch_prober.go:29] interesting pod/e2e-authcfg-http-cfa6c5-openshift-default-679d974589-kjk92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Nov 02 20:36:50.168407 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:50.168218    4463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-cfa6c5-openshift-default-679d974589-kjk92" podUID="877de52e-ed20-431b-a88e-85bca29f9a4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Nov 02 20:36:50.178268 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:50.178239    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-cfa6c5-openshift-default-679d974589-kjk92"
Nov 02 20:36:51.172383 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:51.172352    4463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-cfa6c5-openshift-default-679d974589-kjk92"
Nov 02 20:36:51.173899 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:36:51.173877    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-cfa6c5-openshift-default-679d974589-kjk92"
Nov 02 20:37:04.293789 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:37:04.293747    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 20:37:34.297223 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:37:34.297180    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 20:38:04.293848 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:38:04.293799    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 20:38:21.629116 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:38:21.629082    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-cfa6c5-54b7955cc5-wjfgd"]
Nov 02 20:38:26.629610 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:38:26.629564    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-authcfg-http-cfa6c5-54b7955cc5-wjfgd" podUID="b42b3241-59a2-4493-a057-8b0656d60625" containerName="payload-processing" containerID="cri-o://c15dcfe906098fb8d24e2123f0653eaa959ad6f2e5a36a531253f775ae925faa" gracePeriod=25
Nov 02 20:38:26.798276 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:38:26.798103    4463 generic.go:356] "Generic (PLEG): container finished" podID="b42b3241-59a2-4493-a057-8b0656d60625" containerID="c15dcfe906098fb8d24e2123f0653eaa959ad6f2e5a36a531253f775ae925faa" exitCode=0
Nov 02 20:38:26.798276 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:38:26.798217    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-cfa6c5-54b7955cc5-wjfgd" event={"ID":"b42b3241-59a2-4493-a057-8b0656d60625","Type":"ContainerDied","Data":"c15dcfe906098fb8d24e2123f0653eaa959ad6f2e5a36a531253f775ae925faa"}
Nov 02 20:38:26.881510 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:38:26.881448    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-cfa6c5-54b7955cc5-wjfgd"
Nov 02 20:38:27.053307 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:38:27.053268    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b42b3241-59a2-4493-a057-8b0656d60625-kube-api-access-q87ww\" (UniqueName: \"kubernetes.io/projected/b42b3241-59a2-4493-a057-8b0656d60625-kube-api-access-q87ww\") pod \"b42b3241-59a2-4493-a057-8b0656d60625\" (UID: \"b42b3241-59a2-4493-a057-8b0656d60625\") "
Nov 02 20:38:27.053481 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:38:27.053322    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b42b3241-59a2-4493-a057-8b0656d60625-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b42b3241-59a2-4493-a057-8b0656d60625-plugins-config-volume\") pod \"b42b3241-59a2-4493-a057-8b0656d60625\" (UID: \"b42b3241-59a2-4493-a057-8b0656d60625\") "
Nov 02 20:38:27.053723 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:38:27.053701    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b42b3241-59a2-4493-a057-8b0656d60625-plugins-config-volume" pod "b42b3241-59a2-4493-a057-8b0656d60625" (UID: "b42b3241-59a2-4493-a057-8b0656d60625"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:38:27.055445 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:38:27.055424    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b42b3241-59a2-4493-a057-8b0656d60625-kube-api-access-q87ww" pod "b42b3241-59a2-4493-a057-8b0656d60625" (UID: "b42b3241-59a2-4493-a057-8b0656d60625"). InnerVolumeSpecName "kube-api-access-q87ww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:38:27.154670 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:38:27.154589    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q87ww\" (UniqueName: \"kubernetes.io/projected/b42b3241-59a2-4493-a057-8b0656d60625-kube-api-access-q87ww\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:38:27.154670 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:38:27.154619    4463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b42b3241-59a2-4493-a057-8b0656d60625-plugins-config-volume\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:38:27.803919 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:38:27.803886    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-cfa6c5-54b7955cc5-wjfgd" event={"ID":"b42b3241-59a2-4493-a057-8b0656d60625","Type":"ContainerDied","Data":"a9fddf2213452ce46393d119239f8fb52ccc43f8b396ae1c587fd88bb59b3fa6"}
Nov 02 20:38:27.803919 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:38:27.803904    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-cfa6c5-54b7955cc5-wjfgd"
Nov 02 20:38:27.804440 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:38:27.803953    4463 scope.go:122] "RemoveContainer" containerID="c15dcfe906098fb8d24e2123f0653eaa959ad6f2e5a36a531253f775ae925faa"
Nov 02 20:38:27.956036 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:38:27.955992    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-cfa6c5-54b7955cc5-wjfgd"]
Nov 02 20:38:27.957643 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:38:27.957620    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-cfa6c5-54b7955cc5-wjfgd"]
Nov 02 20:38:29.583297 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:38:29.583263    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b42b3241-59a2-4493-a057-8b0656d60625" path="/var/lib/kubelet/pods/b42b3241-59a2-4493-a057-8b0656d60625/volumes"
Nov 02 20:38:30.885792 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:38:30.885758    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-cfa6c5-558d8df998-xnjhm"]
Nov 02 20:38:30.886268 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:38:30.885939    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-cfa6c5-558d8df998-xnjhm" podUID="9f0c2f2b-202a-4391-83e0-8c84bb587119" containerName="maas-api" containerID="cri-o://b5e80842a07be0234177e25c0d2818ff7d0a344628d2c0e58778f4f6471f32c1" gracePeriod=30
Nov 02 20:38:31.136698 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:38:31.136634    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-cfa6c5-558d8df998-xnjhm"
Nov 02 20:38:31.299206 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:38:31.299176    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f0c2f2b-202a-4391-83e0-8c84bb587119-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f0c2f2b-202a-4391-83e0-8c84bb587119-metrics-certs\") pod \"9f0c2f2b-202a-4391-83e0-8c84bb587119\" (UID: \"9f0c2f2b-202a-4391-83e0-8c84bb587119\") "
Nov 02 20:38:31.299413 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:38:31.299240    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f0c2f2b-202a-4391-83e0-8c84bb587119-kube-api-access-4lflv\" (UniqueName: \"kubernetes.io/projected/9f0c2f2b-202a-4391-83e0-8c84bb587119-kube-api-access-4lflv\") pod \"9f0c2f2b-202a-4391-83e0-8c84bb587119\" (UID: \"9f0c2f2b-202a-4391-83e0-8c84bb587119\") "
Nov 02 20:38:31.299413 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:38:31.299277    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f0c2f2b-202a-4391-83e0-8c84bb587119-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f0c2f2b-202a-4391-83e0-8c84bb587119-openshift-service-ca\") pod \"9f0c2f2b-202a-4391-83e0-8c84bb587119\" (UID: \"9f0c2f2b-202a-4391-83e0-8c84bb587119\") "
Nov 02 20:38:31.299413 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:38:31.299323    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f0c2f2b-202a-4391-83e0-8c84bb587119-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f0c2f2b-202a-4391-83e0-8c84bb587119-maas-api-tls\") pod \"9f0c2f2b-202a-4391-83e0-8c84bb587119\" (UID: \"9f0c2f2b-202a-4391-83e0-8c84bb587119\") "
Nov 02 20:38:31.299636 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:38:31.299610    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f0c2f2b-202a-4391-83e0-8c84bb587119-openshift-service-ca" pod "9f0c2f2b-202a-4391-83e0-8c84bb587119" (UID: "9f0c2f2b-202a-4391-83e0-8c84bb587119"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:38:31.299749 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:38:31.299733    4463 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f0c2f2b-202a-4391-83e0-8c84bb587119-openshift-service-ca\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:38:31.301448 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:38:31.301411    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f0c2f2b-202a-4391-83e0-8c84bb587119-maas-api-tls" pod "9f0c2f2b-202a-4391-83e0-8c84bb587119" (UID: "9f0c2f2b-202a-4391-83e0-8c84bb587119"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:38:31.301568 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:38:31.301462    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f0c2f2b-202a-4391-83e0-8c84bb587119-metrics-certs" pod "9f0c2f2b-202a-4391-83e0-8c84bb587119" (UID: "9f0c2f2b-202a-4391-83e0-8c84bb587119"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:38:31.301568 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:38:31.301499    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f0c2f2b-202a-4391-83e0-8c84bb587119-kube-api-access-4lflv" pod "9f0c2f2b-202a-4391-83e0-8c84bb587119" (UID: "9f0c2f2b-202a-4391-83e0-8c84bb587119"). InnerVolumeSpecName "kube-api-access-4lflv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:38:31.401219 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:38:31.401132    4463 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f0c2f2b-202a-4391-83e0-8c84bb587119-metrics-certs\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:38:31.401219 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:38:31.401163    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lflv\" (UniqueName: \"kubernetes.io/projected/9f0c2f2b-202a-4391-83e0-8c84bb587119-kube-api-access-4lflv\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:38:31.401219 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:38:31.401177    4463 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f0c2f2b-202a-4391-83e0-8c84bb587119-maas-api-tls\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:38:31.823630 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:38:31.823596    4463 generic.go:356] "Generic (PLEG): container finished" podID="9f0c2f2b-202a-4391-83e0-8c84bb587119" containerID="b5e80842a07be0234177e25c0d2818ff7d0a344628d2c0e58778f4f6471f32c1" exitCode=0
Nov 02 20:38:31.823809 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:38:31.823645    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-cfa6c5-558d8df998-xnjhm"
Nov 02 20:38:31.823809 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:38:31.823680    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-cfa6c5-558d8df998-xnjhm" event={"ID":"9f0c2f2b-202a-4391-83e0-8c84bb587119","Type":"ContainerDied","Data":"b5e80842a07be0234177e25c0d2818ff7d0a344628d2c0e58778f4f6471f32c1"}
Nov 02 20:38:31.823809 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:38:31.823735    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-cfa6c5-558d8df998-xnjhm" event={"ID":"9f0c2f2b-202a-4391-83e0-8c84bb587119","Type":"ContainerDied","Data":"3f0fa9fde5d51890fbf543c20128b78197741baccdd86138aaf5abf8b30eebb6"}
Nov 02 20:38:31.823809 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:38:31.823765    4463 scope.go:122] "RemoveContainer" containerID="b5e80842a07be0234177e25c0d2818ff7d0a344628d2c0e58778f4f6471f32c1"
Nov 02 20:38:31.832496 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:38:31.832478    4463 scope.go:122] "RemoveContainer" containerID="b5e80842a07be0234177e25c0d2818ff7d0a344628d2c0e58778f4f6471f32c1"
Nov 02 20:38:31.832758 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:38:31.832742    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5e80842a07be0234177e25c0d2818ff7d0a344628d2c0e58778f4f6471f32c1\": container with ID starting with b5e80842a07be0234177e25c0d2818ff7d0a344628d2c0e58778f4f6471f32c1 not found: ID does not exist" containerID="b5e80842a07be0234177e25c0d2818ff7d0a344628d2c0e58778f4f6471f32c1"
Nov 02 20:38:31.832807 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:38:31.832765    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5e80842a07be0234177e25c0d2818ff7d0a344628d2c0e58778f4f6471f32c1"} err="failed to get container status \"b5e80842a07be0234177e25c0d2818ff7d0a344628d2c0e58778f4f6471f32c1\": rpc error: code = NotFound desc = could not find container \"b5e80842a07be0234177e25c0d2818ff7d0a344628d2c0e58778f4f6471f32c1\": container with ID starting with b5e80842a07be0234177e25c0d2818ff7d0a344628d2c0e58778f4f6471f32c1 not found: ID does not exist"
Nov 02 20:38:31.975355 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:38:31.975326    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-cfa6c5-558d8df998-xnjhm"]
Nov 02 20:38:32.021836 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:38:32.021803    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-cfa6c5-558d8df998-xnjhm"]
Nov 02 20:38:33.582972 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:38:33.582936    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f0c2f2b-202a-4391-83e0-8c84bb587119" path="/var/lib/kubelet/pods/9f0c2f2b-202a-4391-83e0-8c84bb587119/volumes"
Nov 02 20:38:34.293894 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:38:34.293856    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 20:38:35.138815 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:38:35.138776    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 20:38:39.515689 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:38:39.515654    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-cfa6c5-openshift-default-679d974589-kjk92"]
Nov 02 20:38:39.516211 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:38:39.515912    4463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-authcfg-http-cfa6c5-openshift-default-679d974589-kjk92" podUID="877de52e-ed20-431b-a88e-85bca29f9a4a" containerName="istio-proxy" containerID="cri-o://7f73cd71842c61ac6071ad3341b3c83045235b396df50addfdc2f38d6d9b7eaf" gracePeriod=30
Nov 02 20:38:44.753844 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:38:44.753800    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-cfa6c5-openshift-default-679d974589-kjk92"
Nov 02 20:38:44.822263 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:38:44.822204    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/877de52e-ed20-431b-a88e-85bca29f9a4a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/877de52e-ed20-431b-a88e-85bca29f9a4a-istio-envoy\") pod \"877de52e-ed20-431b-a88e-85bca29f9a4a\" (UID: \"877de52e-ed20-431b-a88e-85bca29f9a4a\") "
Nov 02 20:38:44.822408 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:38:44.822273    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/877de52e-ed20-431b-a88e-85bca29f9a4a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/877de52e-ed20-431b-a88e-85bca29f9a4a-credential-socket\") pod \"877de52e-ed20-431b-a88e-85bca29f9a4a\" (UID: \"877de52e-ed20-431b-a88e-85bca29f9a4a\") "
Nov 02 20:38:44.822408 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:38:44.822304    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/877de52e-ed20-431b-a88e-85bca29f9a4a-istio-token\" (UniqueName: \"kubernetes.io/projected/877de52e-ed20-431b-a88e-85bca29f9a4a-istio-token\") pod \"877de52e-ed20-431b-a88e-85bca29f9a4a\" (UID: \"877de52e-ed20-431b-a88e-85bca29f9a4a\") "
Nov 02 20:38:44.822408 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:38:44.822345    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/877de52e-ed20-431b-a88e-85bca29f9a4a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/877de52e-ed20-431b-a88e-85bca29f9a4a-istio-data\") pod \"877de52e-ed20-431b-a88e-85bca29f9a4a\" (UID: \"877de52e-ed20-431b-a88e-85bca29f9a4a\") "
Nov 02 20:38:44.822586 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:38:44.822434    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/877de52e-ed20-431b-a88e-85bca29f9a4a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/877de52e-ed20-431b-a88e-85bca29f9a4a-workload-socket\") pod \"877de52e-ed20-431b-a88e-85bca29f9a4a\" (UID: \"877de52e-ed20-431b-a88e-85bca29f9a4a\") "
Nov 02 20:38:44.822586 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:38:44.822476    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/877de52e-ed20-431b-a88e-85bca29f9a4a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/877de52e-ed20-431b-a88e-85bca29f9a4a-istiod-ca-cert\") pod \"877de52e-ed20-431b-a88e-85bca29f9a4a\" (UID: \"877de52e-ed20-431b-a88e-85bca29f9a4a\") "
Nov 02 20:38:44.822586 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:38:44.822502    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/877de52e-ed20-431b-a88e-85bca29f9a4a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/877de52e-ed20-431b-a88e-85bca29f9a4a-istio-podinfo\") pod \"877de52e-ed20-431b-a88e-85bca29f9a4a\" (UID: \"877de52e-ed20-431b-a88e-85bca29f9a4a\") "
Nov 02 20:38:44.822586 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:38:44.822529    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/877de52e-ed20-431b-a88e-85bca29f9a4a-kube-api-access-lx4mb\" (UniqueName: \"kubernetes.io/projected/877de52e-ed20-431b-a88e-85bca29f9a4a-kube-api-access-lx4mb\") pod \"877de52e-ed20-431b-a88e-85bca29f9a4a\" (UID: \"877de52e-ed20-431b-a88e-85bca29f9a4a\") "
Nov 02 20:38:44.822586 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:38:44.822561    4463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/877de52e-ed20-431b-a88e-85bca29f9a4a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/877de52e-ed20-431b-a88e-85bca29f9a4a-workload-certs\") pod \"877de52e-ed20-431b-a88e-85bca29f9a4a\" (UID: \"877de52e-ed20-431b-a88e-85bca29f9a4a\") "
Nov 02 20:38:44.822853 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:38:44.822659    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/877de52e-ed20-431b-a88e-85bca29f9a4a-istio-data" pod "877de52e-ed20-431b-a88e-85bca29f9a4a" (UID: "877de52e-ed20-431b-a88e-85bca29f9a4a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:38:44.822853 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:38:44.822718    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/877de52e-ed20-431b-a88e-85bca29f9a4a-credential-socket" pod "877de52e-ed20-431b-a88e-85bca29f9a4a" (UID: "877de52e-ed20-431b-a88e-85bca29f9a4a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:38:44.822853 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:38:44.822754    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/877de52e-ed20-431b-a88e-85bca29f9a4a-workload-socket" pod "877de52e-ed20-431b-a88e-85bca29f9a4a" (UID: "877de52e-ed20-431b-a88e-85bca29f9a4a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:38:44.823041 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:38:44.822861    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/877de52e-ed20-431b-a88e-85bca29f9a4a-istiod-ca-cert" pod "877de52e-ed20-431b-a88e-85bca29f9a4a" (UID: "877de52e-ed20-431b-a88e-85bca29f9a4a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:38:44.823041 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:38:44.822876    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/877de52e-ed20-431b-a88e-85bca29f9a4a-workload-certs" pod "877de52e-ed20-431b-a88e-85bca29f9a4a" (UID: "877de52e-ed20-431b-a88e-85bca29f9a4a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:38:44.823041 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:38:44.822988    4463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/877de52e-ed20-431b-a88e-85bca29f9a4a-istiod-ca-cert\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:38:44.823041 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:38:44.823008    4463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/877de52e-ed20-431b-a88e-85bca29f9a4a-workload-certs\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:38:44.823041 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:38:44.823036    4463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/877de52e-ed20-431b-a88e-85bca29f9a4a-credential-socket\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:38:44.823041 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:38:44.823045    4463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/877de52e-ed20-431b-a88e-85bca29f9a4a-istio-data\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:38:44.823365 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:38:44.823053    4463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/877de52e-ed20-431b-a88e-85bca29f9a4a-workload-socket\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:38:44.824600 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:38:44.824578    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/877de52e-ed20-431b-a88e-85bca29f9a4a-istio-token" pod "877de52e-ed20-431b-a88e-85bca29f9a4a" (UID: "877de52e-ed20-431b-a88e-85bca29f9a4a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:38:44.824717 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:38:44.824595    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/877de52e-ed20-431b-a88e-85bca29f9a4a-istio-envoy" pod "877de52e-ed20-431b-a88e-85bca29f9a4a" (UID: "877de52e-ed20-431b-a88e-85bca29f9a4a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:38:44.824717 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:38:44.824619    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/877de52e-ed20-431b-a88e-85bca29f9a4a-istio-podinfo" pod "877de52e-ed20-431b-a88e-85bca29f9a4a" (UID: "877de52e-ed20-431b-a88e-85bca29f9a4a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:38:44.824848 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:38:44.824830    4463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/877de52e-ed20-431b-a88e-85bca29f9a4a-kube-api-access-lx4mb" pod "877de52e-ed20-431b-a88e-85bca29f9a4a" (UID: "877de52e-ed20-431b-a88e-85bca29f9a4a"). InnerVolumeSpecName "kube-api-access-lx4mb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:38:44.881799 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:38:44.881768    4463 generic.go:356] "Generic (PLEG): container finished" podID="877de52e-ed20-431b-a88e-85bca29f9a4a" containerID="7f73cd71842c61ac6071ad3341b3c83045235b396df50addfdc2f38d6d9b7eaf" exitCode=0
Nov 02 20:38:44.881926 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:38:44.881828    4463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-cfa6c5-openshift-default-679d974589-kjk92"
Nov 02 20:38:44.881926 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:38:44.881849    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-cfa6c5-openshift-default-679d974589-kjk92" event={"ID":"877de52e-ed20-431b-a88e-85bca29f9a4a","Type":"ContainerDied","Data":"7f73cd71842c61ac6071ad3341b3c83045235b396df50addfdc2f38d6d9b7eaf"}
Nov 02 20:38:44.882000 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:38:44.881936    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-cfa6c5-openshift-default-679d974589-kjk92" event={"ID":"877de52e-ed20-431b-a88e-85bca29f9a4a","Type":"ContainerDied","Data":"9902492635dc39b5824448cfee5d81c1a373d3640a82131470a7133612b58763"}
Nov 02 20:38:44.882000 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:38:44.881959    4463 scope.go:122] "RemoveContainer" containerID="7f73cd71842c61ac6071ad3341b3c83045235b396df50addfdc2f38d6d9b7eaf"
Nov 02 20:38:44.891940 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:38:44.891921    4463 scope.go:122] "RemoveContainer" containerID="73e23dbc75b058c5fae76e5915aefae48eff00b322048fb60286d4365be8f6af"
Nov 02 20:38:44.899517 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:38:44.899497    4463 scope.go:122] "RemoveContainer" containerID="7f73cd71842c61ac6071ad3341b3c83045235b396df50addfdc2f38d6d9b7eaf"
Nov 02 20:38:44.899783 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:38:44.899764    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f73cd71842c61ac6071ad3341b3c83045235b396df50addfdc2f38d6d9b7eaf\": container with ID starting with 7f73cd71842c61ac6071ad3341b3c83045235b396df50addfdc2f38d6d9b7eaf not found: ID does not exist" containerID="7f73cd71842c61ac6071ad3341b3c83045235b396df50addfdc2f38d6d9b7eaf"
Nov 02 20:38:44.899865 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:38:44.899796    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f73cd71842c61ac6071ad3341b3c83045235b396df50addfdc2f38d6d9b7eaf"} err="failed to get container status \"7f73cd71842c61ac6071ad3341b3c83045235b396df50addfdc2f38d6d9b7eaf\": rpc error: code = NotFound desc = could not find container \"7f73cd71842c61ac6071ad3341b3c83045235b396df50addfdc2f38d6d9b7eaf\": container with ID starting with 7f73cd71842c61ac6071ad3341b3c83045235b396df50addfdc2f38d6d9b7eaf not found: ID does not exist"
Nov 02 20:38:44.899865 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:38:44.899820    4463 scope.go:122] "RemoveContainer" containerID="73e23dbc75b058c5fae76e5915aefae48eff00b322048fb60286d4365be8f6af"
Nov 02 20:38:44.900103 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:38:44.900074    4463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73e23dbc75b058c5fae76e5915aefae48eff00b322048fb60286d4365be8f6af\": container with ID starting with 73e23dbc75b058c5fae76e5915aefae48eff00b322048fb60286d4365be8f6af not found: ID does not exist" containerID="73e23dbc75b058c5fae76e5915aefae48eff00b322048fb60286d4365be8f6af"
Nov 02 20:38:44.900150 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:38:44.900114    4463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73e23dbc75b058c5fae76e5915aefae48eff00b322048fb60286d4365be8f6af"} err="failed to get container status \"73e23dbc75b058c5fae76e5915aefae48eff00b322048fb60286d4365be8f6af\": rpc error: code = NotFound desc = could not find container \"73e23dbc75b058c5fae76e5915aefae48eff00b322048fb60286d4365be8f6af\": container with ID starting with 73e23dbc75b058c5fae76e5915aefae48eff00b322048fb60286d4365be8f6af not found: ID does not exist"
Nov 02 20:38:44.923940 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:38:44.923920    4463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/877de52e-ed20-431b-a88e-85bca29f9a4a-istio-envoy\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:38:44.923940 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:38:44.923941    4463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/877de52e-ed20-431b-a88e-85bca29f9a4a-istio-token\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:38:44.924074 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:38:44.923950    4463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/877de52e-ed20-431b-a88e-85bca29f9a4a-istio-podinfo\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:38:44.924074 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:38:44.923959    4463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lx4mb\" (UniqueName: \"kubernetes.io/projected/877de52e-ed20-431b-a88e-85bca29f9a4a-kube-api-access-lx4mb\") on node \"ip-10-0-133-9.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:38:45.040665 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:38:45.040632    4463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-cfa6c5-openshift-default-679d974589-kjk92"]
Nov 02 20:38:45.046475 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:38:45.046446    4463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-authcfg-http-cfa6c5-openshift-default-679d974589-kjk92"]
Nov 02 20:38:45.583773 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:38:45.583736    4463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="877de52e-ed20-431b-a88e-85bca29f9a4a" path="/var/lib/kubelet/pods/877de52e-ed20-431b-a88e-85bca29f9a4a/volumes"
Nov 02 20:38:47.827744 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:38:47.827708    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 20:38:58.139053 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:38:58.139005    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 20:39:04.293858 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:39:04.293814    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 20:39:19.134474 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:39:19.134439    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 20:39:34.293487 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:39:34.293409    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 20:39:56.918583 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:39:56.918544    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p4mx5"]
Nov 02 20:39:58.705034 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:39:58.704981    4463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-fm6qk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 20:39:58.705559 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:39:58.705062    4463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fm6qk" podUID="ae15f3c4-bd17-4252-837c-4483162c3542" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:40:04.293778 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:40:04.293727    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 20:40:28.447183 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:40:28.447147    4463 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-phk29_00b5dca8-d734-47c4-99d6-00824c07793b/server/0.log"
Nov 02 20:40:29.653926 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:40:29.653888    4463 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-f6584b448-n45m7_32e8cbc9-818f-44ce-a617-1551fbc35930/maas-api/0.log"
Nov 02 20:40:32.785024 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:40:32.784976    4463 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-25nw5_01019899-b2a2-4b23-ac2f-33dd143c6476/manager/0.log"
Nov 02 20:40:33.287205 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:40:33.287177    4463 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-jjs9d_760eeed1-48ec-4029-b3c1-837009e77fbf/kuadrant-console-plugin/0.log"
Nov 02 20:40:33.906416 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:40:33.906387    4463 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-p4mx5_ae3ddaea-c756-486f-bd85-aef7fc1f66b3/limitador/0.log"
Nov 02 20:40:34.293162 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:40:34.293120    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 20:40:35.470348 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:40:35.470313    4463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-65657466b-9hgsh_a394c32e-69b7-4a96-8e08-e0080be53c80/kube-auth-proxy/0.log"
Nov 02 20:40:46.888391 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:40:46.888362    4463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-dw7ld_a85b7c93-b1b1-4bc3-b840-0de225a8ec58/istio-proxy/0.log"
Nov 02 20:40:47.105829 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:40:47.105799    4463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-fm6qk_ae15f3c4-bd17-4252-837c-4483162c3542/istio-proxy/0.log"
Nov 02 20:40:47.774095 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:40:47.774064    4463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-khwqc_2bec6162-6f9a-4871-8185-cc3592f4075c/istio-proxy/0.log"
Nov 02 20:40:48.678146 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:40:48.678114    4463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-slt99_5b33fed1-e127-4f1f-8b7f-d56427da0155/payload-processing/0.log"
Nov 02 20:40:50.002936 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:40:50.002899    4463 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-9b2dc_365a63f7-ce01-4701-95b3-fd3862930ca6/main/0.log"
Nov 02 20:40:50.256363 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:40:50.256269    4463 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-ffswj_bf00bfe7-3062-4f29-a243-367a3e6c13b7/main/0.log"
Nov 02 20:40:50.705642 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:40:50.705605    4463 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-kjt8s_6536ec7e-0d3f-42f2-9e97-180f1e0a35a7/main/0.log"
Nov 02 20:40:51.125587 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:40:51.125518    4463 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-cthc4_27b07c53-9c7a-4cc1-a0cd-8a9cf982d73c/main/0.log"
Nov 02 20:40:51.338639 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:40:51.338609    4463 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-f8bw8_98da7e93-4099-4f42-bcfb-8d820ee124db/main/0.log"
Nov 02 20:41:02.876857 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:41:02.876828    4463 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-phk29_00b5dca8-d734-47c4-99d6-00824c07793b/server/0.log"
Nov 02 20:41:03.873359 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:41:03.873323    4463 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-f6584b448-n45m7_32e8cbc9-818f-44ce-a617-1551fbc35930/maas-api/0.log"
Nov 02 20:41:04.293616 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:41:04.293576    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 20:41:16.622125 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:41:16.622095    4463 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-25nw5_01019899-b2a2-4b23-ac2f-33dd143c6476/manager/0.log"
Nov 02 20:41:17.155243 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:41:17.155214    4463 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-jjs9d_760eeed1-48ec-4029-b3c1-837009e77fbf/kuadrant-console-plugin/0.log"
Nov 02 20:41:17.720581 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:41:17.720554    4463 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-p4mx5_ae3ddaea-c756-486f-bd85-aef7fc1f66b3/limitador/0.log"
Nov 02 20:41:19.146393 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:41:19.146350    4463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-65657466b-9hgsh_a394c32e-69b7-4a96-8e08-e0080be53c80/kube-auth-proxy/0.log"
Nov 02 20:41:20.596264 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:41:20.596232    4463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-dw7ld_a85b7c93-b1b1-4bc3-b840-0de225a8ec58/istio-proxy/0.log"
Nov 02 20:41:20.818156 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:41:20.818124    4463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-fm6qk_ae15f3c4-bd17-4252-837c-4483162c3542/istio-proxy/0.log"
Nov 02 20:41:21.480395 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:41:21.480364    4463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-khwqc_2bec6162-6f9a-4871-8185-cc3592f4075c/istio-proxy/0.log"
Nov 02 20:41:22.388065 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:41:22.388038    4463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-slt99_5b33fed1-e127-4f1f-8b7f-d56427da0155/payload-processing/0.log"
Nov 02 20:41:23.248245 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:41:23.248215    4463 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-9b2dc_365a63f7-ce01-4701-95b3-fd3862930ca6/main/0.log"
Nov 02 20:41:23.502375 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:41:23.502306    4463 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-ffswj_bf00bfe7-3062-4f29-a243-367a3e6c13b7/main/0.log"
Nov 02 20:41:23.935869 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:41:23.935844    4463 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-kjt8s_6536ec7e-0d3f-42f2-9e97-180f1e0a35a7/main/0.log"
Nov 02 20:41:24.377357 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:41:24.377275    4463 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-cthc4_27b07c53-9c7a-4cc1-a0cd-8a9cf982d73c/main/0.log"
Nov 02 20:41:24.610421 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:41:24.610386    4463 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-f8bw8_98da7e93-4099-4f42-bcfb-8d820ee124db/main/0.log"
Nov 02 20:41:34.293516 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:41:34.293479    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 20:42:02.641040 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:42:02.640984    4463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-slt99_5b33fed1-e127-4f1f-8b7f-d56427da0155/payload-processing/0.log"
Nov 02 20:42:02.864034 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:42:02.863989    4463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-slt99_5b33fed1-e127-4f1f-8b7f-d56427da0155/payload-processing/0.log"
Nov 02 20:42:04.294225 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:42:04.294187    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 20:42:34.293182 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:42:34.293103    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 20:42:47.406625 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:42:47.406595    4463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-slt99_5b33fed1-e127-4f1f-8b7f-d56427da0155/payload-processing/0.log"
Nov 02 20:42:47.614396 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:42:47.614366    4463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-slt99_5b33fed1-e127-4f1f-8b7f-d56427da0155/payload-processing/0.log"
Nov 02 20:42:58.684071 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:42:58.684035    4463 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-4b2pr_0b2d7938-0c97-4f07-a7d2-b5c35d2bc8f9/global-pull-secret-syncer/0.log"
Nov 02 20:43:01.553795 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:43:01.553769    4463 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-jgfjj_3fabcb64-faf2-49f9-813a-ddafde0fd159/konnectivity-agent/0.log"
Nov 02 20:43:03.136993 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:43:03.136956    4463 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-9.us-west-2.compute.internal_d6e1f694aea2cef1694653a075ee3dd5/haproxy/0.log"
Nov 02 20:43:04.293675 ip-10-0-133-9 kubenswrapper[4463]: E1102 20:43:04.293632    4463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bh65q" containerName="extractor"
Nov 02 20:43:07.600824 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:43:07.600795    4463 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-25nw5_01019899-b2a2-4b23-ac2f-33dd143c6476/manager/0.log"
Nov 02 20:43:08.378058 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:43:08.378032    4463 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-jjs9d_760eeed1-48ec-4029-b3c1-837009e77fbf/kuadrant-console-plugin/0.log"
Nov 02 20:43:08.407180 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:43:08.407152    4463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-pjlh5/perf-node-gather-daemonset-cqllv"]
Nov 02 20:43:08.407518 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:43:08.407506    4463 cpu_manager.go:414] "containerMap: removing container" podUID="b42b3241-59a2-4493-a057-8b0656d60625" containerName="payload-processing"
Nov 02 20:43:08.407568 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:43:08.407519    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b42b3241-59a2-4493-a057-8b0656d60625" containerName="payload-processing"
Nov 02 20:43:08.407568 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:43:08.407532    4463 cpu_manager.go:414] "containerMap: removing container" podUID="877de52e-ed20-431b-a88e-85bca29f9a4a" containerName="istio-proxy"
Nov 02 20:43:08.407568 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:43:08.407537    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="877de52e-ed20-431b-a88e-85bca29f9a4a" containerName="istio-proxy"
Nov 02 20:43:08.407568 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:43:08.407558    4463 cpu_manager.go:414] "containerMap: removing container" podUID="9f0c2f2b-202a-4391-83e0-8c84bb587119" containerName="maas-api"
Nov 02 20:43:08.407568 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:43:08.407564    4463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f0c2f2b-202a-4391-83e0-8c84bb587119" containerName="maas-api"
Nov 02 20:43:08.407720 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:43:08.407618    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f0c2f2b-202a-4391-83e0-8c84bb587119" containerName="maas-api"
Nov 02 20:43:08.407720 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:43:08.407627    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="877de52e-ed20-431b-a88e-85bca29f9a4a" containerName="istio-proxy"
Nov 02 20:43:08.407720 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:43:08.407635    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="b42b3241-59a2-4493-a057-8b0656d60625" containerName="payload-processing"
Nov 02 20:43:08.407720 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:43:08.407641    4463 memory_manager.go:365] "RemoveStaleState removing state" podUID="877de52e-ed20-431b-a88e-85bca29f9a4a" containerName="istio-proxy"
Nov 02 20:43:08.410816 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:43:08.410797    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pjlh5/perf-node-gather-daemonset-cqllv"
Nov 02 20:43:08.478850 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:43:08.478826    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c90f5032-eab0-4b85-9145-e8d1248ae994-lib-modules\") pod \"perf-node-gather-daemonset-cqllv\" (UID: \"c90f5032-eab0-4b85-9145-e8d1248ae994\") " pod="openshift-must-gather-pjlh5/perf-node-gather-daemonset-cqllv"
Nov 02 20:43:08.478970 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:43:08.478856    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c90f5032-eab0-4b85-9145-e8d1248ae994-sys\") pod \"perf-node-gather-daemonset-cqllv\" (UID: \"c90f5032-eab0-4b85-9145-e8d1248ae994\") " pod="openshift-must-gather-pjlh5/perf-node-gather-daemonset-cqllv"
Nov 02 20:43:08.478970 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:43:08.478885    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c90f5032-eab0-4b85-9145-e8d1248ae994-podres\") pod \"perf-node-gather-daemonset-cqllv\" (UID: \"c90f5032-eab0-4b85-9145-e8d1248ae994\") " pod="openshift-must-gather-pjlh5/perf-node-gather-daemonset-cqllv"
Nov 02 20:43:08.479119 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:43:08.479046    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c90f5032-eab0-4b85-9145-e8d1248ae994-proc\") pod \"perf-node-gather-daemonset-cqllv\" (UID: \"c90f5032-eab0-4b85-9145-e8d1248ae994\") " pod="openshift-must-gather-pjlh5/perf-node-gather-daemonset-cqllv"
Nov 02 20:43:08.479170 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:43:08.479135    4463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpqtp\" (UniqueName: \"kubernetes.io/projected/c90f5032-eab0-4b85-9145-e8d1248ae994-kube-api-access-rpqtp\") pod \"perf-node-gather-daemonset-cqllv\" (UID: \"c90f5032-eab0-4b85-9145-e8d1248ae994\") " pod="openshift-must-gather-pjlh5/perf-node-gather-daemonset-cqllv"
Nov 02 20:43:08.507214 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:43:08.507189    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pjlh5\"/\"openshift-service-ca.crt\""
Nov 02 20:43:08.507214 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:43:08.507201    4463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-pjlh5\"/\"default-dockercfg-2k858\""
Nov 02 20:43:08.507356 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:43:08.507264    4463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pjlh5\"/\"kube-root-ca.crt\""
Nov 02 20:43:08.508894 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:43:08.508872    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pjlh5/perf-node-gather-daemonset-cqllv"]
Nov 02 20:43:08.580180 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:43:08.580155    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c90f5032-eab0-4b85-9145-e8d1248ae994-proc\") pod \"perf-node-gather-daemonset-cqllv\" (UID: \"c90f5032-eab0-4b85-9145-e8d1248ae994\") " pod="openshift-must-gather-pjlh5/perf-node-gather-daemonset-cqllv"
Nov 02 20:43:08.580277 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:43:08.580198    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpqtp\" (UniqueName: \"kubernetes.io/projected/c90f5032-eab0-4b85-9145-e8d1248ae994-kube-api-access-rpqtp\") pod \"perf-node-gather-daemonset-cqllv\" (UID: \"c90f5032-eab0-4b85-9145-e8d1248ae994\") " pod="openshift-must-gather-pjlh5/perf-node-gather-daemonset-cqllv"
Nov 02 20:43:08.580277 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:43:08.580242    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c90f5032-eab0-4b85-9145-e8d1248ae994-lib-modules\") pod \"perf-node-gather-daemonset-cqllv\" (UID: \"c90f5032-eab0-4b85-9145-e8d1248ae994\") " pod="openshift-must-gather-pjlh5/perf-node-gather-daemonset-cqllv"
Nov 02 20:43:08.580277 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:43:08.580260    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c90f5032-eab0-4b85-9145-e8d1248ae994-sys\") pod \"perf-node-gather-daemonset-cqllv\" (UID: \"c90f5032-eab0-4b85-9145-e8d1248ae994\") " pod="openshift-must-gather-pjlh5/perf-node-gather-daemonset-cqllv"
Nov 02 20:43:08.580442 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:43:08.580273    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c90f5032-eab0-4b85-9145-e8d1248ae994-proc\") pod \"perf-node-gather-daemonset-cqllv\" (UID: \"c90f5032-eab0-4b85-9145-e8d1248ae994\") " pod="openshift-must-gather-pjlh5/perf-node-gather-daemonset-cqllv"
Nov 02 20:43:08.580442 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:43:08.580287    4463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c90f5032-eab0-4b85-9145-e8d1248ae994-podres\") pod \"perf-node-gather-daemonset-cqllv\" (UID: \"c90f5032-eab0-4b85-9145-e8d1248ae994\") " pod="openshift-must-gather-pjlh5/perf-node-gather-daemonset-cqllv"
Nov 02 20:43:08.580442 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:43:08.580359    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c90f5032-eab0-4b85-9145-e8d1248ae994-sys\") pod \"perf-node-gather-daemonset-cqllv\" (UID: \"c90f5032-eab0-4b85-9145-e8d1248ae994\") " pod="openshift-must-gather-pjlh5/perf-node-gather-daemonset-cqllv"
Nov 02 20:43:08.580442 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:43:08.580373    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c90f5032-eab0-4b85-9145-e8d1248ae994-lib-modules\") pod \"perf-node-gather-daemonset-cqllv\" (UID: \"c90f5032-eab0-4b85-9145-e8d1248ae994\") " pod="openshift-must-gather-pjlh5/perf-node-gather-daemonset-cqllv"
Nov 02 20:43:08.580584 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:43:08.580454    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c90f5032-eab0-4b85-9145-e8d1248ae994-podres\") pod \"perf-node-gather-daemonset-cqllv\" (UID: \"c90f5032-eab0-4b85-9145-e8d1248ae994\") " pod="openshift-must-gather-pjlh5/perf-node-gather-daemonset-cqllv"
Nov 02 20:43:08.633693 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:43:08.633643    4463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpqtp\" (UniqueName: \"kubernetes.io/projected/c90f5032-eab0-4b85-9145-e8d1248ae994-kube-api-access-rpqtp\") pod \"perf-node-gather-daemonset-cqllv\" (UID: \"c90f5032-eab0-4b85-9145-e8d1248ae994\") " pod="openshift-must-gather-pjlh5/perf-node-gather-daemonset-cqllv"
Nov 02 20:43:08.722591 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:43:08.722572    4463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pjlh5/perf-node-gather-daemonset-cqllv"
Nov 02 20:43:09.089001 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:43:09.088976    4463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pjlh5/perf-node-gather-daemonset-cqllv"]
Nov 02 20:43:09.091602 ip-10-0-133-9 kubenswrapper[4463]: W1102 20:43:09.091576    4463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc90f5032_eab0_4b85_9145_e8d1248ae994.slice/crio-932190e55db257d32b7d5853c36d3d130f677992af4ff68a03ce86eae03c681c WatchSource:0}: Error finding container 932190e55db257d32b7d5853c36d3d130f677992af4ff68a03ce86eae03c681c: Status 404 returned error can't find the container with id 932190e55db257d32b7d5853c36d3d130f677992af4ff68a03ce86eae03c681c
Nov 02 20:43:09.111675 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:43:09.111642    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pjlh5/perf-node-gather-daemonset-cqllv" event={"ID":"c90f5032-eab0-4b85-9145-e8d1248ae994","Type":"ContainerStarted","Data":"932190e55db257d32b7d5853c36d3d130f677992af4ff68a03ce86eae03c681c"}
Nov 02 20:43:09.254886 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:43:09.254863    4463 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-p4mx5_ae3ddaea-c756-486f-bd85-aef7fc1f66b3/limitador/0.log"
Nov 02 20:43:10.116954 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:43:10.116917    4463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pjlh5/perf-node-gather-daemonset-cqllv" event={"ID":"c90f5032-eab0-4b85-9145-e8d1248ae994","Type":"ContainerStarted","Data":"6f43764af13be9df3467ea3620216264105bd5ca0aa550772c2155b08665eb68"}
Nov 02 20:43:10.117349 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:43:10.117050    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-pjlh5/perf-node-gather-daemonset-cqllv"
Nov 02 20:43:10.222933 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:43:10.222888    4463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-pjlh5/perf-node-gather-daemonset-cqllv" podStartSLOduration=2.222874607 podStartE2EDuration="2.222874607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 20:43:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:43:10.221489451 +0000 UTC m=+4496.828523219" watchObservedRunningTime="2026-11-02 20:43:10.222874607 +0000 UTC m=+4496.829908376"
Nov 02 20:43:16.131303 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:43:16.131276    4463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-pjlh5/perf-node-gather-daemonset-cqllv"
Nov 02 20:43:18.352199 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:43:18.352176    4463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gfk8j_7ffd9b32-1bf0-4c92-959e-237289d6f609/node-exporter/0.log"
Nov 02 20:43:18.737856 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:43:18.737831    4463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gfk8j_7ffd9b32-1bf0-4c92-959e-237289d6f609/kube-rbac-proxy/0.log"
Nov 02 20:43:19.124283 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:43:19.124211    4463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gfk8j_7ffd9b32-1bf0-4c92-959e-237289d6f609/init-textfile/0.log"
Nov 02 20:43:20.686171 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:43:20.686146    4463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-ckhs2_bef314aa-0c49-4545-b518-cd0015b5b159/kube-rbac-proxy-main/0.log"
Nov 02 20:43:21.073474 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:43:21.073399    4463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-ckhs2_bef314aa-0c49-4545-b518-cd0015b5b159/kube-rbac-proxy-self/0.log"
Nov 02 20:43:21.488299 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:43:21.488273    4463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-ckhs2_bef314aa-0c49-4545-b518-cd0015b5b159/openshift-state-metrics/0.log"
Nov 02 20:43:21.886736 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:43:21.886663    4463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d91b3696-0787-4913-9bbc-4db403c0139b/prometheus/0.log"
Nov 02 20:43:22.271701 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:43:22.271665    4463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d91b3696-0787-4913-9bbc-4db403c0139b/config-reloader/0.log"
Nov 02 20:43:22.656963 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:43:22.656885    4463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d91b3696-0787-4913-9bbc-4db403c0139b/thanos-sidecar/0.log"
Nov 02 20:43:23.042651 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:43:23.042625    4463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d91b3696-0787-4913-9bbc-4db403c0139b/kube-rbac-proxy-web/0.log"
Nov 02 20:43:23.429839 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:43:23.429804    4463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d91b3696-0787-4913-9bbc-4db403c0139b/kube-rbac-proxy/0.log"
Nov 02 20:43:23.815515 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:43:23.815443    4463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d91b3696-0787-4913-9bbc-4db403c0139b/kube-rbac-proxy-thanos/0.log"
Nov 02 20:43:24.200551 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:43:24.200525    4463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d91b3696-0787-4913-9bbc-4db403c0139b/init-config-reloader/0.log"
Nov 02 20:43:31.410962 ip-10-0-133-9 kubenswrapper[4463]: I1102 20:43:31.410935    4463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-65657466b-9hgsh_a394c32e-69b7-4a96-8e08-e0080be53c80/kube-auth-proxy/0.log"